fix: Saturate NativeBuffer growth instead of overflowing past half of int.MaxValue (#419) ## What's Changed `Grow` doubled the current length in a `checked` context without saturating: ```csharp int newCount = Math.Max(newElementCount, checked(Length * 2)); ``` So once a buffer passed half of the addressable maximum, its **next** grow threw `OverflowException` however small the requested increase, and even though the requested size still fit. For a `NativeBuffer<byte, …>` that is a hard ceiling near 1 GiB, with no way for a caller to work around it: asking for a smaller increment does not help, because the overflow is in the doubling rather than in the request. Growth now saturates at the largest addressable element count, so it stays amortised right up to the ceiling. A request that genuinely cannot be addressed still fails at the byte-size calculation, as it did before — behaviour is unchanged for anything that could not have worked. This is what the `TODO` those lines carried proposed: > There might be a size that's big enough to work for this case but not too big to overflow. We could > use that instead of blindly doubling. ### On testing it Reaching the boundary through `Grow` means allocating more than a gigabyte, which does not belong in a unit test. The count arithmetic is extracted to `ComputeGrowCount` so the boundary can be tested directly and exhaustively, including the per-element-size ceiling — the limit is a byte count, so a wider element type saturates at proportionally fewer elements. Verified the new tests fail against the previous arithmetic before fixing it: with `checked(Length * 2)` restored, `ComputeGrowCountSaturatesInsteadOfOverflowing` and `ComputeGrowCountSaturatesPerElementSize` both fail; the rest pass either way. `Apache.Arrow.Tests` is green on net8.0: 1870 passed, 28 skipped (the Python interop cases). ### Scope This does not change the 2 GiB ceiling on `ArrowBuffer` itself (`ReadOnlyMemory<byte>`, `int Length`) — it only stops buffers failing at half of it. Closes #418. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
An implementation of Arrow targeting .NET Standard.
See our current feature matrix for currently available features.
using System.Diagnostics; using System.IO; using System.Threading.Tasks; using Apache.Arrow; using Apache.Arrow.Ipc; public static async Task<RecordBatch> ReadArrowAsync(string filename) { using (var stream = File.OpenRead(filename)) using (var reader = new ArrowFileReader(stream)) { var recordBatch = await reader.ReadNextRecordBatchAsync(); Debug.WriteLine("Read record batch with {0} column(s)", recordBatch.ColumnCount); return recordBatch; } }
Apache.Arrow.Compression package. When reading compressed data, you must pass an Apache.Arrow.Compression.CompressionCodecFactory instance to the ArrowFileReader or ArrowStreamReader constructor, and when writing compressed data a CompressionCodecFactory must be set in the IpcOptions. Alternatively, a custom implementation of ICompressionCodecFactory can be used.Install the latest .NET Core SDK from https://dotnet.microsoft.com/download.
dotnet build
To build the NuGet package run the following command to build a debug flavor, preview package into the artifacts folder.
dotnet pack
When building the officially released version run: (see Note below about current git repository)
dotnet pack -c Release
Which will build the final/stable package.
NOTE: When building the officially released version, ensure that your git repository has the origin remote set to https://github.com/apache/arrow.git, which will ensure Source Link is set correctly. See https://github.com/dotnet/sourcelink/blob/main/docs/README.md for more information.
There are two output artifacts:
Apache.Arrow.<version>.nupkg - this contains the executable assembliesApache.Arrow.<version>.snupkg - this contains the debug symbols filesBoth of these artifacts can then be uploaded to https://www.nuget.org/packages/manage/upload.
Build from the Apache Arrow project root.
docker build -f csharp/build/docker/Dockerfile .
dotnet test
All build artifacts are placed in the artifacts folder in the project root.
This project follows the coding style specified in Coding Style.
See https://flatbuffers.dev/languages/c_sharp/ for how to get the flatc executable.
Run flatc --csharp on each .fbs file in the format folder. And replace the checked in .cs files under FlatBuf with the generated files.
Update the non-generated FlatBuffers .cs files with the files from the google/flatbuffers repo.