GH-44800: [C#] Implement Flight SQL Client (#44783) GH-44800: [C#] Implement Flight SQL Client ## Rationale for this Change This pull request introduces a **new implementation of `FlightSqlClient` and `PreparedStatement` in C#**. Previously, there was no C# client for Flight SQL, leaving a significant gap for .NET developers who wished to interact with Flight SQL servers. The implementation aligns with the existing **C++ Flight SQL client** API, ensuring consistent and familiar behavior across languages and providing a robust client for the Apache Arrow ecosystem in .NET. --- ## What's Included in this PR? ### Key Features 1. **`FlightSqlClient`**: - Provides query execution (`ExecuteAsync`, `ExecuteUpdateAsync`) and schema retrieval (`GetCatalogsAsync`, `GetDbSchemasAsync`, etc.). - Implements metadata operations for catalogs, schemas, tables, and more. - Fully integrated with gRPC and Apache Arrow ecosystems. - Supports extensibility for advanced features like transactions. 2. **`PreparedStatement`**: - Implements parameterized query execution (`SetParameters`, `ExecuteAsync`, and `ExecuteUpdateAsync`). - Supports lifecycle management (`CloseAsync`) for effective resource handling. - Aligns with the prepared statement design in the C++ client. ### API Consistency This implementation mirrors the **C++ Flight SQL client** to ensure API alignment across supported languages: - Consistent naming conventions and parameter semantics. - Ensures .NET developers can work seamlessly with existing Flight SQL servers. --- ## Are These Changes Tested? ### Testing Overview 1. **Unit Tests**: - Added tests for query execution and parameter binding in `PreparedStatement`. - Verified schema retrieval methods like `GetCatalogsAsync` and `GetDbSchemasAsync`. 2. **Integration Tests**: - Tested against a live Flight SQL server to validate query execution, schema retrieval, and metadata operations. 3. **End-to-End Tests**: - Covered real-world scenarios for parameterized updates and queries, ensuring robustness. ### Example Test Cases - Verify that parameterized queries return correct results with valid input. - Ensure schema retrieval throws appropriate exceptions for invalid descriptors. - Validate row counts after `ExecuteUpdateAsync`. --- ## Are There Any Breaking Changes? This PR introduces **new functionality** and does not affect any existing features. There are **no breaking changes**. --- ## Are There Any User-Facing Changes? ### New Capabilities 1. **FlightSqlClient**: - Query execution and schema retrieval for SQL queries on Flight SQL servers. - Metadata retrieval for catalogs, schemas, tables, and more. 2. **PreparedStatement**: - Supports parameterized queries with proper parameter binding. - Provides robust lifecycle management and execution. ### API Consistency - Aligns with the C++ Flight SQL client for interoperability and familiar API design. --- ## Additional Notes - This PR **does not include transaction support** at this stage, as it requires additional server-side capabilities. - All methods follow idiomatic C# practices, including `async/await` for non-blocking operations. - Extensible for future enhancements, including advanced features like savepoints. --- ## Resources - **C++ Flight SQL Client Reference**: [Apache Arrow Flight SQL Documentation](https://arrow.apache.org/docs/) - **Apache Arrow Contribution Guide**: [Contributing to Apache Arrow](https://arrow.apache.org/docs/dev/developers/guide/) --- ## Feedback and Suggestions Thank you for reviewing this contribution! Suggestions and feedback are welcome to ensure the implementation meets the project's standards and requirements. * GitHub Issue: #44800 Lead-authored-by: HackPoint <genashm@ge.com> Co-authored-by: Genady Shmunik <genady.shmunik@ge.com> Co-authored-by: HackP0!nt <genashm@gmail.com> Co-authored-by: HackPoint <genashm@gmail.com> Co-authored-by: Curt Hagenlocher <curt@hagenlocher.org> Signed-off-by: Curt Hagenlocher <curt@hagenlocher.org>
An implementation of Arrow targeting .NET Standard.
See our current feature matrix for currently available features.
using System.Diagnostics;
using System.IO;
using System.Threading.Tasks;
using Apache.Arrow;
using Apache.Arrow.Ipc;
public static async Task<RecordBatch> ReadArrowAsync(string filename)
{
using (var stream = File.OpenRead(filename))
using (var reader = new ArrowFileReader(stream))
{
var recordBatch = await reader.ReadNextRecordBatchAsync();
Debug.WriteLine("Read record batch with {0} column(s)", recordBatch.ColumnCount);
return recordBatch;
}
}
Apache.Arrow.Compression package. When reading compressed data, you must pass an Apache.Arrow.Compression.CompressionCodecFactory instance to the ArrowFileReader or ArrowStreamReader constructor, and when writing compressed data a CompressionCodecFactory must be set in the IpcOptions. Alternatively, a custom implementation of ICompressionCodecFactory can be used.Install the latest .NET Core SDK from https://dotnet.microsoft.com/download.
dotnet build
To build the NuGet package run the following command to build a debug flavor, preview package into the artifacts folder.
dotnet pack
When building the officially released version run: (see Note below about current git repository)
dotnet pack -c Release
Which will build the final/stable package.
NOTE: When building the officially released version, ensure that your git repository has the origin remote set to https://github.com/apache/arrow.git, which will ensure Source Link is set correctly. See https://github.com/dotnet/sourcelink/blob/main/docs/README.md for more information.
There are two output artifacts:
Apache.Arrow.<version>.nupkg - this contains the executable assembliesApache.Arrow.<version>.snupkg - this contains the debug symbols filesBoth of these artifacts can then be uploaded to https://www.nuget.org/packages/manage/upload.
Build from the Apache Arrow project root.
docker build -f csharp/build/docker/Dockerfile .
dotnet test
All build artifacts are placed in the artifacts folder in the project root.
This project follows the coding style specified in Coding Style.
See https://google.github.io/flatbuffers/flatbuffers_guide_use_java_c-sharp.html for how to get the flatc executable.
Run flatc --csharp on each .fbs file in the format folder. And replace the checked in .cs files under FlatBuf with the generated files.
Update the non-generated FlatBuffers .cs files with the files from the google/flatbuffers repo.