Apache DataFusion Java

Java bindings for Apache DataFusion. Queries run in native Rust and results return to the JVM as Apache Arrow batches via the Arrow C Data Interface.

Early development: no releases yet, API will change. Bug reports and contributions welcome.

Quickstart

import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.datafusion.DataFrame;
import org.apache.datafusion.SessionContext;

try (var allocator = new RootAllocator();
     var ctx = new SessionContext()) {

    ctx.registerParquet("orders", "/path/to/orders.parquet");

    try (DataFrame df = ctx.sql(
            "SELECT o_orderpriority, COUNT(*) AS n " +
            "FROM orders GROUP BY o_orderpriority");
         ArrowReader reader = df.collect(allocator)) {
        while (reader.loadNextBatch()) {
            var batch = reader.getVectorSchemaRoot();
            // ...
        }
    }
}

SessionContext and DataFrame are AutoCloseable and not thread-safe.

Project status

Query interfaces:

[x] SQL: SessionContext.sql(String)
[x] DataFrame: select, filter (other transformations TBD)
[x] DataFusion-Proto LogicalPlanNode: SessionContext.fromProto(byte[]). The datafusion-proto Java classes are generated by the build.

Data sources:

[x] Parquet via registerParquet / readParquet, with ParquetReadOptions
[x] CSV via registerCsv / readCsv, with CsvReadOptions
[ ] JSON, Avro
[ ] Custom catalog and table providers

Results:

[x] DataFrame.collect(allocator) — Arrow C Data Interface stream
[x] DataFrame.count(), show(), show(int)
[x] SessionContext.tableSchema(String)

Not yet:

[ ] SessionConfig / RuntimeEnv knobs
[ ] Java UDFs
[ ] write_* outputs

Requirements

JDK 17+. Building from source: see CONTRIBUTING.md.

Layout

src/ — Java sources and tests
native/ — Rust crate (JNI + Arrow C Data Interface)

Contributing

Open an issue to discuss non-trivial changes before sending a PR. See CONTRIBUTING.md.

License

Apache License 2.0. See LICENSE.txt and NOTICE.txt.