chore(deps): bump DataFusion to 55.0.0 (#117)

## Which issue does this PR close?

N/A — routine dependency bump; no tracking issue was filed.

## Rationale for this change

Keeps the binding current with upstream DataFusion. 55.0.0 is the latest
release, and staying close to it keeps the next bump small and makes
upstream fixes available to Java callers.

## What changes are included in this PR?

Bumps `datafusion`, `datafusion-proto`, `datafusion-spark` and
`datafusion-substrait` from 54.1.0 to 55.0.0, and `arrow` from 58 to 59
to match what DataFusion 55 resolves to. `object_store` stays on 0.13 —
DataFusion 55 still depends on `^0.13.2` — so the pin comment still
holds.

The `datafusion.version` Maven property moves in lock step, since it
selects the upstream tag the `datafusion.proto` /
`datafusion_common.proto` definitions are downloaded from — the
generated Java protobuf classes must match what `datafusion-proto`
55.0.0 decodes.

Adapting to the upstream API changes the bump requires:

- **The `datafusion.proto` schema moved** from `datafusion/proto/proto/`
to `datafusion/proto-models/proto/`, following the new
`datafusion-proto-models` crate. The `download-maven-plugin` URL is
updated, as is the recipe in
`docs/source/contributor-guide/updating-datafusion-version.md` so the
next bump does not rediscover this. Both pinned sha512 digests are
updated; each was verified to match the copy vendored in the published
`datafusion-proto-common` / `datafusion-proto-models` 55.0.0 crates,
independently of the GitHub download.
- **`ListingTableScanNode.target_partitions` was removed** upstream (now
`reserved 9`) — the partition count comes from the session config
instead. The field is dropped from the `fromProtoListingScanMatchesSql`
fixture. Note this is the *scan node's* field; the binding's own
`SessionOptionsProto.target_partitions`, which backs
`SessionContextBuilder.targetPartitions`, is unrelated and unchanged.
- **The concrete `Default*Cache` types were collapsed** into a single
generic `DefaultCache<K, V>`, so `DefaultFileStatisticsCache`,
`DefaultFilesMetadataCache` and `DefaultListFilesCache` no longer exist.
Rather than name the replacement, `cache_manager` now only sets the
cache *limits* (and the list-files TTL) and leaves the cache slots
`None`. `CacheManager::try_new` constructs the built-in cache for any
slot left `None` whose limit is non-zero — it already did so on 54 as
well — so the explicit construction was redundant on both versions and
this is behavior-preserving. It also keeps the binding off upstream's
concrete cache types entirely, which is the intent of the "configure the
built-in caches, don't swap in a custom one" contract.
- **`ExecutionPlan` gained a required `apply_expressions`.**
`JavaScanExec` is a leaf scan whose rows are produced on the Java side
and which holds no `PhysicalExpr`, so it returns
`TreeNodeRecursion::Continue` — the no-expression form upstream
documents for nodes like `EmptyExec`.
- **`datafusion_substrait::serializer::deserialize_bytes` is now
synchronous** and takes `&[u8]` rather than an owned `Vec<u8>`.

## Are these changes tested?

Covered by the existing suites — this is a dependency bump, so the value
is in the current tests continuing to pass against the new version
rather than in new assertions.

- `./mvnw test` — 349 tests, 0 failures, 0 skipped. Run with the
`substrait` Cargo feature enabled (`cargo build --features substrait`)
and with TPC-H SF1 data present, so the Substrait and data-gated tests
execute rather than skip. In particular `fromProtoListingScanMatchesSql`
— the test the `target_partitions` removal touches — asserts the
proto-built plan returns the same result as the equivalent SQL, and
passes.
- `cargo test --workspace --features substrait` — all passing.
- `cargo clippy --all-targets --workspace --features substrait -- -D
warnings` — clean.
- `cargo fmt --all` and `./mvnw spotless:apply` — clean.
- `cargo build --workspace --all-features` with `RUSTFLAGS="--cfg
tokio_unstable"`, to cover the optional `substrait` and
`runtime-metrics` features the default build does not compile.

## Are there any user-facing changes?

No API changes, and no behavior changes originating in this repo — the
cache-manager rework above is deliberately behavior-preserving. Callers
inherit whatever upstream changed between DataFusion 54.1.0 and 55.0.0.
9 files changed
tree: 031a5fc62354fee735c304765f640cca103bae58
  1. .cargo/
  2. .github/
  3. .mvn/
  4. core/
  5. dev/
  6. docs/
  7. examples/
  8. native/
  9. native-common/
  10. proto/
  11. .asf.yaml
  12. .gitignore
  13. Cargo.lock
  14. Cargo.toml
  15. CONTRIBUTING.md
  16. LICENSE.txt
  17. Makefile
  18. mvnw
  19. mvnw.cmd
  20. NOTICE.txt
  21. pom.xml
  22. README.md
README.md

Apache DataFusion Java

Java bindings for Apache DataFusion. Queries run in native Rust and results return to the JVM as Apache Arrow batches via the Arrow C Data Interface.

Early development: the API will change between releases. Bug reports and contributions welcome.

Install

Released to Maven Central. The JAR bundles the native library for Linux and macOS on x86_64 and aarch64. Windows users need to build from source.

Maven:

<dependency>
    <groupId>org.apache.datafusion</groupId>
    <artifactId>datafusion-java</artifactId>
    <version>0.1.0</version>
</dependency>

Gradle:

implementation("org.apache.datafusion:datafusion-java:0.1.0")

Arrow needs --add-opens=java.base/java.nio=ALL-UNNAMED on the JVM command line. See the installation guide for details and for building from source.

Quickstart

import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.datafusion.DataFrame;
import org.apache.datafusion.SessionContext;

try (var allocator = new RootAllocator();
     var ctx = new SessionContext()) {

    ctx.registerParquet("orders", "/path/to/orders.parquet");

    try (DataFrame df = ctx.sql(
            "SELECT o_orderpriority, COUNT(*) AS n " +
            "FROM orders GROUP BY o_orderpriority");
         ArrowReader reader = df.collect(allocator)) {
        while (reader.loadNextBatch()) {
            var batch = reader.getVectorSchemaRoot();
            // ...
        }
    }
}

SessionContext and DataFrame are AutoCloseable and not thread-safe.

Documentation

The full documentation lives under docs/source/ and is built with Sphinx (see docs/README.md for the build steps):

  • User guide — installation, the DataFrame and SQL APIs, Parquet ingestion.
  • Contributor guide — build, test, code style, and how to bump the DataFusion version.

Requirements

JDK 17+. Building from source: see docs/source/contributor-guide/development.md.

Contributing

Open an issue to discuss non-trivial changes before sending a PR. See the contributor guide.

License

Apache License 2.0. See LICENSE.txt and NOTICE.txt.