tree 8869c973b7bcd32d28baa675fcaf31717bcdfb3a
parent ab57cb91cb64037a6d1e7319e9dd3d40849ed7f9
author Andy Grove <agrove@apache.org> 1786026128 -0600
committer GitHub <noreply@github.com> 1786026128 -0600
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsFcBAABCAAQBQJqdJiQCRC1aQ7uu5UhlAAAJG8QAG8VvyUBQpiTP0OWWQcGnTyT
 +J4PXJ2TpMNCla+ZmlucGiuiNiKQyOw1kel/mWsrQneh5AozpvZMejrsKtJm2AE2
 yf+4qWF/mkG/3yUCof7wMXPxMzkMiI6lZUnKPYjMRe00igBh+rSjiB5IKJvnvpw3
 QShhnAVqKrhmBQIZeryWOEFasnPvqe7Zp4foCP3EuVdY6aOoqY7+v6uTPVnHyuWA
 zWBuUJyfbGidh3SE3MRWQ4OiTpAtOvDv6IfaDUe9Tf3hBy3fOnXbelLSIL96CQQC
 lo9bSWy+avlJXaw0rf7Ac2OUpcgLWgi0e00Cc9tQzSxy/UHPT2rRWQwYPIx0q8mh
 F23fz4FN6TZPtL8M7sKiVjcQ5jP9YWWlH8i43xgWVlZWpv5Fx7Ivf5S9zUuzdMnh
 OECNw2+YSHdqQBM9S9pPQijGA2RxEiH/BChPceq+scHbkxNF45AzsQSSsICqTLpN
 GYQG8MZcTl9B1SXGz/7lG4NjryiuTrUbJ0/ZebZGG2Rfj501rRarl6Mx+x9c0+H0
 zvxVuCToJBCf0zNmPNj9X+o/YX5e+DQQNSwuRsac23uOZcdw5V9IqrX62q6i8o4j
 jPWkgkOpoBTS+/vf/MXoYMzJWjXkUhpc+ok/8xBDHUZTdswgnl3/U0ZB96k2+oCk
 TeO3viHNVcSXW+FpLoQL
 =ZBwe
 -----END PGP SIGNATURE-----
 

chore(deps): bump DataFusion to 54.1.0 (#114)

## Which issue does this PR close?

N/A — routine dependency bump; no tracking issue was filed.

## Rationale for this change

Keeps the binding current with upstream DataFusion. 54.1.0 is the latest
release
line, and staying close to it keeps the next bump small and makes
upstream fixes
available to Java callers.

## What changes are included in this PR?

Bumps `datafusion`, `datafusion-proto`, `datafusion-spark` and
`datafusion-substrait` from 53.1.0 to 54.1.0. The `datafusion.version`
Maven
property moves in lock step, since it selects the upstream tag the
`datafusion.proto` / `datafusion_common.proto` definitions are
downloaded from —
the generated Java protobuf classes must match what `datafusion-proto`
54.1.0
decodes. The pinned sha512 digests for both protos are updated
accordingly; each
was verified to match the copy vendored in the published
`datafusion-proto`
/ `datafusion-proto-common` 54.1.0 crates, independently of the GitHub
download.

`arrow` (58) and `object_store` (0.13) are unchanged — 54.1.0 resolves
to the
same majors, so the `object_store` pin comment still holds.

Adapting to the upstream API changes the bump requires:

- `TableProvider`, `ExecutionPlan` and `ScalarUDFImpl` now take `Any` as
a
supertrait, so the manual `as_any` overrides are no longer trait members
and
  are removed.
- `MemoryPool` gained a `name()` method and a `Display` supertrait.
`TrackingMemoryPool` implements both, following upstream's wrapper
convention:
name the wrapper, add the counters it exists to expose, and defer to the
inner
  pool for the usage detail.
- `CacheManagerConfig::table_files_statistics_cache` is renamed to
`file_statistics_cache`, and the accompanying limit is now the on/off
switch —
  `CacheManager::try_new` installs a default statistics cache whenever
`file_statistics_cache_limit > 0`, even when the cache slot is `None`,
and the
default limit is non-zero. An explicit `fileStatisticsCache(false)` from
the
Java surface therefore has to zero the limit as well; otherwise upstream
would
install the very cache the caller asked us to skip. This is the one
place the
  bump would have silently changed observable Java behavior.
- The `cache_unit` module is gone; the default cache impls now live in
  `cache::file_statistics_cache` and `cache`.
- `DataFusionError::AvroError` is gone — DataFusion 54 reads Avro
through
`arrow-avro` rather than `apache-avro`. Avro decode failures now arrive
as
`ArrowError::AvroError`, which the exception classifier already routed
to
`ExecutionException` alongside the `CsvError` / `JsonError` decoder
variants,
so the mapping stays coherent. The dead arm is dropped, along with the
`avro`
  feature on `datafusion-jni-common` that existed only to gate it.

One test fixture is also corrected. `SessionContextSubstraitTest` built
plans
whose base schema declared both columns `NULLABILITY_REQUIRED`, while
the tests
register a CSV — whose inferred schema is always nullable. DataFusion
54's
Substrait consumer now validates that a field a plan declares
non-nullable
really is non-nullable in the table, and rejects the mismatch. That
check is
correct: a plan built around a "never null" assumption must not run
against data
that can contain nulls. The fixture is fixed to declare nullable
columns; it
only passed before because 53 did not check.

## Are these changes tested?

Covered by the existing suites — this is a dependency bump, so the value
is in
the current tests continuing to pass against the new version rather than
in new
assertions.

- `./mvnw test` — 349 tests, 0 failures. Run with the `substrait` Cargo
feature
  enabled (`cargo build -p datafusion-jni --features substrait`) so the
  Substrait tests execute rather than skip.
- `cargo test --workspace` — all passing.
- `cargo clippy --workspace --all-targets -- -D warnings` — clean.
- `cargo fmt --all -- --check` and `./mvnw -q spotless:check` — clean.
- `cargo build --workspace --all-features` with `RUSTFLAGS="--cfg
tokio_unstable"`,
to cover the optional `substrait` and `runtime-metrics` features that
the
  default build does not compile.

## Are there any user-facing changes?

No API changes. Two behavioral notes, both inherited from upstream:

- Avro decode failures now surface as `ExecutionException` rather than
`IoException`, following the move to `arrow-avro`. The `IoException`
javadoc
  is updated to match.
- DataFusion 54 enables the file statistics and list files caches by
default.
Callers who never configured `CacheManagerOptions` pick up upstream's
new
defaults; an explicit `fileStatisticsCache(false)` continues to disable
the
  cache, as described above.