chore(deps): bump DataFusion to 55.0.0 (#117) ## Which issue does this PR close? N/A — routine dependency bump; no tracking issue was filed. ## Rationale for this change Keeps the binding current with upstream DataFusion. 55.0.0 is the latest release, and staying close to it keeps the next bump small and makes upstream fixes available to Java callers. ## What changes are included in this PR? Bumps `datafusion`, `datafusion-proto`, `datafusion-spark` and `datafusion-substrait` from 54.1.0 to 55.0.0, and `arrow` from 58 to 59 to match what DataFusion 55 resolves to. `object_store` stays on 0.13 — DataFusion 55 still depends on `^0.13.2` — so the pin comment still holds. The `datafusion.version` Maven property moves in lock step, since it selects the upstream tag the `datafusion.proto` / `datafusion_common.proto` definitions are downloaded from — the generated Java protobuf classes must match what `datafusion-proto` 55.0.0 decodes. Adapting to the upstream API changes the bump requires: - **The `datafusion.proto` schema moved** from `datafusion/proto/proto/` to `datafusion/proto-models/proto/`, following the new `datafusion-proto-models` crate. The `download-maven-plugin` URL is updated, as is the recipe in `docs/source/contributor-guide/updating-datafusion-version.md` so the next bump does not rediscover this. Both pinned sha512 digests are updated; each was verified to match the copy vendored in the published `datafusion-proto-common` / `datafusion-proto-models` 55.0.0 crates, independently of the GitHub download. - **`ListingTableScanNode.target_partitions` was removed** upstream (now `reserved 9`) — the partition count comes from the session config instead. The field is dropped from the `fromProtoListingScanMatchesSql` fixture. Note this is the *scan node's* field; the binding's own `SessionOptionsProto.target_partitions`, which backs `SessionContextBuilder.targetPartitions`, is unrelated and unchanged. - **The concrete `Default*Cache` types were collapsed** into a single generic `DefaultCache<K, V>`, so `DefaultFileStatisticsCache`, `DefaultFilesMetadataCache` and `DefaultListFilesCache` no longer exist. Rather than name the replacement, `cache_manager` now only sets the cache *limits* (and the list-files TTL) and leaves the cache slots `None`. `CacheManager::try_new` constructs the built-in cache for any slot left `None` whose limit is non-zero — it already did so on 54 as well — so the explicit construction was redundant on both versions and this is behavior-preserving. It also keeps the binding off upstream's concrete cache types entirely, which is the intent of the "configure the built-in caches, don't swap in a custom one" contract. - **`ExecutionPlan` gained a required `apply_expressions`.** `JavaScanExec` is a leaf scan whose rows are produced on the Java side and which holds no `PhysicalExpr`, so it returns `TreeNodeRecursion::Continue` — the no-expression form upstream documents for nodes like `EmptyExec`. - **`datafusion_substrait::serializer::deserialize_bytes` is now synchronous** and takes `&[u8]` rather than an owned `Vec<u8>`. ## Are these changes tested? Covered by the existing suites — this is a dependency bump, so the value is in the current tests continuing to pass against the new version rather than in new assertions. - `./mvnw test` — 349 tests, 0 failures, 0 skipped. Run with the `substrait` Cargo feature enabled (`cargo build --features substrait`) and with TPC-H SF1 data present, so the Substrait and data-gated tests execute rather than skip. In particular `fromProtoListingScanMatchesSql` — the test the `target_partitions` removal touches — asserts the proto-built plan returns the same result as the equivalent SQL, and passes. - `cargo test --workspace --features substrait` — all passing. - `cargo clippy --all-targets --workspace --features substrait -- -D warnings` — clean. - `cargo fmt --all` and `./mvnw spotless:apply` — clean. - `cargo build --workspace --all-features` with `RUSTFLAGS="--cfg tokio_unstable"`, to cover the optional `substrait` and `runtime-metrics` features the default build does not compile. ## Are there any user-facing changes? No API changes, and no behavior changes originating in this repo — the cache-manager rework above is deliberately behavior-preserving. Callers inherit whatever upstream changed between DataFusion 54.1.0 and 55.0.0.
diff --git a/Cargo.lock b/Cargo.lock index ab96620..68f12ee 100644 --- a/Cargo.lock +++ b/Cargo.lock
@@ -90,9 +90,9 @@ [[package]] name = "arrow" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "378530e55cd479eda3c14eb345310799717e6f76d0c332041e8487022166b471" +checksum = "61d285d16bce7d0be61912f7928342b673067b6b7d7ef6cc179258ba7de1fecf" dependencies = [ "arrow-arith", "arrow-array", @@ -111,9 +111,9 @@ [[package]] name = "arrow-arith" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0ab212d2c1886e802f51c5212d78ebbcbb0bec980fff9dadc1eb8d45cd0b738" +checksum = "757ef1836251e88222542a7da2623bc1c9cb9e20afefa6db2c41e79991cd91d4" dependencies = [ "arrow-array", "arrow-buffer", @@ -125,9 +125,9 @@ [[package]] name = "arrow-array" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfd33d3e92f207444098c75b42de99d329562be0cf686b307b097cc52b4e999e" +checksum = "bc9a4a4b2b5ecd0e04df03471661cb61f28bed3c7fd50994715129b01b2edb97" dependencies = [ "ahash", "arrow-buffer", @@ -137,6 +137,7 @@ "chrono-tz", "half", "hashbrown 0.17.1", + "libc", "num-complex", "num-integer", "num-traits", @@ -144,9 +145,9 @@ [[package]] name = "arrow-avro" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "049230728cd6e093088c8d231b4beede184e35cad7777c1505c0d5a8571f4376" +checksum = "9fb45cd6bd2b25c0965793b83200eaca82214273a8030fbbc2d783e4c7c65a61" dependencies = [ "arrow-array", "arrow-buffer", @@ -168,21 +169,21 @@ [[package]] name = "arrow-buffer" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c6cd424c2693bcdbc150d843dc9d4d137dd2de4782ce6df491ad11a3a0416c0" +checksum = "c12b576ef18c1deb80925a248b25ad84f419198d791b8e293fc6aaa60441fe90" dependencies = [ "bytes", "half", - "num-bigint", + "num-bigint 0.5.1", "num-traits", ] [[package]] name = "arrow-cast" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c5aefb56a2c02e9e2b30746241058b85f8983f0fcff2ba0c6d09006e1cded7f" +checksum = "68338a9096a5dc9bc11927c58c43a8526d96bf6abd2012ef6c0c9f505991cc79" dependencies = [ "arrow-array", "arrow-buffer", @@ -191,7 +192,7 @@ "arrow-schema", "arrow-select", "atoi", - "base64", + "base64 0.23.1", "chrono", "comfy-table", "half", @@ -202,9 +203,9 @@ [[package]] name = "arrow-csv" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e94e8cf7e517657a52b91ea1263acf38c4ca62a84655d72458a3359b12ab97de" +checksum = "25011b52b346407d497ef0030e12b45e4f2d0cc279efc09c4f3d09106db30e36" dependencies = [ "arrow-array", "arrow-cast", @@ -217,9 +218,9 @@ [[package]] name = "arrow-data" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c88210023a2bfee1896af366309a3028fc3bcbd6515fa29a7990ee1baa08ee0" +checksum = "723fe4aeed7604e00b9883a465af4ff0a0e6c44c03e41a68c3d1cbc403e0e44d" dependencies = [ "arrow-buffer", "arrow-schema", @@ -230,9 +231,9 @@ [[package]] name = "arrow-ipc" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "238438f0834483703d88896db6fe5a7138b2230debc31b34c0336c2996e3c64f" +checksum = "149437b14371f5b9ec60f5ddc751483ae99d7a7072653c0075e5e469156eea7b" dependencies = [ "arrow-array", "arrow-buffer", @@ -246,9 +247,9 @@ [[package]] name = "arrow-json" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "205ca2119e6d679d5c133c6f30e68f027738d95ed948cf77677ea69c7800036b" +checksum = "f18b9123ccfec418a663f821c9a034af339711678c11ffe00d3ec07da5ff9f7e" dependencies = [ "arrow-array", "arrow-buffer", @@ -271,9 +272,9 @@ [[package]] name = "arrow-ord" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bffd8fd2579286a5d63bac898159873e5094a79009940bcb42bbfce4f19f1d0" +checksum = "e6c08dff0686cf23ca4f562803f191ccbeb726dbae6309cd4b4aaf65e0f2c979" dependencies = [ "arrow-array", "arrow-buffer", @@ -284,9 +285,9 @@ [[package]] name = "arrow-row" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bab5994731204603c73ba69267616c50f80780774c6bb0476f1f830625115e0c" +checksum = "bbec439386df71ad570e6758a946111322b9e9dc8db83b5527321f0b4c9119c2" dependencies = [ "arrow-array", "arrow-buffer", @@ -297,9 +298,9 @@ [[package]] name = "arrow-schema" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f633dbfdf39c039ada1bf9e34c694816eb71fbb7dc78f613993b7245e078a1ed" +checksum = "e6fed2ca0d1eade57e811cbe73b98ad50cc08a1183e13b2d2aa43a7df593f40e" dependencies = [ "bitflags", "serde_core", @@ -308,9 +309,9 @@ [[package]] name = "arrow-select" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8cd065c54172ac787cf3f2f8d4107e0d3fdc26edba76fdf4f4cc170258942222" +checksum = "466b19cf75130b891dc1b23a84b343c714c62c64c9c62e365c76aa0ff90a53fb" dependencies = [ "ahash", "arrow-array", @@ -322,9 +323,9 @@ [[package]] name = "arrow-string" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29dd7cda3ab9692f43a2e4acc444d760cc17b12bb6d8232ddf64e9bab7c06b42" +checksum = "c838a25bb3691e919e0f617616ac51a4ff8517a952e29ca133cf0c22b2ce65b1" dependencies = [ "arrow-array", "arrow-buffer", @@ -357,7 +358,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -368,7 +369,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -399,6 +400,12 @@ checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] +name = "base64" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" + +[[package]] name = "bigdecimal" version = "0.4.10" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -406,7 +413,7 @@ dependencies = [ "autocfg", "libm", - "num-bigint", + "num-bigint 0.4.6", "num-integer", "num-traits", ] @@ -486,12 +493,6 @@ checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" [[package]] -name = "byteorder" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" - -[[package]] name = "bytes" version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -768,9 +769,9 @@ [[package]] name = "datafusion" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "754ef4e8f073922a26f5b23133b9db4829342362b09be0bc94309cf261c2f098" +checksum = "96f76f0167ed0842b29a3d1e41be3c034c0a46409a3a703cc4cc84ee8c24abf4" dependencies = [ "arrow", "arrow-schema", @@ -806,7 +807,7 @@ "flate2", "futures", "indexmap", - "itertools", + "itertools 0.15.0", "liblzma", "log", "object_store", @@ -822,9 +823,9 @@ [[package]] name = "datafusion-catalog" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06afd1e38dd27bbb1258685a1fc6524df6aff4e07b25b393a47de59635178d99" +checksum = "d79ec3460f6ed5c58f9b3f2d873fbc77748b82653bff1b4cdaf06de33bb4e05f" dependencies = [ "arrow", "async-trait", @@ -838,7 +839,7 @@ "datafusion-physical-plan", "datafusion-session", "futures", - "itertools", + "itertools 0.15.0", "log", "object_store", "parking_lot", @@ -847,9 +848,9 @@ [[package]] name = "datafusion-catalog-listing" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0668fb32c12065ec242be0e5b4bc62bd7a06a0be3ecd83791ef877e4be67e02" +checksum = "b48cef241e2efcfd496fe05ae4d0d5de20793451862faefe406c397a467e12d4" dependencies = [ "arrow", "async-trait", @@ -863,16 +864,17 @@ "datafusion-physical-expr-common", "datafusion-physical-plan", "futures", - "itertools", + "itertools 0.15.0", "log", "object_store", + "percent-encoding", ] [[package]] name = "datafusion-common" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ca43b263cdff57042cfa8fb817fb3469f4878933380dccff25f5e793580abbf9" +checksum = "3f72810485975c258f1b4d00baab31728470676c60c5546f366ebd0d99f05ab6" dependencies = [ "arrow", "arrow-ipc", @@ -882,9 +884,10 @@ "half", "hashbrown 0.17.1", "indexmap", - "itertools", + "itertools 0.15.0", "libc", "log", + "num-traits", "object_store", "parquet", "recursive", @@ -896,9 +899,9 @@ [[package]] name = "datafusion-common-runtime" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05f0ba2b864792bdca4d76c59a1de0ab6e1b61946596b9936888dbd6360035f2" +checksum = "533c28e75dba52f41bde187d23a1cb24ab91c7c097966824fa471e67b60320ea" dependencies = [ "futures", "log", @@ -907,9 +910,9 @@ [[package]] name = "datafusion-datasource" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b840a8bce0bcbf5afad02946d438591e7c373f7afccaf3d874c04485772514dd" +checksum = "5b00a1fa0da26f6087136a82fea7f13c76a672cbab452d4086952a7cf770a19b" dependencies = [ "arrow", "async-compression", @@ -925,11 +928,12 @@ "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-session", "flate2", "futures", "glob", - "itertools", + "itertools 0.15.0", "liblzma", "log", "object_store", @@ -943,9 +947,9 @@ [[package]] name = "datafusion-datasource-arrow" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a24cc0b9cf6e367f27f27406eff13abf48a11b72446aaa40b3105c0ded5c17d9" +checksum = "5ad17ec881bff2ed7768b4bfe971d3efbf3473f2fd1f9d365447bccbdf908678" dependencies = [ "arrow", "arrow-ipc", @@ -958,18 +962,19 @@ "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-session", "futures", - "itertools", + "itertools 0.15.0", "object_store", "tokio", ] [[package]] name = "datafusion-datasource-avro" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f26a312bb06528c17b4bb86b798b0c4321e7656d85e0ef65c2e9adf07b0a518d" +checksum = "c1c9587cff8163f9bfcb186e8664ba85cb327031b8ff14330307f961ea2fc196" dependencies = [ "arrow", "arrow-avro", @@ -986,9 +991,9 @@ [[package]] name = "datafusion-datasource-csv" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1abe56b2a7a2d1d6de5117dd1a203181e267f28529faa5da546947621b697d7" +checksum = "b5345285b0c3eaab412e7539b706973c083bd7e5bce575de5e0a3da488d08d1d" dependencies = [ "arrow", "async-trait", @@ -1000,6 +1005,7 @@ "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-session", "futures", "object_store", @@ -1009,9 +1015,9 @@ [[package]] name = "datafusion-datasource-json" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3e467f0611ad7bdd5aad17c63c9bb6182d04e5282e5496d897ea2b49c024ba" +checksum = "da02fb9324f56bd8c53f1ee2e949547425cb66f76adc6832b10d44f80a1221d2" dependencies = [ "arrow", "async-trait", @@ -1023,6 +1029,7 @@ "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-session", "futures", "object_store", @@ -1032,11 +1039,12 @@ [[package]] name = "datafusion-datasource-parquet" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4cc35b92cd560082155e80d9c826929c852d3c51543f4affd3a51c464a0aab3a" +checksum = "3c0b0dc1453952952fd5c69ad1c7f6042176e69ed233011d47e07cf74ed0949e" dependencies = [ "arrow", + "arrow-schema", "async-trait", "bytes", "datafusion-common", @@ -1050,10 +1058,11 @@ "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-pruning", "datafusion-session", "futures", - "itertools", + "itertools 0.15.0", "log", "object_store", "parking_lot", @@ -1063,19 +1072,20 @@ [[package]] name = "datafusion-doc" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d69bb69d8769e34f76839c960dbde24c1ac0c885a79b6c3c2287bdc56ec67891" +checksum = "a88fd985bc0550c36f557db69543cc9d6393b1509783520b30e902f23c555da6" [[package]] name = "datafusion-execution" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d8eac0a09bc8d263f52025cad9e001da4d8138d633fa288edda4d06b1772eae6" +checksum = "a98f1052f91b4991f0bf2ce1e4e36dfbdcda454a956b8c8d562c7c845e8fce1d" dependencies = [ "arrow", "arrow-buffer", "async-trait", + "bytes", "dashmap", "datafusion-common", "datafusion-expr", @@ -1084,16 +1094,19 @@ "log", "object_store", "parking_lot", + "pin-project-lite", "rand 0.9.4", "tempfile", + "tokio", + "tokio-util", "url", ] [[package]] name = "datafusion-expr" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eeb14d374767ee0fc62dc79a5ba8bcf8a63c14e993c7d992d0e63adfa23d77d3" +checksum = "464625a1f0e4b9df552d894fafcc8aac953ebbc8b0fa0acdaf20975fd615040e" dependencies = [ "arrow", "arrow-schema", @@ -1105,8 +1118,10 @@ "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr-common", + "datafusion-proto-common", + "datafusion-proto-models", "indexmap", - "itertools", + "itertools 0.15.0", "recursive", "serde_json", "sqlparser", @@ -1114,25 +1129,25 @@ [[package]] name = "datafusion-expr-common" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7b19a8c95522bee8cbb313d74263b85e355d2b52f42e67ef5694bf5de9e9356" +checksum = "2604994999d5aeca1d1df645ffc98bc787447aaff05dde27aad0342b48fc1fe0" dependencies = [ "arrow", "datafusion-common", "indexmap", - "itertools", + "itertools 0.15.0", ] [[package]] name = "datafusion-functions" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f64c983bbbdcb729d921a2b2ac3375598719b5cc0c30345ad664936f3176fc7" +checksum = "051e97533e6af53e4aa0a0667cadc886abcaf36c4a5925019c55c0aa4c218fde" dependencies = [ "arrow", "arrow-buffer", - "base64", + "base64 0.23.1", "blake2", "blake3", "chrono", @@ -1145,7 +1160,7 @@ "datafusion-macros", "datafusion-physical-expr-common", "hex", - "itertools", + "itertools 0.15.0", "log", "md-5 0.11.0", "memchr", @@ -1158,9 +1173,9 @@ [[package]] name = "datafusion-functions-aggregate" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89bc17041e424a47ed062f43df24d84aab8b57c4c3221e5c1a5eef46d6c5718b" +checksum = "2d0f1bb166d3572b6ed40e1afb2faaacade962abc08c2fcf04babee74681c56b" dependencies = [ "arrow", "datafusion-common", @@ -1171,17 +1186,17 @@ "datafusion-macros", "datafusion-physical-expr", "datafusion-physical-expr-common", - "foldhash 0.2.0", "half", + "hashbrown 0.17.1", "log", "num-traits", ] [[package]] name = "datafusion-functions-aggregate-common" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97dd2a9e865c6108059f5b37b77934f84b50bfb108f837bd0e5c9536e03f0545" +checksum = "7ed756770f5f98369e181d692fd5ee6b1127ffd7322caba92f3730f9f5c92333" dependencies = [ "arrow", "datafusion-common", @@ -1191,9 +1206,9 @@ [[package]] name = "datafusion-functions-nested" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75f0bdfeef16d96417b9632ef855645376b242e9006a126dfd0bedfc54a93f5f" +checksum = "91173fdb5c0ff2a41169a8ffa1b385b8844f18728747bb0a37e35ad7d5772a4f" dependencies = [ "arrow", "arrow-ord", @@ -1208,7 +1223,7 @@ "datafusion-macros", "datafusion-physical-expr-common", "hashbrown 0.17.1", - "itertools", + "itertools 0.15.0", "itoa", "log", "memchr", @@ -1216,9 +1231,9 @@ [[package]] name = "datafusion-functions-table" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f4e4941673c917819616877e9993da4503e4f4739812be0bc32c5356184c6383" +checksum = "b1bcdfb286a745461b126719c32700777e83df4f17cc44db5d71ebce5731e840" dependencies = [ "arrow", "async-trait", @@ -1232,9 +1247,9 @@ [[package]] name = "datafusion-functions-window" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "12dd2e16c12b84b6f6b41b19f55b366dd1c46876bb35b86896c6349067379e8d" +checksum = "9ec4b508f1f93f00038ba3e737e894ec6c775528b4369413386655ae6125f0fc" dependencies = [ "arrow", "datafusion-common", @@ -1249,9 +1264,9 @@ [[package]] name = "datafusion-functions-window-common" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4cdc5e4b6f8b6ef823cc1c761f85088ad4c884fe8df64df3cbcc6b2b84698441" +checksum = "0b352020834140073fbf5b46ee0ceb926e5074a9d0bcae1dbd91d0586d999cde" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -1291,20 +1306,20 @@ [[package]] name = "datafusion-macros" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a3614234dd93578c92428cb4f408e020874f0d2b7e6c90c928d9d28b5df2ceb" +checksum = "15192effab05d38cce10e92a6fb48c967b5f166b27b7195a165a72b232569c58" dependencies = [ "datafusion-doc", "quote", - "syn", + "syn 3.0.4", ] [[package]] name = "datafusion-optimizer" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0635620b050b81bb92764e99250868f654e2cd5ad1bece413283b3f73c83179" +checksum = "854445d9f7847e1e46089cf61b8d341a64382f14484e912c83a0f23b31216896" dependencies = [ "arrow", "chrono", @@ -1313,7 +1328,7 @@ "datafusion-expr-common", "datafusion-physical-expr", "indexmap", - "itertools", + "itertools 0.15.0", "log", "recursive", "regex", @@ -1322,9 +1337,9 @@ [[package]] name = "datafusion-physical-expr" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8cabf7a86eb70b816729e33c81bf7767c936ee1226f607a114f5dac2decac8d0" +checksum = "671558dad1d2aa253c39c0a4c52515958b99eb91abf649f4b88d5e69cc55282f" dependencies = [ "arrow", "datafusion-common", @@ -1332,10 +1347,11 @@ "datafusion-expr-common", "datafusion-functions-aggregate-common", "datafusion-physical-expr-common", + "datafusion-proto-models", "half", "hashbrown 0.17.1", "indexmap", - "itertools", + "itertools 0.15.0", "parking_lot", "petgraph", "recursive", @@ -1344,9 +1360,9 @@ [[package]] name = "datafusion-physical-expr-adapter" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de222e04f7e6744501555a54ab0abe26bfdfebee380af79a9bdc175704246859" +checksum = "ffae3d78c2da80ecc829cb58536cc5aca2e99cf1365eda694fc75bfe288861e0" dependencies = [ "arrow", "datafusion-common", @@ -1354,31 +1370,32 @@ "datafusion-functions", "datafusion-physical-expr", "datafusion-physical-expr-common", - "itertools", + "itertools 0.15.0", ] [[package]] name = "datafusion-physical-expr-common" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72d0d0057fc5a502d45c870cb6d47c66eb7bdd5edb1bd71ad6f3f724975ac2a8" +checksum = "3d9092ed15e7203fbd0903215172f7c9d18f10d94cba35137f3b3836f7c46f16" dependencies = [ "arrow", "chrono", "datafusion-common", "datafusion-expr-common", + "datafusion-proto-models", "hashbrown 0.17.1", "indexmap", - "itertools", + "itertools 0.15.0", "parking_lot", "pin-project", ] [[package]] name = "datafusion-physical-optimizer" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "86046eed10950c5f9aaed9acfd148e9bd2e1dfdfe4f9aef607d1447b271e4183" +checksum = "9005b6cf50b57b72d476c6ed4662b04be7ca6be5320ba9127c6d0b7e4218095b" dependencies = [ "arrow", "datafusion-common", @@ -1389,15 +1406,16 @@ "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-pruning", - "itertools", + "datafusion-session", + "itertools 0.15.0", "recursive", ] [[package]] name = "datafusion-physical-plan" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9bc84da934c903407ba297971ebcc020c4c1a38aafd765d6c144c76eff3fa6a1" +checksum = "5787e4fcff4adc4fce8948441103a99705018b49c8dff0720b650bd7a15da112" dependencies = [ "arrow", "arrow-data", @@ -1405,6 +1423,7 @@ "arrow-ord", "arrow-schema", "async-trait", + "bytes", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", @@ -1414,26 +1433,28 @@ "datafusion-functions-window-common", "datafusion-physical-expr", "datafusion-physical-expr-common", + "datafusion-proto-common", + "datafusion-proto-models", "futures", "half", "hashbrown 0.17.1", "indexmap", - "itertools", + "itertools 0.15.0", "log", "num-traits", "parking_lot", "pin-project-lite", + "serde_json", "tokio", ] [[package]] name = "datafusion-proto" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67791dcfacd142a9d95f8f73c2a161094cc936d231d80c235f5017dacf24e84e" +checksum = "0df0504eb9028d5e01f481af3519cde3f97ab650fd50ce7b787e94b69a7a193c" dependencies = [ "arrow", - "chrono", "datafusion-catalog", "datafusion-catalog-listing", "datafusion-common", @@ -1449,15 +1470,16 @@ "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-proto-common", + "datafusion-proto-models", "object_store", "prost", ] [[package]] name = "datafusion-proto-common" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8cd9e80d637891645d074db0f6c650b591117367247deb313fdfb78dff559cb" +checksum = "6b92415a2442964f180d39cdcd8ff1edd99f9260c1499ba80a396499c2154d11" dependencies = [ "arrow", "datafusion-common", @@ -1465,10 +1487,21 @@ ] [[package]] -name = "datafusion-pruning" -version = "54.1.0" +name = "datafusion-proto-models" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eb63eeac6de19be40f487b65dd84e546195f783c5a9928618e0c4f2a3569b0d7" +checksum = "62e4c0bd6af4fcabdbe201ee86fbeef2ac423a44e1d8fda56d994c9e2d1d3ad2" +dependencies = [ + "datafusion-common", + "datafusion-proto-common", + "prost", +] + +[[package]] +name = "datafusion-pruning" +version = "55.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e651c8df0b90daed6a7be5921ec0ee379e6909705f063eeff70fd4e35010e4c" dependencies = [ "arrow", "datafusion-common", @@ -1482,10 +1515,11 @@ [[package]] name = "datafusion-session" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f961d209177f91bd014db5cbb2c33b7d28a2597b9003e77f17aeb712964315a" +checksum = "fb56667ee38217efab19b895d9a936052cfb47ed438a19663351bdc42a6214a1" dependencies = [ + "arrow-schema", "async-trait", "datafusion-common", "datafusion-execution", @@ -1496,9 +1530,9 @@ [[package]] name = "datafusion-spark" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1ccd16a6949503e56c084df1b90c8889db826ec9347d2f0f51a7837d6fa011e" +checksum = "992dd0b954f24cb576cbeee3555c3083b9cf7a5a5f2448ea25f7a437d444163f" dependencies = [ "arrow", "bigdecimal", @@ -1526,9 +1560,9 @@ [[package]] name = "datafusion-sql" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b1d71cb454da682b2af7488e1fc1ddd72ee1b28f19297b8ccad73f0a21ee9a69" +checksum = "9c29067cb9d32f8e603c45e15d61ea18f1069f96ceafeceb4e18466b8e5b31d9" dependencies = [ "arrow", "bigdecimal", @@ -1541,20 +1575,21 @@ "recursive", "regex", "sqlparser", + "stacker", ] [[package]] name = "datafusion-substrait" -version = "54.1.0" +version = "55.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f047a6fbf967b6b523758a48d2377bb5f9373a20e7ae4c4326d3c569f80ba3d7" +checksum = "f8c026ddbbc33c34d0fac95a9620367c805ad6a7174cac52b57f360746c3e282" dependencies = [ "async-recursion", "async-trait", "chrono", "datafusion", "half", - "itertools", + "itertools 0.15.0", "object_store", "pbjson-types", "prost", @@ -1593,7 +1628,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -1746,7 +1781,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -2012,7 +2047,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" dependencies = [ - "base64", + "base64 0.22.1", "bytes", "futures-channel", "futures-util", @@ -2175,12 +2210,6 @@ ] [[package]] -name = "integer-encoding" -version = "3.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" - -[[package]] name = "ipnet" version = "2.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -2196,6 +2225,15 @@ ] [[package]] +name = "itertools" +version = "0.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b4baf93f58d4425749ca49a51c50ebab072c5df6994d08fed93541c331481dc" +dependencies = [ + "either", +] + +[[package]] name = "itoa" version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -2242,7 +2280,7 @@ checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" dependencies = [ "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -2402,9 +2440,9 @@ [[package]] name = "lz4_flex" -version = "0.13.1" +version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" +checksum = "ecbdfe44b1bd960b68170b417450a628c43f7cf56bb3c5317e61cb230ee7f226" dependencies = [ "twox-hash", ] @@ -2473,6 +2511,16 @@ ] [[package]] +name = "num-bigint" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93e7820bc0a80a0238e650327316f929ba18d5be054b647490a3a6a339f3e7c0" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] name = "num-complex" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -2516,7 +2564,7 @@ checksum = "622acbc9100d3c10e2ee15804b0caa40e55c933d5aa53814cd520805b7958a49" dependencies = [ "async-trait", - "base64", + "base64 0.22.1", "bytes", "chrono", "form_urlencoded", @@ -2527,7 +2575,7 @@ "http-body-util", "humantime", "hyper", - "itertools", + "itertools 0.14.0", "md-5 0.10.6", "parking_lot", "percent-encoding", @@ -2561,15 +2609,6 @@ checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" [[package]] -name = "ordered-float" -version = "2.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68f19d67e5a2795c94e73e0bb1cc1a7edeb2e28efd39e2e1c9b7a40c1108b11c" -dependencies = [ - "num-traits", -] - -[[package]] name = "parking_lot" version = "0.12.5" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -2594,9 +2633,9 @@ [[package]] name = "parquet" -version = "58.3.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5dafa7d01085b62a47dd0c1829550a0a36710ea9c4fe358a05a85477cec8a908" +checksum = "7065842956a20c2a536924ce8e4d9955f7422451511b9eb7500d7bfe5077e59c" dependencies = [ "ahash", "arrow-array", @@ -2605,7 +2644,7 @@ "arrow-ipc", "arrow-schema", "arrow-select", - "base64", + "base64 0.23.1", "brotli", "bytes", "chrono", @@ -2614,33 +2653,25 @@ "half", "hashbrown 0.17.1", "lz4_flex", - "num-bigint", + "num-bigint 0.5.1", "num-integer", "num-traits", "object_store", - "paste", "seq-macro", "simdutf8", "snap", - "thrift", "tokio", "twox-hash", "zstd", ] [[package]] -name = "paste" -version = "1.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" - -[[package]] name = "pbjson" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "898bac3fa00d0ba57a4e8289837e965baa2dee8c3749f3b11d45a64b4223d9c3" dependencies = [ - "base64", + "base64 0.22.1", "serde", ] @@ -2651,7 +2682,7 @@ checksum = "af22d08a625a2213a78dbb0ffa253318c5c79ce3133d32d296655a7bdfb02095" dependencies = [ "heck", - "itertools", + "itertools 0.14.0", "prost", "prost-types", ] @@ -2724,7 +2755,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -2764,7 +2795,7 @@ checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn", + "syn 2.0.117", ] [[package]] @@ -2793,7 +2824,7 @@ checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" dependencies = [ "heck", - "itertools", + "itertools 0.14.0", "log", "multimap", "petgraph", @@ -2801,7 +2832,7 @@ "prost", "prost-types", "regex", - "syn", + "syn 2.0.117", "tempfile", ] @@ -2812,10 +2843,10 @@ checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" dependencies = [ "anyhow", - "itertools", + "itertools 0.14.0", "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3059,7 +3090,7 @@ checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3116,7 +3147,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" dependencies = [ - "base64", + "base64 0.22.1", "bytes", "futures-core", "futures-util", @@ -3292,7 +3323,7 @@ "proc-macro2", "quote", "serde_derive_internals", - "syn", + "syn 2.0.117", ] [[package]] @@ -3367,7 +3398,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3378,7 +3409,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3404,7 +3435,7 @@ "proc-macro2", "quote", "serde", - "syn", + "syn 2.0.117", ] [[package]] @@ -3525,7 +3556,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3556,7 +3587,7 @@ "heck", "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3581,7 +3612,7 @@ "serde", "serde_json", "serde_yaml", - "syn", + "syn 2.0.117", "typify", "walkdir", ] @@ -3604,6 +3635,17 @@ ] [[package]] +name = "syn" +version = "3.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6275cddf4610d1775e6d1fe9469b2e77d0f39fd98fb7450901b821e0c53649f" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] name = "sync_wrapper" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -3620,7 +3662,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3662,7 +3704,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3673,18 +3715,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", -] - -[[package]] -name = "thrift" -version = "0.17.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" -dependencies = [ - "byteorder", - "integer-encoding", - "ordered-float", + "syn 2.0.117", ] [[package]] @@ -3744,7 +3775,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3858,7 +3889,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -3916,7 +3947,7 @@ "semver", "serde", "serde_json", - "syn", + "syn 2.0.117", "thiserror 2.0.18", "unicode-ident", ] @@ -3934,7 +3965,7 @@ "serde", "serde_json", "serde_tokenstream", - "syn", + "syn 2.0.117", "typify-impl", ] @@ -4094,7 +4125,7 @@ "bumpalo", "proc-macro2", "quote", - "syn", + "syn 2.0.117", "wasm-bindgen-shared", ] @@ -4204,7 +4235,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -4215,7 +4246,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -4500,7 +4531,7 @@ "heck", "indexmap", "prettyplease", - "syn", + "syn 2.0.117", "wasm-metadata", "wit-bindgen-core", "wit-component", @@ -4516,7 +4547,7 @@ "prettyplease", "proc-macro2", "quote", - "syn", + "syn 2.0.117", "wit-bindgen-core", "wit-bindgen-rust", ] @@ -4583,7 +4614,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", "synstructure", ] @@ -4604,7 +4635,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -4624,7 +4655,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", "synstructure", ] @@ -4664,7 +4695,7 @@ dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]]
diff --git a/Cargo.toml b/Cargo.toml index 57256e6..5ae6078 100644 --- a/Cargo.toml +++ b/Cargo.toml
@@ -37,15 +37,15 @@ # and add per-crate flags (optional, features, default-features) at the use # site. [workspace.dependencies] -arrow = { version = "58", features = ["ffi"] } +arrow = { version = "59", features = ["ffi"] } async-trait = "0.1" -datafusion = { version = "54.1.0" } -datafusion-proto = "54.1.0" -datafusion-spark = "54.1.0" -datafusion-substrait = "54.1.0" +datafusion = { version = "55.0.0" } +datafusion-proto = "55.0.0" +datafusion-spark = "55.0.0" +datafusion-substrait = "55.0.0" futures = "0.3" jni = "0.21" -# Pinned to the major DataFusion 54.1 pulls in transitively (0.13.x) so we +# Pinned to the major DataFusion 55.0 pulls in transitively (0.13.x) so we # share the same `dyn ObjectStore` vtable and don't double-link. object_store = { version = "0.13", default-features = false } prost = "0.14"
diff --git a/core/pom.xml b/core/pom.xml index ee2cb65..21512d0 100644 --- a/core/pom.xml +++ b/core/pom.xml
@@ -129,7 +129,7 @@ <url>https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto-common/proto/datafusion_common.proto</url> <outputDirectory>${project.build.directory}/proto/datafusion/proto-common/proto</outputDirectory> <outputFileName>datafusion_common.proto</outputFileName> - <sha512>e6a20115badec14f641b9237b0faa3f553162992444208322558ea3fcfaa24ca15334e4945827259dcb7c1de7f9aed659d496974eb8eeb1cfcf54ad4d7f6d20f</sha512> + <sha512>0e6db862693eb687b5a77ca80dc0465d64a40d396cfaf39d864f6062b174c3818adbf68f04c4a7b03b7aad6aadcf319303dcd1c8c41929e36e00cea5e9e2dc08</sha512> </configuration> </execution> <execution> @@ -137,10 +137,10 @@ <phase>generate-sources</phase> <goals><goal>wget</goal></goals> <configuration> - <url>https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto/proto/datafusion.proto</url> + <url>https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto-models/proto/datafusion.proto</url> <outputDirectory>${project.build.directory}/proto/datafusion/proto/proto</outputDirectory> <outputFileName>datafusion.proto</outputFileName> - <sha512>bdae5361d32b4d4f0418677ce650ee8cc212fddfe9390b2c033ee4ab3bb3c969a6148b7b318b9bb268ee9384f18d4254ebe6e0701f15639059a8f2800d6e7847</sha512> + <sha512>05f98d5245ac46b81f9a897d16add727538697dce252796586020220c240679673ec6489242ad0d30240d911cd18caa867577baddd018495ac0e5a86086d3728</sha512> </configuration> </execution> </executions>
diff --git a/core/src/test/java/org/apache/datafusion/proto/SessionContextProtoTest.java b/core/src/test/java/org/apache/datafusion/proto/SessionContextProtoTest.java index 7bb65e6..ebb304b 100644 --- a/core/src/test/java/org/apache/datafusion/proto/SessionContextProtoTest.java +++ b/core/src/test/java/org/apache/datafusion/proto/SessionContextProtoTest.java
@@ -151,7 +151,11 @@ .build()) .setParquet( DatafusionCommon.ParquetFormat.getDefaultInstance()) - .setTargetPartitions(1) + // No target_partitions: DataFusion 55 + // removed the field from + // ListingTableScanNode (now + // `reserved 9`) and takes the partition + // count from the session config. .build()) .build()) .addExpr(
diff --git a/docs/source/contributor-guide/updating-datafusion-version.md b/docs/source/contributor-guide/updating-datafusion-version.md index 6e3b90b..4c14129 100644 --- a/docs/source/contributor-guide/updating-datafusion-version.md +++ b/docs/source/contributor-guide/updating-datafusion-version.md
@@ -46,7 +46,7 @@ # <sha512> elements in pom.xml. NEW=$(grep -m1 -oE '<datafusion.version>[^<]+' pom.xml | cut -d'>' -f2) curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto-common/proto/datafusion_common.proto" | shasum -a 512 | awk '{print $1}' -curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto/proto/datafusion.proto" | shasum -a 512 | awk '{print $1}' +curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto-models/proto/datafusion.proto" | shasum -a 512 | awk '{print $1}' $EDITOR pom.xml # paste the two hashes into the <sha512> elements # Drop the local download cache so the next build re-downloads against
diff --git a/native/src/cache_manager.rs b/native/src/cache_manager.rs index 4be7070..f4fc3e2 100644 --- a/native/src/cache_manager.rs +++ b/native/src/cache_manager.rs
@@ -20,17 +20,18 @@ //! //! Each of the three caches is independent; an unset proto field leaves the //! corresponding upstream default in place (no cache for list-files / stats, -//! a `DefaultFilesMetadataCache` with the default limit for file metadata). -//! When a setter *is* present, the JNI layer always installs a fresh -//! `Default*Cache` impl -- the v1 contract is "configure the built-in -//! caches", not "swap in a custom one". +//! a default file-metadata cache with the default limit). +//! +//! We only ever set the *limits* (and the list-files TTL), never the cache +//! slots themselves: `CacheManager::try_new` constructs the built-in +//! `DefaultCache` for any slot left `None` whose limit is non-zero. That is +//! the v1 contract -- "configure the built-in caches", not "swap in a custom +//! one" -- and it keeps us off the concrete cache types, which DataFusion 55 +//! collapsed into a single generic `DefaultCache<K, V>`. -use std::sync::Arc; use std::time::Duration; use datafusion::execution::cache::cache_manager::CacheManagerConfig; -use datafusion::execution::cache::file_statistics_cache::DefaultFileStatisticsCache; -use datafusion::execution::cache::{DefaultFilesMetadataCache, DefaultListFilesCache}; use crate::proto_gen::CacheManagerOptionsProto; use datafusion_jni_common::errors::JniResult; @@ -56,9 +57,7 @@ let mut config = CacheManagerConfig::default(); if let Some(max_bytes) = opts.file_metadata_cache_max_bytes { - let max = max_bytes as usize; - config.file_metadata_cache = Some(Arc::new(DefaultFilesMetadataCache::new(max))); - config.metadata_cache_limit = max; + config.metadata_cache_limit = max_bytes as usize; } if let Some(lfc) = &opts.list_files_cache { @@ -69,23 +68,19 @@ let max = lfc.max_bytes.map(|v| v as usize).unwrap_or(default_limit); let ttl = lfc.ttl_millis.map(Duration::from_millis); - config.list_files_cache = Some(Arc::new(DefaultListFilesCache::new(max, ttl))); config.list_files_cache_limit = max; config.list_files_cache_ttl = ttl; } // DataFusion 54 renamed `table_files_statistics_cache` to // `file_statistics_cache` and made the *limit* the on/off switch: - // `CacheManager::try_new` now builds a `DefaultFileStatisticsCache` - // whenever `file_statistics_cache_limit > 0`, even when the cache slot is - // `None`. The default limit is non-zero, so an explicit `false` from the - // Java surface has to zero the limit -- otherwise upstream would install a + // `CacheManager::try_new` builds a default file-statistics cache whenever + // `file_statistics_cache_limit > 0`, even when the cache slot is `None`. + // The default limit is non-zero, so an explicit `false` from the Java + // surface has to zero the limit -- otherwise upstream would install a // stats cache the caller just asked us not to. if let Some(enabled) = opts.file_statistics_cache_enabled { - if enabled { - config.file_statistics_cache = Some(Arc::new(DefaultFileStatisticsCache::default())); - } else { - config.file_statistics_cache = None; + if !enabled { config.file_statistics_cache_limit = 0; } }
diff --git a/native/src/proto.rs b/native/src/proto.rs index c1315f9..bccaa2c 100644 --- a/native/src/proto.rs +++ b/native/src/proto.rs
@@ -80,7 +80,7 @@ let bytes: Vec<u8> = env.convert_byte_array(&plan_bytes)?; let plan = runtime().block_on(async { - let substrait_plan = datafusion_substrait::serializer::deserialize_bytes(bytes).await?; + let substrait_plan = datafusion_substrait::serializer::deserialize_bytes(&bytes)?; datafusion_substrait::logical_plan::consumer::from_substrait_plan( &ctx.state(), &substrait_plan,
diff --git a/native/src/table_provider.rs b/native/src/table_provider.rs index 51c892b..4f54e30 100644 --- a/native/src/table_provider.rs +++ b/native/src/table_provider.rs
@@ -30,11 +30,12 @@ use datafusion::arrow::ffi_stream::{ArrowArrayStreamReader, FFI_ArrowArrayStream}; use datafusion::arrow::record_batch::{RecordBatch, RecordBatchReader}; use datafusion::catalog::Session; +use datafusion::common::tree_node::TreeNodeRecursion; use datafusion::datasource::{TableProvider, TableType}; use datafusion::error::{DataFusionError, Result}; use datafusion::execution::context::TaskContext; use datafusion::logical_expr::Expr; -use datafusion::physical_expr::EquivalenceProperties; +use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr}; use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType}; use datafusion::physical_plan::stream::RecordBatchStreamAdapter; use datafusion::physical_plan::{ @@ -155,6 +156,15 @@ vec![] } + /// A leaf scan: the Java side owns the row production, so there are no + /// `PhysicalExpr`s on this node for the visitor to descend into. + fn apply_expressions( + &self, + _f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>, + ) -> Result<TreeNodeRecursion> { + Ok(TreeNodeRecursion::Continue) + } + fn with_new_children( self: Arc<Self>, _children: Vec<Arc<dyn ExecutionPlan>>,
diff --git a/pom.xml b/pom.xml index d9f3212..f561357 100644 --- a/pom.xml +++ b/pom.xml
@@ -40,7 +40,7 @@ <maven.compiler.target>17</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <junit.version>5.11.3</junit.version> - <datafusion.version>54.1.0</datafusion.version> + <datafusion.version>55.0.0</datafusion.version> <protobuf.version>3.25.5</protobuf.version> <arrow.version>19.0.0</arrow.version> <avro.version>1.12.0</avro.version>