blob: 6e3b90b9d0688245001102d7641005b6d5d596e7 [file] [view]
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Updating the DataFusion / protobuf schema version
Three things must move together when bumping DataFusion:
1. `Cargo.toml` (workspace root) — the `datafusion`, `datafusion-proto`,
`datafusion-spark`, and `datafusion-substrait` entries in
`[workspace.dependencies]`. Members inherit from there.
2. `pom.xml` — the `<datafusion.version>` Maven property. **Must equal
the Cargo version**; a mismatch means JVM-built protobuf plans won't
deserialize on the native side.
3. `pom.xml` — the `<sha512>` checksums on the two `download-maven-plugin`
executions. These pin the downloaded `.proto` files; the build fails
if upstream silently re-tags them, which is the desired behavior.
## Recipe
```sh
# 1. Bump the workspace dep
$EDITOR Cargo.toml # set datafusion = "<new>" in [workspace.dependencies]
cargo update -p datafusion
# 2. Bump the Maven property to match
$EDITOR pom.xml # set <datafusion.version>
# 3. Compute the new SHA-512 hashes for both `.proto` files from the
# upstream tag you just set in step 2, then paste them into the two
# <sha512> elements in pom.xml.
NEW=$(grep -m1 -oE '<datafusion.version>[^<]+' pom.xml | cut -d'>' -f2)
curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto-common/proto/datafusion_common.proto" | shasum -a 512 | awk '{print $1}'
curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto/proto/datafusion.proto" | shasum -a 512 | awk '{print $1}'
$EDITOR pom.xml # paste the two hashes into the <sha512> elements
# Drop the local download cache so the next build re-downloads against
# the new hashes.
rm -rf ~/.m2/repository/.cache/download-maven-plugin target/proto
# 4. Verify
make && make test
```
## Why the protobuf runtime version is separate
The protobuf runtime version (`<protobuf.version>` in `pom.xml`) tracks
the Java ecosystem (security and JDK compatibility), not DataFusion.
Bump it independently when there is a reason.