blob: d9e93ac877ac987d4d934bfed1d741512211f583 [file] [view]
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Contributing to Apache DataFusion Java
Bug reports, design discussion, and patches are welcome. This project follows
the Apache DataFusion contribution model.
## Filing issues and discussing changes
- File bugs and feature requests on [GitHub issues](https://github.com/apache/datafusion-java/issues).
- For larger or design-level discussion, the mailing list is
[dev@datafusion.apache.org](mailto:dev@datafusion.apache.org).
- Please open an issue before sending a PR for any significant change so the
approach can be agreed on first.
## Development workflow
Branch from `main`, write changes with [conventional commit](https://www.conventionalcommits.org/)
messages in the imperative mood (e.g. `feat: add foo`, `fix(native): handle bar`),
and open a pull request targeting `main`.
The first build in a fresh checkout reaches out to `raw.githubusercontent.com`
to fetch the DataFusion protobuf schemas (see *Updating the DataFusion /
protobuf schema version* below). Subsequent builds are offline — the
`download-maven-plugin` cache under `~/.m2/repository/.cache/` satisfies them.
## Build prerequisites
- JDK 17 or newer.
- Rust toolchain (stable, installed via [rustup]).
- [`tpchgen-cli`] — only needed to generate test data for the Parquet
integration test (`cargo install tpchgen-cli`).
Maven is bundled via the `./mvnw` wrapper; no separate Maven install required.
[rustup]: https://rustup.rs/
[`tpchgen-cli`]: https://github.com/clflushopt/tpchgen-rs
## Build and test
make test
This builds the native Rust crate and runs the JUnit tests. The steps can be
run individually:
cd native && cargo build
./mvnw test
The native library must be built before running JVM tests.
## Test data
The Parquet integration test reads TPC-H SF1 data (~345 MB across 8 tables in
Snappy-compressed Parquet). Generate it once with:
make tpch-data
Tests that need this data skip cleanly if it is missing. `make clean` does
**not** remove `tpch-data/` — delete it manually to reclaim the disk space.
## Code style
- Java: run `./mvnw spotless:apply` before committing. CI fails the build if
formatting drifts.
- Rust: run `cargo fmt` and `cargo clippy --all-targets -- -D warnings` inside
`native/`.
- New source files need the Apache 2.0 license header. Apache RAT enforces this
during `verify`.
## Updating the DataFusion / protobuf schema version
Three things must move together when bumping DataFusion:
1. `native/Cargo.toml` — the `datafusion` crate dependency.
2. `pom.xml` — the `<datafusion.version>` Maven property. **Must equal the
Cargo version**; a mismatch means JVM-built protobuf plans won't deserialize
on the native side.
3. `pom.xml` — the `<sha512>` checksums on the two `download-maven-plugin`
executions. These pin the downloaded `.proto` files; the build fails if
upstream silently re-tags them, which is the desired behavior.
Recipe:
```sh
# 1. Bump the Cargo dep
$EDITOR native/Cargo.toml # set datafusion = "<new>"
(cd native && cargo update -p datafusion)
# 2. Bump the Maven property to match
$EDITOR pom.xml # set <datafusion.version>
# 3. Compute the new SHA-512 hashes for both `.proto` files from the upstream
# tag you just set in step 2, then paste them into the two <sha512> elements
# in pom.xml.
NEW=$(grep -m1 -oE '<datafusion.version>[^<]+' pom.xml | cut -d'>' -f2)
curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto-common/proto/datafusion_common.proto" | shasum -a 512 | awk '{print $1}'
curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto/proto/datafusion.proto" | shasum -a 512 | awk '{print $1}'
$EDITOR pom.xml # paste the two hashes into the <sha512> elements
# Drop the local download cache so the next build re-downloads against the new hashes.
rm -rf ~/.m2/repository/.cache/download-maven-plugin target/proto
# 4. Verify
make && make test
```
The protobuf runtime version (`<protobuf.version>` in `pom.xml`) tracks the
Java ecosystem (security and JDK compatibility), not DataFusion. Bump it
independently when there is a reason.