This guide is for maintainers creating a release of Apache DataFusion Iceberg.
As part of the Apache governance model, official releases consist of signed source tarballs approved by the DataFusion PMC. The iceberg-datafusion crate is then published to crates.io from the approved source.
apache repoThe instructions below assume the upstream git repo git@github.com:apache/datafusion-iceberg.git in remote apache.
git remote add apache git@github.com:apache/datafusion-iceberg.git
gpg and svn to sign and upload release candidates.java and python3 for the license check that create-tarball.sh runs.uv, or the PyGithub Python package, for the changelog script.A personal access token (PAT) is needed for the changelog script. If you do not already have one, create a token with repo access by navigating to the GitHub Developer Settings page, and follow these steps.
KEYS fileDataFusion subprojects share one KEYS file. If you will be releasing the final tarball, your GPG public key must be present in:
See https://infra.apache.org/release-signing.html#generate for instructions on generating keys.
Committers can add signing keys using the Subversion client and their ASF account:
$ svn co https://dist.apache.org/repos/dist/dev/datafusion $ cd datafusion $ editor KEYS # add your key here $ svn ci KEYS # commit changes
Follow the instructions in the header of the KEYS file to append your key. Here is an example:
(gpg --list-sigs "John Doe" && gpg --armor --export "John Doe") >> KEYS svn commit KEYS -m "Add key for John Doe"
The examples below release version 0.11.0.
crates.io rejects crates with git dependencies, so every dependency of iceberg-datafusion must come from crates.io. In particular, iceberg and iceberg-catalog-rest in the root Cargo.toml must point to a released iceberg-rust version, not a git revision. Check with:
cargo publish --dry-run -p iceberg-datafusion
Update version under [workspace.package] in the root Cargo.toml. It must be higher than the latest version of iceberg-datafusion on crates.io, which includes releases made from apache/iceberg-rust. Then update Cargo.lock:
cargo check --workspace
Commit the changes and create a PR against main.
Each release has its own changelog file in dev/changelog/, such as dev/changelog/0.11.0.md, listing all changes since the previous release.
Generate it with generate-changelog.py. Pass the previous release tag, the commit to release, and the new version. For the first release from this repository, which has no earlier tag, pass the commit to start from instead of a tag. Run the script from the repository root with the GITHUB_TOKEN environment variable set; without a token, GitHub's API rate limit causes 403 errors.
export GITHUB_TOKEN=<your-token-here> uv run dev/release/generate-changelog.py <previous-release-tag> apache/main 0.11.0 > dev/changelog/0.11.0.md
Without uv, install PyGithub and run the script with python3 instead.
The script groups PRs using their labels and Conventional Commits title prefixes (feat:, fix:, docs:, perf:, and ! for breaking changes).
Commit the changelog and create a PR against main.
You must be a committer to run these steps because they upload to the Apache SVN distribution servers.
Pick numbers in sequential order, with 1 for rc1, 2 for rc2, etc.
The official release artifacts are signed tarballs, but we also tag the commit they were created from. Release tags look like 0.11.0, and release candidate tags look like 0.11.0-rc1.
git fetch apache git tag 0.11.0-rc1 apache/main git push apache 0.11.0-rc1
./dev/release/create-tarball.sh 0.11.0 1
The script:
Cargo.toml has the release version.git archive and runs the RAT license check on it.Send the email printed by create-tarball.sh to dev@datafusion.apache.org.
To become official, the release needs at least three +1 votes from PMC members and no -1 votes. The vote must stay open for at least 72 hours so everyone has a chance to review the release candidate.
verify-release-candidate.sh downloads a release candidate and checks its signature and checksums. It then builds and tests it in a sandbox, using the toolchain pinned in rust-toolchain.toml, and runs cargo publish --dry-run:
./dev/release/verify-release-candidate.sh 0.11.0 1
To check license headers as well, run ./dev/release/run-rat.sh on the downloaded tarball.
Merge the fixes to main, update the changelog, and start again from step 4 with the next RC number.
Reply to the vote thread with the [RESULT] prefix added to the subject line. For example:
The vote has passed with <NUMBER> +1 votes. Thank you to all who helped with the release verification.
Only PMC members can do this step, after the release is approved.
Move the artifacts to the release location in SVN, e.g. https://dist.apache.org/repos/dist/release/datafusion/datafusion-iceberg-0.11.0/, using release-tarball.sh:
./dev/release/release-tarball.sh 0.11.0 1
Congratulations! The release is now official!
Tag the same commit as the approved release candidate:
git checkout 0.11.0-rc1 git tag 0.11.0 git push apache 0.11.0
Only approved releases of the tarball may be published to crates.io, in order to conform to Apache Software Foundation governance standards.
Follow these instructions to create an account and log in to crates.io. Then ask an existing owner of iceberg-datafusion to add you; cargo owner --list iceberg-datafusion shows the current owners.
Download and unpack the official release tarball, check that its Cargo.toml has the release version, and publish from the unpacked directory:
cargo publish -p iceberg-datafusion
iceberg-datafusion is the only published crate. iceberg-sqllogictest and iceberg-playground set publish = false.
Add the release to Apache Reporter, following the examples from previous releases. The reporter system should send you a reminder email. The release information is used to generate a template for the DataFusion board report.
See the ASF documentation on when to archive for more information.
Release candidates should be deleted once the release is published.
To list DataFusion Iceberg release candidates:
svn ls https://dist.apache.org/repos/dist/dev/datafusion | grep datafusion-iceberg
To delete a release candidate:
svn delete -m "delete old DataFusion Iceberg RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-iceberg-0.11.0-rc1/
Only the latest release should be available from the release area. Delete old releases after publishing a new one.
To list DataFusion Iceberg releases:
svn ls https://dist.apache.org/repos/dist/release/datafusion | grep datafusion-iceberg
To delete a release:
svn delete -m "delete old DataFusion Iceberg release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-iceberg-0.11.0
CI runs the Apache Release Audit Tool (RAT) on every pull request. To run it locally, which requires java and python3:
./dev/release/run-rat.sh .
Files that cannot carry a license header are listed in dev/release/rat_exclude_files.txt.