DataFusion typically has major releases every two weeks, including breaking API changes.
Patch releases are made on an adhoc basis, but we try and avoid them given the frequent major releases.
branch-18 in the Apache repository (not in a fork)branch-* branch?If you would like to propose your change for inclusion in a release branch
main branch and wait for its approval and merges.main, branch from most recent release branch (e.g. branch-18), cherry-pick the commit and create a PR to release branch.git@github.com:apache/arrow-datafusion.git add as git remote apache.repo accessSee instructions at https://infra.apache.org/release-signing.html#generate for generating keys.
Committers can add signing keys in Subversion client with their ASF account. e.g.:
$ svn co https://dist.apache.org/repos/dist/dev/arrow $ cd arrow $ editor KEYS $ svn ci KEYS
Follow the instructions in the header of the KEYS file to append your key. Here is an example:
(gpg --list-sigs "John Doe" && gpg --armor --export "John Doe") >> KEYS svn commit KEYS -m "Add key for John Doe"
As part of the Apache governance model, official releases consist of signed source tarballs approved by the PMC.
We then use the code in the approved artifacts to release to crates.io and PyPI.
We maintain a CHANGELOG.md so our users know what has been changed between releases.
The CHANGELOG is managed automatically using update_change_log.sh
This script creates a changelog using GitHub PRs and issues based on the labels associated with them.
Prepare a PR to update CHANGELOG.md and versions to reflect the planned release.
See #801 for an example.
Here are the commands that could be used to prepare the 5.1.0 release:
Checkout the main commit to be released
git fetch apache git checkout apache/main
Update datafusion version in datafusion/Cargo.toml to 5.1.0:
./dev/update_datafusion_versions.py 5.1.0
Lastly commit the version change:
git commit -a -m 'Update version'
Define release branch (e.g. main), base version tag (e.g. 7.0.0) and future version tag (e.g. 8.0.0). Commits between the base version tag and the release branch will be used to populate the changelog content.
You will need a GitHub Personal Access Token for the following steps. Follow these instructions to generate one if you do not already have one.
# create the changelog CHANGELOG_GITHUB_TOKEN=<TOKEN> ./dev/release/update_change_log-datafusion.sh main 8.0.0 7.0.0 # review change log / edit issues and labels if needed, rerun until you are happy with the result git commit -a -m 'Create changelog for release'
If you see the error "You have exceeded a secondary rate limit" when running this script, try reducing the CPU allocation to slow the process down and throttle the number of GitHub requests made per minute, by modifying the value of the --cpus argument in the update_change_log.sh script.
You can add invalid or development-process label to exclude items from release notes.
Send a PR to get these changes merged into main branch. If new commits that could change the change log content landed in the main branch before you could merge the PR, you need to rerun the changelog update script to regenerate the changelog and update the PR accordingly.
After the PR gets merged, you are ready to create release artifacts based off the merged commit.
(Note you need to be a committer to run these scripts as they upload to the apache svn distribution servers)
Pick numbers in sequential order, with 0 for rc0, 1 for rc1, etc.
While the official release artifacts are signed tarballs and zip files, we also tag the commit it was created for convenience and code archaeology.
Using a string such as 5.1.0 as the <version>, create and push the tag by running these commands:
git fetch apache git tag <version>-<rc> apache/main # push tag to Github remote git push apache <version>
Run create-tarball.sh with the <version> tag and <rc> and you found in previous steps:
GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 5.1.0 0
The create-tarball.sh script
creates and uploads all release candidate artifacts to the arrow dev location on the apache distribution svn server
provide you an email template to send to dev@arrow.apache.org for release voting.
Send the email output from the script to dev@arrow.apache.org. The email should look like
To: dev@arrow.apache.org Subject: [VOTE][DataFusion] Release Apache Arrow DataFusion 5.1.0 RC0 Hi, I would like to propose a release of Apache Arrow DataFusion Implementation, version 5.1.0. This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1] The proposed release artifacts and signatures are hosted at [2]. The changelog is located at [3]. Please download, verify checksums and signatures, run the unit tests, and vote on the release. The vote will be open for at least 72 hours. [ ] +1 Release this as Apache Arrow DataFusion 5.1.0 [ ] +0 [ ] -1 Do not release this as Apache Arrow DataFusion 5.1.0 because... Here is my vote: +1 [1]: https://github.com/apache/arrow-datafusion/tree/a5dd428f57e62db20a945e8b1895de91405958c4 [2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-datafusion-5.1.0 [3]: https://github.com/apache/arrow-datafusion/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
For the release to become “official” it needs at least three PMC members to vote +1 on it.
The dev/release/verify-release-candidate.sh is a script in this repository that can assist in the verification process. Run it like:
./dev/release/verify-release-candidate.sh 5.1.0 0
If the release is not approved, fix whatever the problem is, merge changelog changes into main if there is any and try again with the next RC number.
NOTE: steps in this section can only be done by PMC members.
Move artifacts to the release location in SVN, e.g. https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-5.1.0/, using the release-tarball.sh script:
./dev/release/release-tarball.sh 5.1.0 0
Congratulations! The release is now official!
Tag the same release candidate commit with the final release tag
git co apache/5.1.0-rc0 git tag 5.1.0 git push apache 5.1.0
Only approved releases of the tarball should be published to crates.io, in order to conform to Apache Software Foundation governance standards.
An Arrow committer can publish this crate after an official project release has been made to crates.io using the following instructions.
Follow these instructions to create an account and login to crates.io before asking to be added as an owner of the following crates:
Download and unpack the official release tarball
Verify that the Cargo.toml in the tarball contains the correct version (e.g. version = "5.1.0") and then publish the crates by running the script release-crates.sh in a directory extracted from the source tarball that was voted on. Note that this script doesn't work if run in a Git repo.
Alternatively the crates can be published one at a time with the following commands. Crates need to be published in the correct order as shown in this diagram.
To update this diagram, manually edit the dependencies in crate-deps.dot and then run:
dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg
(cd datafusion/common && cargo publish) (cd datafusion/expr && cargo publish) (cd datafusion/sql && cargo publish) (cd datafusion/jit && cargo publish) (cd datafusion/row && cargo publish) (cd datafusion/physical-expr && cargo publish) (cd datafusion/optimizer && cargo publish) (cd datafusion/execution && cargo publish) (cd datafusion/core && cargo publish) (cd datafusion/proto && cargo publish) (cd datafusion/substrait && cargo publish)
The CLI needs a --no-verify argument because build.rs generates source into the src directory.
(cd datafusion-cli && cargo publish --no-verify)
Run publish_homebrew.sh to publish datafusion-cli on Homebrew. In order to do so it is necessary to fork the homebrew-core repo https://github.com/Homebrew/homebrew-core/, have Homebrew installed on your macOS/Linux/WSL2 and properly configured and have a Github Personal Access Token that has permission to file pull requests in the homebrew-core repo.
homebrew-core repoGo to https://github.com/Homebrew/homebrew-core/ and fork the repo.
Please visit https://brew.sh/ to obtain Homebrew. In addition to that please check out https://docs.brew.sh/Homebrew-on-Linux if you are on Linux or WSL2.
Before running the script make sure that you can run the following command in your bash to make sure that brew has been installed and configured properly:
brew --version
To create a Github Personal Access Token, please visit https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token for instructions.
homebrew-core in the list of repos under Selected repositories.After all of the above is complete execute the following command:
dev/release/publish_homebrew.sh <version> <github-user> <github-token> <homebrew-default-branch-name>
Note that sometimes someone else has already submitted a PR to update the datafusion formula in homebrew. In this case you will get an error with a message that your PR is a duplicate of an existing one. In this case no further action is required.
Alternatively manually submit a simple PR to update tag and commit hash for the datafusion formula in homebrew-core. Here is an example PR: https://github.com/Homebrew/homebrew-core/pull/89562.
Call the vote on the Arrow dev list by replying to the RC voting thread. The reply should have a new subject constructed by adding [RESULT] prefix to the old subject line.
Sample announcement template:
The vote has passed with <NUMBER> +1 votes. Thank you to all who helped with the release verification.
You can include mention crates.io and PyPI version URLs in the email if applicable.
We have published new versions of DataFusion to crates.io: https://crates.io/crates/datafusion/8.0.0 https://crates.io/crates/datafusion-cli/8.0.0 https://crates.io/crates/datafusion-common/8.0.0 https://crates.io/crates/datafusion-expr/8.0.0 https://crates.io/crates/datafusion-jit/8.0.0 https://crates.io/crates/datafusion-optimizer/8.0.0 https://crates.io/crates/datafusion-physical-expr/8.0.0 https://crates.io/crates/datafusion-proto/8.0.0 https://crates.io/crates/datafusion-row/8.0.0 https://crates.io/crates/datafusion-sql/8.0.0
Add the release to https://reporter.apache.org/addrelease.html?arrow with a version name prefixed with RS-DATAFUSION-, for example RS-DATAFUSION-14.0.0.
The release information is used to generate a template for a board report (see example here).
See the ASF documentation on when to archive for more information.
dev svnRelease candidates should be deleted once the release is published.
Get a list of DataFusion release candidates:
svn ls https://dist.apache.org/repos/dist/dev/arrow | grep datafusion
Delete a release candidate:
svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-datafusion-7.1.0-rc1/
release svnOnly the latest release should be available. Delete old releases after publishing the new release.
Get a list of DataFusion releases:
svn ls https://dist.apache.org/repos/dist/release/arrow | grep datafusion
Delete a release:
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-7.0.0
build.sh in the docs directory from the release tarball.asf-site branchdocs/build/html/* to the datafusion directory in arrow-siteasf-site branch (example)We typically crowdsource release announcements by collaborating on a Google document, usually starting with a copy of the previous release announcement.
Run the following commands to get the number of commits and number of unique contributors for inclusion in the blog post.
git log --pretty=oneline 10.0.0..11.0.0 datafusion datafusion-cli datafusion-examples | wc -l git shortlog -sn 10.0.0..11.0.0 datafusion datafusion-cli datafusion-examples | wc -l
Once there is consensus on the contents of the post, create a PR to add a blog post to the arrow-site repository. Note that there is no need for a formal PMC vote on the blog post contents since this isn't considered to be a “release”.
Here is an example blog post PR:
Once the PR is merged, a GitHub action will publish the new blog post to https://arrow.apache.org/blog/.