The version number in Cargo.toml should be increased, according to semver.
Define release branch (e.g. main), base version tag (e.g. 0.6.0) and future version tag (e.g. 0.7.0). Commits between the base version tag and the release branch will be used to populate the changelog content.
You will need a GitHub Personal Access Token for the following steps. Follow these instructions to generate one if you do not already have one.
# create the changelog CHANGELOG_GITHUB_TOKEN=<TOKEN> ./dev/release/update_change_log-datafusion-python.sh main 0.7.0 0.6.0 # review change log / edit issues and labels if needed, rerun until you are happy with the result git commit -a -m 'Create changelog for release'
If you see the error "You have exceeded a secondary rate limit" when running this script, try reducing the CPU allocation to slow the process down and throttle the number of GitHub requests made per minute, by modifying the value of the --cpus argument in the update_change_log.sh script.
You can add invalid or development-process label to exclude items from release notes.
Send a PR to get these changes merged into main branch. If new commits that could change the change log content landed in the main branch before you could merge the PR, you need to rerun the changelog update script to regenerate the changelog and update the PR accordingly.
git tag 0.7.0-rc1 git push apache 0.7.0-rc1
./dev/release/create_tarball 0.7.0 1
This will also create the email template to send to the mailing list. Here is an example:
To: dev@arrow.apache.org Subject: [VOTE][RUST][DataFusion] Release DataFusion Python Bindings 0.7.0 RC2 Hi, I would like to propose a release of Apache Arrow DataFusion Python Bindings, version 0.7.0. This release candidate is based on commit: bd1b78b6d444b7ab172c6aec23fa58c842a592d7 [1] The proposed release tarball and signatures are hosted at [2]. The changelog is located at [3]. The Python wheels are located at [4]. Please download, verify checksums and signatures, run the unit tests, and vote on the release. The vote will be open for at least 72 hours. Only votes from PMC members are binding, but all members of the community are encouraged to test the release and vote with "(non-binding)". The standard verification procedure is documented at https://github.com/apache/arrow-datafusion-python/blob/main/dev/release/README.md#verifying-release-candidates. [ ] +1 Release this as Apache Arrow DataFusion Python 0.7.0 [ ] +0 [ ] -1 Do not release this as Apache Arrow DataFusion Python 0.7.0 because... Here is my vote: +1 [1]: https://github.com/apache/arrow-datafusion-python/tree/bd1b78b6d444b7ab172c6aec23fa58c842a592d7 [2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-datafusion-python-0.7.0-rc2 [3]: https://github.com/apache/arrow-datafusion-python/blob/bd1b78b6d444b7ab172c6aec23fa58c842a592d7/CHANGELOG.md [4]: https://test.pypi.org/project/datafusion/0.7.0/
Create a draft email using this content, but do not send until after completing the next step.
To securely upload your project, you’ll need a PyPI API token. Create one at https://test.pypi.org/manage/account/#api-tokens, setting the “Scope” to “Entire account”.
You will also need access to the datafusion project on testpypi.
This section assumes some familiary with publishing Python packages to PyPi. For more information, refer to
this tutorial.
Pushing an rc tag to main will cause a GitHub Workflow to run that will build the Python wheels.
Go to https://github.com/apache/arrow-datafusion-python/actions and look for an action named “Python Release Build” that has run against the pushed tag.
Click on the action and scroll down to the bottom of the page titled “Artifacts”. Download dist.zip.
Upload the wheels to testpypi.
unzip dist.zip python3 -m pip install --upgrade setuptools twine build python3 -m twine upload --repository testpypi datafusion-0.7.0-cp37-abi3-*.whl
When prompted for username, enter __token__. When prompted for a password, enter a valid GitHub Personal Access Token
Download the source tarball created in the previous step, untar it, and run:
python3 -m build
This will create a file named dist/datafusion-0.7.0.tar.gz. Upload this to testpypi:
python3 -m twine upload --repository testpypi dist/datafusion-0.7.0.tar.gz
Publishing artifacts to Anaconda is similar to PyPi. First, Download the source tarball created in the previous step and untar it.
# Assuming you have an existing conda environment named `datafusion-dev` if not see root README for instructions conda activate datafusion-dev conda build .
This will setup a virtual conda environment and build the artifacts inside of that virtual env. This step can take a few minutes as the entire build, host, and runtime environments are setup. Once complete a local filesystem path will be emitted for the location of the resulting package. Observe that path and copy to your clipboard.
Ex: /home/conda/envs/datafusion/conda-bld/linux-64/datafusion-0.7.0.tar.bz2
Now you are ready to publish this resulting package to anaconda.org. This can be accomplished in a few simple steps.
# First login to Anaconda with the datafusion credentials anaconda login # Upload the package anaconda upload /home/conda/envs/datafusion/conda-bld/linux-64/datafusion-0.7.0.tar.bz2
Send the email to start the vote.
Install the release from testpypi:
pip install --extra-index-url https://test.pypi.org/simple/ datafusion==0.7.0
Try running one of the examples from the top-level README, or write some custom Python code to query some available data files.
Once the vote passes, we can publish the release.
Create the source release tarball:
./dev/release/release-tarball.sh 0.7.0 1
Go to the Test PyPI page of Datafusion, and download all published artifacts under dist-release/ directory. Then proceed uploading them using twine:
twine upload --repository pypi dist-release/*
git checkout 0.7.0-rc1 git tag 0.7.0 git push apache 0.7.0