| <!-- | |
| - Licensed to the Apache Software Foundation (ASF) under one | |
| - or more contributor license agreements. See the NOTICE file | |
| - distributed with this work for additional information | |
| - regarding copyright ownership. The ASF licenses this file | |
| - to you under the Apache License, Version 2.0 (the | |
| - "License"); you may not use this file except in compliance | |
| - with the License. You may obtain a copy of the License at | |
| - | |
| - http://www.apache.org/licenses/LICENSE-2.0 | |
| - | |
| - Unless required by applicable law or agreed to in writing, | |
| - software distributed under the License is distributed on an | |
| - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | |
| - KIND, either express or implied. See the License for the | |
| - specific language governing permissions and limitations | |
| - under the License. | |
| --> | |
| # How to Release | |
| This guide outlines the process for releasing PyIceberg in accordance with the [Apache Release Process](https://infra.apache.org/release-publishing.html). The steps include: | |
| 1. Preparing for a release | |
| 2. Publishing a Release Candidate (RC) | |
| 3. Community Voting and Validation | |
| 4. Publishing the Final Release (if the vote passes) | |
| 5. Post-Release Step | |
| ## Requirements | |
| * A GPG key must be registered and published in the [Apache Iceberg KEYS file](https://downloads.apache.org/iceberg/KEYS). Follow [the instructions for setting up a GPG key and uploading it to the KEYS file](#set-up-gpg-key-and-upload-to-apache-iceberg-keys-file). | |
| * Permission to update the `KEYS` artifact in the [Apache release distribution](https://dist.apache.org/repos/dist/release/iceberg/) (requires Iceberg PMC privileges). | |
| * SVN Access | |
| * Permission to upload artifacts to the [Apache development distribution](https://dist.apache.org/repos/dist/dev/iceberg/) (requires Apache Committer access). | |
| * Permission to upload artifacts to the [Apache release distribution](https://dist.apache.org/repos/dist/release/iceberg/) (requires Apache PMC access). | |
| * GitHub Access | |
| * Write access to the [apache/iceberg-python](https://github.com/apache/iceberg-python) repository for triggering Dependabot (requires Apache Committer access). | |
| * PyPI Access | |
| * The `twine` package must be installed for uploading releases to PyPi. | |
| * A PyPI account with publishing permissions for the [pyiceberg project](https://pypi.org/project/pyiceberg/). | |
| ## Preparing for a Release | |
| ### Remove Deprecated APIs | |
| Before running the release candidate, we want to remove any APIs that were marked for removal under the `@deprecated` tag for this release. See [#1269](https://github.com/apache/iceberg-python/pull/1269). | |
| For example, the API with the following deprecation tag should be removed when preparing for the 0.2.0 release. | |
| ```python | |
| @deprecated( | |
| deprecated_in="0.1.0", | |
| removed_in="0.2.0", | |
| help_message="Please use load_something_else() instead", | |
| ) | |
| ``` | |
| We also have the `deprecation_message` function. We need to change the behavior according to what is noted in the message of that deprecation. | |
| ```python | |
| deprecation_message( | |
| deprecated_in="0.1.0", | |
| removed_in="0.2.0", | |
| help_message="The old_property is deprecated. Please use the something_else property instead.", | |
| ) | |
| ``` | |
| ### Update Dependencies | |
| <!-- markdown-link-check-disable-next-line --> | |
| Dependabot runs monthly to keep the noise down, so the pinned dependencies in `uv.lock` may be stale by the time of a release. Before cutting the release candidate, go to the [Dependabot page](https://github.com/apache/iceberg-python/network/updates) in the repository's Insights tab and trigger a manual check for both the `uv` and `github-actions` ecosystems. Review and merge the resulting PRs so the release ships with up-to-date dependencies. | |
| <!-- prettier-ignore-start --> | |
| !!! note | |
| Only a committer with write access to the repository can trigger Dependabot manually. Please work with a committer if you do not have write access. | |
| <!-- prettier-ignore-end --> | |
| ### Update Library Version | |
| Update the release version by running `uv version <release-version>`, which updates both `pyproject.toml` and `uv.lock`. Then update the version in `pyiceberg/__init__.py` to match. | |
| ## Publishing a Release Candidate (RC) | |
| ### Release Types | |
| #### Major/Minor Release | |
| * Use the `main` branch for the release. | |
| * Includes new features, enhancements, and any necessary backward-compatible changes. | |
| * Examples: `0.8.0`, `0.9.0`, `1.0.0`. | |
| #### Patch Release | |
| * Use the branch corresponding to the patch version, such as `pyiceberg-0.8.x`. | |
| * Focuses on critical bug fixes or security patches that maintain backward compatibility. | |
| * Examples: `0.8.1`, `0.8.2`. | |
| To create a patch branch from the latest release tag: | |
| ```bash | |
| # Fetch all tags | |
| git fetch --tags | |
| # Assuming 0.8.0 is the latest release tag | |
| git checkout -b pyiceberg-0.8.x pyiceberg-0.8.0 | |
| # Cherry-pick commits for the upcoming patch release | |
| git cherry-pick <commit> | |
| # Push the new branch | |
| git push git@github.com:apache/iceberg-python.git pyiceberg-0.8.x | |
| ``` | |
| ### Create Tag | |
| Ensure you are on the correct branch: | |
| * For a major/minor release, use the `main` branch | |
| * For a patch release, use the branch corresponding to the patch version, i.e. `pyiceberg-0.6.x`. | |
| Create a signed tag: | |
| Replace `VERSION` and `RC` with the appropriate values for the release. | |
| ```bash | |
| export VERSION=0.7.0 | |
| export RC=1 | |
| export VERSION_WITH_RC=${VERSION}rc${RC} | |
| export GIT_TAG=pyiceberg-${VERSION_WITH_RC} | |
| git tag -s ${GIT_TAG} -m "PyIceberg ${VERSION_WITH_RC}" | |
| git push git@github.com:apache/iceberg-python.git ${GIT_TAG} | |
| ``` | |
| ### Create Artifacts | |
| The [`Python Build Release Candidate` Github Action](https://github.com/apache/iceberg-python/actions/workflows/python-release.yml) will run automatically upon tag push. | |
| This action will generate artifacts that will include both source distribution (`sdist`) and binary distributions (`wheels` using [`cibuildwheel`](https://github.com/pypa/cibuildwheel)) for each architectures. | |
| This action will generate two final artifacts: | |
| * `svn-release-candidate-${VERSION}rc${RC}` for SVN | |
| * `pypi-release-candidate-${VERSION}rc${RC}` for PyPi | |
| If `gh` is available, watch the GitHub Action progress using: | |
| ```bash | |
| : "${GIT_TAG:?ERROR: GIT_TAG is not set or is empty}" | |
| RUN_ID=$(gh run list --repo apache/iceberg-python --workflow "Python Build Release Candidate" --branch "${GIT_TAG}" --event push --json databaseId -q '.[0].databaseId') | |
| : "${RUN_ID:?ERROR: RUN_ID could not be determined}" | |
| echo "Waiting for workflow to complete, this will take several minutes..." | |
| gh run watch $RUN_ID --repo apache/iceberg-python | |
| ``` | |
| and download the artifacts using: | |
| ```bash | |
| : "${RUN_ID:?ERROR: RUN_ID is not set or is empty}" | |
| gh run download $RUN_ID --repo apache/iceberg-python | |
| ``` | |
| ### Publish Release Candidate (RC) | |
| #### Upload to Apache Dev SVN | |
| ##### Download Artifacts, Sign, and Generate Checksums | |
| Download the SVN artifact from the GitHub Action and unzip it. | |
| Navigate to the artifact directory. Generate signature and checksum files: | |
| * `.asc` files: GPG-signed versions of each artifact to ensure authenticity. | |
| * `.sha512` files: SHA-512 checksums for verifying file integrity. | |
| ```bash | |
| : "${VERSION:?ERROR: VERSION is not set or is empty}" | |
| : "${RC:?ERROR: RC is not set or is empty}" | |
| ( | |
| cd svn-release-candidate-${VERSION}rc${RC} | |
| for name in $(ls pyiceberg-*.whl pyiceberg-*.tar.gz) | |
| do | |
| gpg --yes --armor --output "${name}.asc" --detach-sig "${name}" | |
| shasum -a 512 "${name}" > "${name}.sha512" | |
| done | |
| ) | |
| ``` | |
| The parentheses `()` create a subshell. Any changes to the directory (`cd`) are limited to this subshell, so the current directory in the parent shell remains unchanged. | |
| ##### Upload Artifacts to Apache Dev SVN | |
| Now, upload the files from the same directory: | |
| ```bash | |
| : "${VERSION:?ERROR: VERSION is not set or is empty}" | |
| : "${VERSION_WITH_RC:?ERROR: VERSION_WITH_RC is not set or is empty}" | |
| : "${RC:?ERROR: RC is not set or is empty}" | |
| svn import "svn-release-candidate-${VERSION}rc${RC}" "https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-${VERSION_WITH_RC}" -m "PyIceberg ${VERSION_WITH_RC}" | |
| ``` | |
| Verify the artifact is uploaded to [https://dist.apache.org/repos/dist/dev/iceberg](https://dist.apache.org/repos/dist/dev/iceberg/). | |
| ##### Remove Old Artifacts From Apache Dev SVN | |
| Clean up old RC artifacts: | |
| ```bash | |
| svn delete https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-<OLD_RC_VERSION> -m "Remove old RC artifacts" | |
| ``` | |
| #### Upload to PyPi | |
| ##### Download Artifacts | |
| Download the PyPi artifact from the GitHub Action and unzip it. | |
| ##### Upload Artifacts to PyPi | |
| Update the artifact directory to PyPi using `twine`. This **won't** bump the version for everyone that hasn't pinned their version, since it is set to an RC [pre-release and those are ignored](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#pre-release-versioning). | |
| <!-- prettier-ignore-start --> | |
| !!! note | |
| `twine` might require an PyPi API token. | |
| <!-- prettier-ignore-end --> | |
| ```bash | |
| : "${VERSION:?ERROR: VERSION is not set or is empty}" | |
| : "${RC:?ERROR: RC is not set or is empty}" | |
| twine upload pypi-release-candidate-${VERSION}rc${RC}/* | |
| ``` | |
| Verify the artifact is uploaded to [PyPi](https://pypi.org/project/pyiceberg/#history). | |
| ## Vote | |
| ### Generate Vote Email | |
| Final step is to generate the email to the dev mail list: | |
| ```bash | |
| : "${GIT_TAG:?ERROR: GIT_TAG is not set or is empty}" | |
| : "${VERSION:?ERROR: VERSION is not set or is empty}" | |
| : "${VERSION_WITH_RC:?ERROR: VERSION_WITH_RC is not set or is empty}" | |
| export GIT_TAG_REF=$(git show-ref ${GIT_TAG}) | |
| export GIT_TAG_HASH=${GIT_TAG_REF:0:40} | |
| export LAST_COMMIT_ID=$(git rev-list ${GIT_TAG} 2> /dev/null | head -n 1) | |
| cat << EOF > release-announcement-email.txt | |
| To: dev@iceberg.apache.org | |
| Subject: [VOTE] PyIceberg $VERSION_WITH_RC | |
| Hi Everyone, | |
| I propose that we release the following RC as the official PyIceberg $VERSION release. | |
| A summary of the high level features: | |
| * <Add summary by hand> | |
| The commit ID is $LAST_COMMIT_ID | |
| * This corresponds to the tag: $GIT_TAG ($GIT_TAG_HASH) | |
| * https://github.com/apache/iceberg-python/releases/tag/$GIT_TAG | |
| * https://github.com/apache/iceberg-python/tree/$LAST_COMMIT_ID | |
| The release tarball, signature, and checksums are here: | |
| * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-$VERSION_WITH_RC/ | |
| You can find the KEYS file here: | |
| * https://downloads.apache.org/iceberg/KEYS | |
| Convenience binary artifacts are staged on pypi: | |
| https://pypi.org/project/pyiceberg/$VERSION_WITH_RC/ | |
| And can be installed using: pip3 install pyiceberg==$VERSION_WITH_RC | |
| Instructions for verifying a release can be found here: | |
| * https://py.iceberg.apache.org/verify-release/ | |
| Please download, verify, and test. | |
| Please vote in the next 72 hours. | |
| [ ] +1 Release this as PyIceberg $VERSION | |
| [ ] +0 | |
| [ ] -1 Do not release this because... | |
| EOF | |
| ``` | |
| ### Send Vote Email | |
| Verify the content of `release-announcement-email.txt` and send it to `dev@iceberg.apache.org` with the corresponding subject line. | |
| ## Vote has failed | |
| If there are concerns with the RC, address the issues and generate another RC. | |
| ## Publish the Final Release (Vote has passed) | |
| A minimum of 3 binding +1 votes is required to pass an RC. | |
| Once the vote has been passed, you can close the vote thread by concluding it: | |
| ```text | |
| Thanks everyone for voting! The 72 hours have passed, and a minimum of 3 binding votes have been cast: | |
| +1 Foo Bar (non-binding) | |
| ... | |
| +1 Fokko Driesprong (binding) | |
| The release candidate has been accepted as PyIceberg <VERSION>. Thanks everyone, when all artifacts are published the announcement will be sent out. | |
| Kind regards, | |
| ``` | |
| ### Upload the accepted RC to Apache Release SVN | |
| <!-- prettier-ignore-start --> | |
| !!! note | |
| Only a PMC member has the permission to upload an artifact to the SVN release dist. | |
| <!-- prettier-ignore-end --> | |
| ```bash | |
| : "${VERSION_WITH_RC:?ERROR: VERSION_WITH_RC is not set or is empty}" | |
| : "${VERSION:?ERROR: VERSION is not set or is empty}" | |
| export SVN_DEV_DIR_VERSIONED="https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-${VERSION_WITH_RC}" | |
| export SVN_RELEASE_DIR_VERSIONED="https://dist.apache.org/repos/dist/release/iceberg/pyiceberg-${VERSION}" | |
| svn mv ${SVN_DEV_DIR_VERSIONED} ${SVN_RELEASE_DIR_VERSIONED} -m "PyIceberg: Add release ${VERSION}" | |
| ``` | |
| Verify the artifact is uploaded to [https://dist.apache.org/repos/dist/release/iceberg](https://dist.apache.org/repos/dist/release/iceberg/). | |
| ### Remove Old Artifacts From Apache Release SVN | |
| We only want to host the latest release. Clean up old release artifacts: | |
| ```bash | |
| svn delete https://dist.apache.org/repos/dist/release/iceberg/pyiceberg-<OLD_RELEASE_VERSION> -m "Remove old release artifacts" | |
| ``` | |
| ### Upload the accepted release to PyPi | |
| The latest version can be pushed to PyPi. Check out the Apache SVN and make sure to publish the right version with `twine`: | |
| <!-- prettier-ignore-start --> | |
| !!! note | |
| `twine` might require an PyPi API token. | |
| <!-- prettier-ignore-end --> | |
| ```bash | |
| : "${VERSION:?ERROR: VERSION is not set or is empty}" | |
| svn checkout https://dist.apache.org/repos/dist/release/iceberg/pyiceberg-${VERSION} /tmp/iceberg-dist-release/pyiceberg-${VERSION} | |
| cd /tmp/iceberg-dist-release/pyiceberg-${VERSION} | |
| twine upload pyiceberg-*.whl pyiceberg-*.tar.gz | |
| ``` | |
| Verify the artifact is uploaded to [PyPi](https://pypi.org/project/pyiceberg/#history). | |
| ## Post Release | |
| ### Send out Release Announcement Email | |
| Send out an announcement on the dev mail list: | |
| ```text | |
| To: dev@iceberg.apache.org | |
| Subject: [ANNOUNCE] PyIceberg <VERSION> | |
| I'm pleased to announce the release of PyIceberg <VERSION>! | |
| Apache Iceberg is an open table format for huge analytic datasets. Iceberg | |
| delivers high query performance for tables with tens of petabytes of data, | |
| along with atomic commits, concurrent writes, and SQL-compatible table | |
| evolution. | |
| This Python release can be downloaded from: https://pypi.org/project/pyiceberg/<VERSION>/ | |
| Thanks to everyone for contributing! | |
| ``` | |
| ### Create a Github Release Note | |
| Create a [new Release Note](https://github.com/apache/iceberg-python/releases/new) on the iceberg-python Github repository. | |
| Input the tag in **Choose a tag** with the newly approved released version (e.g. `0.7.0`) and set it to **Create new tag** on publish. Pick the target commit version as the commit ID the release was approved on. | |
| For example: | |
|  | |
| Then, select the previous release version as the **Previous tag** to use the diff between the two versions in generating the release notes. | |
| **Generate release notes**. | |
| **Set as the latest release** and **Publish**. | |
| Make sure to check the `changelog` label on GitHub to see if anything needs to be highlighted. | |
| ### Release the docs | |
| Run the [`Release Docs` Github Action](https://github.com/apache/iceberg-python/actions/workflows/python-release-docs.yml). | |
| ### Update the Github template | |
| Make sure to create a PR to update the [GitHub issues template](https://github.com/apache/iceberg-python/blob/main/.github/ISSUE_TEMPLATE/iceberg_bug_report.yml) with the latest version. | |
| ## Misc | |
| ### Set up GPG key and Upload to Apache Iceberg KEYS file | |
| To set up GPG key locally, see the [instructions](http://www.apache.org/dev/openpgp.html#key-gen-generate-key). | |
| To install gpg on a M1 based Mac, a couple of additional steps are required: <https://gist.github.com/phortuin/cf24b1cca3258720c71ad42977e1ba57>. | |
| Then, published GPG key to the [Apache Iceberg KEYS file](https://downloads.apache.org/iceberg/KEYS): | |
| ```bash | |
| svn co https://dist.apache.org/repos/dist/release/iceberg icebergsvn | |
| cd icebergsvn | |
| echo "" >> KEYS # append a newline | |
| gpg --list-sigs <YOUR KEY ID HERE> >> KEYS # append signatures | |
| gpg --armor --export <YOUR KEY ID HERE> >> KEYS # append public key block | |
| svn commit -m "add key for <YOUR NAME HERE>" # this requires Iceberg PMC privileges | |
| ``` | |
| <!-- prettier-ignore-start --> | |
| !!! note | |
| Updating the `KEYS` artifact in the `release/` distribution requires Iceberg PMC privileges. Please work with a PMC member to update the file. | |
| <!-- prettier-ignore-end --> |