Post-commit tests validate that Beam works correctly in a live environment. The tests also catch errors that are hard to predict in the design and implementation stages.
Even though post-commit tests run after the code is merged into the repository, it is important that the tests pass reliably. Jenkins executes post-commit tests against the HEAD of the master
branch. If post-commit tests fail, there is a problem with the HEAD build. In addition, post-commit tests are time consuming to run, and it is often hard to triage test failures.
To ensure that Beam's post-commit tests are reliable and healthy, the Beam community follows these post-commit test policies:
When a post-commit test fails, follow the provided steps for your situation.
Note: Rollback is always the first course of action. If a fix is trivial, open a pull request with the proposed fix while doing rollback.
After rollback there is time for deeper investigation. Start by looking at the JIRA issue to see the background information for the rollback. These scenarios are all common:
These are all valid reasons for rollback. Maintaining clear signal is the highest priority.
The high level steps are the same:
If the bug is not in your code, here is how to “create a fix”: