| .. Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| .. http://www.apache.org/licenses/LICENSE-2.0 |
| |
| .. Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| |
| .. contents:: :local: |
| |
| CI Environment |
| ============== |
| |
| Continuous Integration is important component of making Apache Airflow robust and stable. We are running |
| a lot of tests for every pull request, for main and v2-*-test branches and regularly as scheduled jobs. |
| |
| Our execution environment for CI is `GitHub Actions <https://github.com/features/actions>`_. GitHub Actions |
| (GA) are very well integrated with GitHub code and Workflow and it has evolved fast in 2019/202 to become |
| a fully-fledged CI environment, easy to use and develop for, so we decided to switch to it. Our previous |
| CI system was Travis CI. |
| |
| However part of the philosophy we have is that we are not tightly coupled with any of the CI |
| environments we use. Most of our CI jobs are written as bash scripts which are executed as steps in |
| the CI jobs. And we have a number of variables determine build behaviour. |
| |
| You can also take a look at the `CI Sequence Diagrams <CI_DIAGRAMS.md>`_ for more graphical overview |
| of how Airflow CI works. |
| |
| GitHub Actions runs |
| ------------------- |
| |
| Our builds on CI are highly optimized. They utilise some of the latest features provided by GitHub Actions |
| environment that make it possible to reuse parts of the build process across different Jobs. |
| |
| Big part of our CI runs use Container Images. Airflow has a lot of dependencies and in order to make |
| sure that we are running tests in a well configured and repeatable environment, most of the tests, |
| documentation building, and some more sophisticated static checks are run inside a docker container |
| environment. This environment consist of two types of images: CI images and PROD images. CI Images |
| are used for most of the tests and checks where PROD images are used in the Kubernetes tests. |
| |
| In order to run the tests, we need to make sure that the images are built using latest sources and that it |
| is done quickly (full rebuild of such image from scratch might take ~15 minutes). Therefore optimisation |
| techniques have been implemented that use efficiently cache from the GitHub Docker registry - in most cases |
| this brings down the time needed to rebuild the image to ~4 minutes. In some cases (when dependencies change) |
| it can be ~6-7 minutes and in case base image of Python releases new patch-level, it can be ~12 minutes. |
| |
| Container Registry used as cache |
| -------------------------------- |
| |
| We are using GitHub Container Registry to store the results of the ``Build Images`` |
| workflow which is used in the ``Tests`` workflow. |
| |
| Currently in main version of Airflow we run tests in all versions of Python supported, |
| which means that we have to build multiple images (one CI and one PROD for each Python version). |
| Yet we run many jobs (>15) - for each of the CI images. That is a lot of time to just build the |
| environment to run. Therefore we are utilising ``pull_request_target`` feature of GitHub Actions. |
| |
| This feature allows to run a separate, independent workflow, when the main workflow is run - |
| this separate workflow is different than the main one, because by default it runs using ``main`` version |
| of the sources but also - and most of all - that it has WRITE access to the GitHub Container Image registry. |
| |
| This is especially important in our case where Pull Requests to Airflow might come from any repository, |
| and it would be a huge security issue if anyone from outside could |
| utilise the WRITE access to the Container Image Registry via external Pull Request. |
| |
| Thanks to the WRITE access and fact that the ``pull_request_target`` by default uses the ``main`` version of the |
| sources, we can safely run some logic there will checkout the incoming Pull Request, build the container |
| image from the sources from the incoming PR and push such image to an GitHub Docker Registry - so that |
| this image can be built only once and used by all the jobs running tests. The image is tagged with unique |
| ``COMMIT_SHA`` of the incoming Pull Request and the tests run in the Pull Request can simply pull such image |
| rather than build it from the scratch. Pulling such image takes ~ 1 minute, thanks to that we are saving |
| a lot of precious time for jobs. |
| |
| We use `GitHub Container Registry <https://docs.github.com/en/packages/guides/about-github-container-registry>`_. |
| ``GITHUB_TOKEN`` is needed to push to the registry and we configured scopes of the tokens in our jobs |
| to be able to write to the registry. |
| |
| The latest cache is kept as ``:cache-amd64`` and ``:cache-arm64`` tagged cache (suitable for |
| ``--cache-from`` directive of buildx - it contains metadata and cache for all segments in the image, |
| and cache is separately kept for different platform. |
| |
| The ``latest`` images of CI and PROD are ``amd64`` only images for CI, because there is no very easy way |
| to push multiplatform images without merging the manifests and it is not really needed nor used for cache. |
| |
| |
| Naming conventions for stored images |
| ==================================== |
| |
| The images produced during the ``Build Images`` workflow of CI jobs are stored in the |
| `GitHub Container Registry <https://github.com/orgs/apache/packages?repo_name=airflow>`_ |
| |
| The images are stored with both "latest" tag (for last main push image that passes all the tests as well |
| with the COMMIT_SHA id for images that were used in particular build. |
| |
| The image names follow the patterns (except the Python image, all the images are stored in |
| https://ghcr.io/ in ``apache`` organization. |
| |
| The packages are available under (CONTAINER_NAME is url-encoded name of the image). Note that "/" are |
| supported now in the ``ghcr.io`` as apart of the image name within ``apache`` organization, but they |
| have to be percent-encoded when you access them via UI (/ = %2F) |
| |
| ``https://github.com/apache/airflow/pkgs/container/<CONTAINER_NAME>`` |
| |
| +--------------+----------------------------------------------------------+----------------------------------------------------------+ |
| | Image | Name:tag (both cases latest version and per-build) | Description | |
| +==============+==========================================================+==========================================================+ |
| | Python image | python:<X.Y>-slim-bullseye | Base Python image used by both production and CI image. | |
| | (DockerHub) | | Python maintainer release new versions of those image | |
| | | | with security fixes every few weeks in DockerHub. | |
| +--------------+----------------------------------------------------------+----------------------------------------------------------+ |
| | Airflow | airflow/<BRANCH>/python:<X.Y>-slim-bullseye | Version of python base image used in Airflow Builds | |
| | python base | | We keep the "latest" version only to mark last "good" | |
| | image | | python base that went through testing and was pushed. | |
| +--------------+----------------------------------------------------------+----------------------------------------------------------+ |
| | PROD Build | airflow/<BRANCH>/prod-build/python<X.Y>:latest | Production Build image - this is the "build" stage of | |
| | image | | production image. It contains build-essentials and all | |
| | | | necessary apt packages to build/install PIP packages. | |
| | | | We keep the "latest" version only to speed up builds. | |
| +--------------+----------------------------------------------------------+----------------------------------------------------------+ |
| | Manifest | airflow/<BRANCH>/ci-manifest/python<X.Y>:latest | CI manifest image - this is the image used to optimize | |
| | CI image | | pulls and builds for Breeze development environment | |
| | | | They store hash indicating whether the image will be | |
| | | | faster to build or pull. | |
| | | | We keep the "latest" version only to help breeze to | |
| | | | check if new image should be pulled. | |
| +--------------+----------------------------------------------------------+----------------------------------------------------------+ |
| | CI image | airflow/<BRANCH>/ci/python<X.Y>:latest | CI image - this is the image used for most of the tests. | |
| | | or | Contains all provider dependencies and tools useful | |
| | | airflow/<BRANCH>/ci/python<X.Y>:<COMMIT_SHA> | For testing. This image is used in Breeze. | |
| +--------------+----------------------------------------------------------+----------------------------------------------------------+ |
| | | | faster to build or pull. | |
| | PROD image | airflow/<BRANCH>/prod/python<X.Y>:latest | Production image. This is the actual production image | |
| | | or | optimized for size. | |
| | | airflow/<BRANCH>/prod/python<X.Y>:<COMMIT_SHA> | It contains only compiled libraries and minimal set of | |
| | | | dependencies to run Airflow. | |
| +--------------+----------------------------------------------------------+----------------------------------------------------------+ |
| |
| * <BRANCH> might be either "main" or "v2-*-test" |
| * <X.Y> - Python version (Major + Minor).Should be one of ["3.7", "3.8", "3.9"]. |
| * <COMMIT_SHA> - full-length SHA of commit either from the tip of the branch (for pushes/schedule) or |
| commit from the tip of the branch used for the PR. |
| |
| GitHub Registry Variables |
| ========================= |
| |
| Our CI uses GitHub Registry to pull and push images to/from by default. Those variables are set automatically |
| by GitHub Actions when you run Airflow workflows in your fork, so they should automatically use your |
| own repository as GitHub Registry to build and keep the images as build image cache. |
| |
| The variables are automatically set in GitHub actions |
| |
| +--------------------------------+---------------------------+----------------------------------------------+ |
| | Variable | Default | Comment | |
| +================================+===========================+==============================================+ |
| | GITHUB_REPOSITORY | ``apache/airflow`` | Prefix of the image. It indicates which. | |
| | | | registry from GitHub to use for image cache | |
| | | | and to determine the name of the image. | |
| +--------------------------------+---------------------------+----------------------------------------------+ |
| | CONSTRAINTS_GITHUB_REPOSITORY | ``apache/airflow`` | Repository where constraints are stored | |
| +--------------------------------+---------------------------+----------------------------------------------+ |
| | GITHUB_USERNAME | | Username to use to login to GitHub | |
| | | | | |
| +--------------------------------+---------------------------+----------------------------------------------+ |
| | GITHUB_TOKEN | | Token to use to login to GitHub. | |
| | | | Only used when pushing images on CI. | |
| +--------------------------------+---------------------------+----------------------------------------------+ |
| |
| The Variables beginning with ``GITHUB_`` cannot be overridden in GitHub Actions by the workflow. |
| Those variables are set by GitHub Actions automatically and they are immutable. Therefore if |
| you want to override them in your own CI workflow and use ``breeze``, you need to pass the |
| values by corresponding ``breeze`` flags ``--github-repository``, ``--github-username``, |
| ``--github-token`` rather than by setting them as environment variables in your workflow. |
| Unless you want to keep your own copy of constraints in orphaned ``constraints-*`` |
| branches, the ``CONSTRAINTS_GITHUB_REPOSITORY`` should remain ``apache/airflow``, regardless in which |
| repository the CI job is run. |
| |
| One of the variables you might want to override in your own GitHub Actions workflow when using ``breeze`` is |
| ``--github-repository`` - you might want to force it to ``apache/airflow``, because then the cache from |
| ``apache/airflow`` repository will be used and your builds will be much faster. |
| |
| Example command to build your CI image efficiently in your own CI workflow: |
| |
| .. code-block:: bash |
| |
| # GITHUB_REPOSITORY is set automatically in Github Actions so we need to override it with flag |
| # |
| breeze ci-image build --github-repository apache/airflow --python 3.10 |
| docker tag ghcr.io/apache/airflow/main/ci/python3.10 your-image-name:tag |
| |
| |
| Authentication in GitHub Registry |
| ================================= |
| |
| We are using GitHub Container Registry as cache for our images. Authentication uses GITHUB_TOKEN mechanism. |
| Authentication is needed for pushing the images (WRITE) only in "push", "pull_request_target" workflows. |
| When you are running the CI jobs in GitHub Actions, GITHUB_TOKEN is set automatically by the actions. |
| |
| |
| CI run types |
| ============ |
| |
| The following CI Job run types are currently run for Apache Airflow (run by ci.yaml workflow) |
| and each of the run types has different purpose and context. |
| |
| Besides the regular "PR" runs we also have "Canary" runs that are able to detect most of the |
| problems that might impact regular PRs early, without necessarily failing all PRs when those |
| problems happen. This allows to provide much more stable environment for contributors, who |
| contribute their PR, while giving a chance to maintainers to react early on problems that |
| need reaction, when the "canary" builds fail. |
| |
| Pull request run |
| ---------------- |
| |
| Those runs are results of PR from the forks made by contributors. Most builds for Apache Airflow fall |
| into this category. They are executed in the context of the "Fork", not main |
| Airflow Code Repository which means that they have only "read" permission to all the GitHub resources |
| (container registry, code repository). This is necessary as the code in those PRs (including CI job |
| definition) might be modified by people who are not committers for the Apache Airflow Code Repository. |
| |
| The main purpose of those jobs is to check if PR builds cleanly, if the test run properly and if |
| the PR is ready to review and merge. The runs are using cached images from the Private GitHub registry - |
| CI, Production Images as well as base Python images that are also cached in the Private GitHub registry. |
| Also for those builds we only execute Python tests if important files changed (so for example if it is |
| "no-code" change, no tests will be executed. |
| |
| Regular PR builds run in a "stable" environment: |
| |
| * fixed set of constraints (constraints that passed the tests) - except the PRs that change dependencies |
| * limited matrix and set of tests (determined by selective checks based on what changed in the PR) |
| * no ARM image builds are build in the regular PRs |
| * lower probability of flaky tests for non-committer PRs (public runners and less parallelism) |
| |
| Canary run |
| ---------- |
| |
| Those runs are results of direct pushes done by the committers - basically merging of a Pull Request |
| by the committers. Those runs execute in the context of the Apache Airflow Code Repository and have also |
| write permission for GitHub resources (container registry, code repository). |
| |
| The main purpose for the run is to check if the code after merge still holds all the assertions - like |
| whether it still builds, all tests are green. This is a "Canary" build that helps us to detect early |
| problems with dependencies, image building, full matrix of tests in case they passed through selective checks. |
| |
| This is needed because some of the conflicting changes from multiple PRs might cause build and test failures |
| after merge even if they do not fail in isolation. Also those runs are already reviewed and confirmed by the |
| committers so they can be used to do some housekeeping: |
| |
| - pushing most recent image build in the PR to the GitHub Container Registry (for caching) including recent |
| Dockerfile changes and setup.py/setup.cfg changes (Early Cache) |
| - test that image in ``breeze`` command builds quickly |
| - run full matrix of tests to detect any tests that will be mistakenly missed in ``selective checks`` |
| - upgrading to latest constraints and pushing those constraints if all tests succeed |
| - refresh latest Python base images in case new patch-level is released |
| |
| The housekeeping is important - Python base images are refreshed with varying frequency (once every few months |
| usually but sometimes several times per week) with the latest security and bug fixes. |
| |
| Scheduled runs |
| -------------- |
| |
| Those runs are results of (nightly) triggered job - only for ``main`` branch. The |
| main purpose of the job is to check if there was no impact of external dependency changes on the Apache |
| Airflow code (for example transitive dependencies released that fail the build). It also checks if the |
| Docker images can be built from the scratch (again - to see if some dependencies have not changed - for |
| example downloaded package releases etc. |
| |
| All runs consist of the same jobs, but the jobs behave slightly differently or they are skipped in different |
| run categories. Here is a summary of the run categories with regards of the jobs they are running. |
| Those jobs often have matrix run strategy which runs several different variations of the jobs |
| (with different Backend type / Python version, type of the tests to run for example). The following chapter |
| describes the workflows that execute for each run. |
| |
| Those runs and their corresponding ``Build Images`` runs are only executed in main ``apache/airflow`` |
| repository, they are not executed in forks - we want to be nice to the contributors and not use their |
| free build minutes on GitHub Actions. |
| |
| Workflows |
| ========= |
| |
| A general note about cancelling duplicated workflows: for the ``Build Images``, ``Tests`` and ``CodeQL`` |
| workflows we use the ``concurrency`` feature of GitHub actions to automatically cancel "old" workflow runs |
| of each type -- meaning if you push a new commit to a branch or to a pull request and there is a workflow |
| running, GitHub Actions will cancel the old workflow run automatically. |
| |
| Build Images Workflow |
| --------------------- |
| |
| This workflow builds images for the CI Workflow for Pull Requests coming from forks. |
| |
| It's a special type of workflow: ``pull_request_target`` which means that it is triggered when a pull request |
| is opened. This also means that the workflow has Write permission to push to the GitHub registry the images |
| used by CI jobs which means that the images can be built only once and reused by all the CI jobs |
| (including the matrix jobs). We've implemented it so that the ``Tests`` workflow waits |
| until the images are built by the ``Build Images`` workflow before running. |
| |
| Those "Build Image" steps are skipped in case Pull Requests do not come from "forks" (i.e. those |
| are internal PRs for Apache Airflow repository. This is because in case of PRs coming from |
| Apache Airflow (only committers can create those) the "pull_request" workflows have enough |
| permission to push images to GitHub Registry. |
| |
| This workflow is not triggered on normal pushes to our "main" branches, i.e. after a |
| pull request is merged and whenever ``scheduled`` run is triggered. Again in this case the "CI" workflow |
| has enough permissions to push the images. In this case we simply do not run this workflow. |
| |
| The workflow has the following jobs: |
| |
| +---------------------------+---------------------------------------------+ |
| | Job | Description | |
| | | | |
| +===========================+=============================================+ |
| | Build Info | Prints detailed information about the build | |
| +---------------------------+---------------------------------------------+ |
| | Build CI images | Builds all configured CI images | |
| +---------------------------+---------------------------------------------+ |
| | Build PROD images | Builds all configured PROD images | |
| +---------------------------+---------------------------------------------+ |
| |
| The images are stored in the `GitHub Container Registry <https://github.com/orgs/apache/packages?repo_name=airflow>`_ |
| and the names of those images follow the patterns described in |
| `Naming conventions for stored images <#naming-conventions-for-stored-images>`_ |
| |
| Image building is configured in "fail-fast" mode. When any of the images |
| fails to build, it cancels other builds and the source ``Tests`` workflow run |
| that triggered it. |
| |
| |
| Differences for main and release branches |
| ----------------------------------------- |
| |
| There are a few differences of what kind of tests are run, depending on which version/branch the tests are executed for. |
| While all our tests run for the "main" development branch to keep Airflow in check, only a subset of those tests is run |
| in older branches when we are releasing patch-level releases. This is because we never use old branches to release |
| providers and helm charts, we only use them to release Airflow and Airflow image. |
| |
| This behaviour is controlled by ``default-branch`` output of the build-info job. Whenever we create a branch for old version |
| we update the ``AIRFLOW_BRANCH`` in ``airflow_breeze/branch_defaults.py`` to point to the new branch and there are a few |
| places where selection of tests is based on whether this output is ``main``. They are marked as - in the "Release branches" |
| column of the table below. |
| |
| Tests Workflow |
| -------------- |
| |
| This workflow is a regular workflow that performs all checks of Airflow code. |
| |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Job | Description | PR | Canary | Scheduled | Release branches | |
| +=================================+==========================================================+==========+==========+===========+===================+ |
| | Build info | Prints detailed information about the build | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Build CI/PROD images | Builds images in-workflow (not in the build images one) | - | Yes | Yes (1) | Yes (4) | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Check that image builds quickly | Checks that image builds quickly without taking a lot of | - | Yes | - | Yes | |
| | | time for ``pip`` to figure out the right set of deps. | | | | | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Push early cache & images | Pushes early cache/images to GitHub Registry and test | - | Yes | - | - | |
| | | speed of building breeze images from scratch | | | | | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Run breeze tests | Run unit tests for Breeze | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Test OpenAPI client gen | Tests if OpenAPIClient continues to generate | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | React WWW tests | React UI tests for new Airflow UI | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Test image building | Tests if PROD image build examples work | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Test git clone on Windows | Tests if Git clone for for Windows | Yes (5) | - | - | Yes (5) | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Waits for CI Images | Waits for and verify CI Images (2) | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Static checks | Performs full static checks | Yes (6) | Yes | Yes | Yes (7) | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Basic static checks | Performs basic static checks | Yes (6) | - | - | Yes (7) | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Build docs | Builds documentation | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Test Pytest collection | Tests if pytest collection works | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Tests | Run the Pytest unit tests (Backend/Python matrix) | Yes | Yes | Yes | Yes (8) | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Integration tests | Runs integration tests (Postgres/Mysql) | Yes | Yes | Yes | Yes (9) | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Quarantined tests | Runs quarantined tests (with flakiness and side-effects) | Yes | Yes | Yes | Yes (8) | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Tests provider packages | Tests if provider packages can be built and released | Yes | Yes | Yes | - | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Test airflow packages | Tests that Airflow package can be built and released | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Helm tests | Run the Helm integration tests | Yes | Yes | Yes | - | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Summarize warnings | Summarizes warnings from all other tests | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Wait for PROD Images | Waits for and verify PROD Images (2) | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Tests Kubernetes | Run Kubernetes test | Yes | Yes | Yes | - | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Test docker-compose | Tests if quick-start docker compose works | Yes | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Constraints | Upgrade constraints to latest ones (3) | - | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Push cache & images | Pushes cache/images to GitHub Registry (3) | - | Yes | Yes | Yes | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| | Build CI ARM images | Builds CI images for ARM to detect any problems which | Yes (10) | - | Yes | Yes | |
| | | would only appear if we install all dependencies on ARM | | | | | |
| +---------------------------------+----------------------------------------------------------+----------+----------+-----------+-------------------+ |
| |
| ``(1)`` Scheduled jobs builds images from scratch - to test if everything works properly for clean builds |
| |
| ``(2)`` The jobs wait for CI images to be available. |
| |
| ``(3)`` PROD and CI cache & images are pushed as "latest" to GitHub Container registry and constraints are |
| upgraded only if all tests are successful. The images are rebuilt in this step using constraints pushed |
| in the previous step. |
| |
| ``(4)`` In main, PROD image uses locally build providers using "latest" version of the provider code. In the |
| non-main version of the build, the latest released providers from PyPI are used. |
| |
| ``(5)`` Only runs those tests for the builds where public runners are used (so either when non-committer |
| runs it or when ``use public runner`` label is assigned to the PR. |
| |
| ``(6)`` Run full set of static checks when selective-checks determine that they are needed (basically, when |
| Python code has been modified). |
| |
| ``(7)`` On non-main builds some of the static checks that are related to Providers are skipped via selective checks |
| (``skip-pre-commits`` check). |
| |
| ``(8)`` On non-main builds the unit tests for providers are skipped via selective checks removing the |
| "Providers" test type. |
| |
| ``(9)`` On non-main builds the integration tests for providers are skipped via ``skip-provider-tests`` selective |
| check output. |
| |
| ``(10)`` Only run the builds in case dependencies are changed (``upgrade-to-newer-dependencies`` is set). |
| |
| |
| CodeQL scan |
| ----------- |
| |
| The `CodeQL <https://securitylab.github.com/tools/codeql>`_ security scan uses GitHub security scan framework to scan our code for security violations. |
| It is run for JavaScript and Python code. |
| |
| Publishing documentation |
| ------------------------ |
| |
| Documentation from the ``main`` branch is automatically published on Amazon S3. |
| |
| To make this possible, GitHub Action has secrets set up with credentials |
| for an Amazon Web Service account - ``DOCS_AWS_ACCESS_KEY_ID`` and ``DOCS_AWS_SECRET_ACCESS_KEY``. |
| |
| This account has permission to write/list/put objects to bucket ``apache-airflow-docs``. This bucket has |
| public access configured, which means it is accessible through the website endpoint. |
| For more information, see: |
| `Hosting a static website on Amazon S3 <https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html>`_ |
| |
| Website endpoint: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/ |
| |
| |
| Debugging CI Jobs in Github Actions |
| =================================== |
| |
| The CI jobs are notoriously difficult to test, because you can only really see results of it when you run them |
| in CI environment, and the environment in which they run depend on who runs them (they might be either run |
| in our Self-Hosted runners (with 64 GB RAM 8 CPUs) or in the GitHub Public runners (6 GB of RAM, 2 CPUs) and |
| the results will vastly differ depending on which environment is used. We are utilizing parallelism to make |
| use of all the available CPU/Memory but sometimes you need to enable debugging and force certain environments. |
| Additional difficulty is that ``Build Images`` workflow is ``pull-request-target`` type, which means that it |
| will always run using the ``main`` version - no matter what is in your Pull Request. |
| |
| There are several ways how you can debug the CI jobs when you are maintainer. |
| |
| * When you want to tests the build with all combinations of all python, backends etc on regular PR, |
| add ``full tests needed`` label to the PR. |
| * When you want to test maintainer PR using public runners, add ``public runners`` label to the PR |
| * When you want to see resources used by the run, add ``debug ci resources`` label to the PR |
| * When you want to test changes to breeze that include changes to how images are build you should push |
| your PR to ``apache`` repository not to your fork. This will run the images as part of the ``CI`` workflow |
| rather than using ``Build images`` workflow and use the same breeze version for building image and testing |
| * When you want to test changes to ``build-images.yml`` workflow you should push your branch as ``main`` |
| branch in your local fork. This will run changed ``build-images.yml`` workflow as it will be in ``main`` |
| branch of your fork |
| |
| Replicating the CI Jobs locally |
| =============================== |
| |
| The main goal of the CI philosophy we have that no matter how complex the test and integration |
| infrastructure, as a developer you should be able to reproduce and re-run any of the failed checks |
| locally. One part of it are pre-commit checks, that allow you to run the same static checks in CI |
| and locally, but another part is the CI environment which is replicated locally with Breeze. |
| |
| You can read more about Breeze in `BREEZE.rst <BREEZE.rst>`_ but in essence it is a script that allows |
| you to re-create CI environment in your local development instance and interact with it. In its basic |
| form, when you do development you can run all the same tests that will be run in CI - but locally, |
| before you submit them as PR. Another use case where Breeze is useful is when tests fail on CI. You can |
| take the full ``COMMIT_SHA`` of the failed build pass it as ``--image-tag`` parameter of Breeze and it will |
| download the very same version of image that was used in CI and run it locally. This way, you can very |
| easily reproduce any failed test that happens in CI - even if you do not check out the sources |
| connected with the run. |
| |
| All our CI jobs are executed via ``breeze`` commands. You can replicate exactly what our CI is doing |
| by running the sequence of corresponding ``breeze`` command. Make sure however that you look at both: |
| |
| * flags passed to ``breeze`` commands |
| * environment variables used when ``breeze`` command is run - this is useful when we want |
| to set a common flag for all ``breeze`` commands in the same job or even the whole workflow. For |
| example ``VERBOSE`` variable is set to ``true`` for all our workflows so that more detailed information |
| about internal commands executed in CI is printed. |
| |
| In the output of the CI jobs, you will find both - the flags passed and environment variables set. |
| |
| You can read more about it in `BREEZE.rst <BREEZE.rst>`_ and `TESTING.rst <TESTING.rst>`_ |
| |
| Since we store images from every CI run, you should be able easily reproduce any of the CI tests problems |
| locally. You can do it by pulling and using the right image and running it with the right docker command, |
| For example knowing that the CI job was for commit ``cd27124534b46c9688a1d89e75fcd137ab5137e3``: |
| |
| .. code-block:: bash |
| |
| docker pull ghcr.io/apache/airflow/main/ci/python3.7:cd27124534b46c9688a1d89e75fcd137ab5137e3 |
| |
| docker run -it ghcr.io/apache/airflow/main/ci/python3.7:cd27124534b46c9688a1d89e75fcd137ab5137e3 |
| |
| |
| But you usually need to pass more variables and complex setup if you want to connect to a database or |
| enable some integrations. Therefore it is easiest to use `Breeze <BREEZE.rst>`_ for that. For example if |
| you need to reproduce a MySQL environment in python 3.8 environment you can run: |
| |
| .. code-block:: bash |
| |
| breeze --image-tag cd27124534b46c9688a1d89e75fcd137ab5137e3 --python 3.8 --backend mysql |
| |
| You will be dropped into a shell with the exact version that was used during the CI run and you will |
| be able to run pytest tests manually, easily reproducing the environment that was used in CI. Note that in |
| this case, you do not need to checkout the sources that were used for that run - they are already part of |
| the image - but remember that any changes you make in those sources are lost when you leave the image as |
| the sources are not mapped from your host machine. |
| |
| Depending whether the scripts are run locally via `Breeze <BREEZE.rst>`_ or whether they |
| are run in ``Build Images`` or ``Tests`` workflows they can take different values. |
| |
| You can use those variables when you try to reproduce the build locally (alternatively you can pass |
| those via command line flags passed to ``breeze`` command. |
| |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | Variable | Local | Build Images | CI | Comment | |
| | | development | workflow | Workflow | | |
| +=========================================+=============+==============+============+=================================================+ |
| | Basic variables | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``PYTHON_MAJOR_MINOR_VERSION`` | | | | Major/Minor version of Python used. | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``DB_RESET`` | false | true | true | Determines whether database should be reset | |
| | | | | | at the container entry. By default locally | |
| | | | | | the database is not reset, which allows to | |
| | | | | | keep the database content between runs in | |
| | | | | | case of Postgres or MySQL. However, | |
| | | | | | it requires to perform manual init/reset | |
| | | | | | if you stop the environment. | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | Forcing answer | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``ANSWER`` | | yes | yes | This variable determines if answer to questions | |
| | | | | | during the build process should be | |
| | | | | | automatically given. For local development, | |
| | | | | | the user is occasionally asked to provide | |
| | | | | | answers to questions such as - whether | |
| | | | | | the image should be rebuilt. By default | |
| | | | | | the user has to answer but in the CI | |
| | | | | | environment, we force "yes" answer. | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | Host variables | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``HOST_USER_ID`` | | | | User id of the host user. | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``HOST_GROUP_ID`` | | | | Group id of the host user. | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``HOST_OS`` | | linux | linux | OS of the Host (darwin/linux/windows). | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | Git variables | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``COMMIT_SHA`` | | GITHUB_SHA | GITHUB_SHA | SHA of the commit of the build is run | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | In container environment initialization | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``SKIP_ENVIRONMENT_INITIALIZATION`` | false\* | false\* | false\* | Skip initialization of test environment | |
| | | | | | | |
| | | | | | \* set to true in pre-commits | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``SKIP_PROVIDER_TESTS`` | false\* | false\* | false\* | Skip running provider integration tests | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``SKIP_SSH_SETUP`` | false\* | false\* | false\* | Skip setting up SSH server for tests. | |
| | | | | | | |
| | | | | | \* set to true in GitHub CodeSpaces | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``VERBOSE_COMMANDS`` | false | false | false | Determines whether every command | |
| | | | | | executed in docker should also be printed | |
| | | | | | before execution. This is a low-level | |
| | | | | | debugging feature of bash (set -x) enabled in | |
| | | | | | entrypoint and it should only be used if you | |
| | | | | | need to debug the bash scripts in container. | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | Image build variables | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| | ``UPGRADE_TO_NEWER_DEPENDENCIES`` | false | false | false\* | Determines whether the build should | |
| | | | | | attempt to upgrade Python base image and all | |
| | | | | | PIP dependencies to latest ones matching | |
| | | | | | ``setup.py`` limits. This tries to replicate | |
| | | | | | the situation of "fresh" user who just installs | |
| | | | | | airflow and uses latest version of matching | |
| | | | | | dependencies. By default we are using a | |
| | | | | | tested set of dependency constraints | |
| | | | | | stored in separated "orphan" branches | |
| | | | | | of the airflow repository | |
| | | | | | ("constraints-main, "constraints-2-0") | |
| | | | | | but when this flag is set to anything but false | |
| | | | | | (for example random value), they are not used | |
| | | | | | used and "eager" upgrade strategy is used | |
| | | | | | when installing dependencies. We set it | |
| | | | | | to true in case of direct pushes (merges) | |
| | | | | | to main and scheduled builds so that | |
| | | | | | the constraints are tested. In those builds, | |
| | | | | | in case we determine that the tests pass | |
| | | | | | we automatically push latest set of | |
| | | | | | "tested" constraints to the repository. | |
| | | | | | | |
| | | | | | Setting the value to random value is best way | |
| | | | | | to assure that constraints are upgraded even if | |
| | | | | | there is no change to setup.py | |
| | | | | | | |
| | | | | | This way our constraints are automatically | |
| | | | | | tested and updated whenever new versions | |
| | | | | | of libraries are released. | |
| | | | | | | |
| | | | | | \* true in case of direct pushes and | |
| | | | | | scheduled builds | |
| +-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+ |
| |
| Adding new Python versions to CI |
| ================================ |
| |
| In order to add a new version the following operations should be done (example uses Python 3.10) |
| |
| * copy the latest constraints in ``constraints-main`` branch from previous versions and name it |
| using the new Python version (``constraints-3.10.txt``). Commit and push |
| |
| * build image locally for both prod and CI locally using Breeze: |
| |
| .. code-block:: bash |
| |
| breeze ci-image build --python 3.10 |
| |
| * Find the 2 new images (prod, ci) created in |
| `GitHub Container registry <https://github.com/orgs/apache/packages?tab=packages&ecosystem=container&q=airflow>`_ |
| go to Package Settings and turn on ``Public Visibility`` and set "Inherit access from Repository" flag. |
