docs/README.md - airflow - Git at Google


            
            <!--

            
             Licensed to the Apache Software Foundation (ASF) under one

            
             or more contributor license agreements.  See the NOTICE file

            
             distributed with this work for additional information

            
             regarding copyright ownership.  The ASF licenses this file

            
             to you under the Apache License, Version 2.0 (the

            
             "License"); you may not use this file except in compliance

            
             with the License.  You may obtain a copy of the License at


               http://www.apache.org/licenses/LICENSE-2.0


             Unless required by applicable law or agreed to in writing,

            
             software distributed under the License is distributed on an

            
             "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

            
             KIND, either express or implied.  See the License for the

            
             specific language governing permissions and limitations

            
             under the License.

            
             -->


            <!-- START doctoc generated TOC please keep comment here to allow auto update -->

            
            <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

            
            **Table of contents**


            - [Documentation configuration](#documentation-configuration)

            
            - [Architecture of documentation for Airflow](#architecture-of-documentation-for-airflow)

            
            - [Diagrams of the documentation architecture](#diagrams-of-the-documentation-architecture)

            
            - [Staging site](#staging-site)

            
            - [Typical workflows](#typical-workflows)

            
              - [Publishing the documentation by the release manager](#publishing-the-documentation-by-the-release-manager)

            
              - [Publishing changes to the website (including theme)](#publishing-changes-to-the-website-including-theme)

            
            - [Publishing changes manually](#publishing-changes-manually)

            
            - [Fixing historical documentation](#fixing-historical-documentation)

            
              - [Manually publishing documentation directly to S3](#manually-publishing-documentation-directly-to-s3)

            
              - [Manually publishing documentation via `apache-airflow-site-archive` repo](#manually-publishing-documentation-via-apache-airflow-site-archive-repo)


            <!-- END doctoc generated TOC please keep comment here to allow auto update -->


            # Documentation configuration


            This directory used to contain all the documentation files for the project. The documentation has

            
            been split into separate folders; the documentation is now stored in the folders of the sub-projects it refers to.


            If you are looking for the documentation, it is stored as follows:


            Documentation in separate distributions:


            * `airflow-core/docs` - documentation for Airflow Core

            
            * `providers/**/docs` - documentation for Providers

            
            * `chart/docs` - documentation for the Helm Chart

            
            * `task-sdk/docs` - documentation for Task SDK (new format not yet published)

            
            * `airflow-ctl/docs` - documentation for Airflow CLI

            
            * `dev/mypy/docs` - documentation for the Apache Airflow Mypy plugins


            Documentation for a general overview and summaries not connected with any specific distribution:


            * `docker-stack-docs` - documentation for Docker Stack

            
            * `providers-summary-docs` - documentation for the provider summary page


            # Architecture of documentation for Airflow


            Building documentation for Airflow is optimized for speed and for the convenience workflows of the release

            
            managers and committers who publish and fix the documentation - that's why it's a little complex, as we have

            
            multiple repositories and multiple sources of documentation involved.


            There are a few repositories under `apache` organization that are used to build the documentation for Airflow:


            * `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,

            
               provider distributions, providers' summary, and docker summary: [apache-airflow](https://github.com/apache/airflow).

            
               From here, we publish the documentation to an S3 bucket where the documentation is hosted.

            
            * `airflow-site` - the repository with the website theme and content where we keep sources of the website

            
               structure, navigation, and theme for the website [airflow-site](https://github.com/apache/airflow-site). From here,

            
               we publish the website to the ASF servers so they are published as the [official website](https://airflow.apache.org)

            
            * `airflow-site-archive` - here we keep the archived historical versions of the generated documentation

            
               of all the documentation packages that we keep on S3. This repository is automatically synchronized from

            
               the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here, only

            
               generated `html`, `css`, `js`, and `images` files are kept; no sources of the documentation are kept here.


            We have two S3 buckets where we can publish the documentation generated from the `apache-airflow` repository:


            * `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)

            
            * `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://airflow.staged.apache.org/docs/)


            Note that those S3 buckets are not served directly to Apache Server, but they are served via Cloudfront

            
            in order to provide caching and automated resolution of folders into index.html files.


            Our CloudFront distributions are:


            * Live CloudFront url: https://d7fnmbhf26p21.cloudfront.net

            
            * Staging CloudFront url: https://d3a2du7x0n8ydr.cloudfront.net


            Those cloudfront caches are automatically invalidated when we publish new documentation to S3 using

            
            GitHub Actions workflows, but you can also manually invalidate them using the AWS Console if needed.


            # Diagrams of the documentation architecture


            This is the diagram of live documentation architecture:


            ![Live documentation architecture](images/documentation_architecture.png)


            # Staging site


            Staging documentation architecture is similar, but uses staging bucket and staging Apache Website. The main

            
            differences are:


            * The staging bucket is `s3://staging-docs-airflow-apache-org/docs/`

            
            * The staging website is `https://airflow.staged.apache.org/docs/`

            
            * The staging site is deployed by merging PR or pushing the `staging` branch in the `airflow-site` repository

            
              rather than `main`. The `staging` branch should be periodically rebased to the `main` branch, but while

            
              some changes are developed in `staging`, it can diverge from the `main` branch.

            
            * Merging into the `staging` branch of the `airflow-site` repository or pushing the `staging` branch

            
              will automatically trigger the build of the website and publish it to the `publish-staging` branch and

            
              effectively to the staging site.


            Documentation of pre-release versions of Airflow distributions should be published to the staging S3

            
            bucket so that we can test the documentation before we publish it to the `live` bucket.


            # Typical workflows


            There are a few typical workflows that we support:


            ## Publishing the documentation by the release manager


            The release manager publishes the documentation using GitHub Actions workflow

            
            [Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).

            
            The same workflow can be used to publish Airflow, Helm chart, and providers' documentation.


            This workflow is used twice:


            * when pre-release distributions are prepared (alpha/beta/rc) - the documentation should be published to

            
              the `staging` bucket and `staging` site should be built and published.

            
            * when final releases of distributions are prepared - the documentation should be published to the `live`

            
              bucket and the `live` website should be built and published.


            When the release manager publishes the documentation, they choose `auto` destination by default - depending on the

            
            tag they use - `staging` will be used to publish from pre-release tag and `live` will be used to publish

            
            from the release tag.


            You can also specify whether documentation should be published to `live` or `staging`, thereby overriding the auto-detection.


            The person who triggers the build (release manager) should specify the tag name of the docs to be published

            
            and the list of documentation packages to be published. Usually it is:


            * Airflow: `apache-airflow docker-stack task-sdk apache-airflow-ctl apache-airflow-mypy`

            
            * Helm chart: `helm-chart`

            
            * Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.


            Optionally - specifically if we run `all-providers` and the release manager wants to exclude some providers,

            
            they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages

            
            specified to be published without exclusions.


            Example screenshot of the workflow triggered from the GitHub UI:


            ![Publishing airflow or providers](images/publish_airflow.png)


            Note that this just publishes the documentation but does not update the "site" with version numbers or

            
            stable links to providers and airflow - if you release a new documentation version, it will be available

            
            with direct URL (say https://airflow.apache.org/docs/apache-airflow/3.0.1/), but the main site will still

            
            point to the previous version of the documentation as `stable` and the version drop-downs will not be updated.


            In order to do it, you need to run the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)

            
            workflow in `airflow-site` repository.


            For `live` site you should run the workflow in `main` branch. For `staging` site it should be staging branch.

            
            This will build the website and publish it to the `publish` branch of `airflow-site` repository (for `live`

            
            site) or `publish-staging` branch (for `staging` site). The workflow will also update the website,

            
            including refreshing the version numbers in the drop-downs and stable links.


            The `staging` documentation is produced automatically with `staging` watermark added.


            ![Publishing site](images/publish_site.png)


            This workflow also invalidates cache in Fastly that Apache Software Foundation uses to serve the website,

            
            so you should always run it after you modify the documentation for the website. Fastly is configured with a

            
            3600-second TTL, which means that changes may take up to ~1 hour to propagate to the website.


            Shortly after the workflow succeeds and documentation is published, in the live bucket, the [airflow-site-archive](https://github.com/apache/airflow-site-archive)

            
            repository is automatically synchronized with the live S3 bucket. TODO: IMPLEMENT THIS, FOR NOW IT HAS

            
            TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)

            
            workflow in `airflow-site-archive` repository. The `airflow-site-archive` essentially keeps the history of

            
            snapshots of the `live` documentation.


            ## Publishing changes to the website (including theme)


            The workflows in `apache-airflow` only update the documentation for the packages (Airflow, Helm chart,

            
            Providers, Docker Stack) that we publish from airflow sources. If we want to publish changes to the website

            
            itself or to the theme (css, javascript) we need to do it in `airflow-site` repository.


            Publishing of airflow-site happens automatically when a PR from `airflow-site` is merged to `main` or when

            
            the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) workflow is triggered

            
            manually in the main branch of `airflow-site` repository. The workflow builds the website and publishes it to

            
            `publish` branch of `airflow-site` repository, which in turn gets picked up by the ASF servers and is

            
            published as the official website. This includes any changes to `.htaccess` of the website.


            Such a main build also publishes the latest "sphinx-airflow-theme" package to GitHub so that the next build

            
            of documentation can automatically pick it up from there. This means that if you want to make changes to

            
            `javascript` or `css` that are part of the theme, you need to do it in `airflow-site` repository and

            
            merge it to `main` branch in order to be able to run the documentation build in `apache-airflow` repository

            
            and pick up the latest version of the theme.


            The version of sphinx theme is fixed in both repositories:


            * https://github.com/apache/airflow-site/blob/main/sphinx_airflow_theme/sphinx_airflow_theme/__init__.py#L21

            
            * https://github.com/apache/airflow/blob/main/devel-common/pyproject.toml#L77 in "docs" section


            In case of bigger changes to the theme, we can first iterate on the website and merge a new theme version,

            
            and only after that can we switch to the new version of the theme.


            # Publishing changes manually


            Sometimes you do not want to use Publishing workflows to publish individual files and caches might get

            
            into the way as both Cloudfront and Fastly caches might take some time to invalidate. In such a case, when

            
            you manually upload the files to S3 bucket, you can immediately invalidate the caches:


            1) Manually run invalidation request in Cloudfront for the documentation S3 bucket you uploaded the files

            
               to via AWS Console or AWS CLI (You can use `/*` to invalidate all files).


            ![Cloudfront invalidation](images/cloudfront_invalidation.png)


            2) Run the `Build docs` workflow in `airflow-site` repository to invalidate Fastly cache for the website.

            
               Use `main` branch to rebuild site for `live` site and `staging` to rebuild the `staging` site:


            ![Build docs](images/build-docs.png)


            # Fixing historical documentation


            Sometimes we need to update historical documentation (modify generated `html`) - for example, when we find

            
            bad links or when we change some of the structure in the documentation. This can be done via the

            
            `airflow-site-archive` repository. The workflow is as follows:


            1. Get the latest version of the documentation from S3 to `airflow-site-archive` repository using

            
               `Sync S3 to GitHub` workflow. This will download the latest version of the documentation from S3 to

            
               `airflow-site-archive` repository (this should normally not be needed, if automated synchronization works).

            
            2. Make the changes to the documentation in `airflow-site-archive` repository. This can be done using any

            
               text editor, script, etc. Those files are generated as `html` files and are not meant to be regenerated,

            
               they should be modified as `html` files in-place


            > [!IMPORTANT]

            
            > When you modify a file for latest published version of the documentation you should do it in two

            
            > places - in the `stable` version and in the `X.Y.Z` version of the documentation, as `stable` is just a

            
            > copy of the latest version of the documentation.


            3. Commit the changes to `airflow-site-archive` repository and push them to `some` branch of the repository.

            
            4. Create a Pull Request from that branch and merge it to `main`

            
            5. Run `Sync GitHub to S3` workflow in `airflow-site-archive` repository. This will upload the modified

            
               documentation to the S3 bucket. Use `main` branch (default) as "Reference of the commit used

            
               for synchronization". You can choose whether to sync the changes to `live` or `staging` bucket.

            
               The default is `live`:

            
               ![Sync GitHub to S3](images/sync_github_to_s3.png)

            
            6. By default, the workflow will synchronize all documentation modified in a single last commit pushed to

            
               the `main`. You can also specify `full_sync` to synchronize all files in the repository if you want to

            
               make sure that S3 reflects `main`. The workflow might run for a long time (hours) in case of full sync

            
               or many changes to the `.html` files.

            
            7. In case you specify `full_sync`, you can also synchronize `all` docs or only selected documentation

            
               packages (for example `apache-airflow` or `docker-stack` or `amazon` or `helm-chart`) - you can specify

            
               more than one package separated by spaces.

            
            8. The workflow will invalidate Cloudfront cache for "live" or "staging" bucket respectively.

            
            9. Run the `Build docs` workflow in `airflow-site` repository to make sure that Fastly cache of the

            
               https://airflow.apache.org or https://airflow.staged.apache.org/ invalidated. Use `main` to rebuild site

            
               for `live` site and `staging` to rebuild the `staging` site:

            
               ![Build docs](images/build-docs.png)


            ## Manually publishing documentation directly to S3


            The regular publishing workflows involve running a GitHub Actions workflow, and they cover the majority of cases.

            
            However, sometimes, some manual updates and cherry-picks are needed, when we discover problems with the

            
            publishing and doc building code - for example, when we find that we need to fix extensions to sphinx.


            In such a case, the release manager or a committer can build and publish documentation locally - providing that

            
            they configure AWS credentials to be able to upload files to S3. You can ask in

            
            the #internal-airflow-ci-cd channel on Airflow Slack to get your AWS credentials configured.


            You can checkout locally a version of airflow repo that you need and apply any cherry-picks you need before

            
            running publishing.


            This is done using breeze. You also need to have AWS CLI installed and configured credentials to be able

            
            to upload files to S3. You can get credentials from one of the admins of Airflow's AWS account. The

            
            region to set for AWS is `us-east-2`.


            Note that it is advised to add `--dry-run` if you just want to see what would happen. Also, you can use

            
            the `s3://staging-docs-airflow-apache-org/docs/` bucket to test the publishing using staging site.


            ```bash

            
            breeze build-docs "<package_id1>" "<package_id2>"

            
            mkdir /tmp/airflow-site

            
            breeze release-management publish-docs --override-versioned --airflow-site-directory /tmp/airflow-site

            
            breeze release-management publish-docs-to-s3 --source-dir-path /tmp/airflow-site/docs-archive \

            
             --destination-location s3://live-docs-airflow-apache-org/docs/ --stable-versions \

            
             --exclude-docs "<package_id1_to_exclude> <package_id2_to_exclude>" [--dry-run]

            
            ```


            ## Manually publishing documentation via `apache-airflow-site-archive` repo


            If you do not have S3 credentials and want to be careful about publishing the documentation, you can also

            
            use publishing via `apache-airflow-site-archive` repository. This is a little more complex, but it allows

            
            you to publish documentation without having S3 credentials.


            The process is as follows:


            1. Run `Sync s3 to GitHub` workflow in `apache-airflow-site-archive` repository. This will download the

            
               latest version of the documentation from S3 to `airflow-site-archive` repository (this should normally not

            
               be needed, if automated synchronization works).

            
            2. Checkout the `apache-airflow-site-archive` repository and create a branch for your changes.

            
            3. Build documentation locally in `apache-airflow` repo with any cherry-picks and modifications you need, and

            
               publish the docs to the checked out `airflow-site-archive` branch.


            ```bash

            
            breeze build-docs "<package_id1>" "<package_id2>"

            
            breeze release-management publish-docs --override-versioned --airflow-site-directory <PATH_TO_THE_ARCHIVE_REPO>

            
            ```


            4. Commit the changes to `apache-airflow-site-archive` repository and push them to `some` branch of the

            
               repository.

            
            5. Run `Sync GitHub to S3` workflow in `apache-airflow-site-archive` repository. This will upload the modified

            
               documentation to S3 bucket. You can choose whether to sync the changes to `live` or `staging` bucket.

            
               The default is `live`. You can also specify which folders to sync - by default, all modified folders are synced.

            
            6. After you synchronize the changes to S3, the Sync `S3 to GitHub` workflow will be triggered

            
               automatically and the changes will be synchronized to `airflow-site-archive` `main` branch - so there

            
               is no need to merge your changes to `main` branch of `airflow-site-archive` repository. You can safely

            
               delete the branch you created in step 2.
	<!--
	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.
	-->

	<!-- START doctoc generated TOC please keep comment here to allow auto update -->
	<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
	Table of contents

	- [Documentation configuration](#documentation-configuration)
	- [Architecture of documentation for Airflow](#architecture-of-documentation-for-airflow)
	- [Diagrams of the documentation architecture](#diagrams-of-the-documentation-architecture)
	- [Staging site](#staging-site)
	- [Typical workflows](#typical-workflows)
	- [Publishing the documentation by the release manager](#publishing-the-documentation-by-the-release-manager)
	- [Publishing changes to the website (including theme)](#publishing-changes-to-the-website-including-theme)
	- [Publishing changes manually](#publishing-changes-manually)
	- [Fixing historical documentation](#fixing-historical-documentation)
	- [Manually publishing documentation directly to S3](#manually-publishing-documentation-directly-to-s3)
	- [Manually publishing documentation via `apache-airflow-site-archive` repo](#manually-publishing-documentation-via-apache-airflow-site-archive-repo)

	<!-- END doctoc generated TOC please keep comment here to allow auto update -->


	# Documentation configuration

	This directory used to contain all the documentation files for the project. The documentation has
	been split into separate folders; the documentation is now stored in the folders of the sub-projects it refers to.

	If you are looking for the documentation, it is stored as follows:

	Documentation in separate distributions:

	* `airflow-core/docs` - documentation for Airflow Core
	* `providers/**/docs` - documentation for Providers
	* `chart/docs` - documentation for the Helm Chart
	* `task-sdk/docs` - documentation for Task SDK (new format not yet published)
	* `airflow-ctl/docs` - documentation for Airflow CLI
	* `dev/mypy/docs` - documentation for the Apache Airflow Mypy plugins

	Documentation for a general overview and summaries not connected with any specific distribution:

	* `docker-stack-docs` - documentation for Docker Stack
	* `providers-summary-docs` - documentation for the provider summary page

	# Architecture of documentation for Airflow

	Building documentation for Airflow is optimized for speed and for the convenience workflows of the release
	managers and committers who publish and fix the documentation - that's why it's a little complex, as we have
	multiple repositories and multiple sources of documentation involved.

	There are a few repositories under `apache` organization that are used to build the documentation for Airflow:

	* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions,
	provider distributions, providers' summary, and docker summary: [apache-airflow](https://github.com/apache/airflow).
	From here, we publish the documentation to an S3 bucket where the documentation is hosted.
	* `airflow-site` - the repository with the website theme and content where we keep sources of the website
	structure, navigation, and theme for the website [airflow-site](https://github.com/apache/airflow-site). From here,
	we publish the website to the ASF servers so they are published as the [official website](https://airflow.apache.org)
	* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation
	of all the documentation packages that we keep on S3. This repository is automatically synchronized from
	the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here, only
	generated `html`, `css`, `js`, and `images` files are kept; no sources of the documentation are kept here.

	We have two S3 buckets where we can publish the documentation generated from the `apache-airflow` repository:

	* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/)
	* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://airflow.staged.apache.org/docs/)

	Note that those S3 buckets are not served directly to Apache Server, but they are served via Cloudfront
	in order to provide caching and automated resolution of folders into index.html files.

	Our CloudFront distributions are:

	* Live CloudFront url: https://d7fnmbhf26p21.cloudfront.net
	* Staging CloudFront url: https://d3a2du7x0n8ydr.cloudfront.net

	Those cloudfront caches are automatically invalidated when we publish new documentation to S3 using
	GitHub Actions workflows, but you can also manually invalidate them using the AWS Console if needed.


	# Diagrams of the documentation architecture

	This is the diagram of live documentation architecture:

	![Live documentation architecture](images/documentation_architecture.png)

	# Staging site

	Staging documentation architecture is similar, but uses staging bucket and staging Apache Website. The main
	differences are:

	* The staging bucket is `s3://staging-docs-airflow-apache-org/docs/`
	* The staging website is `https://airflow.staged.apache.org/docs/`
	* The staging site is deployed by merging PR or pushing the `staging` branch in the `airflow-site` repository
	rather than `main`. The `staging` branch should be periodically rebased to the `main` branch, but while
	some changes are developed in `staging`, it can diverge from the `main` branch.
	* Merging into the `staging` branch of the `airflow-site` repository or pushing the `staging` branch
	will automatically trigger the build of the website and publish it to the `publish-staging` branch and
	effectively to the staging site.

	Documentation of pre-release versions of Airflow distributions should be published to the staging S3
	bucket so that we can test the documentation before we publish it to the `live` bucket.

	# Typical workflows

	There are a few typical workflows that we support:

	## Publishing the documentation by the release manager

	The release manager publishes the documentation using GitHub Actions workflow
	[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
	The same workflow can be used to publish Airflow, Helm chart, and providers' documentation.

	This workflow is used twice:

	* when pre-release distributions are prepared (alpha/beta/rc) - the documentation should be published to
	the `staging` bucket and `staging` site should be built and published.
	* when final releases of distributions are prepared - the documentation should be published to the `live`
	bucket and the `live` website should be built and published.

	When the release manager publishes the documentation, they choose `auto` destination by default - depending on the
	tag they use - `staging` will be used to publish from pre-release tag and `live` will be used to publish
	from the release tag.

	You can also specify whether documentation should be published to `live` or `staging`, thereby overriding the auto-detection.

	The person who triggers the build (release manager) should specify the tag name of the docs to be published
	and the list of documentation packages to be published. Usually it is:

	* Airflow: `apache-airflow docker-stack task-sdk apache-airflow-ctl apache-airflow-mypy`
	* Helm chart: `helm-chart`
	* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published.

	Optionally - specifically if we run `all-providers` and the release manager wants to exclude some providers,
	they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages
	specified to be published without exclusions.


	Example screenshot of the workflow triggered from the GitHub UI:

	![Publishing airflow or providers](images/publish_airflow.png)

	Note that this just publishes the documentation but does not update the "site" with version numbers or
	stable links to providers and airflow - if you release a new documentation version, it will be available
	with direct URL (say https://airflow.apache.org/docs/apache-airflow/3.0.1/), but the main site will still
	point to the previous version of the documentation as `stable` and the version drop-downs will not be updated.

	In order to do it, you need to run the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
	workflow in `airflow-site` repository.

	For `live` site you should run the workflow in `main` branch. For `staging` site it should be staging branch.
	This will build the website and publish it to the `publish` branch of `airflow-site` repository (for `live`
	site) or `publish-staging` branch (for `staging` site). The workflow will also update the website,
	including refreshing the version numbers in the drop-downs and stable links.

	The `staging` documentation is produced automatically with `staging` watermark added.

	![Publishing site](images/publish_site.png)

	This workflow also invalidates cache in Fastly that Apache Software Foundation uses to serve the website,
	so you should always run it after you modify the documentation for the website. Fastly is configured with a
	3600-second TTL, which means that changes may take up to ~1 hour to propagate to the website.

	Shortly after the workflow succeeds and documentation is published, in the live bucket, the [airflow-site-archive](https://github.com/apache/airflow-site-archive)
	repository is automatically synchronized with the live S3 bucket. TODO: IMPLEMENT THIS, FOR NOW IT HAS
	TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)
	workflow in `airflow-site-archive` repository. The `airflow-site-archive` essentially keeps the history of
	snapshots of the `live` documentation.

	## Publishing changes to the website (including theme)

	The workflows in `apache-airflow` only update the documentation for the packages (Airflow, Helm chart,
	Providers, Docker Stack) that we publish from airflow sources. If we want to publish changes to the website
	itself or to the theme (css, javascript) we need to do it in `airflow-site` repository.

	Publishing of airflow-site happens automatically when a PR from `airflow-site` is merged to `main` or when
	the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) workflow is triggered
	manually in the main branch of `airflow-site` repository. The workflow builds the website and publishes it to
	`publish` branch of `airflow-site` repository, which in turn gets picked up by the ASF servers and is
	published as the official website. This includes any changes to `.htaccess` of the website.

	Such a main build also publishes the latest "sphinx-airflow-theme" package to GitHub so that the next build
	of documentation can automatically pick it up from there. This means that if you want to make changes to
	`javascript` or `css` that are part of the theme, you need to do it in `airflow-site` repository and
	merge it to `main` branch in order to be able to run the documentation build in `apache-airflow` repository
	and pick up the latest version of the theme.

	The version of sphinx theme is fixed in both repositories:

	* https://github.com/apache/airflow-site/blob/main/sphinx_airflow_theme/sphinx_airflow_theme/__init__.py#L21
	* https://github.com/apache/airflow/blob/main/devel-common/pyproject.toml#L77 in "docs" section

	In case of bigger changes to the theme, we can first iterate on the website and merge a new theme version,
	and only after that can we switch to the new version of the theme.

	# Publishing changes manually

	Sometimes you do not want to use Publishing workflows to publish individual files and caches might get
	into the way as both Cloudfront and Fastly caches might take some time to invalidate. In such a case, when
	you manually upload the files to S3 bucket, you can immediately invalidate the caches:

	1) Manually run invalidation request in Cloudfront for the documentation S3 bucket you uploaded the files
	to via AWS Console or AWS CLI (You can use `/*` to invalidate all files).

	![Cloudfront invalidation](images/cloudfront_invalidation.png)

	2) Run the `Build docs` workflow in `airflow-site` repository to invalidate Fastly cache for the website.
	Use `main` branch to rebuild site for `live` site and `staging` to rebuild the `staging` site:

	![Build docs](images/build-docs.png)

	# Fixing historical documentation

	Sometimes we need to update historical documentation (modify generated `html`) - for example, when we find
	bad links or when we change some of the structure in the documentation. This can be done via the
	`airflow-site-archive` repository. The workflow is as follows:

	1. Get the latest version of the documentation from S3 to `airflow-site-archive` repository using
	`Sync S3 to GitHub` workflow. This will download the latest version of the documentation from S3 to
	`airflow-site-archive` repository (this should normally not be needed, if automated synchronization works).
	2. Make the changes to the documentation in `airflow-site-archive` repository. This can be done using any
	text editor, script, etc. Those files are generated as `html` files and are not meant to be regenerated,
	they should be modified as `html` files in-place

	> [!IMPORTANT]
	> When you modify a file for latest published version of the documentation you should do it in two
	> places - in the `stable` version and in the `X.Y.Z` version of the documentation, as `stable` is just a
	> copy of the latest version of the documentation.

	3. Commit the changes to `airflow-site-archive` repository and push them to `some` branch of the repository.
	4. Create a Pull Request from that branch and merge it to `main`
	5. Run `Sync GitHub to S3` workflow in `airflow-site-archive` repository. This will upload the modified
	documentation to the S3 bucket. Use `main` branch (default) as "Reference of the commit used
	for synchronization". You can choose whether to sync the changes to `live` or `staging` bucket.
	The default is `live`:
	![Sync GitHub to S3](images/sync_github_to_s3.png)
	6. By default, the workflow will synchronize all documentation modified in a single last commit pushed to
	the `main`. You can also specify `full_sync` to synchronize all files in the repository if you want to
	make sure that S3 reflects `main`. The workflow might run for a long time (hours) in case of full sync
	or many changes to the `.html` files.
	7. In case you specify `full_sync`, you can also synchronize `all` docs or only selected documentation
	packages (for example `apache-airflow` or `docker-stack` or `amazon` or `helm-chart`) - you can specify
	more than one package separated by spaces.
	8. The workflow will invalidate Cloudfront cache for "live" or "staging" bucket respectively.
	9. Run the `Build docs` workflow in `airflow-site` repository to make sure that Fastly cache of the
	https://airflow.apache.org or https://airflow.staged.apache.org/ invalidated. Use `main` to rebuild site
	for `live` site and `staging` to rebuild the `staging` site:
	![Build docs](images/build-docs.png)


	## Manually publishing documentation directly to S3

	The regular publishing workflows involve running a GitHub Actions workflow, and they cover the majority of cases.
	However, sometimes, some manual updates and cherry-picks are needed, when we discover problems with the
	publishing and doc building code - for example, when we find that we need to fix extensions to sphinx.

	In such a case, the release manager or a committer can build and publish documentation locally - providing that
	they configure AWS credentials to be able to upload files to S3. You can ask in
	the #internal-airflow-ci-cd channel on Airflow Slack to get your AWS credentials configured.

	You can checkout locally a version of airflow repo that you need and apply any cherry-picks you need before
	running publishing.

	This is done using breeze. You also need to have AWS CLI installed and configured credentials to be able
	to upload files to S3. You can get credentials from one of the admins of Airflow's AWS account. The
	region to set for AWS is `us-east-2`.

	Note that it is advised to add `--dry-run` if you just want to see what would happen. Also, you can use
	the `s3://staging-docs-airflow-apache-org/docs/` bucket to test the publishing using staging site.

	```bash
	breeze build-docs "<package_id1>" "<package_id2>"
	mkdir /tmp/airflow-site
	breeze release-management publish-docs --override-versioned --airflow-site-directory /tmp/airflow-site
	breeze release-management publish-docs-to-s3 --source-dir-path /tmp/airflow-site/docs-archive \
	--destination-location s3://live-docs-airflow-apache-org/docs/ --stable-versions \
	--exclude-docs "<package_id1_to_exclude> <package_id2_to_exclude>" [--dry-run]
	```

	## Manually publishing documentation via `apache-airflow-site-archive` repo

	If you do not have S3 credentials and want to be careful about publishing the documentation, you can also
	use publishing via `apache-airflow-site-archive` repository. This is a little more complex, but it allows
	you to publish documentation without having S3 credentials.

	The process is as follows:

	1. Run `Sync s3 to GitHub` workflow in `apache-airflow-site-archive` repository. This will download the
	latest version of the documentation from S3 to `airflow-site-archive` repository (this should normally not
	be needed, if automated synchronization works).
	2. Checkout the `apache-airflow-site-archive` repository and create a branch for your changes.
	3. Build documentation locally in `apache-airflow` repo with any cherry-picks and modifications you need, and
	publish the docs to the checked out `airflow-site-archive` branch.

	```bash
	breeze build-docs "<package_id1>" "<package_id2>"
	breeze release-management publish-docs --override-versioned --airflow-site-directory <PATH_TO_THE_ARCHIVE_REPO>
	```

	4. Commit the changes to `apache-airflow-site-archive` repository and push them to `some` branch of the
	repository.
	5. Run `Sync GitHub to S3` workflow in `apache-airflow-site-archive` repository. This will upload the modified
	documentation to S3 bucket. You can choose whether to sync the changes to `live` or `staging` bucket.
	The default is `live`. You can also specify which folders to sync - by default, all modified folders are synced.
	6. After you synchronize the changes to S3, the Sync `S3 to GitHub` workflow will be triggered
	automatically and the changes will be synchronized to `airflow-site-archive` `main` branch - so there
	is no need to merge your changes to `main` branch of `airflow-site-archive` repository. You can safely
	delete the branch you created in step 2.