| .. Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| .. http://www.apache.org/licenses/LICENSE-2.0 |
| |
| .. Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| |
| What is Airflow? |
| ========================================= |
| |
| `Apache Airflow <https://github.com/apache/airflow>`_ is an open-source platform for developing, scheduling, |
| and monitoring batch-oriented workflows. Airflow's extensible Python framework enables you to build workflows |
| connecting with virtually any technology. A web interface helps manage the state of your workflows. Airflow is |
| deployable in many ways, varying from a single process on your laptop to a distributed setup to support even |
| the biggest workflows. |
| |
| Workflows as code |
| ========================================= |
| The main characteristic of Airflow workflows is that all workflows are defined in Python code. "Workflows as |
| code" serves several purposes: |
| |
| - **Dynamic**: Airflow pipelines are configured as Python code, allowing for dynamic pipeline generation. |
| - **Extensible**: The Airflow framework contains operators to connect with numerous technologies. All Airflow components are extensible to easily adjust to your environment. |
| - **Flexible**: Workflow parameterization is built-in leveraging the `Jinja <https://jinja.palletsprojects.com>`_ templating engine. |
| |
| Take a look at the following snippet of code: |
| |
| .. code-block:: python |
| |
| from datetime import datetime |
| |
| from airflow import DAG |
| from airflow.decorators import task |
| from airflow.operators.bash import BashOperator |
| |
| # A DAG represents a workflow, a collection of tasks |
| with DAG(dag_id="demo", start_date=datetime(2022, 1, 1), schedule="0 0 * * *") as dag: |
| |
| # Tasks are represented as operators |
| hello = BashOperator(task_id="hello", bash_command="echo hello") |
| |
| @task() |
| def airflow(): |
| print("airflow") |
| |
| # Set dependencies between tasks |
| hello >> airflow() |
| |
| |
| Here you see: |
| |
| - A DAG named "demo", starting on Jan 1st 2022 and running once a day. A DAG is Airflow's representation of a workflow. |
| - Two tasks, a BashOperator running a Bash script and a Python function defined using the ``@task`` decorator |
| - ``>>`` between the tasks defines a dependency and controls in which order the tasks will be executed |
| |
| Airflow evaluates this script and executes the tasks at the set interval and in the defined order. The status |
| of the "demo" DAG is visible in the web interface: |
| |
| .. image:: /img/hello_world_graph_view.png |
| :alt: Demo DAG in the Graph View, showing the status of one DAG run |
| |
| This example demonstrates a simple Bash and Python script, but these tasks can run any arbitrary code. Think |
| of running a Spark job, moving data between two buckets, or sending an email. The same structure can also be |
| seen running over time: |
| |
| .. image:: /img/hello_world_grid_view.png |
| :alt: Demo DAG in the Grid View, showing the status of all DAG runs |
| |
| Each column represents one DAG run. These are two of the most used views in Airflow, but there are several |
| other views which allow you to deep dive into the state of your workflows. |
| |
| Why Airflow? |
| ========================================= |
| Airflow is a batch workflow orchestration platform. The Airflow framework contains operators to connect with |
| many technologies and is easily extensible to connect with a new technology. If your workflows have a clear |
| start and end, and run at regular intervals, they can be programmed as an Airflow DAG. |
| |
| If you prefer coding over clicking, Airflow is the tool for you. Workflows are defined as Python code which |
| means: |
| |
| - Workflows can be stored in version control so that you can roll back to previous versions |
| - Workflows can be developed by multiple people simultaneously |
| - Tests can be written to validate functionality |
| - Components are extensible and you can build on a wide collection of existing components |
| |
| Rich scheduling and execution semantics enable you to easily define complex pipelines, running at regular |
| intervals. Backfilling allows you to (re-)run pipelines on historical data after making changes to your logic. |
| And the ability to rerun partial pipelines after resolving an error helps maximize efficiency. |
| |
| Airflow's user interface provides both in-depth views of pipelines and individual tasks, and an overview of |
| pipelines over time. From the interface, you can inspect logs and manage tasks, for example retrying a task in |
| case of failure. |
| |
| The open-source nature of Airflow ensures you work on components developed, tested, and used by many other |
| `companies <https://github.com/apache/airflow/blob/main/INTHEWILD.md>`_ around the world. In the active |
| `community <https://airflow.apache.org/community>`_ you can find plenty of helpful resources in the form of |
| blogs posts, articles, conferences, books, and more. You can connect with other peers via several channels |
| such as `Slack <https://s.apache.org/airflow-slack>`_ and mailing lists. |
| |
| Why not Airflow? |
| ========================================= |
| Airflow was built for finite batch workflows. While the CLI and REST API do allow triggering workflows, |
| Airflow was not built for infinitely-running event-based workflows. Airflow is not a streaming solution. |
| However, a streaming system such as Apache Kafka is often seen working together with Apache Airflow. Kafka can |
| be used for ingestion and processing in real-time, event data is written to a storage location, and Airflow |
| periodically starts a workflow processing a batch of data. |
| |
| If you prefer clicking over coding, Airflow is probably not the right solution. The web interface aims to make |
| managing workflows as easy as possible and the Airflow framework is continuously improved to make the |
| developer experience as smooth as possible. However, the philosophy of Airflow is to define workflows as code |
| so coding will always be required. |
| |
| |
| .. toctree:: |
| :hidden: |
| :caption: Content |
| |
| Overview <self> |
| project |
| license |
| start |
| installation/index |
| upgrading-from-1-10/index |
| tutorial/index |
| howto/index |
| ui |
| concepts/index |
| executor/index |
| dag-run |
| plugins |
| security/index |
| logging-monitoring/index |
| timezone |
| Using the CLI <usage-cli> |
| integration |
| kubernetes |
| lineage |
| listeners |
| dag-serialization |
| modules_management |
| Release Policies <release-process> |
| release_notes |
| best-practices |
| production-deployment |
| faq |
| privacy_notice |
| |
| .. toctree:: |
| :hidden: |
| :caption: References |
| |
| Operators and hooks <operators-and-hooks-ref> |
| CLI <cli-and-env-variables-ref> |
| Templates <templates-ref> |
| Python API <python-api-ref> |
| Stable REST API <stable-rest-api-ref> |
| deprecated-rest-api-ref |
| Configurations <configurations-ref> |
| Extra packages <extra-packages-ref> |
| |
| .. toctree:: |
| :hidden: |
| :caption: Internal DB details |
| |
| Database Migrations <migrations-ref> |
| Database ERD Schema <database-erd-ref> |
