Google Git
Sign in
apache / airflow / refs/tags/2.0.0rc2
tage4e8056ec1fbc53c2971beff46cc5ec635428cfc
taggerAsh Berlin-Taylor <ash@apache.org>Sat Dec 12 15:02:45 2020 +0000
object16d0ae7b7bac594a862fc4d1ae39340b4e4ac63b 
Release 2.0.0rc2

Filtered changelog from rc1

16d0ae7b7 Update sqlalchemy_jsonfield to avoid pkgresources use (#13032)
db027735a Changes release image preparation to use PyPI packages (#12990)
f015296d0 Add more links to navbar for production docs (#12953)
1a56a58a0 Bump ini from 1.3.5 to 1.3.8 in /airflow/www (#13030)
baa68ca51 Adds customized_form_field_behaviours.schema.json to MANIFEST.in (#13031)
5057f56d2 Handle None values properly when CLI output is YAML/JSON format (#13024)
5495ab074 Fix broken build of docs/ by removing unused import (#13007)
08faa9d18 Detect invalid package fiiters (#12996)
2efd9ff85 Fix failing master (#13001)
81a1305bb Trigger provider-yamls check on docs change (#12998)
1fafc8bc4 Display progress for docs build (#13000)
aadecf716 Less verbose output for docs build (#12994)
969d3ea4f Add changes from 1.10.14 (#12993)
57528210e Promote new flags in ./docs/build_docs.py (#12991)
aa58ef150 Download inventories only once (#12989)
2ec03cd92 Update Dockerfile (#12987)
d84faa36a Update Dockerfile.ci (#12988)
fbd8348d0 Allow all default roles to view Profile page + allow editing profile/resetting password for DB-ModelView. (#12971)
db166ba75 Update CI to run tests againt v2-0-test branch (#10891)
4fe156f98 Remove unused pre-commit and Fix CI (#12964)
commit16d0ae7b7bac594a862fc4d1ae39340b4e4ac63b[log] [tgz]
authorAsh Berlin-Taylor <ash_github@firemirror.com>Sat Dec 12 13:45:16 2020 +0000
committerGitHub <noreply@github.com>Sat Dec 12 13:45:16 2020 +0000
tree7aafe604f69df73ae688f23a88c69426d1781abc
parentdb027735a7ee03d2678c8ca8712dfd75de148261 [diff]
Update sqlalchemy_jsonfield to avoid pkgresources use (#13032)

The previous version of sqlalchemy_jsonfield imported pkg_resources
which slowed down a lot of things. They have just released 1.0.0 with
that change.
1 file changed
tree: 7aafe604f69df73ae688f23a88c69426d1781abc
  1. .github/
  2. airflow/
  3. chart/
  4. clients/
  5. dags/
  6. dev/
  7. docker-context-files/
  8. docs/
  9. empty/
  10. hooks/
  11. images/
  12. kubernetes_tests/
  13. license-templates/
  14. licenses/
  15. manifests/
  16. metastore_browser/
  17. provider_packages/
  18. scripts/
  19. tests/
  20. .asf.yaml
  21. .bash_completion
  22. .coveragerc
  23. .dockerignore
  24. .editorconfig
  25. .flake8
  26. .gitignore
  27. .hadolint.yaml
  28. .mailmap
  29. .markdownlint.yml
  30. .pre-commit-config.yaml
  31. .rat-excludes
  32. .readthedocs.yml
  33. breeze
  34. breeze-complete
  35. BREEZE.rst
  36. CHANGELOG.txt
  37. CI.rst
  38. CODE_OF_CONDUCT.md
  39. codecov.yml
  40. confirm
  41. CONTRIBUTING.rst
  42. Dockerfile
  43. Dockerfile.ci
  44. IMAGES.rst
  45. INSTALL
  46. INTHEWILD.md
  47. LICENSE
  48. LOCAL_VIRTUALENV.rst
  49. MANIFEST.in
  50. NOTICE
  51. PULL_REQUEST_WORKFLOW.rst
  52. pylintrc
  53. pyproject.toml
  54. pytest.ini
  55. README.md
  56. setup.cfg
  57. setup.py
  58. STATIC_CODE_CHECKS.rst
  59. TESTING.rst
  60. UPDATING.md
  61. yamllint-config.yml
README.md

Apache Airflow

PyPI version GitHub Build Coverage Status License PyPI - Python Version Docker Pulls Docker Stars PyPI - Downloads Code style: black Twitter Follow Slack Status

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.

When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

Table of contents

Project Focus

Airflow works best with workflows that are mostly static and slowly changing. When DAG structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include Luigi, Oozie and Azkaban.

Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e. results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's Xcom feature). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work.

Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches.

Principles

  • Dynamic: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
  • Extensible: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
  • Elegant: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful Jinja templating engine.
  • Scalable: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers.

Requirements

Apache Airflow is tested with:

Master version (2.0.0dev)Stable version (1.10.14)
Python3.6, 3.7, 3.82.7, 3.5, 3.6, 3.7, 3.8
PostgreSQL9.6, 10, 11, 12, 139.6, 10, 11, 12, 13
MySQL5.7, 85.6, 5.7
SQLitelatest stablelatest stable
Kubernetes1.16.9, 1.17.5, 1.18.61.16.9, 1.17.5, 1.18.6

Note: MariaDB and MySQL 5.x are unable to or have limitations with running multiple schedulers -- please see the “Scheduler” docs.

Note: SQLite is used in Airflow tests. Do not use it in production.

Additional notes on Python version requirements

  • Stable version requires at least Python 3.5.3 when using Python 3

Getting started

Visit the official Airflow website documentation (latest stable release) for help with installing Airflow, getting started, or walking through a more complete tutorial.

Note: If you're looking for documentation for master branch (latest development branch): you can find it on s.apache.org/airflow-docs.

For more information on Airflow's Roadmap or Airflow Improvement Proposals (AIPs), visit the Airflow Wiki.

Official Docker (container) images for Apache Airflow are described in IMAGES.rst.

Installing from PyPI

We publish Apache Airflow as apache-airflow package in PyPI. Installing it however might be sometimes tricky because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open and applications usually pin them, but we should do neither and both at the same time. We decided to keep our dependencies as open as possible (in setup.py) so users can install different versions of libraries if needed. This means that from time to time plain pip install apache-airflow will not work or will produce unusable Airflow installation.

In order to have repeatable installation, however, introduced in Airflow 1.10.10 and updated in Airflow 1.10.12 we also keep a set of “known-to-be-working” constraint files in the orphan constraints-master and constraints-1-10 branches. We keep those “known-to-be-working” constraints files separately per major/minor python version. You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify correct Airflow tag/version/branch and python versions in the URL.

  1. Installing just Airflow:

NOTE!!!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 pip upgrade --pip==20.2.4 or, in case you use Pip 20.3, you need to add option --use-deprecated legacy-resolver to your pip install command.

pip install apache-airflow==1.10.14 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt"
  1. Installing with extras (for example postgres,google)
pip install apache-airflow[postgres,google]==1.10.14 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt"

For information on installing backport providers check [/docs/backport-providers.rst][/docs/backport-providers.rst].

Official source code

Apache Airflow is an Apache Software Foundation (ASF) project, and our official source code releases:

Following the ASF rules, the source packages released must be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools.

Convenience packages

There are other ways of installing and using Airflow. Those are “convenience” methods - they are not “official releases” as stated by the ASF Release Policy, but they can be used by the users who do not want to build the software themselves.

Those are - in the order of most common ways people install Airflow:

  • PyPI releases to install Airflow using standard pip tool
  • Docker Images to install airflow via docker tool, use them in Kubernetes, Helm Charts, docker-compose, docker swarm etc. You can read more about using, customising, and extending the images in the Latest docs, and learn details on the internals in the IMAGES.rst document.
  • Tags in GitHub to retrieve the git project sources that were used to generate official source packages via git

All those artifacts are not official releases, but they are prepared using officially released sources. Some of those artifacts are “development” or “pre-release” ones, and they are clearly marked as such following the ASF Policy.

User Interface

  • DAGs: Overview of all DAGs in your environment.

    DAGs

  • Tree View: Tree representation of a DAG that spans across time.

    Tree View

  • Graph View: Visualization of a DAG's dependencies and their current status for a specific run.

    Graph View

  • Task Duration: Total time spent on different tasks over time.

    Task Duration

  • Gantt View: Duration and overlap of a DAG.

    Gantt View

  • Code View: Quick way to view source code of a DAG.

    Code View

Contributing

Want to help build Apache Airflow? Check out our contributing documentation.

Who uses Apache Airflow?

More than 350 organizations are using Apache Airflow in the wild.

Who Maintains Apache Airflow?

Airflow is the work of the community, but the core committers/maintainers are responsible for reviewing and merging PRs as well as steering conversation around new feature requests. If you would like to become a maintainer, please review the Apache Airflow committer requirements.

Can I use the Apache Airflow logo in my presentation?

Yes! Be sure to abide by the Apache Foundation trademark policies and the Apache Airflow Brandbook. The most up to date logos are found in this repo and on the Apache Software Foundation website.

Airflow merchandise

If you would love to have Apache Airflow stickers, t-shirt etc. then check out Redbubble Shop.

Links