| .. Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| .. http://www.apache.org/licenses/LICENSE-2.0 |
| |
| .. Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| |
| Installation from PyPI |
| ---------------------- |
| |
| This page describes installations using the ``apache-airflow`` package `published in |
| PyPI <https://pypi.org/project/apache-airflow/>`__. |
| |
| Installation tools |
| '''''''''''''''''' |
| |
| Only ``pip`` installation is currently officially supported. |
| |
| While there are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or |
| `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as |
| ``pip`` - especially when it comes to constraint vs. requirements management. |
| Installing via ``Poetry`` or ``pip-tools`` is not currently supported. If you wish to install airflow |
| using those tools you should use the constraints and convert them to appropriate |
| format and workflow that your tool requires. |
| |
| Typical command to install airflow from PyPI looks like below: |
| |
| .. code-block:: |
| |
| pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.7.txt" |
| |
| This is an example, see further for more explanation. |
| |
| .. _installation:constraints: |
| |
| Constraints files |
| ''''''''''''''''' |
| |
| Airflow installation can be tricky sometimes because Airflow is both a library and an application. |
| Libraries usually keep their dependencies open and applications usually pin them, but we should do neither |
| and both at the same time. We decided to keep our dependencies as open as possible |
| (in ``setup.cfg`` and ``setup.py``) so users can install different |
| version of libraries if needed. This means that from time to time plain ``pip install apache-airflow`` will |
| not work or will produce an unusable Airflow installation. |
| |
| In order to have a repeatable installation, we also keep a set of "known-to-be-working" constraint files in the |
| ``constraints-main``, ``constraints-2-0``, ``constraints-2-1`` etc. orphan branches and then we create a tag |
| for each released version e.g. :subst-code:`constraints-|version|`. This way, we keep a tested and working set of dependencies. |
| |
| Those "known-to-be-working" constraints are per major/minor Python version. You can use them as constraint |
| files when installing Airflow from PyPI. Note that you have to specify the correct Airflow |
| and Python versions in the URL. |
| |
| You can create the URL to the file substituting the variables in the template below. |
| |
| .. code-block:: |
| |
| https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt |
| |
| where: |
| |
| - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version |
| - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.7`` |
| |
| There is also a no-providers constraint file, which contains just constraints required to install Airflow core. This allows |
| to install and upgrade airflow separately and independently from providers. |
| |
| You can create the URL to the file substituting the variables in the template below. |
| |
| .. code-block:: |
| |
| https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt |
| |
| You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest" |
| constraints always points to the "latest" released Airflow version constraints: |
| |
| .. code-block:: |
| |
| https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.7.txt |
| |
| Installation and upgrade scenarios |
| '''''''''''''''''''''''''''''''''' |
| |
| In order to simplify the installation, we have prepared examples of how to upgrade Airflow and providers. |
| |
| Installing Airflow with extras and providers |
| ============================================ |
| |
| If you need to install extra dependencies of Airflow, you can use the script below to make an installation |
| a one-liner (the example below installs Postgres and Google providers, as well as ``async`` extra). |
| |
| .. code-block:: bash |
| :substitutions: |
| |
| AIRFLOW_VERSION=|version| |
| PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)" |
| CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt" |
| pip install "apache-airflow[async,postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" |
| |
| Note, that it will install the versions of providers that were available at the moment this version of Airflow |
| has been prepared. You need to follow next steps if you want to upgrade provider packages in case they were |
| released afterwards. |
| |
| |
| Upgrading Airflow with providers |
| ================================ |
| |
| You can upgrade airflow together with extras (providers available at the time of the release of Airflow |
| being installed. |
| |
| |
| .. code-block:: bash |
| :substitutions: |
| |
| AIRFLOW_VERSION=|version| |
| PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)" |
| CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt" |
| pip install "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" |
| |
| Installation and upgrading of Airflow providers separately |
| ========================================================== |
| |
| You can manually install all the providers you need. You can continue using the "providers" constraint files |
| but the 'versioned' airflow constraints installs only the versions of providers that were available in PyPI at |
| the time of preparing of the airflow version. However, usually you can use "main" version of the providers |
| to install latest version of providers. Usually the providers work with most versions of Airflow, if there |
| will be any incompatibilities, it will be captured as package dependencies. |
| |
| Note that "main" is just an example - you might need to choose a specific airflow version to install providers |
| in specific version. |
| |
| .. code-block:: bash |
| |
| PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)" |
| # For example: 3.7 |
| CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-${PYTHON_VERSION}.txt" |
| pip install "apache-airflow-providers-google" --constraint "${CONSTRAINT_URL}" |
| |
| You can also upgrade the providers to latest versions (you need to use main version of constraints for that): |
| |
| .. code-block:: bash |
| |
| PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)" |
| # For example: 3.7 |
| CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-${PYTHON_VERSION}.txt" |
| pip install "apache-airflow-providers-google" --upgrade --constraint "${CONSTRAINT_URL}" |
| |
| |
| Installation and upgrade of Airflow core |
| ======================================== |
| |
| If you don't want to install any extra providers, initially you can use the command set below. |
| |
| .. code-block:: bash |
| :substitutions: |
| |
| AIRFLOW_VERSION=|version| |
| PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)" |
| # For example: 3.7 |
| CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt" |
| # For example: https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-no-providers-3.7.txt |
| pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" |
| |
| |
| Troubleshooting |
| ''''''''''''''' |
| |
| This section describes how to troubleshoot installation issues with PyPI installation. |
| |
| Airflow command is not recognized |
| ================================= |
| |
| If the ``airflow`` command is not getting recognized (can happen on Windows when using WSL), then |
| ensure that ``~/.local/bin`` is in your ``PATH`` environment variable, and add it in if necessary: |
| |
| .. code-block:: bash |
| |
| PATH=$PATH:~/.local/bin |
| |
| You can also start airflow with ``python -m airflow`` |
| |
| Symbol not found: ``_Py_GetArgcArgv`` |
| ===================================== |
| |
| If you see ``Symbol not found: _Py_GetArgcArgv`` while starting or importing Airflow, this may mean that you are using an incompatible version of Python. |
| For a homebrew installed version of Python, this is generally caused by using Python in ``/usr/local/opt/bin`` rather than the Frameworks installation (e.g. for ``python 3.7``: ``/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7``). |
| |
| The crux of the issue is that a library Airflow depends on, ``setproctitle``, uses a non-public Python API |
| which is not available from the standard installation ``/usr/local/opt/`` (which symlinks to a path under ``/usr/local/Cellar``). |
| |
| An easy fix is just to ensure you use a version of Python that has a dylib of the Python library available. For example: |
| |
| .. code-block:: bash |
| |
| # Note: these instructions are for python3.7 but can be loosely modified for other versions |
| brew install python@3.7 |
| virtualenv -p /usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/bin/python3 .toy-venv |
| source .toy-venv/bin/activate |
| pip install apache-airflow |
| python |
| >>> import setproctitle |
| # Success! |
| |
| Alternatively, you can download and install Python directly from the `Python website <https://www.python.org/>`__. |