Google Git
Sign in
apache / airflow-site / f7689e56856a989d8bb9f316d2e95fc25cbe24d7 / . / docs-archive / apache-airflow / 2.0.0 / apache-airflow / stable / _sources / extra-packages-ref.rst.txt
blob: b54733259faa5e6bf188a49b02927c642ed07bb0 [file] [log] [blame]
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Extra Packages Reference
''''''''''''''''''''''''
Here's the list of the :ref:`subpackages <installation:extra_packages>` and what they enable:
**Fundamentals:**
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| subpackage | install command | enables |
+=====================+=====================================================+======================================================================+
| all | ``pip install 'apache-airflow[all]'`` | All Airflow user facing features (no devel and doc requirements) |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` | All databases integrations |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| devel | ``pip install 'apache-airflow[devel]'`` | Minimum dev tools requirements (without ``all``) |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` | Same as ``devel`` + dependencies for developing the Hadoop stack |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| devel_all | ``pip install 'apache-airflow[devel_all]'`` | Everything needed for development (``devel_hadoop`` + ``all``) |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| devel_ci | ``pip install 'apache-airflow[devel_ci]'`` | All dependencies required for CI build. |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| doc | ``pip install 'apache-airflow[doc]'`` | Packages needed to build docs (included in ``devel``) |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| password | ``pip install 'apache-airflow[password]'`` | Password authentication for users |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
**Apache Software:**
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| subpackage | install command | enables |
+=====================+=====================================================+======================================================================+
| atlas | ``pip install 'apache-airflow[apache.atlas]'`` | Apache Atlas to use Data Lineage feature |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| beam | ``pip install 'apache-airflow[apache.beam]'`` | Apache Beam operators & hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| cassandra | ``pip install 'apache-airflow[apache.cassandra]'`` | Cassandra related operators & hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| druid | ``pip install 'apache-airflow[apache.druid]'`` | Druid related operators & hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| hdfs | ``pip install 'apache-airflow[apache.hdfs]'`` | HDFS hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| hive | ``pip install 'apache-airflow[apache.hive]'`` | All Hive related operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| kylin | ``pip install 'apache-airflow[apache.kylin]'`` | All Kylin related operators & hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| livy | ``pip install 'apache-airflow[apache.livy]'`` | All Livy related operators, hooks & sensors |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| pig | ``pip install 'apache-airflow[apache.pig]'`` | All Pig related operators & hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| pinot | ``pip install 'apache-airflow[apache.pinot]'`` | All Pinot related hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| spark | ``pip install 'apache-airflow[apache.spark]'`` | All Spark related operators & hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| sqoop | ``pip install 'apache-airflow[apache.sqoop]'`` | All Sqoop related operators & hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| webhdfs | ``pip install 'apache-airflow[apache.webhdfs]'`` | HDFS hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
**Services:**
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| subpackage | install command | enables |
+=====================+=====================================================+============================================================================+
| aws | ``pip install 'apache-airflow[amazon]'`` | Amazon Web Services |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| azure | ``pip install 'apache-airflow[microsoft.azure]'`` | Microsoft Azure |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| cloudant | ``pip install 'apache-airflow[cloudant]'`` | Cloudant hook |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| databricks | ``pip install 'apache-airflow[databricks]'`` | Databricks hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| datadog | ``pip install 'apache-airflow[datadog]'`` | Datadog hooks and sensors |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| dingding | ``pip install 'apache-airflow[dingding]'`` | Dingding hooks and sensors |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| discord | ``pip install 'apache-airflow[discord]'`` | Discord hooks and sensors |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| facebook | ``pip install 'apache-airflow[facebook]'`` | Facebook Social |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| gcp | ``pip install 'apache-airflow[google]'`` | Google Cloud |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| github_enterprise | ``pip install 'apache-airflow[github_enterprise]'`` | GitHub Enterprise auth backend |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| google_auth | ``pip install 'apache-airflow[google_auth]'`` | Google auth backend |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| hashicorp | ``pip install 'apache-airflow[hashicorp]'`` | Hashicorp Services (Vault) |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| jira | ``pip install 'apache-airflow[jira]'`` | Jira hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| opsgenie | ``pip install 'apache-airflow[opsgenie]'`` | OpsGenie hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| pagerduty | ``pip install 'apache-airflow[pagerduty]'`` | Pagerduty hook |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| plexus | ``pip install 'apache-airflow[plexus]'`` | Plexus service of CoreScientific.com AI platform |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| qds | ``pip install 'apache-airflow[qds]'`` | Enable QDS (Qubole Data Service) support |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| salesforce | ``pip install 'apache-airflow[salesforce]'`` | Salesforce hook |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| sendgrid | ``pip install 'apache-airflow[sendgrid]'`` | Send email using sendgrid |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| segment | ``pip install 'apache-airflow[segment]'`` | Segment hooks and sensors |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| sentry | ``pip install 'apache-airflow[sentry]'`` | Sentry service for application logging and monitoring |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| slack | ``pip install 'apache-airflow[slack]'`` | :class:`airflow.providers.slack.operators.slack.SlackAPIOperator` |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| snowflake | ``pip install 'apache-airflow[snowflake]'`` | Snowflake hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| telegram | ``pip install 'apache-airflow[telegram]'`` | :class:`airflow.providers.telegram.operators.telegram.TelegramOperator` |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| vertica | ``pip install 'apache-airflow[vertica]'`` | Vertica hook support as an Airflow backend |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| yandex | ``pip install 'apache-airflow[yandex]'`` | Yandex.cloud hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| zendesk | ``pip install 'apache-airflow[zendesk]'`` | Zendesk hooks |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
**Software:**
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| subpackage | install command | enables |
+=====================+=====================================================+====================================================================================+
| async | ``pip install 'apache-airflow[async]'`` | Async worker classes for Gunicorn |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| celery | ``pip install 'apache-airflow[celery]'`` | CeleryExecutor |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| dask | ``pip install 'apache-airflow[dask]'`` | DaskExecutor |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| docker | ``pip install 'apache-airflow[docker]'`` | Docker hooks and operators |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| elasticsearch | ``pip install 'apache-airflow[elasticsearch]'`` | Elasticsearch hooks and Log Handler |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| exasol | ``pip install 'apache-airflow[exasol]'`` | Exasol hooks and operators |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| jenkins | ``pip install 'apache-airflow[jenkins]'`` | Jenkins hooks and operators |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| kubernetes | ``pip install 'apache-airflow[cncf.kubernetes]'`` | Kubernetes Executor and operator |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| mongo | ``pip install 'apache-airflow[mongo]'`` | Mongo hooks and operators |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| mssql (deprecated) | ``pip install 'apache-airflow[microsoft.mssql]'`` | Microsoft SQL Server operators and hook, |
| | | support as an Airflow backend. Uses pymssql. |
| | | Will be replaced by subpackage ``odbc``. |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| mysql | ``pip install 'apache-airflow[mysql]'`` | MySQL operators and hook, support as an Airflow |
| | | backend. The version of MySQL server has to be |
| | | 5.6.4+. The exact version upper bound depends |
| | | on version of ``mysqlclient`` package. For |
| | | example, ``mysqlclient`` 1.3.12 can only be |
| | | used with MySQL server 5.6.4 through 5.7. |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| odbc | ``pip install 'apache-airflow[odbc]'`` | ODBC data sources including MS SQL Server. Can use MsSqlOperator, |
| | | or as metastore database backend. Uses pyodbc. |
| | | See :doc:`apache-airflow-providers-odbc:index` for more info. |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| openfaas | ``pip install 'apache-airflow[openfaas]'`` | OpenFaaS hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| oracle | ``pip install 'apache-airflow[oracle]'`` | Oracle hooks and operators |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| postgres | ``pip install 'apache-airflow[postgres]'`` | PostgreSQL operators and hook, support as an |
| | | Airflow backend |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| presto | ``pip install 'apache-airflow[presto]'`` | All Presto related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| rabbitmq | ``pip install 'apache-airflow[rabbitmq]'`` | RabbitMQ support as a Celery backend |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| redis | ``pip install 'apache-airflow[redis]'`` | Redis hooks and sensors |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| samba | ``pip install 'apache-airflow[samba]'`` | :class:`airflow.providers.apache.hive.transfers.hive_to_samba.HiveToSambaOperator` |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| singularity | ``pip install 'apache-airflow[singularity]'`` | Singularity container operator |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| statsd | ``pip install 'apache-airflow[statsd]'`` | Needed by StatsD metrics |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| tableau | ``pip install 'apache-airflow[tableau]'`` | Tableau visualization integration |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
| virtualenv | ``pip install 'apache-airflow[virtualenv]'`` | Running python tasks in local virtualenv |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
**Other:**
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| subpackage | install command | enables |
+=====================+=====================================================+======================================================================+
| cgroups | ``pip install 'apache-airflow[cgroups]'`` | Needed To use CgroupTaskRunner |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| ftp | ``pip install 'apache-airflow[ftp]'`` | FTP hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| grpc | ``pip install 'apache-airflow[grpc]'`` | Grpc hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| http | ``pip install 'apache-airflow[http]'`` | HTTP hooks, operators and sensors |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| imap | ``pip install 'apache-airflow[imap]'`` | IMAP hooks and sensors |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| jdbc | ``pip install 'apache-airflow[jdbc]'`` | JDBC hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| kerberos | ``pip install 'apache-airflow[kerberos]'`` | Kerberos integration for Kerberized Hadoop |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| ldap | ``pip install 'apache-airflow[ldap]'`` | LDAP authentication for users |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| papermill | ``pip install 'apache-airflow[papermill]'`` | Papermill hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| sftp | ``pip install 'apache-airflow[sftp]'`` | SFTP hooks, operators and sensors |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| sqlite | ``pip install 'apache-airflow[sqlite]'`` | SQLite hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| ssh | ``pip install 'apache-airflow[ssh]'`` | SSH hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| winrm | ``pip install 'apache-airflow[microsoft.winrm]'`` | WinRM hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
** Deprecated 1.10 Extras **
Those are the extras that have been deprecated in 2.0 and will be removed
(current plan is to remove them in 2.1):
+---------------------+-----------------------------+
| Deprecated extra | Extra to be used instead |
+=====================+=============================+
| atlas | apache.atlas |
+---------------------+-----------------------------+
| aws | amazon |
+---------------------+-----------------------------+
| azure | microsoft.azure |
+---------------------+-----------------------------+
| cassandra | apache.cassandra |
+---------------------+-----------------------------+
| crypto | - * |
+---------------------+-----------------------------+
| druid | apache.druid |
+---------------------+-----------------------------+
| gcp | google |
+---------------------+-----------------------------+
| gcp_api | google |
+---------------------+-----------------------------+
| hdfs | apache.hdfs |
+---------------------+-----------------------------+
| hive | apache.hive |
+---------------------+-----------------------------+
| kubernetes | cncf.kubernetes |
+---------------------+-----------------------------+
| mssql | microsoft.mssql |
+---------------------+-----------------------------+
| pinot | apache.pinot |
+---------------------+-----------------------------+
| qds | qubole |
+---------------------+-----------------------------+
| s3 | amazon |
+---------------------+-----------------------------+
| webhdfs | apache.webhdfs |
+---------------------+-----------------------------+
| winrm | microsoft.winrm |
+---------------------+-----------------------------+
* crypto extra is not needed as cryptography is installed by default in Airflow 2.0