
 .. Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

 ..   http://www.apache.org/licenses/LICENSE-2.0

 .. Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

Provider packages
-----------------

Apache Airflow 2 is built in modular way. The "Core" of Apache Airflow provides core scheduler
functionality which allow you to write some basic tasks, but the capabilities of Apache Airflow can
be extended by installing additional packages, called ``providers``.

Providers can contain operators, hooks, sensor, and transfer operators to communicate with a
multitude of external systems, but they can also extend Airflow core with new capabilities.

You can install those provider packages separately in order to interface with a given service. The providers
for ``Apache Airflow`` are designed in the way that you can write your own providers easily. The
``Apache Airflow Community`` develops and maintain more than 80 provider packages, but you are free to
develop your own providers - the providers you build have exactly the same capability as the providers
written by the community, so you can release and share those providers with others.


If you want to learn how to build your own custom provider, you can find all the information about it at
:doc:`/howto/create-custom-providers`.


The full list of all the community managed providers is available at
`Providers Index <https://airflow.apache.org/docs/#providers-packages-docs-apache-airflow-providers-index-html>`_.

You can also see index of all the community provider's operators and hooks in
:doc:`/operators-and-hooks-ref/index`

Extending Airflow core functionality
------------------------------------

Providers give you the capability of extending core Airflow with extra capabilities. The Core airflow
provides basic and solid functionality of scheduling, the providers extend its capabilities. Here we
describe all the custom capabilities.

Airflow automatically discovers which providers add those additional capabilities and, once you install
provider package and re-start Airflow, those become automatically available to Airflow Users.

The summary of all the core functionalities that can be extended are available in
:doc:`/core-extensions/index`.

Configuration
'''''''''''''

Providers can have their own configuration options which allow you to configure how they work:

You can see all community-managed providers with their own configuration in
:doc:`/core-extensions/configurations`

Auth backends
'''''''''''''

The providers can add custom authentication backends, that allow you to configure the way how your
web server authenticates your users, integrating it with public or private authentication services.

You can see all the authentication backends available via community-managed providers in
:doc:`/core-extensions/auth-backends`

Custom connections
''''''''''''''''''

The providers can add custom connection types, extending connection form and handling custom form field
behaviour for the connections defined by the provider.

You can see all the custom connections available via community-managed providers in
:doc:`/core-extensions/connections`.

Extra links
'''''''''''

The providers can add extra custom links to operators delivered by the provider. Those will be visible in
task details view of the task.

You can see all the extra links available via community-managed providers in
:doc:`/core-extensions/extra-links`.


Logging
'''''''

The providers can add additional task logging capabilities. By default ``Apache Airflow`` saves logs for
tasks locally and make them available to Airflow UI via internal http server. However, providers
can add extra logging capabilities, where Airflow Logs can be written to a remote service and
retrieved from those services.

You can see all task loggers available via community-managed providers in
:doc:`/core-extensions/logging`.


Secret backends
'''''''''''''''

Airflow has the capability of reading connections, variables and configuration from Secret Backends rather
than from its own Database.

You can see all the secret backends available via community-managed providers in
:doc:`/core-extensions/secrets-backends`.

Notifications
'''''''''''''

The providers can add custom notifications, that allow you to configure the way how you would like to receive
notifications about the status of your tasks/DAGs.

You can see all the notifications available via community-managed providers in
:doc:`/core-extensions/notifications`.


Installing and upgrading providers
----------------------------------

Separate provider packages give the possibilities that were not available in 1.10:

1. You can upgrade to latest version of particular providers without the need of Apache Airflow core upgrade.

2. You can downgrade to previous version of particular provider in case the new version introduces
   some problems, without impacting the main Apache Airflow core package.

3. You can release and upgrade/downgrade provider packages incrementally, independent from each other. This
   means that you can incrementally validate each of the provider package update in your environment,
   following the usual tests you have in your environment.


Types of providers
------------------

Providers have the same capacity - no matter if they are provided by the community or if they are
third-party providers. This chapter explains how community managed providers are versioned and released
and how you can create your own providers.

.. _providers:community-maintained-providers:

Community maintained providers
''''''''''''''''''''''''''''''

From the point of view of the community, Airflow is delivered in multiple, separate packages.
The core of Airflow scheduling system is delivered as ``apache-airflow`` package and there are more than
80 provider packages which can be installed separately as so called ``Airflow Provider packages``.
Those packages are available as ``apache-airflow-providers`` packages - for example there is an
``apache-airflow-providers-amazon`` or ``apache-airflow-providers-google`` package).

Community maintained providers are released and versioned separately from the Airflow releases. We are
following the `Semver <https://semver.org/>`_ versioning scheme for the packages. Some versions of the
provider packages might depend on particular versions of Airflow, but the general approach we have is that
unless there is a good reason, new version of providers should work with recent versions of Airflow 2.x.
Details will vary per-provider and if there is a limitation for particular version of particular provider,
constraining the Airflow version used, it will be included as limitation of dependencies in the provider
package.

Each community provider has corresponding extra which can be used when installing airflow to install the
provider together with ``Apache Airflow`` - for example you can install airflow with those extras:
``apache-airflow[google,amazon]`` (with correct constraints -see :doc:`apache-airflow:installation/index`) and you
will install the appropriate versions of the ``apache-airflow-providers-amazon`` and
``apache-airflow-providers-google`` packages together with ``Apache Airflow``.

Some of the community  providers have cross-provider dependencies as well. Those are not required
dependencies, they might simply enable certain features (for example transfer operators often create
dependency between different providers. Again, the general approach here is that the providers are backwards
compatible, including cross-dependencies. Any kind of breaking changes and requirements on particular versions of other
provider packages are automatically documented in the release notes of every provider.

.. note::
    For Airflow 1.10 We also provided ``apache-airflow-backport-providers`` packages that could be installed
    with those versions Those were the same providers as for 2.0 but automatically back-ported to work for
    Airflow 1.10. The last release of backport providers was done on March 17, 2021 and the backport
    providers will no longer be released, since Airflow 1.10 has reached End-Of-Life as of June 17, 2021.


If you want to contribute to ``Apache Airflow``, you can see how to build and extend community
managed providers in
``https://github.com/apache/airflow/blob/main/airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst``.

.. toctree::
    :hidden:
    :maxdepth: 2

    Providers <self>
    Installing from PyPI <installing-from-pypi>
    Installing from sources <installing-from-sources>
    Create custom providers <howto/create-custom-providers>
    Packages <packages-ref>
    Operators and hooks <operators-and-hooks-ref/index>
    Core Extensions <core-extensions/index>