blob: 5ceaa391669cae52d3d0a6485e68638b0e8c85a6 [file] [log] [blame]
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
AWS Secrets Manager Backend
To enable Secrets Manager, specify :py:class:``
as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``. These ``backend_kwargs`` are parsed as JSON, hence Python
values like the bool False or None will be ignored, taking for those kwargs the default values of the secrets backend.
Here is a sample configuration:
.. code-block:: ini
backend =
backend_kwargs = {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "profile_name": "default", "full_url_mode": false}
To authenticate you can either supply a profile name to reference aws profile, e.g. defined in ``~/.aws/config`` or set
environment variables like ``AWS_ACCESS_KEY_ID``, ``AWS_SECRET_ACCESS_KEY``.
Storing and Retrieving Connections
You can store the different values for a secret in two forms: storing the conn URI in one field (default mode) or using different
fields in Amazon Secrets Manager (setting ``full_url_mode`` as ``false`` in the backend config), as follow:
.. image:: img/aws-secrets-manager.png
By default you must use some of the following words for each kind of field:
* For storing passwords, valid key names are password, pass and key
* Users: user, username, login, user_name
* Host: host, remote_host, server
* Port: port
* You should also specify the type of connection, which can be done naming the key as conn_type, conn_id,
connection_type or engine. Valid values for this field are postgres, mysql, snowflake, google_cloud, mongo...
* For the extra value of the connections, a field called extra must exists. Please note this extra field
should be a valid JSON.
However, more words can be added to the list using the parameter ``extra_conn_words`` in the configuration. This
parameter has to be a dict of lists with the following optional keys: user, password, host, schema, conn_type
As an example, if you have set ``connections_prefix`` as ``airflow/connections``, then for a connection id of ``smtp_default``,
you would want to store your connection at ``airflow/connections/smtp_default``. This can be done through the AWS web
console or through Amazon CLI as shown below:
.. code-block:: bash
aws secretsmanager put-secret-value \
--secret-id airflow/connections/smtp_default \
--secret-string [{"user": "nice_user"}, {"pass": "this_is_the_password"}, {"host": ""}, {"port": "999"}]
Verify that you can get the secret:
.. code-block:: console
❯ aws secretsmanager get-secret-value --secret-id airflow/connections/smtp_default
"ARN": "arn:aws:secretsmanager:us-east-2:314524341751:secret:airflow/connections/smtp_default-7meuul",
"Name": "airflow/connections/smtp_default",
"VersionId": "34f90eff-ea21-455a-9c8f-5ee74b21be672",
"SecretString": "{\n \"user\":\"nice_user\",\n \"pass\":\"this_is_the_password\"\n,
\n \"host\":\"\"\n,\n \"port\":\"999\"\n}\n",
"VersionStages": [
"CreatedDate": "2020-04-08T02:10:35.132000+01:00"
If you don't want to use any ``connections_prefix`` for retrieving connections, set it as an empty string ``""`` in the configuration.
Storing and Retrieving Variables
If you have set ``variables_prefix`` as ``airflow/variables``, then for an Variable key of ``hello``,
you would want to store your Variable at ``airflow/variables/hello``.
Optional lookup
Optionally connections, variables, or config may be looked up exclusive of each other or in any combination.
This will prevent requests being sent to AWS Secrets Manager for the excluded type.
If you want to look up some and not others in AWS Secrets Manager you may do so by setting the relevant ``*_prefix`` parameter of the ones to be excluded as ``null``.
For example, if you want to set parameter ``connections_prefix`` to ``"airflow/connections"`` and not look up variables, your configuration file should look like this:
.. code-block:: ini
backend =
backend_kwargs = {"connections_prefix": "airflow/connections", "variables_prefix": null, "profile_name": "default"}
Example of storing Google Secrets in AWS Secrets Manager
For connecting to a google cloud conn, all the fields must be in the extra field, and their names follow the pattern
``extra_google_cloud_platform__value``. For example:
.. code-block:: ini
{'extra__google_cloud_platform__key_path': '/opt/airflow/service_account.json',
'extra__google_cloud_platform__scope': ''}