| .. Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| .. http://www.apache.org/licenses/LICENSE-2.0 |
| |
| .. Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| |
| .. _google_cloud_secret_manager_backend: |
| |
| Google Cloud Secret Manager Backend |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as |
| a secret backend and how to manage secrets. |
| |
| Before you begin |
| """""""""""""""" |
| |
| Before you start, make sure you have performed the following tasks: |
| |
| 1. Include ``google`` subpackage as an extra of your Airflow installation |
| |
| .. code-block:: bash |
| |
| pip install apache-airflow[google] |
| |
| 2. `Configure Secret Manager and your local environment <https://cloud.google.com/secret-manager/docs/configuring-secret-manager>`__, once per project. |
| |
| Enabling the secret backend |
| """"""""""""""""""""""""""" |
| |
| To enable the secret backend for Google Cloud Secrets Manager to retrieve connection/variables, |
| specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend` |
| as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``. |
| |
| Here is a sample configuration if you want to use it: |
| |
| .. code-block:: ini |
| |
| [secrets] |
| backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend |
| |
| You can also set this with environment variables. |
| |
| .. code-block:: bash |
| |
| export AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend |
| |
| You can verify the correct setting of the configuration options with the ``airflow config get-value`` command. |
| |
| .. code-block:: console |
| |
| $ airflow config get-value secrets backend |
| airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend |
| |
| Backend parameters |
| """""""""""""""""" |
| |
| The next step is to configure backend parameters using the ``backend_kwargs`` options. You can pass |
| the following parameters: |
| |
| * ``connections_prefix``: Specifies the prefix of the secret to read to get Connections. Default: ``"airflow-connections"`` |
| * ``variables_prefix``: Specifies the prefix of the secret to read to get Variables. Default: ``"airflow-variables"`` |
| * ``gcp_key_path``: Path to Google Cloud Service Account Key file (JSON). |
| * ``gcp_keyfile_dict``: Dictionary of keyfile parameters. |
| * ``gcp_scopes``: Comma-separated string containing OAuth2 scopes. |
| * ``sep``: Separator used to concatenate connections_prefix and conn_id. Default: ``"-"`` |
| * ``project_id``: Project ID to read the secrets from. If not passed, the project ID from credentials will be used. |
| |
| All options should be passed as a JSON dictionary. |
| |
| For example, if you want to set parameter ``connections_prefix`` to ``"example-connections-prefix"`` and parameter ``variables_prefix`` to ``"example-variables-prefix"``, your configuration file should look like this: |
| |
| .. code-block:: ini |
| |
| [secrets] |
| backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend |
| backend_kwargs = {"connections_prefix": "example-connections-prefix", "variables_prefix": "example-variables-prefix"} |
| |
| Set-up credentials |
| """""""""""""""""" |
| |
| You can configure the credentials in three ways: |
| |
| * By default, Application Default Credentials (ADC) is used obtain credentials. |
| * ``gcp_key_path`` option in ``backend_kwargs`` option - allows you to configure authorizations with a service account stored in local file. |
| * ``gcp_keyfile_dict`` option in ``backend_kwargs`` option - allows you to configure authorizations with a service account stored in Airflow configuration. |
| |
| .. note:: |
| |
| For more information about the Application Default Credentials (ADC), see: |
| |
| * `google.auth.default <https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.default>`__ |
| * `Setting Up Authentication for Server to Server Production Applications <https://cloud.google.com/docs/authentication/production>`__ |
| |
| Managing secrets |
| """""""""""""""" |
| |
| If you want to configure a connection, you need to save it as a :ref:`connection URI representation <generating_connection_uri>`. |
| Variables should be saved as plain text. |
| |
| In order to manage secrets, you can use the ``gcloud`` tool or other supported tools. For more information, take a look at: |
| `Managing secrets <https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets>`__ in Google Cloud Documentation. |
| |
| The name of the secret must fit the following formats: |
| |
| * for connections: ``[connections_prefix][sep][connection_name]`` |
| * for variables: ``[variables_prefix][sep][variable_name]`` |
| * for Airflow config: ``[config_prefix][sep][config_name]`` |
| |
| where: |
| |
| * ``connections_prefix`` - fixed value defined in the ``connections_prefix`` parameter in backend configuration. Default: ``airflow-connections``. |
| * ``variables_prefix`` - fixed value defined in the ``variables_prefix`` parameter in backend configuration. Default: ``airflow-variables``. |
| * ``config_prefix`` - fixed value defined in the ``config_prefix`` parameter in backend configuration. Default: ``airflow-config``. |
| * ``sep`` - fixed value defined in the ``sep`` parameter in backend configuration. Default: ``-``. |
| |
| The Cloud Secrets Manager secret name should follow the pattern ``^[a-zA-Z0-9-_]*$``. |
| |
| If you have the default backend configuration and you want to create a connection with ``conn_id`` |
| equals ``first-connection``, you should create secret named ``airflow-connections-first-connection``. |
| You can do it with the gcloud tools as in the example below. |
| |
| .. code-block:: bash |
| |
| $ echo "mysql://example.org" | gcloud beta secrets create \ |
| airflow-connections-first-connection \ |
| --data-file=- \ |
| --replication-policy=automatic |
| Created version [1] of the secret [airflow-variables-first-connection]. |
| |
| If you have the default backend configuration and you want to create a variable named ``first-variable``, |
| you should create a secret named ``airflow-variables-first-variable``. You can do it with the gcloud |
| command as in the example below. |
| |
| .. code-block:: bash |
| |
| $ echo "secret_content" | gcloud beta secrets create \ |
| airflow-variables-first-variable \ |
| --data-file=-\ |
| --replication-policy=automatic |
| Created version [1] of the secret [airflow-variables-first-variable]. |
| |
| .. note:: If only key of the connection should be hidden there is an option to store |
| only that key in Cloud Secret Manager and not entire connection. For more details take |
| a look at :ref:`Google Cloud Connection <howto/connection:gcp>`. |
| |
| Checking configuration |
| ====================== |
| |
| You can use the ``airflow connections get`` command to check if the connection is correctly read from the backend secret: |
| |
| .. code-block:: console |
| |
| $ airflow connections get first-connection |
| Id: null |
| Connection Id: first-connection |
| Connection Type: mysql |
| Host: example.org |
| Schema: '' |
| Login: null |
| Password: null |
| Port: null |
| Is Encrypted: null |
| Is Extra Encrypted: null |
| Extra: {} |
| URI: mysql://example.org |
| |
| To check the variables is correctly read from the backend secret, you can use ``airflow variables get``: |
| |
| .. code-block:: console |
| |
| $ airflow variables get first-variable |
| secret_content |
| |
| Clean up |
| ======== |
| |
| To avoid incurring charges to your Google Cloud account for the resources used in this guide, |
| delete secrets by running ``gcloud beta secrets delete``: |
| |
| .. code-block:: bash |
| |
| gcloud beta secrets delete airflow-connections-first-connection |
| gcloud beta secrets delete airflow-variables-first-variable |