blob: 3a87eba531247f3513eb7c03f65489bc193442d3 [file] [log] [blame]
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. _write-logs-azure:
Writing Logs to Azure Blob Storage
Airflow can be configured to read and write task logs in Azure Blob Storage. It uses an existing
Airflow connection to read or write logs. If you don't have a connection properly setup,
this process will fail.
Follow the steps below to enable Azure Blob Storage logging:
#. Airflow's logging system requires a custom ``.py`` file to be located in the :envvar:`PYTHONPATH`, so that it's importable from Airflow. Start by creating a directory to store the config file, ``$AIRFLOW_HOME/config`` is recommended.
#. Create empty files called ``$AIRFLOW_HOME/config/`` and ``$AIRFLOW_HOME/config/``.
#. Copy the contents of ``airflow/config_templates/`` into the ```` file created in ``Step 2``.
#. Customize the following portions of the template:
.. code-block:: ini
# wasb buckets should start with "wasb" just to help Airflow select correct handler
REMOTE_BASE_LOG_FOLDER = 'wasb-<whatever you want here>'
#. Make sure a Azure Blob Storage (Wasb) connection hook has been defined in Airflow. The hook should have read and write access to the Azure Blob Storage bucket defined above in ``REMOTE_BASE_LOG_FOLDER``.
#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain:
.. code-block:: ini
remote_logging = True
logging_config_class = log_config.LOGGING_CONFIG
remote_log_conn_id = <name of the Azure Blob Storage connection>
#. Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.
#. Verify that logs are showing up for newly executed tasks in the bucket you have defined.