| :py:mod:`airflow.providers.microsoft.azure.operators.adls` |
| ========================================================== |
| |
| .. py:module:: airflow.providers.microsoft.azure.operators.adls |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.microsoft.azure.operators.adls.ADLSDeleteOperator |
| airflow.providers.microsoft.azure.operators.adls.ADLSListOperator |
| |
| |
| |
| |
| .. py:class:: ADLSDeleteOperator(*, path, recursive = False, ignore_not_found = True, azure_data_lake_conn_id = 'azure_data_lake_default', **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Delete files in the specified path. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:ADLSDeleteOperator` |
| |
| :param path: A directory or file to remove |
| :param recursive: Whether to loop into directories in the location and remove the files |
| :param ignore_not_found: Whether to raise error if file to delete is not found |
| :param azure_data_lake_conn_id: Reference to the :ref:`Azure Data Lake connection<howto/connection:adl>`. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['path'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #901dd2 |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: ADLSListOperator(*, path, azure_data_lake_conn_id = 'azure_data_lake_default', **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| List all files from the specified path |
| |
| This operator returns a python list with the names of files which can be used by |
| `xcom` in the downstream tasks. |
| |
| :param path: The Azure Data Lake path to find the objects. Supports glob |
| strings (templated) |
| :param azure_data_lake_conn_id: Reference to the :ref:`Azure Data Lake connection<howto/connection:adl>`. |
| |
| **Example**: |
| The following Operator would list all the Parquet files from ``folder/output/`` |
| folder in the specified ADLS account :: |
| |
| adls_files = ADLSListOperator( |
| task_id='adls_files', |
| path='folder/output/*.parquet', |
| azure_data_lake_conn_id='azure_data_lake_default' |
| ) |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['path'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #901dd2 |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |