| :py:mod:`airflow.providers.microsoft.azure.hooks.data_factory` |
| ============================================================== |
| |
| .. py:module:: airflow.providers.microsoft.azure.hooks.data_factory |
| |
| .. autoapi-nested-parse:: |
| |
| .. spelling:: |
| |
| CreateRunResponse |
| DatasetResource |
| LinkedServiceResource |
| LROPoller |
| PipelineResource |
| PipelineRun |
| TriggerResource |
| datafactory |
| mgmt |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.microsoft.azure.hooks.data_factory.PipelineRunInfo |
| airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryPipelineRunStatus |
| airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryHook |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.microsoft.azure.hooks.data_factory.provide_targeted_factory |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.microsoft.azure.hooks.data_factory.Credentials |
| |
| |
| .. py:data:: Credentials |
| |
| |
| |
| |
| .. py:function:: provide_targeted_factory(func) |
| |
| Provide the targeted factory to the decorated function in case it isn't specified. |
| |
| If ``resource_group_name`` or ``factory_name`` is not provided it defaults to the value specified in |
| the connection extras. |
| |
| |
| .. py:class:: PipelineRunInfo |
| |
| Bases: :py:obj:`airflow.typing_compat.TypedDict` |
| |
| Type class for the pipeline run info dictionary. |
| |
| .. py:attribute:: run_id |
| :annotation: :str |
| |
| |
| |
| .. py:attribute:: factory_name |
| :annotation: :Optional[str] |
| |
| |
| |
| .. py:attribute:: resource_group_name |
| :annotation: :Optional[str] |
| |
| |
| |
| |
| .. py:class:: AzureDataFactoryPipelineRunStatus |
| |
| Azure Data Factory pipeline operation statuses. |
| |
| .. py:attribute:: QUEUED |
| :annotation: = Queued |
| |
| |
| |
| .. py:attribute:: IN_PROGRESS |
| :annotation: = InProgress |
| |
| |
| |
| .. py:attribute:: SUCCEEDED |
| :annotation: = Succeeded |
| |
| |
| |
| .. py:attribute:: FAILED |
| :annotation: = Failed |
| |
| |
| |
| .. py:attribute:: CANCELING |
| :annotation: = Canceling |
| |
| |
| |
| .. py:attribute:: CANCELLED |
| :annotation: = Cancelled |
| |
| |
| |
| .. py:attribute:: TERMINAL_STATUSES |
| |
| |
| |
| |
| |
| .. py:exception:: AzureDataFactoryPipelineRunException |
| |
| Bases: :py:obj:`airflow.exceptions.AirflowException` |
| |
| An exception that indicates a pipeline run failed to complete. |
| |
| |
| .. py:class:: AzureDataFactoryHook(azure_data_factory_conn_id = default_conn_name) |
| |
| Bases: :py:obj:`airflow.hooks.base.BaseHook` |
| |
| A hook to interact with Azure Data Factory. |
| |
| :param azure_data_factory_conn_id: The :ref:`Azure Data Factory connection id<howto/connection:adf>`. |
| |
| .. py:attribute:: conn_type |
| :annotation: :str = azure_data_factory |
| |
| |
| |
| .. py:attribute:: conn_name_attr |
| :annotation: :str = azure_data_factory_conn_id |
| |
| |
| |
| .. py:attribute:: default_conn_name |
| :annotation: :str = azure_data_factory_default |
| |
| |
| |
| .. py:attribute:: hook_name |
| :annotation: :str = Azure Data Factory |
| |
| |
| |
| .. py:method:: get_connection_form_widgets() |
| :staticmethod: |
| |
| Returns connection widgets to add to connection form |
| |
| |
| .. py:method:: get_ui_field_behaviour() |
| :staticmethod: |
| |
| Returns custom field behaviour |
| |
| |
| .. py:method:: get_conn(self) |
| |
| Returns connection for the hook. |
| |
| |
| .. py:method:: get_factory(self, resource_group_name = None, factory_name = None, **config) |
| |
| Get the factory. |
| |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: The factory. |
| |
| |
| .. py:method:: update_factory(self, factory, resource_group_name = None, factory_name = None, **config) |
| |
| Update the factory. |
| |
| :param factory: The factory resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the factory does not exist. |
| :return: The factory. |
| |
| |
| .. py:method:: create_factory(self, factory, resource_group_name = None, factory_name = None, **config) |
| |
| Create the factory. |
| |
| :param factory: The factory resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the factory already exists. |
| :return: The factory. |
| |
| |
| .. py:method:: delete_factory(self, resource_group_name = None, factory_name = None, **config) |
| |
| Delete the factory. |
| |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| |
| |
| .. py:method:: get_linked_service(self, linked_service_name, resource_group_name = None, factory_name = None, **config) |
| |
| Get the linked service. |
| |
| :param linked_service_name: The linked service name. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: The linked service. |
| |
| |
| .. py:method:: update_linked_service(self, linked_service_name, linked_service, resource_group_name = None, factory_name = None, **config) |
| |
| Update the linked service. |
| |
| :param linked_service_name: The linked service name. |
| :param linked_service: The linked service resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the linked service does not exist. |
| :return: The linked service. |
| |
| |
| .. py:method:: create_linked_service(self, linked_service_name, linked_service, resource_group_name = None, factory_name = None, **config) |
| |
| Create the linked service. |
| |
| :param linked_service_name: The linked service name. |
| :param linked_service: The linked service resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the linked service already exists. |
| :return: The linked service. |
| |
| |
| .. py:method:: delete_linked_service(self, linked_service_name, resource_group_name = None, factory_name = None, **config) |
| |
| Delete the linked service. |
| |
| :param linked_service_name: The linked service name. |
| :param resource_group_name: The linked service name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| |
| |
| .. py:method:: get_dataset(self, dataset_name, resource_group_name = None, factory_name = None, **config) |
| |
| Get the dataset. |
| |
| :param dataset_name: The dataset name. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: The dataset. |
| |
| |
| .. py:method:: update_dataset(self, dataset_name, dataset, resource_group_name = None, factory_name = None, **config) |
| |
| Update the dataset. |
| |
| :param dataset_name: The dataset name. |
| :param dataset: The dataset resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the dataset does not exist. |
| :return: The dataset. |
| |
| |
| .. py:method:: create_dataset(self, dataset_name, dataset, resource_group_name = None, factory_name = None, **config) |
| |
| Create the dataset. |
| |
| :param dataset_name: The dataset name. |
| :param dataset: The dataset resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the dataset already exists. |
| :return: The dataset. |
| |
| |
| .. py:method:: delete_dataset(self, dataset_name, resource_group_name = None, factory_name = None, **config) |
| |
| Delete the dataset. |
| |
| :param dataset_name: The dataset name. |
| :param resource_group_name: The dataset name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| |
| |
| .. py:method:: get_pipeline(self, pipeline_name, resource_group_name = None, factory_name = None, **config) |
| |
| Get the pipeline. |
| |
| :param pipeline_name: The pipeline name. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: The pipeline. |
| |
| |
| .. py:method:: update_pipeline(self, pipeline_name, pipeline, resource_group_name = None, factory_name = None, **config) |
| |
| Update the pipeline. |
| |
| :param pipeline_name: The pipeline name. |
| :param pipeline: The pipeline resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the pipeline does not exist. |
| :return: The pipeline. |
| |
| |
| .. py:method:: create_pipeline(self, pipeline_name, pipeline, resource_group_name = None, factory_name = None, **config) |
| |
| Create the pipeline. |
| |
| :param pipeline_name: The pipeline name. |
| :param pipeline: The pipeline resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the pipeline already exists. |
| :return: The pipeline. |
| |
| |
| .. py:method:: delete_pipeline(self, pipeline_name, resource_group_name = None, factory_name = None, **config) |
| |
| Delete the pipeline. |
| |
| :param pipeline_name: The pipeline name. |
| :param resource_group_name: The pipeline name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| |
| |
| .. py:method:: run_pipeline(self, pipeline_name, resource_group_name = None, factory_name = None, **config) |
| |
| Run a pipeline. |
| |
| :param pipeline_name: The pipeline name. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: The pipeline run. |
| |
| |
| .. py:method:: get_pipeline_run(self, run_id, resource_group_name = None, factory_name = None, **config) |
| |
| Get the pipeline run. |
| |
| :param run_id: The pipeline run identifier. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: The pipeline run. |
| |
| |
| .. py:method:: get_pipeline_run_status(self, run_id, resource_group_name = None, factory_name = None) |
| |
| Get a pipeline run's current status. |
| |
| :param run_id: The pipeline run identifier. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :return: The status of the pipeline run. |
| |
| |
| .. py:method:: wait_for_pipeline_run_status(self, run_id, expected_statuses, resource_group_name = None, factory_name = None, check_interval = 60, timeout = 60 * 60 * 24 * 7) |
| |
| Waits for a pipeline run to match an expected status. |
| |
| :param run_id: The pipeline run identifier. |
| :param expected_statuses: The desired status(es) to check against a pipeline run's current status. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param check_interval: Time in seconds to check on a pipeline run's status. |
| :param timeout: Time in seconds to wait for a pipeline to reach a terminal status or the expected |
| status. |
| :return: Boolean indicating if the pipeline run has reached the ``expected_status``. |
| |
| |
| .. py:method:: cancel_pipeline_run(self, run_id, resource_group_name = None, factory_name = None, **config) |
| |
| Cancel the pipeline run. |
| |
| :param run_id: The pipeline run identifier. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| |
| |
| .. py:method:: get_trigger(self, trigger_name, resource_group_name = None, factory_name = None, **config) |
| |
| Get the trigger. |
| |
| :param trigger_name: The trigger name. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: The trigger. |
| |
| |
| .. py:method:: update_trigger(self, trigger_name, trigger, resource_group_name = None, factory_name = None, **config) |
| |
| Update the trigger. |
| |
| :param trigger_name: The trigger name. |
| :param trigger: The trigger resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the trigger does not exist. |
| :return: The trigger. |
| |
| |
| .. py:method:: create_trigger(self, trigger_name, trigger, resource_group_name = None, factory_name = None, **config) |
| |
| Create the trigger. |
| |
| :param trigger_name: The trigger name. |
| :param trigger: The trigger resource definition. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :raise AirflowException: If the trigger already exists. |
| :return: The trigger. |
| |
| |
| .. py:method:: delete_trigger(self, trigger_name, resource_group_name = None, factory_name = None, **config) |
| |
| Delete the trigger. |
| |
| :param trigger_name: The trigger name. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| |
| |
| .. py:method:: start_trigger(self, trigger_name, resource_group_name = None, factory_name = None, **config) |
| |
| Start the trigger. |
| |
| :param trigger_name: The trigger name. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: An Azure operation poller. |
| |
| |
| .. py:method:: stop_trigger(self, trigger_name, resource_group_name = None, factory_name = None, **config) |
| |
| Stop the trigger. |
| |
| :param trigger_name: The trigger name. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| :return: An Azure operation poller. |
| |
| |
| .. py:method:: rerun_trigger(self, trigger_name, run_id, resource_group_name = None, factory_name = None, **config) |
| |
| Rerun the trigger. |
| |
| :param trigger_name: The trigger name. |
| :param run_id: The trigger run identifier. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| |
| |
| .. py:method:: cancel_trigger(self, trigger_name, run_id, resource_group_name = None, factory_name = None, **config) |
| |
| Cancel the trigger. |
| |
| :param trigger_name: The trigger name. |
| :param run_id: The trigger run identifier. |
| :param resource_group_name: The resource group name. |
| :param factory_name: The factory name. |
| :param config: Extra parameters for the ADF client. |
| |
| |
| .. py:method:: test_connection(self) |
| |
| Test a configured Azure Data Factory connection. |
| |
| |
| |