blob: a76c4e8d7e440c690aaf0b35f06a7ac46d70f8b6 [file] [log] [blame]
:py:mod:`airflow.providers.microsoft.azure.hooks.data_factory`
==============================================================
.. py:module:: airflow.providers.microsoft.azure.hooks.data_factory
.. autoapi-nested-parse::
.. spelling::
CreateRunResponse
DatasetResource
LinkedServiceResource
LROPoller
PipelineResource
PipelineRun
TriggerResource
datafactory
mgmt
Module Contents
---------------
Classes
~~~~~~~
.. autoapisummary::
airflow.providers.microsoft.azure.hooks.data_factory.PipelineRunInfo
airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryPipelineRunStatus
airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryHook
Functions
~~~~~~~~~
.. autoapisummary::
airflow.providers.microsoft.azure.hooks.data_factory.provide_targeted_factory
Attributes
~~~~~~~~~~
.. autoapisummary::
airflow.providers.microsoft.azure.hooks.data_factory.Credentials
.. py:data:: Credentials
.. py:function:: provide_targeted_factory(func)
Provide the targeted factory to the decorated function in case it isn't specified.
If ``resource_group_name`` or ``factory_name`` is not provided it defaults to the value specified in
the connection extras.
.. py:class:: PipelineRunInfo
Bases: :py:obj:`airflow.typing_compat.TypedDict`
Type class for the pipeline run info dictionary.
.. py:attribute:: run_id
:annotation: :str
.. py:attribute:: factory_name
:annotation: :Optional[str]
.. py:attribute:: resource_group_name
:annotation: :Optional[str]
.. py:class:: AzureDataFactoryPipelineRunStatus
Azure Data Factory pipeline operation statuses.
.. py:attribute:: QUEUED
:annotation: = Queued
.. py:attribute:: IN_PROGRESS
:annotation: = InProgress
.. py:attribute:: SUCCEEDED
:annotation: = Succeeded
.. py:attribute:: FAILED
:annotation: = Failed
.. py:attribute:: CANCELING
:annotation: = Canceling
.. py:attribute:: CANCELLED
:annotation: = Cancelled
.. py:attribute:: TERMINAL_STATUSES
.. py:exception:: AzureDataFactoryPipelineRunException
Bases: :py:obj:`airflow.exceptions.AirflowException`
An exception that indicates a pipeline run failed to complete.
.. py:class:: AzureDataFactoryHook(azure_data_factory_conn_id = default_conn_name)
Bases: :py:obj:`airflow.hooks.base.BaseHook`
A hook to interact with Azure Data Factory.
:param azure_data_factory_conn_id: The :ref:`Azure Data Factory connection id<howto/connection:adf>`.
.. py:attribute:: conn_type
:annotation: :str = azure_data_factory
.. py:attribute:: conn_name_attr
:annotation: :str = azure_data_factory_conn_id
.. py:attribute:: default_conn_name
:annotation: :str = azure_data_factory_default
.. py:attribute:: hook_name
:annotation: :str = Azure Data Factory
.. py:method:: get_connection_form_widgets()
:staticmethod:
Returns connection widgets to add to connection form
.. py:method:: get_ui_field_behaviour()
:staticmethod:
Returns custom field behaviour
.. py:method:: get_conn(self)
Returns connection for the hook.
.. py:method:: get_factory(self, resource_group_name = None, factory_name = None, **config)
Get the factory.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: The factory.
.. py:method:: update_factory(self, factory, resource_group_name = None, factory_name = None, **config)
Update the factory.
:param factory: The factory resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the factory does not exist.
:return: The factory.
.. py:method:: create_factory(self, factory, resource_group_name = None, factory_name = None, **config)
Create the factory.
:param factory: The factory resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the factory already exists.
:return: The factory.
.. py:method:: delete_factory(self, resource_group_name = None, factory_name = None, **config)
Delete the factory.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
.. py:method:: get_linked_service(self, linked_service_name, resource_group_name = None, factory_name = None, **config)
Get the linked service.
:param linked_service_name: The linked service name.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: The linked service.
.. py:method:: update_linked_service(self, linked_service_name, linked_service, resource_group_name = None, factory_name = None, **config)
Update the linked service.
:param linked_service_name: The linked service name.
:param linked_service: The linked service resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the linked service does not exist.
:return: The linked service.
.. py:method:: create_linked_service(self, linked_service_name, linked_service, resource_group_name = None, factory_name = None, **config)
Create the linked service.
:param linked_service_name: The linked service name.
:param linked_service: The linked service resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the linked service already exists.
:return: The linked service.
.. py:method:: delete_linked_service(self, linked_service_name, resource_group_name = None, factory_name = None, **config)
Delete the linked service.
:param linked_service_name: The linked service name.
:param resource_group_name: The linked service name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
.. py:method:: get_dataset(self, dataset_name, resource_group_name = None, factory_name = None, **config)
Get the dataset.
:param dataset_name: The dataset name.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: The dataset.
.. py:method:: update_dataset(self, dataset_name, dataset, resource_group_name = None, factory_name = None, **config)
Update the dataset.
:param dataset_name: The dataset name.
:param dataset: The dataset resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the dataset does not exist.
:return: The dataset.
.. py:method:: create_dataset(self, dataset_name, dataset, resource_group_name = None, factory_name = None, **config)
Create the dataset.
:param dataset_name: The dataset name.
:param dataset: The dataset resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the dataset already exists.
:return: The dataset.
.. py:method:: delete_dataset(self, dataset_name, resource_group_name = None, factory_name = None, **config)
Delete the dataset.
:param dataset_name: The dataset name.
:param resource_group_name: The dataset name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
.. py:method:: get_pipeline(self, pipeline_name, resource_group_name = None, factory_name = None, **config)
Get the pipeline.
:param pipeline_name: The pipeline name.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: The pipeline.
.. py:method:: update_pipeline(self, pipeline_name, pipeline, resource_group_name = None, factory_name = None, **config)
Update the pipeline.
:param pipeline_name: The pipeline name.
:param pipeline: The pipeline resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the pipeline does not exist.
:return: The pipeline.
.. py:method:: create_pipeline(self, pipeline_name, pipeline, resource_group_name = None, factory_name = None, **config)
Create the pipeline.
:param pipeline_name: The pipeline name.
:param pipeline: The pipeline resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the pipeline already exists.
:return: The pipeline.
.. py:method:: delete_pipeline(self, pipeline_name, resource_group_name = None, factory_name = None, **config)
Delete the pipeline.
:param pipeline_name: The pipeline name.
:param resource_group_name: The pipeline name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
.. py:method:: run_pipeline(self, pipeline_name, resource_group_name = None, factory_name = None, **config)
Run a pipeline.
:param pipeline_name: The pipeline name.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: The pipeline run.
.. py:method:: get_pipeline_run(self, run_id, resource_group_name = None, factory_name = None, **config)
Get the pipeline run.
:param run_id: The pipeline run identifier.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: The pipeline run.
.. py:method:: get_pipeline_run_status(self, run_id, resource_group_name = None, factory_name = None)
Get a pipeline run's current status.
:param run_id: The pipeline run identifier.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:return: The status of the pipeline run.
.. py:method:: wait_for_pipeline_run_status(self, run_id, expected_statuses, resource_group_name = None, factory_name = None, check_interval = 60, timeout = 60 * 60 * 24 * 7)
Waits for a pipeline run to match an expected status.
:param run_id: The pipeline run identifier.
:param expected_statuses: The desired status(es) to check against a pipeline run's current status.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param check_interval: Time in seconds to check on a pipeline run's status.
:param timeout: Time in seconds to wait for a pipeline to reach a terminal status or the expected
status.
:return: Boolean indicating if the pipeline run has reached the ``expected_status``.
.. py:method:: cancel_pipeline_run(self, run_id, resource_group_name = None, factory_name = None, **config)
Cancel the pipeline run.
:param run_id: The pipeline run identifier.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
.. py:method:: get_trigger(self, trigger_name, resource_group_name = None, factory_name = None, **config)
Get the trigger.
:param trigger_name: The trigger name.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: The trigger.
.. py:method:: update_trigger(self, trigger_name, trigger, resource_group_name = None, factory_name = None, **config)
Update the trigger.
:param trigger_name: The trigger name.
:param trigger: The trigger resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the trigger does not exist.
:return: The trigger.
.. py:method:: create_trigger(self, trigger_name, trigger, resource_group_name = None, factory_name = None, **config)
Create the trigger.
:param trigger_name: The trigger name.
:param trigger: The trigger resource definition.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:raise AirflowException: If the trigger already exists.
:return: The trigger.
.. py:method:: delete_trigger(self, trigger_name, resource_group_name = None, factory_name = None, **config)
Delete the trigger.
:param trigger_name: The trigger name.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
.. py:method:: start_trigger(self, trigger_name, resource_group_name = None, factory_name = None, **config)
Start the trigger.
:param trigger_name: The trigger name.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: An Azure operation poller.
.. py:method:: stop_trigger(self, trigger_name, resource_group_name = None, factory_name = None, **config)
Stop the trigger.
:param trigger_name: The trigger name.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
:return: An Azure operation poller.
.. py:method:: rerun_trigger(self, trigger_name, run_id, resource_group_name = None, factory_name = None, **config)
Rerun the trigger.
:param trigger_name: The trigger name.
:param run_id: The trigger run identifier.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
.. py:method:: cancel_trigger(self, trigger_name, run_id, resource_group_name = None, factory_name = None, **config)
Cancel the trigger.
:param trigger_name: The trigger name.
:param run_id: The trigger run identifier.
:param resource_group_name: The resource group name.
:param factory_name: The factory name.
:param config: Extra parameters for the ADF client.
.. py:method:: test_connection(self)
Test a configured Azure Data Factory connection.