blob: 787c2543d7a60771bf886ac0da6306283c98d3ac [file] [log] [blame]
:py:mod:`airflow.sensors.external_task`
=======================================
.. py:module:: airflow.sensors.external_task
Module Contents
---------------
Classes
~~~~~~~
.. autoapisummary::
airflow.sensors.external_task.ExternalDagLink
airflow.sensors.external_task.ExternalTaskSensor
airflow.sensors.external_task.ExternalTaskMarker
airflow.sensors.external_task.ExternalTaskSensorLink
.. py:class:: ExternalDagLink
Bases: :py:obj:`airflow.models.baseoperator.BaseOperatorLink`
Operator link for ExternalTaskSensor and ExternalTaskMarker.
It allows users to access DAG waited with ExternalTaskSensor or cleared by ExternalTaskMarker.
.. py:attribute:: name
:annotation: = External DAG
.. py:method:: get_link(operator, dttm)
Link to external system.
Note: The old signature of this function was ``(self, operator, dttm: datetime)``. That is still
supported at runtime but is deprecated.
:param operator: The Airflow operator object this link is associated to.
:param ti_key: TaskInstance ID to return link for.
:return: link to external system
.. py:class:: ExternalTaskSensor(*, external_dag_id, external_task_id = None, external_task_ids = None, external_task_group_id = None, allowed_states = None, failed_states = None, execution_delta = None, execution_date_fn = None, check_existence = False, **kwargs)
Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator`
Waits for a different DAG, a task group, or a task in a different DAG to complete for a
specific logical date.
If both `external_task_group_id` and `external_task_id` are ``None`` (default), the sensor
waits for the DAG.
Values for `external_task_group_id` and `external_task_id` can't be set at the same time.
By default the ExternalTaskSensor will wait for the external task to
succeed, at which point it will also succeed. However, by default it will
*not* fail if the external task fails, but will continue to check the status
until the sensor times out (thus giving you time to retry the external task
without also having to clear the sensor).
It is possible to alter the default behavior by setting states which
cause the sensor to fail, e.g. by setting ``allowed_states=[State.FAILED]``
and ``failed_states=[State.SUCCESS]`` you will flip the behaviour to get a
sensor which goes green when the external task *fails* and immediately goes
red if the external task *succeeds*!
Note that ``soft_fail`` is respected when examining the failed_states. Thus
if the external task enters a failed state and ``soft_fail == True`` the
sensor will _skip_ rather than fail. As a result, setting ``soft_fail=True``
and ``failed_states=[State.SKIPPED]`` will result in the sensor skipping if
the external task skips.
:param external_dag_id: The dag_id that contains the task you want to
wait for
:param external_task_id: The task_id that contains the task you want to
wait for.
:param external_task_ids: The list of task_ids that you want to wait for.
If ``None`` (default value) the sensor waits for the DAG. Either
external_task_id or external_task_ids can be passed to
ExternalTaskSensor, but not both.
:param allowed_states: Iterable of allowed states, default is ``['success']``
:param failed_states: Iterable of failed or dis-allowed states, default is ``None``
:param execution_delta: time difference with the previous execution to
look at, the default is the same logical date as the current task or DAG.
For yesterday, use [positive!] datetime.timedelta(days=1). Either
execution_delta or execution_date_fn can be passed to
ExternalTaskSensor, but not both.
:param execution_date_fn: function that receives the current execution's logical date as the first
positional argument and optionally any number of keyword arguments available in the
context dictionary, and returns the desired logical dates to query.
Either execution_delta or execution_date_fn can be passed to ExternalTaskSensor,
but not both.
:param check_existence: Set to `True` to check if the external task exists (when
external_task_id is not None) or check if the DAG to wait for exists (when
external_task_id is None), and immediately cease waiting if the external task
or DAG does not exist (default value: False).
.. py:attribute:: template_fields
:annotation: = ['external_dag_id', 'external_task_id', 'external_task_ids']
.. py:attribute:: ui_color
:annotation: = #19647e
.. py:attribute:: operator_extra_links
.. py:method:: poke(context, session=None)
Function that the sensors defined while deriving this class should
override.
.. py:method:: get_count(dttm_filter, session, states)
Get the count of records against dttm filter and states
:param dttm_filter: date time filter for execution date
:param session: airflow session object
:param states: task or dag states
:return: count of record against the filters
.. py:method:: get_external_task_group_task_ids(session)
.. py:class:: ExternalTaskMarker(*, external_dag_id, external_task_id, execution_date = '{{ logical_date.isoformat() }}', recursion_depth = 10, **kwargs)
Bases: :py:obj:`airflow.operators.empty.EmptyOperator`
Use this operator to indicate that a task on a different DAG depends on this task.
When this task is cleared with "Recursive" selected, Airflow will clear the task on
the other DAG and its downstream tasks recursively. Transitive dependencies are followed
until the recursion_depth is reached.
:param external_dag_id: The dag_id that contains the dependent task that needs to be cleared.
:param external_task_id: The task_id of the dependent task that needs to be cleared.
:param execution_date: The logical date of the dependent task execution that needs to be cleared.
:param recursion_depth: The maximum level of transitive dependencies allowed. Default is 10.
This is mostly used for preventing cyclic dependencies. It is fine to increase
this number if necessary. However, too many levels of transitive dependencies will make
it slower to clear tasks in the web UI.
.. py:attribute:: template_fields
:annotation: = ['external_dag_id', 'external_task_id', 'execution_date']
.. py:attribute:: ui_color
:annotation: = #19647e
.. py:attribute:: operator_extra_links
.. py:method:: get_serialized_fields()
:classmethod:
Serialized ExternalTaskMarker contain exactly these fields + templated_fields .
.. py:class:: ExternalTaskSensorLink
Bases: :py:obj:`ExternalDagLink`
This external link is deprecated.
Please use :class:`airflow.sensors.external_task.ExternalDagLink`.
.. py:method:: __attrs_post_init__()