| :py:mod:`airflow.sensors.external_task` |
| ======================================= |
| |
| .. py:module:: airflow.sensors.external_task |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.sensors.external_task.ExternalDagLink |
| airflow.sensors.external_task.ExternalTaskSensor |
| airflow.sensors.external_task.ExternalTaskMarker |
| airflow.sensors.external_task.ExternalTaskSensorLink |
| |
| |
| |
| |
| .. py:class:: ExternalDagLink |
| |
| Bases: :py:obj:`airflow.models.baseoperator.BaseOperatorLink` |
| |
| Operator link for ExternalTaskSensor and ExternalTaskMarker. |
| It allows users to access DAG waited with ExternalTaskSensor or cleared by ExternalTaskMarker. |
| |
| .. py:attribute:: name |
| :annotation: = External DAG |
| |
| |
| |
| .. py:method:: get_link(operator, dttm) |
| |
| Link to external system. |
| |
| Note: The old signature of this function was ``(self, operator, dttm: datetime)``. That is still |
| supported at runtime but is deprecated. |
| |
| :param operator: The Airflow operator object this link is associated to. |
| :param ti_key: TaskInstance ID to return link for. |
| :return: link to external system |
| |
| |
| |
| .. py:class:: ExternalTaskSensor(*, external_dag_id, external_task_id = None, external_task_ids = None, external_task_group_id = None, allowed_states = None, failed_states = None, execution_delta = None, execution_date_fn = None, check_existence = False, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Waits for a different DAG, a task group, or a task in a different DAG to complete for a |
| specific logical date. |
| |
| If both `external_task_group_id` and `external_task_id` are ``None`` (default), the sensor |
| waits for the DAG. |
| Values for `external_task_group_id` and `external_task_id` can't be set at the same time. |
| |
| By default the ExternalTaskSensor will wait for the external task to |
| succeed, at which point it will also succeed. However, by default it will |
| *not* fail if the external task fails, but will continue to check the status |
| until the sensor times out (thus giving you time to retry the external task |
| without also having to clear the sensor). |
| |
| It is possible to alter the default behavior by setting states which |
| cause the sensor to fail, e.g. by setting ``allowed_states=[State.FAILED]`` |
| and ``failed_states=[State.SUCCESS]`` you will flip the behaviour to get a |
| sensor which goes green when the external task *fails* and immediately goes |
| red if the external task *succeeds*! |
| |
| Note that ``soft_fail`` is respected when examining the failed_states. Thus |
| if the external task enters a failed state and ``soft_fail == True`` the |
| sensor will _skip_ rather than fail. As a result, setting ``soft_fail=True`` |
| and ``failed_states=[State.SKIPPED]`` will result in the sensor skipping if |
| the external task skips. |
| |
| :param external_dag_id: The dag_id that contains the task you want to |
| wait for |
| :param external_task_id: The task_id that contains the task you want to |
| wait for. |
| :param external_task_ids: The list of task_ids that you want to wait for. |
| If ``None`` (default value) the sensor waits for the DAG. Either |
| external_task_id or external_task_ids can be passed to |
| ExternalTaskSensor, but not both. |
| :param allowed_states: Iterable of allowed states, default is ``['success']`` |
| :param failed_states: Iterable of failed or dis-allowed states, default is ``None`` |
| :param execution_delta: time difference with the previous execution to |
| look at, the default is the same logical date as the current task or DAG. |
| For yesterday, use [positive!] datetime.timedelta(days=1). Either |
| execution_delta or execution_date_fn can be passed to |
| ExternalTaskSensor, but not both. |
| :param execution_date_fn: function that receives the current execution's logical date as the first |
| positional argument and optionally any number of keyword arguments available in the |
| context dictionary, and returns the desired logical dates to query. |
| Either execution_delta or execution_date_fn can be passed to ExternalTaskSensor, |
| but not both. |
| :param check_existence: Set to `True` to check if the external task exists (when |
| external_task_id is not None) or check if the DAG to wait for exists (when |
| external_task_id is None), and immediately cease waiting if the external task |
| or DAG does not exist (default value: False). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['external_dag_id', 'external_task_id', 'external_task_ids'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #19647e |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: poke(context, session=None) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| .. py:method:: get_count(dttm_filter, session, states) |
| |
| Get the count of records against dttm filter and states |
| |
| :param dttm_filter: date time filter for execution date |
| :param session: airflow session object |
| :param states: task or dag states |
| :return: count of record against the filters |
| |
| |
| .. py:method:: get_external_task_group_task_ids(session) |
| |
| |
| |
| .. py:class:: ExternalTaskMarker(*, external_dag_id, external_task_id, execution_date = '{{ logical_date.isoformat() }}', recursion_depth = 10, **kwargs) |
| |
| Bases: :py:obj:`airflow.operators.empty.EmptyOperator` |
| |
| Use this operator to indicate that a task on a different DAG depends on this task. |
| When this task is cleared with "Recursive" selected, Airflow will clear the task on |
| the other DAG and its downstream tasks recursively. Transitive dependencies are followed |
| until the recursion_depth is reached. |
| |
| :param external_dag_id: The dag_id that contains the dependent task that needs to be cleared. |
| :param external_task_id: The task_id of the dependent task that needs to be cleared. |
| :param execution_date: The logical date of the dependent task execution that needs to be cleared. |
| :param recursion_depth: The maximum level of transitive dependencies allowed. Default is 10. |
| This is mostly used for preventing cyclic dependencies. It is fine to increase |
| this number if necessary. However, too many levels of transitive dependencies will make |
| it slower to clear tasks in the web UI. |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['external_dag_id', 'external_task_id', 'execution_date'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #19647e |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: get_serialized_fields() |
| :classmethod: |
| |
| Serialized ExternalTaskMarker contain exactly these fields + templated_fields . |
| |
| |
| |
| .. py:class:: ExternalTaskSensorLink |
| |
| Bases: :py:obj:`ExternalDagLink` |
| |
| This external link is deprecated. |
| Please use :class:`airflow.sensors.external_task.ExternalDagLink`. |
| |
| .. py:method:: __attrs_post_init__() |
| |
| |
| |