| :py:mod:`airflow.sensors.external_task` |
| ======================================= |
| |
| .. py:module:: airflow.sensors.external_task |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.sensors.external_task.ExternalTaskSensorLink |
| airflow.sensors.external_task.ExternalTaskSensor |
| airflow.sensors.external_task.ExternalTaskMarker |
| |
| |
| |
| |
| .. py:class:: ExternalTaskSensorLink |
| |
| Bases: :py:obj:`airflow.models.BaseOperatorLink` |
| |
| Operator link for ExternalTaskSensor. It allows users to access |
| DAG waited with ExternalTaskSensor. |
| |
| .. py:attribute:: name |
| :annotation: = External DAG |
| |
| |
| |
| .. py:method:: get_link(self, operator, dttm) |
| |
| Link to external system. |
| |
| :param operator: airflow operator |
| :param dttm: datetime |
| :return: link to external system |
| |
| |
| |
| .. py:class:: ExternalTaskSensor(*, external_dag_id: str, external_task_id: Optional[str] = None, external_task_ids: Optional[Iterable[str]] = None, allowed_states: Optional[Iterable[str]] = None, failed_states: Optional[Iterable[str]] = None, execution_delta: Optional[datetime.timedelta] = None, execution_date_fn: Optional[Callable] = None, check_existence: bool = False, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Waits for a different DAG or a task in a different DAG to complete for a |
| specific logical date. |
| |
| :param external_dag_id: The dag_id that contains the task you want to |
| wait for |
| :type external_dag_id: str |
| :param external_task_id: The task_id that contains the task you want to |
| wait for. If ``None`` (default value) the sensor waits for the DAG |
| :type external_task_id: str or None |
| :param external_task_ids: The list of task_ids that you want to wait for. |
| If ``None`` (default value) the sensor waits for the DAG. Either |
| external_task_id or external_task_ids can be passed to |
| ExternalTaskSensor, but not both. |
| :type external_task_ids: Iterable of task_ids or None, default is None |
| :param allowed_states: Iterable of allowed states, default is ``['success']`` |
| :type allowed_states: Iterable |
| :param failed_states: Iterable of failed or dis-allowed states, default is ``None`` |
| :type failed_states: Iterable |
| :param execution_delta: time difference with the previous execution to |
| look at, the default is the same logical date as the current task or DAG. |
| For yesterday, use [positive!] datetime.timedelta(days=1). Either |
| execution_delta or execution_date_fn can be passed to |
| ExternalTaskSensor, but not both. |
| :type execution_delta: Optional[datetime.timedelta] |
| :param execution_date_fn: function that receives the current execution's logical date as the first |
| positional argument and optionally any number of keyword arguments available in the |
| context dictionary, and returns the desired logical dates to query. |
| Either execution_delta or execution_date_fn can be passed to ExternalTaskSensor, |
| but not both. |
| :type execution_date_fn: Optional[Callable] |
| :param check_existence: Set to `True` to check if the external task exists (when |
| external_task_id is not None) or check if the DAG to wait for exists (when |
| external_task_id is None), and immediately cease waiting if the external task |
| or DAG does not exist (default value: False). |
| :type check_existence: bool |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['external_dag_id', 'external_task_id'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #19647e |
| |
| |
| |
| .. py:method:: operator_extra_links(self) |
| :property: |
| |
| Return operator extra links |
| |
| |
| .. py:method:: poke(self, context, session=None) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| .. py:method:: get_count(self, dttm_filter, session, states) -> int |
| |
| Get the count of records against dttm filter and states |
| |
| :param dttm_filter: date time filter for execution date |
| :type dttm_filter: list |
| :param session: airflow session object |
| :type session: SASession |
| :param states: task or dag states |
| :type states: list |
| :return: count of record against the filters |
| |
| |
| |
| .. py:class:: ExternalTaskMarker(*, external_dag_id: str, external_task_id: str, execution_date: Optional[Union[str, datetime.datetime]] = '{{ logical_date.isoformat() }}', recursion_depth: int = 10, **kwargs) |
| |
| Bases: :py:obj:`airflow.operators.dummy.DummyOperator` |
| |
| Use this operator to indicate that a task on a different DAG depends on this task. |
| When this task is cleared with "Recursive" selected, Airflow will clear the task on |
| the other DAG and its downstream tasks recursively. Transitive dependencies are followed |
| until the recursion_depth is reached. |
| |
| :param external_dag_id: The dag_id that contains the dependent task that needs to be cleared. |
| :type external_dag_id: str |
| :param external_task_id: The task_id of the dependent task that needs to be cleared. |
| :type external_task_id: str |
| :param execution_date: The logical date of the dependent task execution that needs to be cleared. |
| :type execution_date: str or datetime.datetime |
| :param recursion_depth: The maximum level of transitive dependencies allowed. Default is 10. |
| This is mostly used for preventing cyclic dependencies. It is fine to increase |
| this number if necessary. However, too many levels of transitive dependencies will make |
| it slower to clear tasks in the web UI. |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['external_dag_id', 'external_task_id', 'execution_date'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #19647e |
| |
| |
| |
| .. py:method:: get_serialized_fields(cls) |
| :classmethod: |
| |
| Serialized ExternalTaskMarker contain exactly these fields + templated_fields . |
| |
| |
| |