| :py:mod:`airflow.sensors.base` |
| ============================== |
| |
| .. py:module:: airflow.sensors.base |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.sensors.base.PokeReturnValue |
| airflow.sensors.base.BaseSensorOperator |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.sensors.base.poke_mode_only |
| |
| |
| |
| .. py:class:: PokeReturnValue(is_done, xcom_value = None) |
| |
| Sensors can optionally return an instance of the PokeReturnValue class in the poke method. |
| If an XCom value is supplied when the sensor is done, then the XCom value will be |
| pushed through the operator return value. |
| :param is_done: Set to true to indicate the sensor can stop poking. |
| :param xcom_value: An optional XCOM value to be returned by the operator. |
| |
| .. py:method:: __bool__(self) |
| |
| |
| |
| .. py:class:: BaseSensorOperator(*, poke_interval = 60, timeout = conf.getfloat('sensors', 'default_timeout'), soft_fail = False, mode = 'poke', exponential_backoff = False, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.baseoperator.BaseOperator`, :py:obj:`airflow.models.skipmixin.SkipMixin` |
| |
| Sensor operators are derived from this class and inherit these attributes. |
| |
| Sensor operators keep executing at a time interval and succeed when |
| a criteria is met and fail if and when they time out. |
| |
| :param soft_fail: Set to true to mark the task as SKIPPED on failure |
| :param poke_interval: Time in seconds that the job should wait in |
| between each tries |
| :param timeout: Time, in seconds before the task times out and fails. |
| :param mode: How the sensor operates. |
| Options are: ``{ poke | reschedule }``, default is ``poke``. |
| When set to ``poke`` the sensor is taking up a worker slot for its |
| whole execution time and sleeps between pokes. Use this mode if the |
| expected runtime of the sensor is short or if a short poke interval |
| is required. Note that the sensor will hold onto a worker slot and |
| a pool slot for the duration of the sensor's runtime in this mode. |
| When set to ``reschedule`` the sensor task frees the worker slot when |
| the criteria is not yet met and it's rescheduled at a later time. Use |
| this mode if the time before the criteria is met is expected to be |
| quite long. The poke interval should be more than one minute to |
| prevent too much load on the scheduler. |
| :param exponential_backoff: allow progressive longer waits between |
| pokes by using exponential backoff algorithm |
| |
| .. py:attribute:: ui_color |
| :annotation: :str = #e6f1f2 |
| |
| |
| |
| .. py:attribute:: valid_modes |
| :annotation: :Iterable[str] = ['poke', 'reschedule'] |
| |
| |
| |
| .. py:attribute:: execution_fields |
| :annotation: = ['poke_interval', 'retries', 'execution_timeout', 'timeout', 'email', 'email_on_retry',... |
| |
| |
| |
| .. py:attribute:: deps |
| |
| |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| .. py:method:: is_smart_sensor_compatible(self) |
| |
| Return if this operator can use smart service. Default False. |
| |
| |
| .. py:method:: register_in_sensor_service(self, ti, context) |
| |
| Register ti in smart sensor service |
| |
| :param ti: Task instance object. |
| :param context: TaskInstance template context from the ti. |
| :return: boolean |
| |
| |
| .. py:method:: get_poke_context(self, context) |
| |
| Return a dictionary with all attributes in poke_context_fields. The |
| poke_context with operator class can be used to identify a unique |
| sensor job. |
| |
| :param context: TaskInstance template context. |
| :return: A dictionary with key in poke_context_fields. |
| |
| |
| .. py:method:: get_execution_context(self, context) |
| |
| Return a dictionary with all attributes in execution_fields. The |
| execution_context include execution requirement for each sensor task |
| such as timeout setup, email_alert setup. |
| |
| :param context: TaskInstance template context. |
| :return: A dictionary with key in execution_fields. |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| .. py:method:: prepare_for_execution(self) |
| |
| Lock task for execution to disable custom action in __setattr__ and |
| returns a copy of the task |
| |
| |
| .. py:method:: reschedule(self) |
| :property: |
| |
| Define mode rescheduled sensors. |
| |
| |
| |
| .. py:function:: poke_mode_only(cls) |
| |
| Class Decorator for child classes of BaseSensorOperator to indicate |
| that instances of this class are only safe to use poke mode. |
| |
| Will decorate all methods in the class to assert they did not change |
| the mode from 'poke'. |
| |
| :param cls: BaseSensor class to enforce methods only use 'poke' mode. |
| |
| |