| :py:mod:`airflow.sensors.base` |
| ============================== |
| |
| .. py:module:: airflow.sensors.base |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.sensors.base.BaseSensorOperator |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.sensors.base.poke_mode_only |
| |
| |
| |
| .. py:class:: BaseSensorOperator(*, poke_interval: float = 60, timeout: float = 60 * 60 * 24 * 7, soft_fail: bool = False, mode: str = 'poke', exponential_backoff: bool = False, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator`, :py:obj:`airflow.models.skipmixin.SkipMixin` |
| |
| Sensor operators are derived from this class and inherit these attributes. |
| |
| Sensor operators keep executing at a time interval and succeed when |
| a criteria is met and fail if and when they time out. |
| |
| :param soft_fail: Set to true to mark the task as SKIPPED on failure |
| :type soft_fail: bool |
| :param poke_interval: Time in seconds that the job should wait in |
| between each tries |
| :type poke_interval: float |
| :param timeout: Time, in seconds before the task times out and fails. |
| :type timeout: float |
| :param mode: How the sensor operates. |
| Options are: ``{ poke | reschedule }``, default is ``poke``. |
| When set to ``poke`` the sensor is taking up a worker slot for its |
| whole execution time and sleeps between pokes. Use this mode if the |
| expected runtime of the sensor is short or if a short poke interval |
| is required. Note that the sensor will hold onto a worker slot and |
| a pool slot for the duration of the sensor's runtime in this mode. |
| When set to ``reschedule`` the sensor task frees the worker slot when |
| the criteria is not yet met and it's rescheduled at a later time. Use |
| this mode if the time before the criteria is met is expected to be |
| quite long. The poke interval should be more than one minute to |
| prevent too much load on the scheduler. |
| :type mode: str |
| :param exponential_backoff: allow progressive longer waits between |
| pokes by using exponential backoff algorithm |
| :type exponential_backoff: bool |
| |
| .. py:attribute:: ui_color |
| :annotation: :str = #e6f1f2 |
| |
| |
| |
| .. py:attribute:: valid_modes |
| :annotation: :Iterable[str] = ['poke', 'reschedule'] |
| |
| |
| |
| .. py:attribute:: execution_fields |
| :annotation: = ['poke_interval', 'retries', 'execution_timeout', 'timeout', 'email', 'email_on_retry',... |
| |
| |
| |
| .. py:method:: poke(self, context: Dict) -> bool |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| .. py:method:: is_smart_sensor_compatible(self) |
| |
| Return if this operator can use smart service. Default False. |
| |
| |
| .. py:method:: register_in_sensor_service(self, ti, context) |
| |
| Register ti in smart sensor service |
| |
| :param ti: Task instance object. |
| :param context: TaskInstance template context from the ti. |
| :return: boolean |
| |
| |
| .. py:method:: get_poke_context(self, context) |
| |
| Return a dictionary with all attributes in poke_context_fields. The |
| poke_context with operator class can be used to identify a unique |
| sensor job. |
| |
| :param context: TaskInstance template context. |
| :return: A dictionary with key in poke_context_fields. |
| |
| |
| .. py:method:: get_execution_context(self, context) |
| |
| Return a dictionary with all attributes in execution_fields. The |
| execution_context include execution requirement for each sensor task |
| such as timeout setup, email_alert setup. |
| |
| :param context: TaskInstance template context. |
| :return: A dictionary with key in execution_fields. |
| |
| |
| .. py:method:: execute(self, context: Dict) -> Any |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| .. py:method:: prepare_for_execution(self) -> airflow.models.BaseOperator |
| |
| Lock task for execution to disable custom action in __setattr__ and |
| returns a copy of the task |
| |
| |
| .. py:method:: reschedule(self) |
| :property: |
| |
| Define mode rescheduled sensors. |
| |
| |
| .. py:method:: deps(self) |
| :property: |
| |
| Adds one additional dependency for all sensor operators that |
| checks if a sensor task instance can be rescheduled. |
| |
| |
| |
| .. py:function:: poke_mode_only(cls) |
| |
| Class Decorator for child classes of BaseSensorOperator to indicate |
| that instances of this class are only safe to use poke mode. |
| |
| Will decorate all methods in the class to assert they did not change |
| the mode from 'poke'. |
| |
| :param cls: BaseSensor class to enforce methods only use 'poke' mode. |
| :type cls: type |
| |
| |