| :py:mod:`airflow.providers.qubole.sensors.qubole` |
| ================================================= |
| |
| .. py:module:: airflow.providers.qubole.sensors.qubole |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.qubole.sensors.qubole.QuboleSensor |
| airflow.providers.qubole.sensors.qubole.QuboleFileSensor |
| airflow.providers.qubole.sensors.qubole.QubolePartitionSensor |
| |
| |
| |
| |
| .. py:class:: QuboleSensor(*, data, qubole_conn_id = 'qubole_default', **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Base class for all Qubole Sensors |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['data', 'qubole_conn_id'] |
| |
| |
| |
| .. py:attribute:: template_ext |
| :annotation: :Sequence[str] = ['.txt'] |
| |
| |
| |
| .. py:method:: poke(context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| |
| .. py:class:: QuboleFileSensor(**kwargs) |
| |
| Bases: :py:obj:`QuboleSensor` |
| |
| Wait for a file or folder to be present in cloud storage |
| and check for its presence via QDS APIs |
| |
| .. seealso:: |
| For more information on how to use this sensor, take a look at the guide: |
| :ref:`howto/operator:QuboleFileSensor` |
| |
| :param qubole_conn_id: Connection id which consists of qds auth_token |
| :param data: a JSON object containing payload, whose presence needs to be checked |
| Check this `example <https://github.com/apache/airflow/blob/main /airflow/providers/qubole/example_dags/example_qubole_sensor.py>`_ for sample payload |
| structure. |
| |
| .. note:: Both ``data`` and ``qubole_conn_id`` fields support templating. You can |
| also use ``.txt`` files for template-driven use cases. |
| |
| |
| .. py:class:: QubolePartitionSensor(**kwargs) |
| |
| Bases: :py:obj:`QuboleSensor` |
| |
| Wait for a Hive partition to show up in QHS (Qubole Hive Service) |
| and check for its presence via QDS APIs |
| |
| .. seealso:: |
| For more information on how to use this sensor, take a look at the guide: |
| :ref:`howto/operator:QubolePartitionSensor` |
| |
| :param qubole_conn_id: Connection id which consists of qds auth_token |
| :param data: a JSON object containing payload, whose presence needs to be checked. |
| Check this `example <https://github.com/apache/airflow/blob/main /airflow/providers/qubole/example_dags/example_qubole_sensor.py>`_ for sample payload |
| structure. |
| |
| .. note:: Both ``data`` and ``qubole_conn_id`` fields support templating. You can |
| also use ``.txt`` files for template-driven use cases. |
| |
| |