| :mod:`airflow.sensors.hdfs_sensor` |
| ================================== |
| |
| .. py:module:: airflow.sensors.hdfs_sensor |
| |
| |
| Module Contents |
| --------------- |
| |
| .. data:: log |
| |
| |
| |
| |
| .. py:class:: HdfsSensor(filepath, hdfs_conn_id='hdfs_default', ignored_ext=None, ignore_copying=True, file_size=None, hook=HDFSHook, *args, **kwargs) |
| |
| Bases: :class:`airflow.sensors.base_sensor_operator.BaseSensorOperator` |
| |
| Waits for a file or folder to land in HDFS |
| |
| .. attribute:: template_fields |
| :annotation: = ['filepath'] |
| |
| |
| |
| .. attribute:: ui_color |
| |
| |
| |
| |
| |
| .. staticmethod:: filter_for_filesize(result, size=None) |
| |
| Will test the filepath result and test if its size is at least self.filesize |
| |
| :param result: a list of dicts returned by Snakebite ls |
| :param size: the file size in MB a file should be at least to trigger True |
| :return: (bool) depending on the matching criteria |
| |
| |
| |
| |
| .. staticmethod:: filter_for_ignored_ext(result, ignored_ext, ignore_copying) |
| |
| Will filter if instructed to do so the result to remove matching criteria |
| |
| :param result: list of dicts returned by Snakebite ls |
| :type result: list[dict] |
| :param ignored_ext: list of ignored extensions |
| :type ignored_ext: list |
| :param ignore_copying: shall we ignore ? |
| :type ignore_copying: bool |
| :return: list of dicts which were not removed |
| :rtype: list[dict] |
| |
| |
| |
| |
| .. method:: poke(self, context) |
| |
| |
| |
| |
