| :py:mod:`airflow.providers.amazon.aws.sensors.s3` |
| ================================================= |
| |
| .. py:module:: airflow.providers.amazon.aws.sensors.s3 |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.sensors.s3.S3KeySensor |
| airflow.providers.amazon.aws.sensors.s3.S3KeySizeSensor |
| airflow.providers.amazon.aws.sensors.s3.S3KeysUnchangedSensor |
| airflow.providers.amazon.aws.sensors.s3.S3PrefixSensor |
| |
| |
| |
| |
| .. py:class:: S3KeySensor(*, bucket_key, bucket_name = None, wildcard_match = False, check_fn = None, aws_conn_id = 'aws_default', verify = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Waits for one or multiple keys (a file-like instance on S3) to be present in a S3 bucket. |
| S3 being a key/value it does not support folders. The path is just a key |
| a resource. |
| |
| .. seealso:: |
| For more information on how to use this sensor, take a look at the guide: |
| :ref:`howto/sensor:S3KeySensor` |
| |
| :param bucket_key: The key(s) being waited on. Supports full s3:// style url |
| or relative path from root level. When it's specified as a full s3:// |
| url, please leave bucket_name as `None` |
| :param bucket_name: Name of the S3 bucket. Only needed when ``bucket_key`` |
| is not provided as a full s3:// url. When specified, all the keys passed to ``bucket_key`` |
| refers to this bucket |
| :param wildcard_match: whether the bucket_key should be interpreted as a |
| Unix wildcard pattern |
| :param check_fn: Function that receives the list of the S3 objects, |
| and returns a boolean: |
| - ``True``: the criteria is met |
| - ``False``: the criteria isn't met |
| **Example**: Wait for any S3 object size more than 1 megabyte :: |
| |
| def check_fn(files: List) -> bool: |
| return any(f.get('Size', 0) > 1048576 for f in files) |
| :param aws_conn_id: a reference to the s3 connection |
| :param verify: Whether or not to verify SSL certificates for S3 connection. |
| By default SSL certificates are verified. |
| You can provide the following values: |
| |
| - ``False``: do not validate SSL certificates. SSL will still be used |
| (unless use_ssl is False), but SSL certificates will not be |
| verified. |
| - ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to uses. |
| You can specify this argument if you want to use a different |
| CA cert bundle than the one used by botocore. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['bucket_key', 'bucket_name'] |
| |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| .. py:method:: get_hook(self) |
| |
| Create and return an S3Hook |
| |
| |
| |
| .. py:class:: S3KeySizeSensor(*, check_fn = None, **kwargs) |
| |
| Bases: :py:obj:`S3KeySensor` |
| |
| This class is deprecated. |
| Please use :class:`~airflow.providers.amazon.aws.sensors.s3.S3KeySensor`. |
| |
| .. py:method:: default_check_fn(data) |
| :staticmethod: |
| |
| Default function for checking that S3 Objects have size more than 0 |
| |
| :param data: List of the objects in S3 bucket. |
| |
| |
| |
| .. py:class:: S3KeysUnchangedSensor(*, bucket_name, prefix, aws_conn_id = 'aws_default', verify = None, inactivity_period = 60 * 60, min_objects = 1, previous_objects = None, allow_delete = True, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Checks for changes in the number of objects at prefix in AWS S3 |
| bucket and returns True if the inactivity period has passed with no |
| increase in the number of objects. Note, this sensor will not behave correctly |
| in reschedule mode, as the state of the listed objects in the S3 bucket will |
| be lost between rescheduled invocations. |
| |
| .. seealso:: |
| For more information on how to use this sensor, take a look at the guide: |
| :ref:`howto/sensor:S3KeysUnchangedSensor` |
| |
| :param bucket_name: Name of the S3 bucket |
| :param prefix: The prefix being waited on. Relative path from bucket root level. |
| :param aws_conn_id: a reference to the s3 connection |
| :param verify: Whether or not to verify SSL certificates for S3 connection. |
| By default SSL certificates are verified. |
| You can provide the following values: |
| |
| - ``False``: do not validate SSL certificates. SSL will still be used |
| (unless use_ssl is False), but SSL certificates will not be |
| verified. |
| - ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to uses. |
| You can specify this argument if you want to use a different |
| CA cert bundle than the one used by botocore. |
| :param inactivity_period: The total seconds of inactivity to designate |
| keys unchanged. Note, this mechanism is not real time and |
| this operator may not return until a poke_interval after this period |
| has passed with no additional objects sensed. |
| :param min_objects: The minimum number of objects needed for keys unchanged |
| sensor to be considered valid. |
| :param previous_objects: The set of object ids found during the last poke. |
| :param allow_delete: Should this sensor consider objects being deleted |
| between pokes valid behavior. If true a warning message will be logged |
| when this happens. If false an error will be raised. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['bucket_name', 'prefix'] |
| |
| |
| |
| .. py:method:: hook(self) |
| |
| Returns S3Hook. |
| |
| |
| .. py:method:: is_keys_unchanged(self, current_objects) |
| |
| Checks whether new objects have been uploaded and the inactivity_period |
| has passed and updates the state of the sensor accordingly. |
| |
| :param current_objects: set of object ids in bucket during last poke. |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| |
| .. py:class:: S3PrefixSensor(*, prefix, delimiter = '/', **kwargs) |
| |
| Bases: :py:obj:`S3KeySensor` |
| |
| This class is deprecated. |
| Please use :class:`~airflow.providers.amazon.aws.sensors.s3.S3KeySensor`. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['prefix', 'bucket_name'] |
| |
| |
| |
| |
