:py:mod:`airflow.providers.amazon.aws.hooks.s3`
===============================================

.. py:module:: airflow.providers.amazon.aws.hooks.s3

.. autoapi-nested-parse::

   Interact with AWS S3, using the boto3 library.


Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

   airflow.providers.amazon.aws.hooks.s3.S3Hook


Functions
~~~~~~~~~

.. autoapisummary::

   airflow.providers.amazon.aws.hooks.s3.provide_bucket_name
   airflow.providers.amazon.aws.hooks.s3.unify_bucket_name_and_key


Attributes
~~~~~~~~~~

.. autoapisummary::

   airflow.providers.amazon.aws.hooks.s3.T


.. py:data:: T
   

.. py:function:: provide_bucket_name(func)

   Function decorator that provides a bucket name taken from the connection
   in case no bucket name has been passed to the function.


.. py:function:: unify_bucket_name_and_key(func)

   Function decorator that unifies bucket name and key taken from the key
   in case no bucket name and at least a key has been passed to the function.


.. py:class:: S3Hook(*args, **kwargs)

   Bases: :py:obj:`airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`

   Interact with AWS S3, using the boto3 library.

   Additional arguments (such as ``aws_conn_id``) may be specified and
   are passed down to the underlying AwsBaseHook.

   .. seealso::
       :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`

   .. py:attribute:: conn_type
      :annotation: = s3

      
   .. py:attribute:: hook_name
      :annotation: = Amazon S3

      
   .. py:method:: parse_s3_url(s3url)
      :staticmethod:

      Parses the S3 Url into a bucket name and key.

      :param s3url: The S3 Url to parse.
      :return: the parsed bucket name and key
      :rtype: tuple of str


   .. py:method:: get_s3_bucket_key(bucket, key, bucket_param_name, key_param_name)
      :staticmethod:

      Get the S3 bucket name and key from either:
          - bucket name and key. Return the info as it is after checking `key` is a relative path
          - key. Must be a full s3:// url

      :param bucket: The S3 bucket name
      :param key: The S3 key
      :param bucket_param_name: The parameter name containing the bucket name
      :param key_param_name: The parameter name containing the key name
      :return: the parsed bucket name and key
      :rtype: tuple of str


   .. py:method:: check_for_bucket(self, bucket_name = None)

      Check if bucket_name exists.

      :param bucket_name: the name of the bucket
      :return: True if it exists and False if not.
      :rtype: bool


   .. py:method:: get_bucket(self, bucket_name = None)

      Returns a boto3.S3.Bucket object

      :param bucket_name: the name of the bucket
      :return: the bucket object to the bucket name.
      :rtype: boto3.S3.Bucket


   .. py:method:: create_bucket(self, bucket_name = None, region_name = None)

      Creates an Amazon S3 bucket.

      :param bucket_name: The name of the bucket
      :param region_name: The name of the aws region in which to create the bucket.


   .. py:method:: check_for_prefix(self, prefix, delimiter, bucket_name = None)

      Checks that a prefix exists in a bucket

      :param bucket_name: the name of the bucket
      :param prefix: a key prefix
      :param delimiter: the delimiter marks key hierarchy.
      :return: False if the prefix does not exist in the bucket and True if it does.
      :rtype: bool


   .. py:method:: list_prefixes(self, bucket_name = None, prefix = None, delimiter = None, page_size = None, max_items = None)

      Lists prefixes in a bucket under prefix

      :param bucket_name: the name of the bucket
      :param prefix: a key prefix
      :param delimiter: the delimiter marks key hierarchy.
      :param page_size: pagination size
      :param max_items: maximum items to return
      :return: a list of matched prefixes
      :rtype: list


   .. py:method:: list_keys(self, bucket_name = None, prefix = None, delimiter = None, page_size = None, max_items = None, start_after_key = None, from_datetime = None, to_datetime = None, object_filter = None)

      Lists keys in a bucket under prefix and not containing delimiter

      :param bucket_name: the name of the bucket
      :param prefix: a key prefix
      :param delimiter: the delimiter marks key hierarchy.
      :param page_size: pagination size
      :param max_items: maximum items to return
      :param start_after_key: should return only keys greater than this key
      :param from_datetime: should return only keys with LastModified attr greater than this equal
          from_datetime
      :param to_datetime: should return only keys with LastModified attr less than this to_datetime
      :param object_filter: Function that receives the list of the S3 objects, from_datetime and
          to_datetime and returns the List of matched key.

      **Example**: Returns the list of S3 object with LastModified attr greater than from_datetime
           and less than to_datetime:

      .. code-block:: python

          def object_filter(
              keys: list,
              from_datetime: Optional[datetime] = None,
              to_datetime: Optional[datetime] = None,
          ) -> list:
              def _is_in_period(input_date: datetime) -> bool:
                  if from_datetime is not None and input_date < from_datetime:
                      return False

                  if to_datetime is not None and input_date > to_datetime:
                      return False
                  return True

              return [k["Key"] for k in keys if _is_in_period(k["LastModified"])]

      :return: a list of matched keys
      :rtype: list


   .. py:method:: get_file_metadata(self, prefix, bucket_name = None, page_size = None, max_items = None)

      Lists metadata objects in a bucket under prefix

      :param prefix: a key prefix
      :param bucket_name: the name of the bucket
      :param page_size: pagination size
      :param max_items: maximum items to return
      :return: a list of metadata of objects
      :rtype: list


   .. py:method:: head_object(self, key, bucket_name = None)

      Retrieves metadata of an object

      :param key: S3 key that will point to the file
      :param bucket_name: Name of the bucket in which the file is stored
      :return: metadata of an object
      :rtype: dict


   .. py:method:: check_for_key(self, key, bucket_name = None)

      Checks if a key exists in a bucket

      :param key: S3 key that will point to the file
      :param bucket_name: Name of the bucket in which the file is stored
      :return: True if the key exists and False if not.
      :rtype: bool


   .. py:method:: get_key(self, key, bucket_name = None)

      Returns a boto3.s3.Object

      :param key: the path to the key
      :param bucket_name: the name of the bucket
      :return: the key object from the bucket
      :rtype: boto3.s3.Object


   .. py:method:: read_key(self, key, bucket_name = None)

      Reads a key from S3

      :param key: S3 key that will point to the file
      :param bucket_name: Name of the bucket in which the file is stored
      :return: the content of the key
      :rtype: str


   .. py:method:: select_key(self, key, bucket_name = None, expression = None, expression_type = None, input_serialization = None, output_serialization = None)

      Reads a key with S3 Select.

      :param key: S3 key that will point to the file
      :param bucket_name: Name of the bucket in which the file is stored
      :param expression: S3 Select expression
      :param expression_type: S3 Select expression type
      :param input_serialization: S3 Select input data serialization format
      :param output_serialization: S3 Select output data serialization format
      :return: retrieved subset of original data by S3 Select
      :rtype: str

      .. seealso::
          For more details about S3 Select parameters:
          http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.select_object_content


   .. py:method:: check_for_wildcard_key(self, wildcard_key, bucket_name = None, delimiter = '')

      Checks that a key matching a wildcard expression exists in a bucket

      :param wildcard_key: the path to the key
      :param bucket_name: the name of the bucket
      :param delimiter: the delimiter marks key hierarchy
      :return: True if a key exists and False if not.
      :rtype: bool


   .. py:method:: get_wildcard_key(self, wildcard_key, bucket_name = None, delimiter = '')

      Returns a boto3.s3.Object object matching the wildcard expression

      :param wildcard_key: the path to the key
      :param bucket_name: the name of the bucket
      :param delimiter: the delimiter marks key hierarchy
      :return: the key object from the bucket or None if none has been found.
      :rtype: boto3.s3.Object


   .. py:method:: load_file(self, filename, key, bucket_name = None, replace = False, encrypt = False, gzip = False, acl_policy = None)

      Loads a local file to S3

      :param filename: path to the file to load.
      :param key: S3 key that will point to the file
      :param bucket_name: Name of the bucket in which to store the file
      :param replace: A flag to decide whether or not to overwrite the key
          if it already exists. If replace is False and the key exists, an
          error will be raised.
      :param encrypt: If True, the file will be encrypted on the server-side
          by S3 and will be stored in an encrypted form while at rest in S3.
      :param gzip: If True, the file will be compressed locally
      :param acl_policy: String specifying the canned ACL policy for the file being
          uploaded to the S3 bucket.


   .. py:method:: load_string(self, string_data, key, bucket_name = None, replace = False, encrypt = False, encoding = None, acl_policy = None, compression = None)

      Loads a string to S3

      This is provided as a convenience to drop a string in S3. It uses the
      boto infrastructure to ship a file to s3.

      :param string_data: str to set as content for the key.
      :param key: S3 key that will point to the file
      :param bucket_name: Name of the bucket in which to store the file
      :param replace: A flag to decide whether or not to overwrite the key
          if it already exists
      :param encrypt: If True, the file will be encrypted on the server-side
          by S3 and will be stored in an encrypted form while at rest in S3.
      :param encoding: The string to byte encoding
      :param acl_policy: The string to specify the canned ACL policy for the
          object to be uploaded
      :param compression: Type of compression to use, currently only gzip is supported.


   .. py:method:: load_bytes(self, bytes_data, key, bucket_name = None, replace = False, encrypt = False, acl_policy = None)

      Loads bytes to S3

      This is provided as a convenience to drop bytes data into S3. It uses the
      boto infrastructure to ship a file to s3.

      :param bytes_data: bytes to set as content for the key.
      :param key: S3 key that will point to the file
      :param bucket_name: Name of the bucket in which to store the file
      :param replace: A flag to decide whether or not to overwrite the key
          if it already exists
      :param encrypt: If True, the file will be encrypted on the server-side
          by S3 and will be stored in an encrypted form while at rest in S3.
      :param acl_policy: The string to specify the canned ACL policy for the
          object to be uploaded


   .. py:method:: load_file_obj(self, file_obj, key, bucket_name = None, replace = False, encrypt = False, acl_policy = None)

      Loads a file object to S3

      :param file_obj: The file-like object to set as the content for the S3 key.
      :param key: S3 key that will point to the file
      :param bucket_name: Name of the bucket in which to store the file
      :param replace: A flag that indicates whether to overwrite the key
          if it already exists.
      :param encrypt: If True, S3 encrypts the file on the server,
          and the file is stored in encrypted form at rest in S3.
      :param acl_policy: The string to specify the canned ACL policy for the
          object to be uploaded


   .. py:method:: copy_object(self, source_bucket_key, dest_bucket_key, source_bucket_name = None, dest_bucket_name = None, source_version_id = None, acl_policy = None)

      Creates a copy of an object that is already stored in S3.

      Note: the S3 connection used here needs to have access to both
      source and destination bucket/key.

      :param source_bucket_key: The key of the source object.

          It can be either full s3:// style url or relative path from root level.

          When it's specified as a full s3:// url, please omit source_bucket_name.
      :param dest_bucket_key: The key of the object to copy to.

          The convention to specify `dest_bucket_key` is the same
          as `source_bucket_key`.
      :param source_bucket_name: Name of the S3 bucket where the source object is in.

          It should be omitted when `source_bucket_key` is provided as a full s3:// url.
      :param dest_bucket_name: Name of the S3 bucket to where the object is copied.

          It should be omitted when `dest_bucket_key` is provided as a full s3:// url.
      :param source_version_id: Version ID of the source object (OPTIONAL)
      :param acl_policy: The string to specify the canned ACL policy for the
          object to be copied which is private by default.


   .. py:method:: delete_bucket(self, bucket_name, force_delete = False)

      To delete s3 bucket, delete all s3 bucket objects and then delete the bucket.

      :param bucket_name: Bucket name
      :param force_delete: Enable this to delete bucket even if not empty
      :return: None
      :rtype: None


   .. py:method:: delete_objects(self, bucket, keys)

      Delete keys from the bucket.

      :param bucket: Name of the bucket in which you are going to delete object(s)
      :param keys: The key(s) to delete from S3 bucket.

          When ``keys`` is a string, it's supposed to be the key name of
          the single object to delete.

          When ``keys`` is a list, it's supposed to be the list of the
          keys to delete.


   .. py:method:: download_file(self, key, bucket_name = None, local_path = None)

      Downloads a file from the S3 location to the local file system.

      :param key: The key path in S3.
      :param bucket_name: The specific bucket to use.
      :param local_path: The local path to the downloaded file. If no path is provided it will use the
          system's temporary directory.
      :return: the file name.
      :rtype: str


   .. py:method:: generate_presigned_url(self, client_method, params = None, expires_in = 3600, http_method = None)

      Generate a presigned url given a client, its method, and arguments

      :param client_method: The client method to presign for.
      :param params: The parameters normally passed to ClientMethod.
      :param expires_in: The number of seconds the presigned url is valid for.
          By default it expires in an hour (3600 seconds).
      :param http_method: The http method to use on the generated url.
          By default, the http method is whatever is used in the method's model.
      :return: The presigned url.
      :rtype: str


   .. py:method:: get_bucket_tagging(self, bucket_name = None)

      Gets a List of tags from a bucket.

      :param bucket_name: The name of the bucket.
      :return: A List containing the key/value pairs for the tags
      :rtype: Optional[List[Dict[str, str]]]


   .. py:method:: put_bucket_tagging(self, tag_set = None, key = None, value = None, bucket_name = None)

      Overwrites the existing TagSet with provided tags.  Must provide either a TagSet or a key/value pair.

      :param tag_set: A List containing the key/value pairs for the tags.
      :param key: The Key for the new TagSet entry.
      :param value: The Value for the new TagSet entry.
      :param bucket_name: The name of the bucket.
      :return: None
      :rtype: None


   .. py:method:: delete_bucket_tagging(self, bucket_name = None)

      Deletes all tags from a bucket.

      :param bucket_name: The name of the bucket.
      :return: None
      :rtype: None