:py:mod:`airflow.providers.google.cloud.operators.gcs`
======================================================

.. py:module:: airflow.providers.google.cloud.operators.gcs

.. autoapi-nested-parse::

   This module contains a Google Cloud Storage Bucket operator.


Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

   airflow.providers.google.cloud.operators.gcs.GCSCreateBucketOperator
   airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator
   airflow.providers.google.cloud.operators.gcs.GCSDeleteObjectsOperator
   airflow.providers.google.cloud.operators.gcs.GCSBucketCreateAclEntryOperator
   airflow.providers.google.cloud.operators.gcs.GCSObjectCreateAclEntryOperator
   airflow.providers.google.cloud.operators.gcs.GCSFileTransformOperator
   airflow.providers.google.cloud.operators.gcs.GCSTimeSpanFileTransformOperator
   airflow.providers.google.cloud.operators.gcs.GCSDeleteBucketOperator
   airflow.providers.google.cloud.operators.gcs.GCSSynchronizeBucketsOperator


.. py:class:: GCSCreateBucketOperator(*, bucket_name, resource = None, storage_class = 'MULTI_REGIONAL', location = 'US', project_id = None, labels = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Creates a new bucket. Google Cloud Storage uses a flat namespace,
   so you can't create a bucket with a name that is already in use.

       .. seealso::
           For more information, see Bucket Naming Guidelines:
           https://cloud.google.com/storage/docs/bucketnaming.html#requirements

   :param bucket_name: The name of the bucket. (templated)
   :param resource: An optional dict with parameters for creating the bucket.
           For information on available parameters, see Cloud Storage API doc:
           https://cloud.google.com/storage/docs/json_api/v1/buckets/insert
   :param storage_class: This defines how objects in the bucket are stored
           and determines the SLA and the cost of storage (templated). Values include

           - ``MULTI_REGIONAL``
           - ``REGIONAL``
           - ``STANDARD``
           - ``NEARLINE``
           - ``COLDLINE``.

           If this value is not specified when the bucket is
           created, it will default to STANDARD.
   :param location: The location of the bucket. (templated)
       Object data for objects in the bucket resides in physical storage
       within this region. Defaults to US.

       .. seealso:: https://developers.google.com/storage/docs/bucket-locations

   :param project_id: The ID of the Google Cloud Project. (templated)
   :param labels: User-provided labels, in key/value pairs.
   :param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
   :param delegate_to: The account to impersonate using domain-wide delegation of authority,
       if any. For this to work, the service account making the request must have
       domain-wide delegation enabled.
   :param impersonation_chain: Optional service account to impersonate using short-term
       credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   The following Operator would create a new bucket ``test-bucket``
   with ``MULTI_REGIONAL`` storage class in ``EU`` region

   .. code-block:: python

       CreateBucket = GoogleCloudStorageCreateBucketOperator(
           task_id="CreateNewBucket",
           bucket_name="test-bucket",
           storage_class="MULTI_REGIONAL",
           location="EU",
           labels={"env": "dev", "team": "airflow"},
           gcp_conn_id="airflow-conn-id",
       )


   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['bucket_name', 'storage_class', 'location', 'project_id', 'impersonation_chain']

      
   .. py:attribute:: ui_color
      :annotation: = #f0eee4

      
   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: GCSListObjectsOperator(*, bucket, prefix = None, delimiter = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   List all objects from the bucket with the given string prefix and delimiter in name.

   This operator returns a python list with the name of objects which can be used by
    `xcom` in the downstream task.

   :param bucket: The Google Cloud Storage bucket to find the objects. (templated)
   :param prefix: Prefix string which filters objects whose name begin with
          this prefix. (templated)
   :param delimiter: The delimiter by which you want to filter the objects. (templated)
       For e.g to lists the CSV files from in a directory in GCS you would use
       delimiter='.csv'.
   :param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
   :param delegate_to: The account to impersonate using domain-wide delegation of authority,
       if any. For this to work, the service account making the request must have
       domain-wide delegation enabled.
   :param impersonation_chain: Optional service account to impersonate using short-term
       credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   **Example**:
       The following Operator would list all the Avro files from ``sales/sales-2017``
       folder in ``data`` bucket. ::

           GCS_Files = GoogleCloudStorageListOperator(
               task_id='GCS_Files',
               bucket='data',
               prefix='sales/sales-2017/',
               delimiter='.avro',
               gcp_conn_id=google_cloud_conn_id
           )

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['bucket', 'prefix', 'delimiter', 'impersonation_chain']

      
   .. py:attribute:: ui_color
      :annotation: = #f0eee4

      
   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: GCSDeleteObjectsOperator(*, bucket_name, objects = None, prefix = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Deletes objects from a Google Cloud Storage bucket, either
   from an explicit list of object names or all objects
   matching a prefix.

   :param bucket_name: The GCS bucket to delete from
   :param objects: List of objects to delete. These should be the names
       of objects in the bucket, not including gs://bucket/
   :param prefix: Prefix of objects to delete. All objects matching this
       prefix in the bucket will be deleted.
   :param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
   :param delegate_to: The account to impersonate using domain-wide delegation of authority,
       if any. For this to work, the service account making the request must have
       domain-wide delegation enabled.
   :param impersonation_chain: Optional service account to impersonate using short-term
       credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['bucket_name', 'prefix', 'objects', 'impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: GCSBucketCreateAclEntryOperator(*, bucket, entity, role, user_project = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Creates a new ACL entry on the specified bucket.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:GCSBucketCreateAclEntryOperator`

   :param bucket: Name of a bucket.
   :param entity: The entity holding the permission, in one of the following forms:
       user-userId, user-email, group-groupId, group-email, domain-domain,
       project-team-projectId, allUsers, allAuthenticatedUsers
   :param role: The access permission for the entity.
       Acceptable values are: "OWNER", "READER", "WRITER".
   :param user_project: (Optional) The project to be billed for this request.
       Required for Requester Pays buckets.
   :param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
   :param impersonation_chain: Optional service account to impersonate using short-term
       credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['bucket', 'entity', 'role', 'user_project', 'impersonation_chain']

      
   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: GCSObjectCreateAclEntryOperator(*, bucket, object_name, entity, role, generation = None, user_project = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Creates a new ACL entry on the specified object.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:GCSObjectCreateAclEntryOperator`

   :param bucket: Name of a bucket.
   :param object_name: Name of the object. For information about how to URL encode object
       names to be path safe, see:
       https://cloud.google.com/storage/docs/json_api/#encoding
   :param entity: The entity holding the permission, in one of the following forms:
       user-userId, user-email, group-groupId, group-email, domain-domain,
       project-team-projectId, allUsers, allAuthenticatedUsers
   :param role: The access permission for the entity.
       Acceptable values are: "OWNER", "READER".
   :param generation: Optional. If present, selects a specific revision of this object.
   :param user_project: (Optional) The project to be billed for this request.
       Required for Requester Pays buckets.
   :param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
   :param impersonation_chain: Optional service account to impersonate using short-term
       credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['bucket', 'object_name', 'entity', 'generation', 'role', 'user_project', 'impersonation_chain']

      
   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: GCSFileTransformOperator(*, source_bucket, source_object, transform_script, destination_bucket = None, destination_object = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Copies data from a source GCS location to a temporary location on the
   local filesystem. Runs a transformation on this file as specified by
   the transformation script and uploads the output to a destination bucket.
   If the output bucket is not specified the original file will be
   overwritten.

   The locations of the source and the destination files in the local
   filesystem is provided as an first and second arguments to the
   transformation script. The transformation script is expected to read the
   data from source, transform it and write the output to the local
   destination file.

   :param source_bucket: The bucket to locate the source_object. (templated)
   :param source_object: The key to be retrieved from GCS. (templated)
   :param destination_bucket: The bucket to upload the key after transformation.
       If not provided, source_bucket will be used. (templated)
   :param destination_object: The key to be written in GCS.
       If not provided, source_object will be used. (templated)
   :param transform_script: location of the executable transformation script or list of arguments
       passed to subprocess ex. `['python', 'script.py', 10]`. (templated)
   :param gcp_conn_id: The connection ID to use connecting to Google Cloud.
   :param impersonation_chain: Optional service account to impersonate using short-term
       credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['source_bucket', 'source_object', 'destination_bucket', 'destination_object',...

      
   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: GCSTimeSpanFileTransformOperator(*, source_bucket, source_prefix, source_gcp_conn_id, destination_bucket, destination_prefix, destination_gcp_conn_id, transform_script, source_impersonation_chain = None, destination_impersonation_chain = None, chunk_size = None, download_continue_on_fail = False, download_num_attempts = 1, upload_continue_on_fail = False, upload_num_attempts = 1, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Determines a list of objects that were added or modified at a GCS source
   location during a specific time-span, copies them to a temporary location
   on the local file system, runs a transform on this file as specified by
   the transformation script and uploads the output to the destination bucket.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:GCSTimeSpanFileTransformOperator`

   The locations of the source and the destination files in the local
   filesystem is provided as an first and second arguments to the
   transformation script. The time-span is passed to the transform script as
   third and fourth argument as UTC ISO 8601 string.

   The transformation script is expected to read the
   data from source, transform it and write the output to the local
   destination file.

   :param source_bucket: The bucket to fetch data from. (templated)
   :param source_prefix: Prefix string which filters objects whose name begin with
          this prefix. Can interpolate execution date and time components. (templated)
   :param source_gcp_conn_id: The connection ID to use connecting to Google Cloud
          to download files to be processed.
   :param source_impersonation_chain: Optional service account to impersonate using short-term
       credentials (to download files to be processed), or chained list of accounts required to
       get the access_token of the last account in the list, which will be impersonated in the
       request. If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   :param destination_bucket: The bucket to write data to. (templated)
   :param destination_prefix: Prefix string for the upload location.
       Can interpolate execution date and time components. (templated)
   :param destination_gcp_conn_id: The connection ID to use connecting to Google Cloud
          to upload processed files.
   :param destination_impersonation_chain: Optional service account to impersonate using short-term
       credentials (to upload processed files), or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   :param transform_script: location of the executable transformation script or list of arguments
       passed to subprocess ex. `['python', 'script.py', 10]`. (templated)


   :param chunk_size: The size of a chunk of data when downloading or uploading (in bytes).
       This must be a multiple of 256 KB (per the google clout storage API specification).
   :param download_continue_on_fail: With this set to true, if a download fails the task does not error out
       but will still continue.
   :param upload_chunk_size: The size of a chunk of data when uploading (in bytes).
       This must be a multiple of 256 KB (per the google clout storage API specification).
   :param upload_continue_on_fail: With this set to true, if an upload fails the task does not error out
       but will still continue.
   :param upload_num_attempts: Number of attempts to try to upload a single file.

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['source_bucket', 'source_prefix', 'destination_bucket', 'destination_prefix',...

      
   .. py:attribute:: operator_extra_links
      

   .. py:method:: interpolate_prefix(prefix, dt)
      :staticmethod:

      Interpolate prefix with datetime.

      :param prefix: The prefix to interpolate
      :param dt: The datetime to interpolate


   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: GCSDeleteBucketOperator(*, bucket_name, force = True, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Deletes bucket from a Google Cloud Storage.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:GCSDeleteBucketOperator`

   :param bucket_name: name of the bucket which will be deleted
   :param force: false not allow to delete non empty bucket, set force=True
       allows to delete non empty bucket
   :param gcp_conn_id: The connection ID to use connecting to Google Cloud.
   :param impersonation_chain: Optional service account to impersonate using short-term
       credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['bucket_name', 'gcp_conn_id', 'impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: GCSSynchronizeBucketsOperator(*, source_bucket, destination_bucket, source_object = None, destination_object = None, recursive = True, delete_extra_files = False, allow_overwrite = False, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Synchronizes the contents of the buckets or bucket's directories in the Google Cloud Services.

   Parameters ``source_object`` and ``destination_object`` describe the root sync directory. If they are
   not passed, the entire bucket will be synchronized. They should point to directories.

   .. note::
       The synchronization of individual files is not supported. Only entire directories can be
       synchronized.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:GCSSynchronizeBuckets`

   :param source_bucket: The name of the bucket containing the source objects.
   :param destination_bucket: The name of the bucket containing the destination objects.
   :param source_object: The root sync directory in the source bucket.
   :param destination_object: The root sync directory in the destination bucket.
   :param recursive: If True, subdirectories will be considered
   :param allow_overwrite: if True, the files will be overwritten if a mismatched file is found.
       By default, overwriting files is not allowed
   :param delete_extra_files: if True, deletes additional files from the source that not found in the
       destination. By default extra files are not deleted.

       .. note::
           This option can delete data quickly if you specify the wrong source/destination combination.

   :param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
   :param delegate_to: The account to impersonate using domain-wide delegation of authority,
       if any. For this to work, the service account making the request must have
       domain-wide delegation enabled.
   :param impersonation_chain: Optional service account to impersonate using short-term
       credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['source_bucket', 'destination_bucket', 'source_object', 'destination_object', 'recursive',...

      
   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.