:py:mod:`airflow.providers.google.cloud.operators.cloud_storage_transfer_service`
=================================================================================

.. py:module:: airflow.providers.google.cloud.operators.cloud_storage_transfer_service

.. autoapi-nested-parse::

   This module contains Google Cloud Transfer operators.


Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.TransferJobPreprocessor
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.TransferJobValidator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceCreateJobOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceUpdateJobOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceDeleteJobOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceGetOperationOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceListOperationsOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServicePauseOperationOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceResumeOperationOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceCancelOperationOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceS3ToGCSOperator
   airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceGCSToGCSOperator


.. py:class:: TransferJobPreprocessor(body, aws_conn_id = 'aws_default', default_schedule = False)

   Helper class for preprocess of transfer job body.

   .. py:method:: process_body(self)

      Injects AWS credentials into body if needed and
      reformats schedule information.

      :return: Preprocessed body
      :rtype: dict


.. py:class:: TransferJobValidator(body)

   Helper class for validating transfer job body.

   .. py:method:: validate_body(self)

      Validates the body. Checks if body specifies `transferSpec`
      if yes, then check if AWS credentials are passed correctly and
      no more than 1 data source was selected.

      :raises: AirflowException


.. py:class:: CloudDataTransferServiceCreateJobOperator(*, body, aws_conn_id = 'aws_default', gcp_conn_id = 'google_cloud_default', api_version = 'v1', google_impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Creates a transfer job that runs periodically.

   .. warning::

       This operator is NOT idempotent in the following cases:

       * `name` is not passed in body param
       * transfer job `name` has been soft deleted. In this case,
         each new task will receive a unique suffix

       If you run it many times, many transfer jobs will be created in the Google Cloud.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:CloudDataTransferServiceCreateJobOperator`

   :param body: (Required) The request body, as described in
       https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
       With three additional improvements:

       * dates can be given in the form :class:`datetime.date`
       * times can be given in the form :class:`datetime.time`
       * credentials to Amazon Web Service should be stored in the connection and indicated by the
         aws_conn_id parameter

   :param aws_conn_id: The connection ID used to retrieve credentials to
       Amazon Web Service.
   :param gcp_conn_id: The connection ID used to connect to Google Cloud.
   :param api_version: API version used (e.g. v1).
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['body', 'gcp_conn_id', 'aws_conn_id', 'google_impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServiceUpdateJobOperator(*, job_name, body, aws_conn_id = 'aws_default', gcp_conn_id = 'google_cloud_default', api_version = 'v1', google_impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Updates a transfer job that runs periodically.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:CloudDataTransferServiceUpdateJobOperator`

   :param job_name: (Required) Name of the job to be updated
   :param body: (Required) The request body, as described in
       https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
       With three additional improvements:

       * dates can be given in the form :class:`datetime.date`
       * times can be given in the form :class:`datetime.time`
       * credentials to Amazon Web Service should be stored in the connection and indicated by the
         aws_conn_id parameter

   :param aws_conn_id: The connection ID used to retrieve credentials to
       Amazon Web Service.
   :param gcp_conn_id: The connection ID used to connect to Google Cloud.
   :param api_version: API version used (e.g. v1).
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['job_name', 'body', 'gcp_conn_id', 'aws_conn_id', 'google_impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServiceDeleteJobOperator(*, job_name, gcp_conn_id = 'google_cloud_default', api_version = 'v1', project_id = None, google_impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Delete a transfer job. This is a soft delete. After a transfer job is
   deleted, the job and all the transfer executions are subject to garbage
   collection. Transfer jobs become eligible for garbage collection
   30 days after soft delete.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:CloudDataTransferServiceDeleteJobOperator`

   :param job_name: (Required) Name of the TRANSFER operation
   :param project_id: (Optional) the ID of the project that owns the Transfer
       Job. If set to None or missing, the default project_id from the Google Cloud
       connection is used.
   :param gcp_conn_id: The connection ID used to connect to Google Cloud.
   :param api_version: API version used (e.g. v1).
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['job_name', 'project_id', 'gcp_conn_id', 'api_version', 'google_impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServiceGetOperationOperator(*, operation_name, gcp_conn_id = 'google_cloud_default', api_version = 'v1', google_impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Gets the latest state of a long-running operation in Google Storage Transfer
   Service.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:CloudDataTransferServiceGetOperationOperator`

   :param operation_name: (Required) Name of the transfer operation.
   :param gcp_conn_id: The connection ID used to connect to Google
       Cloud Platform.
   :param api_version: API version used (e.g. v1).
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['operation_name', 'gcp_conn_id', 'google_impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServiceListOperationsOperator(request_filter = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1', google_impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Lists long-running operations in Google Storage Transfer
   Service that match the specified filter.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:CloudDataTransferServiceListOperationsOperator`

   :param request_filter: (Required) A request filter, as described in
           https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
   :param gcp_conn_id: The connection ID used to connect to Google
       Cloud Platform.
   :param api_version: API version used (e.g. v1).
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['filter', 'gcp_conn_id', 'google_impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServicePauseOperationOperator(*, operation_name, gcp_conn_id = 'google_cloud_default', api_version = 'v1', google_impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Pauses a transfer operation in Google Storage Transfer Service.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:CloudDataTransferServicePauseOperationOperator`

   :param operation_name: (Required) Name of the transfer operation.
   :param gcp_conn_id: The connection ID used to connect to Google Cloud.
   :param api_version:  API version used (e.g. v1).
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['operation_name', 'gcp_conn_id', 'api_version', 'google_impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServiceResumeOperationOperator(*, operation_name, gcp_conn_id = 'google_cloud_default', api_version = 'v1', google_impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Resumes a transfer operation in Google Storage Transfer Service.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:CloudDataTransferServiceResumeOperationOperator`

   :param operation_name: (Required) Name of the transfer operation.
   :param gcp_conn_id: The connection ID used to connect to Google Cloud.
   :param api_version: API version used (e.g. v1).
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['operation_name', 'gcp_conn_id', 'api_version', 'google_impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServiceCancelOperationOperator(*, operation_name, gcp_conn_id = 'google_cloud_default', api_version = 'v1', google_impersonation_chain = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Cancels a transfer operation in Google Storage Transfer Service.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:CloudDataTransferServiceCancelOperationOperator`

   :param operation_name: (Required) Name of the transfer operation.
   :param api_version: API version used (e.g. v1).
   :param gcp_conn_id: The connection ID used to connect to Google
       Cloud Platform.
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['operation_name', 'gcp_conn_id', 'api_version', 'google_impersonation_chain']

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServiceS3ToGCSOperator(*, s3_bucket, gcs_bucket, s3_path = None, gcs_path = None, project_id = None, aws_conn_id = 'aws_default', gcp_conn_id = 'google_cloud_default', delegate_to = None, description = None, schedule = None, object_conditions = None, transfer_options = None, wait = True, timeout = None, google_impersonation_chain = None, delete_job_after_completion = False, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Synchronizes an S3 bucket with a Google Cloud Storage bucket using the
   Google Cloud Storage Transfer Service.

   .. warning::

       This operator is NOT idempotent. If you run it many times, many transfer
       jobs will be created in the Google Cloud.

   **Example**:

   .. code-block:: python

      s3_to_gcs_transfer_op = S3ToGoogleCloudStorageTransferOperator(
          task_id="s3_to_gcs_transfer_example",
          s3_bucket="my-s3-bucket",
          project_id="my-gcp-project",
          gcs_bucket="my-gcs-bucket",
          dag=my_dag,
      )

   :param s3_bucket: The S3 bucket where to find the objects. (templated)
   :param gcs_bucket: The destination Google Cloud Storage bucket
       where you want to store the files. (templated)
   :param s3_path: Optional root path where the source objects are. (templated)
   :param gcs_path: Optional root path for transferred objects. (templated)
   :param project_id: Optional ID of the Google Cloud Console project that
       owns the job
   :param aws_conn_id: The source S3 connection
   :param gcp_conn_id: The destination connection ID to use
       when connecting to Google Cloud Storage.
   :param delegate_to: Google account to impersonate using domain-wide delegation of authority,
       if any. For this to work, the service account making the request must have
       domain-wide delegation enabled.
   :param description: Optional transfer service job description
   :param schedule: Optional transfer service schedule;
       If not set, run transfer job once as soon as the operator runs
       The format is described
       https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs.
       With two additional improvements:

       * dates they can be passed as :class:`datetime.date`
       * times they can be passed as :class:`datetime.time`

   :param object_conditions: Optional transfer service object conditions; see
       https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec
   :param transfer_options: Optional transfer service transfer options; see
       https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec
   :param wait: Wait for transfer to finish. It must be set to True, if
       'delete_job_after_completion' is set to True.
   :param timeout: Time to wait for the operation to end in seconds. Defaults to 60 seconds if not specified.
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).
   :param delete_job_after_completion: If True, delete the job after complete.
       If set to True, 'wait' must be set to True.

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['gcp_conn_id', 's3_bucket', 'gcs_bucket', 's3_path', 'gcs_path', 'description',...

      
   .. py:attribute:: ui_color
      :annotation: = #e09411

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.


.. py:class:: CloudDataTransferServiceGCSToGCSOperator(*, source_bucket, destination_bucket, source_path = None, destination_path = None, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, description = None, schedule = None, object_conditions = None, transfer_options = None, wait = True, timeout = None, google_impersonation_chain = None, delete_job_after_completion = False, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`

   Copies objects from a bucket to another using the Google Cloud Storage Transfer Service.

   .. warning::

       This operator is NOT idempotent. If you run it many times, many transfer
       jobs will be created in the Google Cloud.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:GCSToGCSOperator`

   **Example**:

   .. code-block:: python

      gcs_to_gcs_transfer_op = GoogleCloudStorageToGoogleCloudStorageTransferOperator(
          task_id="gcs_to_gcs_transfer_example",
          source_bucket="my-source-bucket",
          destination_bucket="my-destination-bucket",
          project_id="my-gcp-project",
          dag=my_dag,
      )

   :param source_bucket: The source Google Cloud Storage bucket where the
        object is. (templated)
   :param destination_bucket: The destination Google Cloud Storage bucket
       where the object should be. (templated)
   :param source_path: Optional root path where the source objects are. (templated)
   :param destination_path: Optional root path for transferred objects. (templated)
   :param project_id: The ID of the Google Cloud Console project that
       owns the job
   :param gcp_conn_id: Optional connection ID to use when connecting to Google Cloud
       Storage.
   :param delegate_to: Google account to impersonate using domain-wide delegation of authority,
       if any. For this to work, the service account making the request must have
       domain-wide delegation enabled.
   :param description: Optional transfer service job description
   :param schedule: Optional transfer service schedule;
       If not set, run transfer job once as soon as the operator runs
       See:
       https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs.
       With two additional improvements:

       * dates they can be passed as :class:`datetime.date`
       * times they can be passed as :class:`datetime.time`

   :param object_conditions: Optional transfer service object conditions; see
       https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec#ObjectConditions
   :param transfer_options: Optional transfer service transfer options; see
       https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec#TransferOptions
   :param wait: Wait for transfer to finish. It must be set to True, if
       'delete_job_after_completion' is set to True.
   :param timeout: Time to wait for the operation to end in seconds. Defaults to 60 seconds if not specified.
   :param google_impersonation_chain: Optional Google service account to impersonate using
       short-term credentials, or chained list of accounts required to get the access_token
       of the last account in the list, which will be impersonated in the request.
       If set as a string, the account must grant the originating account
       the Service Account Token Creator IAM role.
       If set as a sequence, the identities from the list must grant
       Service Account Token Creator IAM role to the directly preceding identity, with first
       account from the list granting this role to the originating account (templated).
   :param delete_job_after_completion: If True, delete the job after complete.
       If set to True, 'wait' must be set to True.

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['gcp_conn_id', 'source_bucket', 'destination_bucket', 'source_path', 'destination_path',...

      
   .. py:attribute:: ui_color
      :annotation: = #e09411

      
   .. py:method:: execute(self, context)

      This is the main method to derive when creating an operator.
      Context is the same dictionary used as when rendering jinja templates.

      Refer to get_template_context for more context.