docs-archive/1.10.13/_sources/_api/airflow/contrib/operators/s3_to_gcs_operator/index.rst.txt

:mod:`airflow.contrib.operators.s3_to_gcs_operator`
===================================================

.. py:module:: airflow.contrib.operators.s3_to_gcs_operator


Module Contents
---------------

.. py:class:: S3ToGoogleCloudStorageOperator(bucket, prefix='', delimiter='', aws_conn_id='aws_default', verify=None, dest_gcs_conn_id=None, dest_gcs=None, delegate_to=None, replace=False, gzip=False, *args, **kwargs)

   Bases: :class:`airflow.contrib.operators.s3_list_operator.S3ListOperator`

   Synchronizes an S3 key, possibly a prefix, with a Google Cloud Storage
   destination path.

   :param bucket: The S3 bucket where to find the objects. (templated)
   :type bucket: str
   :param prefix: Prefix string which filters objects whose name begin with
       such prefix. (templated)
   :type prefix: str
   :param delimiter: the delimiter marks key hierarchy. (templated)
   :type delimiter: str
   :param aws_conn_id: The source S3 connection
   :type aws_conn_id: str
   :param verify: Whether or not to verify SSL certificates for S3 connection.
       By default SSL certificates are verified.
       You can provide the following values:

       - ``False``: do not validate SSL certificates. SSL will still be used
                (unless use_ssl is False), but SSL certificates will not be
                verified.
       - ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to uses.
                You can specify this argument if you want to use a different
                CA cert bundle than the one used by botocore.
   :type verify: bool or str
   :param dest_gcs_conn_id: The destination connection ID to use
       when connecting to Google Cloud Storage.
   :type dest_gcs_conn_id: str
   :param dest_gcs: The destination Google Cloud Storage bucket and prefix
       where you want to store the files. (templated)
   :type dest_gcs: str
   :param delegate_to: The account to impersonate, if any.
       For this to work, the service account making the request must have
       domain-wide delegation enabled.
   :type delegate_to: str
   :param replace: Whether you want to replace existing destination files
       or not.
   :type replace: bool
   :param gzip: Option to compress file for upload
   :type gzip: bool


   **Example**:

   .. code-block:: python

      s3_to_gcs_op = S3ToGoogleCloudStorageOperator(
           task_id='s3_to_gcs_example',
           bucket='my-s3-bucket',
           prefix='data/customers-201804',
           dest_gcs_conn_id='google_cloud_default',
           dest_gcs='gs://my.gcs.bucket/some/customers/',
           replace=False,
           gzip=True,
           dag=my-dag)

   Note that ``bucket``, ``prefix``, ``delimiter`` and ``dest_gcs`` are
   templated, so you can use variables in them if you wish.

   .. attribute:: template_fields
      :annotation: = ['bucket', 'prefix', 'delimiter', 'dest_gcs']

      

   .. attribute:: ui_color
      :annotation: = #e09411

      

   
   .. method:: execute(self, context)



   
   .. staticmethod:: _gcs_object_is_directory(object)