| :py:mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3` |
| ================================================================= |
| |
| .. py:module:: airflow.providers.amazon.aws.transfers.google_api_to_s3 |
| |
| .. autoapi-nested-parse:: |
| |
| This module allows you to transfer data from any Google API endpoint into a S3 Bucket. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.transfers.google_api_to_s3.GoogleApiToS3Operator |
| |
| |
| |
| |
| .. py:class:: GoogleApiToS3Operator(*, google_api_service_name, google_api_service_version, google_api_endpoint_path, google_api_endpoint_params, s3_destination_key, google_api_response_via_xcom = None, google_api_endpoint_params_via_xcom = None, google_api_endpoint_params_via_xcom_task_ids = None, google_api_pagination = False, google_api_num_retries = 0, s3_overwrite = False, gcp_conn_id = 'google_cloud_default', delegate_to = None, aws_conn_id = 'aws_default', google_impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Basic class for transferring data from a Google API endpoint into a S3 Bucket. |
| |
| This discovery-based operator use |
| :class:`~airflow.providers.google.common.hooks.discovery_api.GoogleDiscoveryApiHook` to communicate |
| with Google Services via the |
| `Google API Python Client <https://github.com/googleapis/google-api-python-client>`__. |
| Please note that this library is in maintenance mode hence it won't fully support Google Cloud in |
| the future. |
| Therefore it is recommended that you use the custom Google Cloud Service Operators for working |
| with the Google Cloud Platform. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:GoogleApiToS3Operator` |
| |
| :param google_api_service_name: The specific API service that is being requested. |
| :param google_api_service_version: The version of the API that is being requested. |
| :param google_api_endpoint_path: The client libraries path to the api call's executing method. |
| For example: 'analyticsreporting.reports.batchGet' |
| |
| .. note:: See https://developers.google.com/apis-explorer |
| for more information on which methods are available. |
| |
| :param google_api_endpoint_params: The params to control the corresponding endpoint result. |
| :param s3_destination_key: The url where to put the data retrieved from the endpoint in S3. |
| :param google_api_response_via_xcom: Can be set to expose the google api response to xcom. |
| :param google_api_endpoint_params_via_xcom: If set to a value this value will be used as a key |
| for pulling from xcom and updating the google api endpoint params. |
| :param google_api_endpoint_params_via_xcom_task_ids: Task ids to filter xcom by. |
| :param google_api_pagination: If set to True Pagination will be enabled for this request |
| to retrieve all data. |
| |
| .. note:: This means the response will be a list of responses. |
| |
| :param google_api_num_retries: Define the number of retries for the google api requests being made |
| if it fails. |
| :param s3_overwrite: Specifies whether the s3 file will be overwritten if exists. |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: Google account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param aws_conn_id: The connection id specifying the authentication information for the S3 Bucket. |
| :param google_impersonation_chain: Optional Google service account to impersonate using |
| short-term credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['google_api_endpoint_params', 's3_destination_key', 'google_impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: template_ext |
| :annotation: :Sequence[str] = [] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #cc181e |
| |
| |
| |
| .. py:method:: execute(context) |
| |
| Transfers Google APIs json data to S3. |
| |
| :param context: The context that is being provided when executing. |
| |
| |
| |