| :py:mod:`airflow.providers.google.common.hooks.base_google` |
| =========================================================== |
| |
| .. py:module:: airflow.providers.google.common.hooks.base_google |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains a Google Cloud API base hook. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.common.hooks.base_google.retry_if_temporary_quota |
| airflow.providers.google.common.hooks.base_google.retry_if_operation_in_progress |
| airflow.providers.google.common.hooks.base_google.GoogleBaseHook |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.common.hooks.base_google.is_soft_quota_exception |
| airflow.providers.google.common.hooks.base_google.is_operation_in_progress_exception |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.common.hooks.base_google.log |
| airflow.providers.google.common.hooks.base_google.INVALID_KEYS |
| airflow.providers.google.common.hooks.base_google.INVALID_REASONS |
| airflow.providers.google.common.hooks.base_google.PROVIDE_PROJECT_ID |
| airflow.providers.google.common.hooks.base_google.T |
| airflow.providers.google.common.hooks.base_google.RT |
| |
| |
| .. py:data:: log |
| |
| |
| |
| |
| .. py:data:: INVALID_KEYS |
| :annotation: = ['DefaultRequestsPerMinutePerProject', 'DefaultRequestsPerMinutePerUser',... |
| |
| |
| |
| .. py:data:: INVALID_REASONS |
| :annotation: = ['userRateLimitExceeded'] |
| |
| |
| |
| .. py:function:: is_soft_quota_exception(exception) |
| |
| API for Google services does not have a standardized way to report quota violation errors. |
| The function has been adapted by trial and error to the following services: |
| |
| * Google Translate |
| * Google Vision |
| * Google Text-to-Speech |
| * Google Speech-to-Text |
| * Google Natural Language |
| * Google Video Intelligence |
| |
| |
| .. py:function:: is_operation_in_progress_exception(exception) |
| |
| Some of the calls return 429 (too many requests!) or 409 errors (Conflict) |
| in case of operation in progress. |
| |
| * Google Cloud SQL |
| |
| |
| .. py:class:: retry_if_temporary_quota |
| |
| Bases: :py:obj:`tenacity.retry_if_exception` |
| |
| Retries if there was an exception for exceeding the temporary quote limit. |
| |
| |
| .. py:class:: retry_if_operation_in_progress |
| |
| Bases: :py:obj:`tenacity.retry_if_exception` |
| |
| Retries if there was an exception for exceeding the temporary quote limit. |
| |
| |
| .. py:data:: PROVIDE_PROJECT_ID |
| :annotation: :str |
| |
| |
| |
| .. py:data:: T |
| |
| |
| |
| |
| .. py:data:: RT |
| |
| |
| |
| |
| .. py:class:: GoogleBaseHook(gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None) |
| |
| Bases: :py:obj:`airflow.hooks.base.BaseHook` |
| |
| A base hook for Google cloud-related hooks. Google cloud has a shared REST |
| API client that is built in the same way no matter which service you use. |
| This class helps construct and authorize the credentials needed to then |
| call googleapiclient.discovery.build() to actually discover and build a client |
| for a Google cloud service. |
| |
| The class also contains some miscellaneous helper functions. |
| |
| All hook derived from this base hook use the 'Google Cloud' connection |
| type. Three ways of authentication are supported: |
| |
| Default credentials: Only the 'Project Id' is required. You'll need to |
| have set up default credentials, such as by the |
| ``GOOGLE_APPLICATION_DEFAULT`` environment variable or from the metadata |
| server on Google Compute Engine. |
| |
| JSON key file: Specify 'Project Id', 'Keyfile Path' and 'Scope'. |
| |
| Legacy P12 key files are not supported. |
| |
| JSON data provided in the UI: Specify 'Keyfile JSON'. |
| |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account. |
| |
| .. py:attribute:: conn_name_attr |
| :annotation: = gcp_conn_id |
| |
| |
| |
| .. py:attribute:: default_conn_name |
| :annotation: = google_cloud_default |
| |
| |
| |
| .. py:attribute:: conn_type |
| :annotation: = google_cloud_platform |
| |
| |
| |
| .. py:attribute:: hook_name |
| :annotation: = Google Cloud |
| |
| |
| |
| .. py:method:: get_connection_form_widgets() |
| :staticmethod: |
| |
| Returns connection widgets to add to connection form |
| |
| |
| .. py:method:: get_ui_field_behaviour() |
| :staticmethod: |
| |
| Returns custom field behaviour |
| |
| |
| .. py:method:: project_id(self) |
| :property: |
| |
| Returns project id. |
| |
| :return: id of the project |
| :rtype: str |
| |
| |
| .. py:method:: num_retries(self) |
| :property: |
| |
| Returns num_retries from Connection. |
| |
| :return: the number of times each API request should be retried |
| :rtype: int |
| |
| |
| .. py:method:: client_info(self) |
| :property: |
| |
| Return client information used to generate a user-agent for API calls. |
| |
| It allows for better errors tracking. |
| |
| This object is only used by the google-cloud-* libraries that are built specifically for |
| the Google Cloud. It is not supported by The Google APIs Python Client that use Discovery |
| based APIs. |
| |
| |
| .. py:method:: scopes(self) |
| :property: |
| |
| Return OAuth 2.0 scopes. |
| |
| :return: Returns the scope defined in the connection configuration, or the default scope |
| :rtype: Sequence[str] |
| |
| |
| .. py:method:: quota_retry(*args, **kwargs) |
| :staticmethod: |
| |
| A decorator that provides a mechanism to repeat requests in response to exceeding a temporary quote |
| limit. |
| |
| |
| .. py:method:: operation_in_progress_retry(*args, **kwargs) |
| :staticmethod: |
| |
| A decorator that provides a mechanism to repeat requests in response to |
| operation in progress (HTTP 409) |
| limit. |
| |
| |
| .. py:method:: fallback_to_default_project_id(func) |
| :staticmethod: |
| |
| Decorator that provides fallback for Google Cloud project id. If |
| the project is None it will be replaced with the project_id from the |
| service account the Hook is authenticated with. Project id can be specified |
| either via project_id kwarg or via first parameter in positional args. |
| |
| :param func: function to wrap |
| :return: result of the function call |
| |
| |
| .. py:method:: provide_gcp_credential_file(func) |
| :staticmethod: |
| |
| Function decorator that provides a Google Cloud credentials for application supporting Application |
| Default Credentials (ADC) strategy. |
| |
| It is recommended to use ``provide_gcp_credential_file_as_context`` context manager to limit the |
| scope when authorization data is available. Using context manager also |
| makes it easier to use multiple connection in one function. |
| |
| |
| .. py:method:: provide_gcp_credential_file_as_context(self) |
| |
| Context manager that provides a Google Cloud credentials for application supporting `Application |
| Default Credentials (ADC) strategy <https://cloud.google.com/docs/authentication/production>`__. |
| |
| It can be used to provide credentials for external programs (e.g. gcloud) that expect authorization |
| file in ``GOOGLE_APPLICATION_CREDENTIALS`` environment variable. |
| |
| |
| .. py:method:: provide_authorized_gcloud(self) |
| |
| Provides a separate gcloud configuration with current credentials. |
| |
| The gcloud tool allows you to login to Google Cloud only - ``gcloud auth login`` and |
| for the needs of Application Default Credentials ``gcloud auth application-default login``. |
| In our case, we want all commands to use only the credentials from ADCm so |
| we need to configure the credentials in gcloud manually. |
| |
| |
| .. py:method:: download_content_from_request(file_handle, request, chunk_size) |
| :staticmethod: |
| |
| Download media resources. |
| Note that the Python file object is compatible with io.Base and can be used with this class also. |
| |
| :param file_handle: io.Base or file object. The stream in which to write the downloaded |
| bytes. |
| :param request: googleapiclient.http.HttpRequest, the media request to perform in chunks. |
| :param chunk_size: int, File will be downloaded in chunks of this many bytes. |
| |
| |
| |