| :mod:`airflow.contrib.hooks.gcp_api_base_hook` |
| ============================================== |
| |
| .. py:module:: airflow.contrib.hooks.gcp_api_base_hook |
| |
| |
| Module Contents |
| --------------- |
| |
| .. data:: log |
| |
| |
| |
| |
| .. data:: _DEFAULT_SCOPES |
| :annotation: = ['https://www.googleapis.com/auth/cloud-platform'] |
| |
| |
| |
| .. data:: INVALID_KEYS |
| :annotation: = ['DefaultRequestsPerMinutePerProject', 'DefaultRequestsPerMinutePerUser', 'RequestsPerMinutePerProject', 'Resource has been exhausted (e.g. check quota).'] |
| |
| |
| |
| .. data:: INVALID_REASONS |
| :annotation: = ['userRateLimitExceeded'] |
| |
| |
| |
| .. function:: is_soft_quota_exception(exception) |
| API for Google services does not have a standardized way to report quota violation errors. |
| |
| The function has been adapted by trial and error to the following services: |
| |
| * Google Translate |
| * Google Vision |
| * Google Text-to-Speech |
| * Google Speech-to-Text |
| * Google Natural Language |
| * Google Video Intelligence |
| |
| |
| .. py:class:: retry_if_temporary_quota |
| |
| Bases: :class:`tenacity.retry_if_exception` |
| |
| Retries if there was an exception for exceeding the temporary quote limit. |
| |
| |
| .. py:class:: GoogleCloudBaseHook(gcp_conn_id='google_cloud_default', delegate_to=None) |
| |
| Bases: :class:`airflow.hooks.base_hook.BaseHook` |
| |
| A base hook for Google cloud-related hooks. Google cloud has a shared REST |
| API client that is built in the same way no matter which service you use. |
| This class helps construct and authorize the credentials needed to then |
| call googleapiclient.discovery.build() to actually discover and build a client |
| for a Google cloud service. |
| |
| The class also contains some miscellaneous helper functions. |
| |
| All hook derived from this base hook use the 'Google Cloud Platform' connection |
| type. Three ways of authentication are supported: |
| |
| Default credentials: Only the 'Project Id' is required. You'll need to |
| have set up default credentials, such as by the |
| ``GOOGLE_APPLICATION_DEFAULT`` environment variable or from the metadata |
| server on Google Compute Engine. |
| |
| JSON key file: Specify 'Project Id', 'Keyfile Path' and 'Scope'. |
| |
| Legacy P12 key files are not supported. |
| |
| JSON data provided in the UI: Specify 'Keyfile JSON'. |
| |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :type gcp_conn_id: str |
| :param delegate_to: The account to impersonate, if any. |
| For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :type delegate_to: str |
| |
| .. py:class:: _Decorators |
| |
| Bases: :class:`object` |
| |
| A private inner class for keeping all decorator methods. |
| |
| |
| .. staticmethod:: provide_gcp_credential_file(func) |
| |
| Function decorator that provides a GOOGLE_APPLICATION_CREDENTIALS |
| environment variable, pointing to file path of a JSON file of service |
| account key. |
| |
| |
| |
| |
| .. attribute:: project_id |
| |
| |
| |
| |
| .. attribute:: num_retries |
| |
| |
| Returns num_retries from Connection. |
| |
| :return: the number of times each API request should be retried |
| :rtype: int |
| |
| |
| |
| .. method:: _get_credentials(self) |
| |
| Returns the Credentials object for Google API |
| |
| |
| |
| |
| .. method:: _get_access_token(self) |
| |
| Returns a valid access token from Google API Credentials |
| |
| |
| |
| |
| .. method:: _authorize(self) |
| |
| Returns an authorized HTTP object to be used to build a Google cloud |
| service hook connection. |
| |
| |
| |
| |
| .. method:: _get_field(self, f, default=None) |
| |
| Fetches a field from extras, and returns it. This is some Airflow |
| magic. The google_cloud_platform hook type adds custom UI elements |
| to the hook page, which allow admins to specify service_account, |
| key_path, etc. They get formatted as shown below. |
| |
| |
| |
| |
| .. staticmethod:: quota_retry(*args, **kwargs) |
| |
| Function decorator that provides a mechanism to repeat requests in response to exceeding a temporary |
| quote limit. |
| |
| |
| |
| |
| .. staticmethod:: catch_http_exception(func) |
| |
| Function decorator that intercepts HTTP Errors and raises AirflowException |
| with more informative message. |
| |
| |
| |
| |
| .. staticmethod:: fallback_to_default_project_id(func) |
| |
| Decorator that provides fallback for Google Cloud Platform project id. If |
| the project is None it will be replaced with the project_id from the |
| service account the Hook is authenticated with. Project id can be specified |
| either via project_id kwarg or via first parameter in positional args. |
| |
| :param func: function to wrap |
| :return: result of the function call |
| |
| |
| |
| |
| .. method:: _get_project_id(self, project_id) |
| |
| In case project_id is None, overrides it with default project_id from |
| the service account that is authorized. |
| |
| :param project_id: project id to |
| :type project_id: str |
| :return: the project_id specified or default project id if project_id is None |
| |
| |
| |
| |