| :py:mod:`airflow.providers.google.cloud.hooks.dataproc_metastore` |
| ================================================================= |
| |
| .. py:module:: airflow.providers.google.cloud.hooks.dataproc_metastore |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains a Google Cloud Dataproc Metastore hook. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.dataproc_metastore.DataprocMetastoreHook |
| |
| |
| |
| |
| .. py:class:: DataprocMetastoreHook(gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None) |
| |
| Bases: :py:obj:`airflow.providers.google.common.hooks.base_google.GoogleBaseHook` |
| |
| Hook for Google Cloud Dataproc Metastore APIs. |
| |
| .. py:method:: get_dataproc_metastore_client(self) |
| |
| Returns DataprocMetastoreClient. |
| |
| |
| .. py:method:: wait_for_operation(self, timeout, operation) |
| |
| Waits for long-lasting operation to complete. |
| |
| |
| .. py:method:: create_backup(self, project_id, region, service_id, backup, backup_id, request_id = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Creates a new backup in a given project and location. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param backup: Required. The backup to create. The ``name`` field is ignored. The ID of the created |
| backup must be provided in the request's ``backup_id`` field. |
| |
| This corresponds to the ``backup`` field on the ``request`` instance; if ``request`` is provided, |
| this should not be set. |
| :param backup_id: Required. The ID of the backup, which is used as the final component of the |
| backup's name. This value must be between 1 and 64 characters long, begin with a letter, end with |
| a letter or number, and consist of alphanumeric ASCII characters or hyphens. |
| |
| This corresponds to the ``backup_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param request_id: Optional. A unique id used to identify the request. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: create_metadata_import(self, project_id, region, service_id, metadata_import, metadata_import_id, request_id = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Creates a new MetadataImport in a given project and location. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param metadata_import: Required. The metadata import to create. The ``name`` field is ignored. The |
| ID of the created metadata import must be provided in the request's ``metadata_import_id`` field. |
| |
| This corresponds to the ``metadata_import`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param metadata_import_id: Required. The ID of the metadata import, which is used as the final |
| component of the metadata import's name. This value must be between 1 and 64 characters long, |
| begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``metadata_import_id`` field on the ``request`` instance; if ``request`` |
| is provided, this should not be set. |
| :param request_id: Optional. A unique id used to identify the request. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: create_service(self, region, project_id, service, service_id, request_id = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Creates a metastore service in a project and location. |
| |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param service: Required. The Metastore service to create. The ``name`` field is ignored. The ID of |
| the created metastore service must be provided in the request's ``service_id`` field. |
| |
| This corresponds to the ``service`` field on the ``request`` instance; if ``request`` is provided, |
| this should not be set. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param request_id: Optional. A unique id used to identify the request. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: delete_backup(self, project_id, region, service_id, backup_id, request_id = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Deletes a single backup. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param backup_id: Required. The ID of the backup, which is used as the final component of the |
| backup's name. This value must be between 1 and 64 characters long, begin with a letter, end with |
| a letter or number, and consist of alphanumeric ASCII characters or hyphens. |
| |
| This corresponds to the ``backup_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param request_id: Optional. A unique id used to identify the request. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: delete_service(self, project_id, region, service_id, request_id = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Deletes a single service. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param request_id: Optional. A unique id used to identify the request. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: export_metadata(self, destination_gcs_folder, project_id, region, service_id, request_id = None, database_dump_type = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Exports metadata from a service. |
| |
| :param destination_gcs_folder: A Cloud Storage URI of a folder, in the format |
| ``gs://<bucket_name>/<path_inside_bucket>``. A sub-folder |
| ``<export_folder>`` containing exported files will be |
| created below it. |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param request_id: Optional. A unique id used to identify the request. |
| :param database_dump_type: Optional. The type of the database dump. If unspecified, |
| defaults to ``MYSQL``. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: get_service(self, project_id, region, service_id, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Gets the details of a single service. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: get_backup(self, project_id, region, service_id, backup_id, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Get backup from a service. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param backup_id: Required. The ID of the metastore service backup to restore from |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: list_backups(self, project_id, region, service_id, page_size = None, page_token = None, filter = None, order_by = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Lists backups in a service. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param page_size: Optional. The maximum number of backups to |
| return. The response may contain less than the |
| maximum number. If unspecified, no more than 500 |
| backups are returned. The maximum value is 1000; |
| values above 1000 are changed to 1000. |
| :param page_token: Optional. A page token, received from a previous |
| [DataprocMetastore.ListBackups][google.cloud.metastore.v1.DataprocMetastore.ListBackups] |
| call. Provide this token to retrieve the subsequent page. |
| To retrieve the first page, supply an empty page token. |
| When paginating, other parameters provided to |
| [DataprocMetastore.ListBackups][google.cloud.metastore.v1.DataprocMetastore.ListBackups] |
| must match the call that provided the page token. |
| :param filter: Optional. The filter to apply to list |
| results. |
| :param order_by: Optional. Specify the ordering of results as described in |
| `Sorting |
| Order <https://cloud.google.com/apis/design/design_patterns#sorting_order>`__. |
| If not specified, the results will be sorted in the default |
| order. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: restore_service(self, project_id, region, service_id, backup_project_id, backup_region, backup_service_id, backup_id, restore_type = None, request_id = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Restores a service from a backup. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param backup_project_id: Required. The ID of the Google Cloud project that the metastore service |
| backup to restore from. |
| :param backup_region: Required. The ID of the Google Cloud region that the metastore |
| service backup to restore from. |
| :param backup_service_id: Required. The ID of the metastore service backup to restore from, |
| which is used as the final component of the metastore service's name. This value must be |
| between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, |
| and consist of alphanumeric ASCII characters or hyphens. |
| :param backup_id: Required. The ID of the metastore service backup to restore from |
| :param restore_type: Optional. The type of restore. If unspecified, defaults to |
| ``METADATA_ONLY`` |
| :param request_id: Optional. A unique id used to identify the request. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: update_service(self, project_id, region, service_id, service, update_mask, request_id = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Updates the parameters of a single service. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param service_id: Required. The ID of the metastore service, which is used as the final component of |
| the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin |
| with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or |
| hyphens. |
| |
| This corresponds to the ``service_id`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param service: Required. The metastore service to update. The server only merges fields in the |
| service if they are specified in ``update_mask``. |
| |
| The metastore service's ``name`` field is used to identify the metastore service to be updated. |
| |
| This corresponds to the ``service`` field on the ``request`` instance; if ``request`` is provided, |
| this should not be set. |
| :param update_mask: Required. A field mask used to specify the fields to be overwritten in the |
| metastore service resource by the update. Fields specified in the ``update_mask`` are relative to |
| the resource (not to the full request). A field is overwritten if it is in the mask. |
| |
| This corresponds to the ``update_mask`` field on the ``request`` instance; if ``request`` is |
| provided, this should not be set. |
| :param request_id: Optional. A unique id used to identify the request. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| |