| :py:mod:`airflow.providers.google.cloud.operators.vertex_ai.dataset` |
| ==================================================================== |
| |
| .. py:module:: airflow.providers.google.cloud.operators.vertex_ai.dataset |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains Google Vertex AI operators. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.operators.vertex_ai.dataset.CreateDatasetOperator |
| airflow.providers.google.cloud.operators.vertex_ai.dataset.GetDatasetOperator |
| airflow.providers.google.cloud.operators.vertex_ai.dataset.DeleteDatasetOperator |
| airflow.providers.google.cloud.operators.vertex_ai.dataset.ExportDataOperator |
| airflow.providers.google.cloud.operators.vertex_ai.dataset.ImportDataOperator |
| airflow.providers.google.cloud.operators.vertex_ai.dataset.ListDatasetsOperator |
| airflow.providers.google.cloud.operators.vertex_ai.dataset.UpdateDatasetOperator |
| |
| |
| |
| |
| .. py:class:: CreateDatasetOperator(*, region, project_id, dataset, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Creates a Dataset. |
| |
| :param project_id: Required. The ID of the Google Cloud project the cluster belongs to. |
| :param region: Required. The Cloud Dataproc region in which to handle the request. |
| :param dataset: Required. The Dataset to create. This corresponds to the ``dataset`` field on the |
| ``request`` instance; if ``request`` is provided, this should not be set. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: GetDatasetOperator(*, region, project_id, dataset_id, read_mask = None, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Get a Dataset. |
| |
| :param project_id: Required. The ID of the Google Cloud project the cluster belongs to. |
| :param region: Required. The Cloud Dataproc region in which to handle the request. |
| :param dataset_id: Required. The ID of the Dataset to get. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'dataset_id', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: DeleteDatasetOperator(*, region, project_id, dataset_id, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Deletes a Dataset. |
| |
| :param project_id: Required. The ID of the Google Cloud project the cluster belongs to. |
| :param region: Required. The Cloud Dataproc region in which to handle the request. |
| :param dataset_id: Required. The ID of the Dataset to delete. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'dataset_id', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: ExportDataOperator(*, region, project_id, dataset_id, export_config, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Exports data from a Dataset. |
| |
| :param project_id: Required. The ID of the Google Cloud project the cluster belongs to. |
| :param region: Required. The Cloud Dataproc region in which to handle the request. |
| :param dataset_id: Required. The ID of the Dataset to delete. |
| :param export_config: Required. The desired output location. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'dataset_id', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: ImportDataOperator(*, region, project_id, dataset_id, import_configs, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Imports data into a Dataset. |
| |
| :param project_id: Required. The ID of the Google Cloud project the cluster belongs to. |
| :param region: Required. The Cloud Dataproc region in which to handle the request. |
| :param dataset_id: Required. The ID of the Dataset to delete. |
| :param import_configs: Required. The desired input locations. The contents of all input locations will be |
| imported in one batch. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'dataset_id', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: ListDatasetsOperator(*, region, project_id, filter = None, page_size = None, page_token = None, read_mask = None, order_by = None, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Lists Datasets in a Location. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param filter: The standard list filter. |
| :param page_size: The standard list page size. |
| :param page_token: The standard list page token. |
| :param read_mask: Mask specifying which fields to read. |
| :param order_by: A comma-separated list of fields to order by, sorted in ascending order. Use "desc" |
| after a field name for descending. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: UpdateDatasetOperator(*, project_id, region, dataset_id, dataset, update_mask, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Updates a Dataset. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param dataset_id: Required. The ID of the Dataset to update. |
| :param dataset: Required. The Dataset which replaces the resource on the server. |
| :param update_mask: Required. The update mask applies to the resource. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'dataset_id', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |