| :py:mod:`airflow.providers.google.cloud.operators.datastore` |
| ============================================================ |
| |
| .. py:module:: airflow.providers.google.cloud.operators.datastore |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains Google Datastore operators. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreExportEntitiesOperator |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreImportEntitiesOperator |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreAllocateIdsOperator |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreBeginTransactionOperator |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreCommitOperator |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreRollbackOperator |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreRunQueryOperator |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreGetOperationOperator |
| airflow.providers.google.cloud.operators.datastore.CloudDatastoreDeleteOperationOperator |
| |
| |
| |
| |
| .. py:class:: CloudDatastoreExportEntitiesOperator(*, bucket, namespace = None, datastore_conn_id = 'google_cloud_default', cloud_storage_conn_id = 'google_cloud_default', delegate_to = None, entity_filter = None, labels = None, polling_interval_in_seconds = 10, overwrite_existing = False, project_id = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Export entities from Google Cloud Datastore to Cloud Storage |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreExportEntitiesOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/export-import-entities |
| |
| :param bucket: name of the cloud storage bucket to backup data |
| :param namespace: optional namespace path in the specified Cloud Storage bucket |
| to backup data. If this namespace does not exist in GCS, it will be created. |
| :param datastore_conn_id: the name of the Datastore connection id to use |
| :param cloud_storage_conn_id: the name of the cloud storage connection id to |
| force-write backup |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param entity_filter: description of what data from the project is included in the |
| export, refer to |
| https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter |
| :param labels: client-assigned labels for cloud storage |
| :param polling_interval_in_seconds: number of seconds to wait before polling for |
| execution status again |
| :param overwrite_existing: if the storage bucket + namespace is not empty, it will be |
| emptied prior to exports. This enables overwriting existing backups. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['bucket', 'namespace', 'entity_filter', 'labels', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudDatastoreImportEntitiesOperator(*, bucket, file, namespace = None, entity_filter = None, labels = None, datastore_conn_id = 'google_cloud_default', delegate_to = None, polling_interval_in_seconds = 10, project_id = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Import entities from Cloud Storage to Google Cloud Datastore |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreImportEntitiesOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/export-import-entities |
| |
| :param bucket: container in Cloud Storage to store data |
| :param file: path of the backup metadata file in the specified Cloud Storage bucket. |
| It should have the extension .overall_export_metadata |
| :param namespace: optional namespace of the backup metadata file in |
| the specified Cloud Storage bucket. |
| :param entity_filter: description of what data from the project is included in |
| the export, refer to |
| https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter |
| :param labels: client-assigned labels for cloud storage |
| :param datastore_conn_id: the name of the connection id to use |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param polling_interval_in_seconds: number of seconds to wait before polling for |
| execution status again |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['bucket', 'file', 'namespace', 'entity_filter', 'labels', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudDatastoreAllocateIdsOperator(*, partial_keys, project_id = None, delegate_to = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Allocate IDs for incomplete keys. Return list of keys. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreAllocateIdsOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds |
| |
| :param partial_keys: a list of partial keys. |
| :param project_id: Google Cloud project ID against which to make the request. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['partial_keys', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudDatastoreBeginTransactionOperator(*, transaction_options, project_id = None, delegate_to = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Begins a new transaction. Returns a transaction handle. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreBeginTransactionOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction |
| |
| :param transaction_options: Options for a new transaction. |
| :param project_id: Google Cloud project ID against which to make the request. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['transaction_options', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudDatastoreCommitOperator(*, body, project_id = None, delegate_to = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Commit a transaction, optionally creating, deleting or modifying some entities. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreCommitOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit |
| |
| :param body: the body of the commit request. |
| :param project_id: Google Cloud project ID against which to make the request. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['body', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudDatastoreRollbackOperator(*, transaction, project_id = None, delegate_to = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Roll back a transaction. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreRollbackOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback |
| |
| :param transaction: the transaction to roll back. |
| :param project_id: Google Cloud project ID against which to make the request. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['transaction', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudDatastoreRunQueryOperator(*, body, project_id = None, delegate_to = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Run a query for entities. Returns the batch of query results. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreRunQueryOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery |
| |
| :param body: the body of the query request. |
| :param project_id: Google Cloud project ID against which to make the request. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['body', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudDatastoreGetOperationOperator(*, name, delegate_to = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Gets the latest state of a long-running operation. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreGetOperationOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects.operations/get |
| |
| :param name: the name of the operation resource. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['name', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudDatastoreDeleteOperationOperator(*, name, delegate_to = None, gcp_conn_id = 'google_cloud_default', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Deletes the long-running operation. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudDatastoreDeleteOperationOperator` |
| |
| .. seealso:: |
| https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects.operations/delete |
| |
| :param name: the name of the operation resource. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['name', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
