| :py:mod:`airflow.providers.google.cloud.operators.cloud_sql` |
| ============================================================ |
| |
| .. py:module:: airflow.providers.google.cloud.operators.cloud_sql |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains Google Cloud SQL operators. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLBaseOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLCreateInstanceOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLInstancePatchOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLDeleteInstanceOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLCreateInstanceDatabaseOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLPatchInstanceDatabaseOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLDeleteInstanceDatabaseOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLExportInstanceOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLImportInstanceOperator |
| airflow.providers.google.cloud.operators.cloud_sql.CloudSQLExecuteQueryOperator |
| |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.operators.cloud_sql.SETTINGS |
| airflow.providers.google.cloud.operators.cloud_sql.SETTINGS_VERSION |
| airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_CREATE_VALIDATION |
| airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_EXPORT_VALIDATION |
| airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_IMPORT_VALIDATION |
| airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_DATABASE_CREATE_VALIDATION |
| airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_DATABASE_PATCH_VALIDATION |
| |
| |
| .. py:data:: SETTINGS |
| :annotation: = settings |
| |
| |
| |
| .. py:data:: SETTINGS_VERSION |
| :annotation: = settingsVersion |
| |
| |
| |
| .. py:data:: CLOUD_SQL_CREATE_VALIDATION |
| |
| |
| |
| |
| .. py:data:: CLOUD_SQL_EXPORT_VALIDATION |
| |
| |
| |
| |
| .. py:data:: CLOUD_SQL_IMPORT_VALIDATION |
| |
| |
| |
| |
| .. py:data:: CLOUD_SQL_DATABASE_CREATE_VALIDATION |
| |
| |
| |
| |
| .. py:data:: CLOUD_SQL_DATABASE_PATCH_VALIDATION |
| |
| |
| |
| |
| .. py:class:: CloudSQLBaseOperator(*, instance, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Abstract base operator for Google Cloud SQL operators to inherit from. |
| |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :param project_id: Optional, Google Cloud Project ID. f set to None or missing, |
| the default project_id from the Google Cloud connection is used. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLCreateInstanceOperator(*, body, instance, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', validate_body = True, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`CloudSQLBaseOperator` |
| |
| Creates a new Cloud SQL instance. |
| If an instance with the same name exists, no action will be taken and |
| the operator will succeed. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLCreateInstanceOperator` |
| |
| :param body: Body required by the Cloud SQL insert API, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/insert |
| #request-body |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :param project_id: Optional, Google Cloud Project ID. If set to None or missing, |
| the default project_id from the Google Cloud connection is used. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param validate_body: True if body should be validated, False otherwise. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #FADBDA |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLInstancePatchOperator(*, body, instance, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`CloudSQLBaseOperator` |
| |
| Updates settings of a Cloud SQL instance. |
| |
| Caution: This is a partial update, so only included values for the settings will be |
| updated. |
| |
| In the request body, supply the relevant portions of an instance resource, according |
| to the rules of patch semantics. |
| https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLInstancePatchOperator` |
| |
| :param body: Body required by the Cloud SQL patch API, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/patch#request-body |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :param project_id: Optional, Google Cloud Project ID. If set to None or missing, |
| the default project_id from the Google Cloud connection is used. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #FBDAC8 |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLDeleteInstanceOperator(*, instance, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`CloudSQLBaseOperator` |
| |
| Deletes a Cloud SQL instance. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLDeleteInstanceOperator` |
| |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :param project_id: Optional, Google Cloud Project ID. If set to None or missing, |
| the default project_id from the Google Cloud connection is used. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['project_id', 'instance', 'gcp_conn_id', 'api_version', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #FEECD2 |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLCreateInstanceDatabaseOperator(*, instance, body, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', validate_body = True, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`CloudSQLBaseOperator` |
| |
| Creates a new database inside a Cloud SQL instance. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLCreateInstanceDatabaseOperator` |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body |
| :param project_id: Optional, Google Cloud Project ID. If set to None or missing, |
| the default project_id from the Google Cloud connection is used. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param validate_body: Whether the body should be validated. Defaults to True. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #FFFCDB |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLPatchInstanceDatabaseOperator(*, instance, database, body, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', validate_body = True, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`CloudSQLBaseOperator` |
| |
| Updates a resource containing information about a database inside a Cloud SQL |
| instance using patch semantics. |
| See: https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLPatchInstanceDatabaseOperator` |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :param database: Name of the database to be updated in the instance. |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/patch#request-body |
| :param project_id: Optional, Google Cloud Project ID. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param validate_body: Whether the body should be validated. Defaults to True. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['project_id', 'instance', 'body', 'database', 'gcp_conn_id', 'api_version', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #ECF4D9 |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLDeleteInstanceDatabaseOperator(*, instance, database, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`CloudSQLBaseOperator` |
| |
| Deletes a database from a Cloud SQL instance. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLDeleteInstanceDatabaseOperator` |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :param database: Name of the database to be deleted in the instance. |
| :param project_id: Optional, Google Cloud Project ID. If set to None or missing, |
| the default project_id from the Google Cloud connection is used. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['project_id', 'instance', 'database', 'gcp_conn_id', 'api_version', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #D5EAD8 |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLExportInstanceOperator(*, instance, body, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', validate_body = True, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`CloudSQLBaseOperator` |
| |
| Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump |
| or CSV file. |
| |
| Note: This operator is idempotent. If executed multiple times with the same |
| export file URI, the export file in GCS will simply be overridden. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLExportInstanceOperator` |
| |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body |
| :param project_id: Optional, Google Cloud Project ID. If set to None or missing, |
| the default project_id from the Google Cloud connection is used. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param validate_body: Whether the body should be validated. Defaults to True. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #D4ECEA |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLImportInstanceOperator(*, instance, body, project_id = None, gcp_conn_id = 'google_cloud_default', api_version = 'v1beta4', validate_body = True, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`CloudSQLBaseOperator` |
| |
| Imports data into a Cloud SQL instance from a SQL dump or CSV file in Cloud Storage. |
| |
| CSV IMPORT: |
| |
| This operator is NOT idempotent for a CSV import. If the same file is imported |
| multiple times, the imported data will be duplicated in the database. |
| Moreover, if there are any unique constraints the duplicate import may result in an |
| error. |
| |
| SQL IMPORT: |
| |
| This operator is idempotent for a SQL import if it was also exported by Cloud SQL. |
| The exported SQL contains 'DROP TABLE IF EXISTS' statements for all tables |
| to be imported. |
| |
| If the import file was generated in a different way, idempotence is not guaranteed. |
| It has to be ensured on the SQL file level. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLImportInstanceOperator` |
| |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/import#request-body |
| :param project_id: Optional, Google Cloud Project ID. If set to None or missing, |
| the default project_id from the Google Cloud connection is used. |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud. |
| :param api_version: API version used (e.g. v1beta4). |
| :param validate_body: Whether the body should be validated. Defaults to True. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #D3EDFB |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: CloudSQLExecuteQueryOperator(*, sql, autocommit = False, parameters = None, gcp_conn_id = 'google_cloud_default', gcp_cloudsql_conn_id = 'google_cloud_sql_default', **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Performs DML or DDL query on an existing Cloud Sql instance. It optionally uses |
| cloud-sql-proxy to establish secure connection with the database. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSQLExecuteQueryOperator` |
| |
| :param sql: SQL query or list of queries to run (should be DML or DDL query - |
| this operator does not return any data from the database, |
| so it is useless to pass it DQL queries. Note that it is responsibility of the |
| author of the queries to make sure that the queries are idempotent. For example |
| you can use CREATE TABLE IF NOT EXISTS to create a table. |
| :param parameters: (optional) the parameters to render the SQL query with. |
| :param autocommit: if True, each command is automatically committed. |
| (default value: False) |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud for |
| cloud-sql-proxy authentication. |
| :param gcp_cloudsql_conn_id: The connection ID used to connect to Google Cloud SQL |
| its schema should be gcpcloudsql://. |
| See :class:`~airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook` for |
| details on how to define ``gcpcloudsql://`` connection. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['sql', 'gcp_cloudsql_conn_id', 'gcp_conn_id'] |
| |
| |
| |
| .. py:attribute:: template_ext |
| :annotation: :Sequence[str] = ['.sql'] |
| |
| |
| |
| .. py:attribute:: template_fields_renderers |
| |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #D3DEF1 |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
