| :mod:`airflow.contrib.operators.gcp_sql_operator` |
| ================================================= |
| |
| .. py:module:: airflow.contrib.operators.gcp_sql_operator |
| |
| |
| Module Contents |
| --------------- |
| |
| .. data:: SETTINGS |
| :annotation: = settings |
| |
| |
| |
| .. data:: SETTINGS_VERSION |
| :annotation: = settingsVersion |
| |
| |
| |
| .. data:: CLOUD_SQL_CREATE_VALIDATION |
| |
| |
| |
| |
| .. data:: CLOUD_SQL_EXPORT_VALIDATION |
| |
| |
| |
| |
| .. data:: CLOUD_SQL_IMPORT_VALIDATION |
| |
| |
| |
| |
| .. data:: CLOUD_SQL_DATABASE_CREATE_VALIDATION |
| |
| |
| |
| |
| .. data:: CLOUD_SQL_DATABASE_PATCH_VALIDATION |
| |
| |
| |
| |
| .. py:class:: CloudSqlBaseOperator(instance, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Abstract base operator for Google Cloud SQL operators to inherit from. |
| |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :type instance: str |
| :param project_id: Optional, Google Cloud Platform Project ID. f set to None or missing, |
| the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: _check_if_instance_exists(self, instance) |
| |
| |
| |
| |
| .. method:: _check_if_db_exists(self, db_name) |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. staticmethod:: _get_settings_version(instance) |
| |
| |
| |
| |
| .. py:class:: CloudSqlInstanceCreateOperator(body, instance, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs) |
| |
| Bases: :class:`airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator` |
| |
| Creates a new Cloud SQL instance. |
| If an instance with the same name exists, no action will be taken and |
| the operator will succeed. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlInstanceCreateOperator` |
| |
| :param body: Body required by the Cloud SQL insert API, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/insert |
| #request-body |
| :type body: dict |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :type instance: str |
| :param project_id: Optional, Google Cloud Platform Project ID. If set to None or missing, |
| the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| :param validate_body: True if body should be validated, False otherwise. |
| :type validate_body: bool |
| |
| .. attribute:: template_fields |
| :annotation: = ['project_id', 'instance', 'gcp_conn_id', 'api_version'] |
| |
| |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: _validate_body_fields(self) |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: CloudSqlInstancePatchOperator(body, instance, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', *args, **kwargs) |
| |
| Bases: :class:`airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator` |
| |
| Updates settings of a Cloud SQL instance. |
| |
| Caution: This is a partial update, so only included values for the settings will be |
| updated. |
| |
| In the request body, supply the relevant portions of an instance resource, according |
| to the rules of patch semantics. |
| https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlInstancePatchOperator` |
| |
| :param body: Body required by the Cloud SQL patch API, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/patch#request-body |
| :type body: dict |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :type instance: str |
| :param project_id: Optional, Google Cloud Platform Project ID. If set to None or missing, |
| the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| |
| .. attribute:: template_fields |
| :annotation: = ['project_id', 'instance', 'gcp_conn_id', 'api_version'] |
| |
| |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: CloudSqlInstanceDeleteOperator(instance, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', *args, **kwargs) |
| |
| Bases: :class:`airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator` |
| |
| Deletes a Cloud SQL instance. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlInstanceDeleteOperator` |
| |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :type instance: str |
| :param project_id: Optional, Google Cloud Platform Project ID. If set to None or missing, |
| the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| |
| .. attribute:: template_fields |
| :annotation: = ['project_id', 'instance', 'gcp_conn_id', 'api_version'] |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: CloudSqlInstanceDatabaseCreateOperator(instance, body, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs) |
| |
| Bases: :class:`airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator` |
| |
| Creates a new database inside a Cloud SQL instance. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlInstanceDatabaseCreateOperator` |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :type instance: str |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body |
| :type body: dict |
| :param project_id: Optional, Google Cloud Platform Project ID. If set to None or missing, |
| the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| :param validate_body: Whether the body should be validated. Defaults to True. |
| :type validate_body: bool |
| |
| .. attribute:: template_fields |
| :annotation: = ['project_id', 'instance', 'gcp_conn_id', 'api_version'] |
| |
| |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: _validate_body_fields(self) |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: CloudSqlInstanceDatabasePatchOperator(instance, database, body, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs) |
| |
| Bases: :class:`airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator` |
| |
| Updates a resource containing information about a database inside a Cloud SQL |
| instance using patch semantics. |
| See: https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlInstanceDatabasePatchOperator` |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :type instance: str |
| :param database: Name of the database to be updated in the instance. |
| :type database: str |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/patch#request-body |
| :type body: dict |
| :param project_id: Optional, Google Cloud Platform Project ID. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| :param validate_body: Whether the body should be validated. Defaults to True. |
| :type validate_body: bool |
| |
| .. attribute:: template_fields |
| :annotation: = ['project_id', 'instance', 'database', 'gcp_conn_id', 'api_version'] |
| |
| |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: _validate_body_fields(self) |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: CloudSqlInstanceDatabaseDeleteOperator(instance, database, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', *args, **kwargs) |
| |
| Bases: :class:`airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator` |
| |
| Deletes a database from a Cloud SQL instance. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlInstanceDatabaseDeleteOperator` |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :type instance: str |
| :param database: Name of the database to be deleted in the instance. |
| :type database: str |
| :param project_id: Optional, Google Cloud Platform Project ID. If set to None or missing, |
| the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| |
| .. attribute:: template_fields |
| :annotation: = ['project_id', 'instance', 'database', 'gcp_conn_id', 'api_version'] |
| |
| |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: CloudSqlInstanceExportOperator(instance, body, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs) |
| |
| Bases: :class:`airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator` |
| |
| Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump |
| or CSV file. |
| |
| Note: This operator is idempotent. If executed multiple times with the same |
| export file URI, the export file in GCS will simply be overridden. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlInstanceImportOperator` |
| |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :type instance: str |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body |
| :type body: dict |
| :param project_id: Optional, Google Cloud Platform Project ID. If set to None or missing, |
| the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| :param validate_body: Whether the body should be validated. Defaults to True. |
| :type validate_body: bool |
| |
| .. attribute:: template_fields |
| :annotation: = ['project_id', 'instance', 'gcp_conn_id', 'api_version'] |
| |
| |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: _validate_body_fields(self) |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: CloudSqlInstanceImportOperator(instance, body, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs) |
| |
| Bases: :class:`airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator` |
| |
| Imports data into a Cloud SQL instance from a SQL dump or CSV file in Cloud Storage. |
| |
| CSV IMPORT: |
| |
| This operator is NOT idempotent for a CSV import. If the same file is imported |
| multiple times, the imported data will be duplicated in the database. |
| Moreover, if there are any unique constraints the duplicate import may result in an |
| error. |
| |
| SQL IMPORT: |
| |
| This operator is idempotent for a SQL import if it was also exported by Cloud SQL. |
| The exported SQL contains 'DROP TABLE IF EXISTS' statements for all tables |
| to be imported. |
| |
| If the import file was generated in a different way, idempotence is not guaranteed. |
| It has to be ensured on the SQL file level. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlInstanceImportOperator` |
| |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :type instance: str |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body |
| :type body: dict |
| :param project_id: Optional, Google Cloud Platform Project ID. If set to None or missing, |
| the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform. |
| :type gcp_conn_id: str |
| :param api_version: API version used (e.g. v1beta4). |
| :type api_version: str |
| :param validate_body: Whether the body should be validated. Defaults to True. |
| :type validate_body: bool |
| |
| .. attribute:: template_fields |
| :annotation: = ['project_id', 'instance', 'gcp_conn_id', 'api_version', 'body'] |
| |
| |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: _validate_body_fields(self) |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: CloudSqlQueryOperator(sql, autocommit=False, parameters=None, gcp_conn_id='google_cloud_default', gcp_cloudsql_conn_id='google_cloud_sql_default', *args, **kwargs) |
| |
| Bases: :class:`airflow.models.BaseOperator` |
| |
| Performs DML or DDL query on an existing Cloud Sql instance. It optionally uses |
| cloud-sql-proxy to establish secure connection with the database. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:CloudSqlQueryOperator` |
| |
| :param sql: SQL query or list of queries to run (should be DML or DDL query - |
| this operator does not return any data from the database, |
| so it is useless to pass it DQL queries. Note that it is responsibility of the |
| author of the queries to make sure that the queries are idempotent. For example |
| you can use CREATE TABLE IF NOT EXISTS to create a table. |
| :type sql: str or list[str] |
| :param parameters: (optional) the parameters to render the SQL query with. |
| :type parameters: mapping or iterable |
| :param autocommit: if True, each command is automatically committed. |
| (default value: False) |
| :type autocommit: bool |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform for |
| cloud-sql-proxy authentication. |
| :type gcp_conn_id: str |
| :param gcp_cloudsql_conn_id: The connection ID used to connect to Google Cloud SQL |
| its schema should be gcpcloudsql://. |
| See :class:`~airflow.contrib.hooks.gcp_sql_hook.CloudSqlDatabaseHook` for |
| details on how to define gcpcloudsql:// connection. |
| :type gcp_cloudsql_conn_id: str |
| |
| .. attribute:: template_fields |
| :annotation: = ['sql', 'gcp_cloudsql_conn_id', 'gcp_conn_id'] |
| |
| |
| |
| .. attribute:: template_ext |
| :annotation: = ['.sql'] |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |