| :py:mod:`airflow.providers.google.cloud.hooks.cloud_sql` |
| ======================================================== |
| |
| .. py:module:: airflow.providers.google.cloud.hooks.cloud_sql |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains a Google Cloud SQL Hook. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.cloud_sql.CloudSqlOperationStatus |
| airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook |
| airflow.providers.google.cloud.hooks.cloud_sql.CloudSqlProxyRunner |
| airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook |
| |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.cloud_sql.UNIX_PATH_MAX |
| airflow.providers.google.cloud.hooks.cloud_sql.TIME_TO_SLEEP_IN_SECONDS |
| airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_PROXY_DOWNLOAD_URL |
| airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL |
| airflow.providers.google.cloud.hooks.cloud_sql.GCP_CREDENTIALS_KEY_PATH |
| airflow.providers.google.cloud.hooks.cloud_sql.GCP_CREDENTIALS_KEYFILE_DICT |
| airflow.providers.google.cloud.hooks.cloud_sql.CONNECTION_URIS |
| airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_VALID_DATABASE_TYPES |
| |
| |
| .. py:data:: UNIX_PATH_MAX |
| :annotation: = 108 |
| |
| |
| |
| .. py:data:: TIME_TO_SLEEP_IN_SECONDS |
| :annotation: = 20 |
| |
| |
| |
| .. py:class:: CloudSqlOperationStatus |
| |
| Helper class with operation statuses. |
| |
| .. py:attribute:: PENDING |
| :annotation: = PENDING |
| |
| |
| |
| .. py:attribute:: RUNNING |
| :annotation: = RUNNING |
| |
| |
| |
| .. py:attribute:: DONE |
| :annotation: = DONE |
| |
| |
| |
| .. py:attribute:: UNKNOWN |
| :annotation: = UNKNOWN |
| |
| |
| |
| |
| .. py:class:: CloudSQLHook(api_version, gcp_conn_id = default_conn_name, delegate_to = None, impersonation_chain = None) |
| |
| Bases: :py:obj:`airflow.providers.google.common.hooks.base_google.GoogleBaseHook` |
| |
| Hook for Google Cloud SQL APIs. |
| |
| All the methods in the hook where project_id is used must be called with |
| keyword arguments rather than positional. |
| |
| :param api_version: This is the version of the api. |
| :param gcp_conn_id: The Airflow connection used for GCP credentials. |
| :param delegate_to: This performs a task on one host with reference to other hosts. |
| :param impersonation_chain: This is the optional service account to impersonate using short term |
| credentials. |
| |
| .. py:attribute:: conn_name_attr |
| :annotation: = gcp_conn_id |
| |
| |
| |
| .. py:attribute:: default_conn_name |
| :annotation: = google_cloud_sql_default |
| |
| |
| |
| .. py:attribute:: conn_type |
| :annotation: = gcpcloudsql |
| |
| |
| |
| .. py:attribute:: hook_name |
| :annotation: = Google Cloud SQL |
| |
| |
| |
| .. py:method:: get_conn() |
| |
| Retrieves connection to Cloud SQL. |
| |
| :return: Google Cloud SQL services object. |
| :rtype: dict |
| |
| |
| .. py:method:: get_instance(instance, project_id) |
| |
| Retrieves a resource containing information about a Cloud SQL instance. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: A Cloud SQL instance resource. |
| :rtype: dict |
| |
| |
| .. py:method:: create_instance(body, project_id) |
| |
| Creates a new Cloud SQL instance. |
| |
| :param body: Body required by the Cloud SQL insert API, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/insert#request-body. |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: None |
| |
| |
| .. py:method:: patch_instance(body, instance, project_id) |
| |
| Updates settings of a Cloud SQL instance. |
| |
| Caution: This is not a partial update, so you must include values for |
| all the settings that you want to retain. |
| |
| :param body: Body required by the Cloud SQL patch API, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/patch#request-body. |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: None |
| |
| |
| .. py:method:: delete_instance(instance, project_id) |
| |
| Deletes a Cloud SQL instance. |
| |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :return: None |
| |
| |
| .. py:method:: get_database(instance, database, project_id) |
| |
| Retrieves a database resource from a Cloud SQL instance. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :param database: Name of the database in the instance. |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: A Cloud SQL database resource, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases#resource. |
| :rtype: dict |
| |
| |
| .. py:method:: create_database(instance, body, project_id) |
| |
| Creates a new database inside a Cloud SQL instance. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body. |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: None |
| |
| |
| .. py:method:: patch_database(instance, database, body, project_id) |
| |
| Updates a database resource inside a Cloud SQL instance. |
| |
| This method supports patch semantics. |
| See https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :param database: Name of the database to be updated in the instance. |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body. |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: None |
| |
| |
| .. py:method:: delete_database(instance, database, project_id) |
| |
| Deletes a database from a Cloud SQL instance. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :param database: Name of the database to be deleted in the instance. |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: None |
| |
| |
| .. py:method:: export_instance(instance, body, project_id) |
| |
| Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump |
| or CSV file. |
| |
| :param instance: Database instance ID of the Cloud SQL instance. This does not include the |
| project ID. |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: None |
| |
| |
| .. py:method:: import_instance(instance, body, project_id) |
| |
| Imports data into a Cloud SQL instance from a SQL dump or CSV file in |
| Cloud Storage. |
| |
| :param instance: Database instance ID. This does not include the |
| project ID. |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/import#request-body |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the Google Cloud connection is used. |
| :return: None |
| |
| |
| |
| .. py:data:: CLOUD_SQL_PROXY_DOWNLOAD_URL |
| :annotation: = https://dl.google.com/cloudsql/cloud_sql_proxy.{}.{} |
| |
| |
| |
| .. py:data:: CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL |
| :annotation: = https://storage.googleapis.com/cloudsql-proxy/{}/cloud_sql_proxy.{}.{} |
| |
| |
| |
| .. py:data:: GCP_CREDENTIALS_KEY_PATH |
| :annotation: = extra__google_cloud_platform__key_path |
| |
| |
| |
| .. py:data:: GCP_CREDENTIALS_KEYFILE_DICT |
| :annotation: = extra__google_cloud_platform__keyfile_dict |
| |
| |
| |
| .. py:class:: CloudSqlProxyRunner(path_prefix, instance_specification, gcp_conn_id = 'google_cloud_default', project_id = None, sql_proxy_version = None, sql_proxy_binary_path = None) |
| |
| Bases: :py:obj:`airflow.utils.log.logging_mixin.LoggingMixin` |
| |
| Downloads and runs cloud-sql-proxy as subprocess of the Python process. |
| |
| The cloud-sql-proxy needs to be downloaded and started before we can connect |
| to the Google Cloud SQL instance via database connection. It establishes |
| secure tunnel connection to the database. It authorizes using the |
| Google Cloud credentials that are passed by the configuration. |
| |
| More details about the proxy can be found here: |
| https://cloud.google.com/sql/docs/mysql/sql-proxy |
| |
| :param path_prefix: Unique path prefix where proxy will be downloaded and |
| directories created for unix sockets. |
| :param instance_specification: Specification of the instance to connect the |
| proxy to. It should be specified in the form that is described in |
| https://cloud.google.com/sql/docs/mysql/sql-proxy#multiple-instances in |
| -instances parameter (typically in the form of ``<project>:<region>:<instance>`` |
| for UNIX socket connections and in the form of |
| ``<project>:<region>:<instance>=tcp:<port>`` for TCP connections. |
| :param gcp_conn_id: Id of Google Cloud connection to use for |
| authentication |
| :param project_id: Optional id of the Google Cloud project to connect to - it overwrites |
| default project id taken from the Google Cloud connection. |
| :param sql_proxy_version: Specific version of SQL proxy to download |
| (for example 'v1.13'). By default latest version is downloaded. |
| :param sql_proxy_binary_path: If specified, then proxy will be |
| used from the path specified rather than dynamically generated. This means |
| that if the binary is not present in that path it will also be downloaded. |
| |
| .. py:method:: start_proxy() |
| |
| Starts Cloud SQL Proxy. |
| |
| You have to remember to stop the proxy if you started it! |
| |
| |
| .. py:method:: stop_proxy() |
| |
| Stops running proxy. |
| |
| You should stop the proxy after you stop using it. |
| |
| |
| .. py:method:: get_proxy_version() |
| |
| Returns version of the Cloud SQL Proxy. |
| |
| |
| .. py:method:: get_socket_path() |
| |
| Retrieves UNIX socket path used by Cloud SQL Proxy. |
| |
| :return: The dynamically generated path for the socket created by the proxy. |
| :rtype: str |
| |
| |
| |
| .. py:data:: CONNECTION_URIS |
| :annotation: :Dict[str, Dict[str, Dict[str, str]]] |
| |
| |
| |
| .. py:data:: CLOUD_SQL_VALID_DATABASE_TYPES |
| :annotation: = ['postgres', 'mysql'] |
| |
| |
| |
| .. py:class:: CloudSQLDatabaseHook(gcp_cloudsql_conn_id = 'google_cloud_sql_default', gcp_conn_id = 'google_cloud_default', default_gcp_project_id = None) |
| |
| Bases: :py:obj:`airflow.hooks.base.BaseHook` |
| |
| Serves DB connection configuration for Google Cloud SQL (Connections |
| of *gcpcloudsqldb://* type). |
| |
| The hook is a "meta" one. It does not perform an actual connection. |
| It is there to retrieve all the parameters configured in gcpcloudsql:// connection, |
| start/stop Cloud SQL Proxy if needed, dynamically generate Postgres or MySQL |
| connection in the database and return an actual Postgres or MySQL hook. |
| The returned Postgres/MySQL hooks are using direct connection or Cloud SQL |
| Proxy socket/TCP as configured. |
| |
| Main parameters of the hook are retrieved from the standard URI components: |
| |
| * **user** - User name to authenticate to the database (from login of the URI). |
| * **password** - Password to authenticate to the database (from password of the URI). |
| * **public_ip** - IP to connect to for public connection (from host of the URI). |
| * **public_port** - Port to connect to for public connection (from port of the URI). |
| * **database** - Database to connect to (from schema of the URI). |
| |
| Remaining parameters are retrieved from the extras (URI query parameters): |
| |
| * **project_id** - Optional, Google Cloud project where the Cloud SQL |
| instance exists. If missing, default project id passed is used. |
| * **instance** - Name of the instance of the Cloud SQL database instance. |
| * **location** - The location of the Cloud SQL instance (for example europe-west1). |
| * **database_type** - The type of the database instance (MySQL or Postgres). |
| * **use_proxy** - (default False) Whether SQL proxy should be used to connect to Cloud |
| SQL DB. |
| * **use_ssl** - (default False) Whether SSL should be used to connect to Cloud SQL DB. |
| You cannot use proxy and SSL together. |
| * **sql_proxy_use_tcp** - (default False) If set to true, TCP is used to connect via |
| proxy, otherwise UNIX sockets are used. |
| * **sql_proxy_binary_path** - Optional path to Cloud SQL Proxy binary. If the binary |
| is not specified or the binary is not present, it is automatically downloaded. |
| * **sql_proxy_version** - Specific version of the proxy to download (for example |
| v1.13). If not specified, the latest version is downloaded. |
| * **sslcert** - Path to client certificate to authenticate when SSL is used. |
| * **sslkey** - Path to client private key to authenticate when SSL is used. |
| * **sslrootcert** - Path to server's certificate to authenticate when SSL is used. |
| |
| :param gcp_cloudsql_conn_id: URL of the connection |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud for |
| cloud-sql-proxy authentication. |
| :param default_gcp_project_id: Default project id used if project_id not specified |
| in the connection URL |
| |
| .. py:attribute:: conn_name_attr |
| :annotation: = gcp_cloudsql_conn_id |
| |
| |
| |
| .. py:attribute:: default_conn_name |
| :annotation: = google_cloud_sqldb_default |
| |
| |
| |
| .. py:attribute:: conn_type |
| :annotation: = gcpcloudsqldb |
| |
| |
| |
| .. py:attribute:: hook_name |
| :annotation: = Google Cloud SQL Database |
| |
| |
| |
| .. py:method:: validate_ssl_certs() |
| |
| SSL certificates validator. |
| |
| :return: None |
| |
| |
| .. py:method:: validate_socket_path_length() |
| |
| Validates sockets path length. |
| |
| :return: None or rises AirflowException |
| |
| |
| .. py:method:: create_connection() |
| |
| Create Connection object, according to whether it uses proxy, TCP, UNIX sockets, SSL. |
| Connection ID will be randomly generated. |
| |
| |
| .. py:method:: get_sqlproxy_runner() |
| |
| Retrieve Cloud SQL Proxy runner. It is used to manage the proxy |
| lifecycle per task. |
| |
| :return: The Cloud SQL Proxy runner. |
| :rtype: CloudSqlProxyRunner |
| |
| |
| .. py:method:: get_database_hook(connection) |
| |
| Retrieve database hook. This is the actual Postgres or MySQL database hook |
| that uses proxy or connects directly to the Google Cloud SQL database. |
| |
| |
| .. py:method:: cleanup_database_hook() |
| |
| Clean up database hook after it was used. |
| |
| |
| .. py:method:: reserve_free_tcp_port() |
| |
| Reserve free TCP port to be used by Cloud SQL Proxy |
| |
| |
| .. py:method:: free_reserved_port() |
| |
| Free TCP port. Makes it immediately ready to be used by Cloud SQL Proxy. |
| |
| |
| |
