| :mod:`airflow.contrib.hooks.gcp_sql_hook` |
| ========================================= |
| |
| .. py:module:: airflow.contrib.hooks.gcp_sql_hook |
| |
| |
| Module Contents |
| --------------- |
| |
| .. data:: UNIX_PATH_MAX |
| :annotation: = 108 |
| |
| |
| |
| .. data:: TIME_TO_SLEEP_IN_SECONDS |
| :annotation: = 1 |
| |
| |
| |
| .. py:class:: CloudSqlOperationStatus |
| |
| .. attribute:: PENDING |
| :annotation: = PENDING |
| |
| |
| |
| .. attribute:: RUNNING |
| :annotation: = RUNNING |
| |
| |
| |
| .. attribute:: DONE |
| :annotation: = DONE |
| |
| |
| |
| .. attribute:: UNKNOWN |
| :annotation: = UNKNOWN |
| |
| |
| |
| |
| .. py:class:: CloudSqlHook(api_version, gcp_conn_id='google_cloud_default', delegate_to=None) |
| |
| Bases: :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook` |
| |
| Hook for Google Cloud SQL APIs. |
| |
| All the methods in the hook where project_id is used must be called with |
| keyword arguments rather than positional. |
| |
| .. attribute:: _conn |
| |
| |
| |
| |
| |
| .. method:: get_conn(self) |
| |
| Retrieves connection to Cloud SQL. |
| |
| :return: Google Cloud SQL services object. |
| :rtype: dict |
| |
| |
| |
| |
| .. method:: get_instance(self, instance, project_id=None) |
| |
| Retrieves a resource containing information about a Cloud SQL instance. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :type instance: str |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: A Cloud SQL instance resource. |
| :rtype: dict |
| |
| |
| |
| |
| .. method:: create_instance(self, body, project_id=None) |
| |
| Creates a new Cloud SQL instance. |
| |
| :param body: Body required by the Cloud SQL insert API, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/insert#request-body. |
| :type body: dict |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: None |
| |
| |
| |
| |
| .. method:: patch_instance(self, body, instance, project_id=None) |
| |
| Updates settings of a Cloud SQL instance. |
| |
| Caution: This is not a partial update, so you must include values for |
| all the settings that you want to retain. |
| |
| :param body: Body required by the Cloud SQL patch API, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/patch#request-body. |
| :type body: dict |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :type instance: str |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: None |
| |
| |
| |
| |
| .. method:: delete_instance(self, instance, project_id=None) |
| |
| Deletes a Cloud SQL instance. |
| |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :param instance: Cloud SQL instance ID. This does not include the project ID. |
| :type instance: str |
| :return: None |
| |
| |
| |
| |
| .. method:: get_database(self, instance, database, project_id=None) |
| |
| Retrieves a database resource from a Cloud SQL instance. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :type instance: str |
| :param database: Name of the database in the instance. |
| :type database: str |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: A Cloud SQL database resource, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases#resource. |
| :rtype: dict |
| |
| |
| |
| |
| .. method:: create_database(self, instance, body, project_id=None) |
| |
| Creates a new database inside a Cloud SQL instance. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :type instance: str |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body. |
| :type body: dict |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: None |
| |
| |
| |
| |
| .. method:: patch_database(self, instance, database, body, project_id=None) |
| |
| Updates a database resource inside a Cloud SQL instance. |
| |
| This method supports patch semantics. |
| See https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :type instance: str |
| :param database: Name of the database to be updated in the instance. |
| :type database: str |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body. |
| :type body: dict |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: None |
| |
| |
| |
| |
| .. method:: delete_database(self, instance, database, project_id=None) |
| |
| Deletes a database from a Cloud SQL instance. |
| |
| :param instance: Database instance ID. This does not include the project ID. |
| :type instance: str |
| :param database: Name of the database to be deleted in the instance. |
| :type database: str |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: None |
| |
| |
| |
| |
| .. method:: export_instance(self, instance, body, project_id=None) |
| |
| Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump |
| or CSV file. |
| |
| :param instance: Database instance ID of the Cloud SQL instance. This does not include the |
| project ID. |
| :type instance: str |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body |
| :type body: dict |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: None |
| |
| |
| |
| |
| .. method:: import_instance(self, instance, body, project_id=None) |
| |
| Imports data into a Cloud SQL instance from a SQL dump or CSV file in |
| Cloud Storage. |
| |
| :param instance: Database instance ID. This does not include the |
| project ID. |
| :type instance: str |
| :param body: The request body, as described in |
| https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body |
| :type body: dict |
| :param project_id: Project ID of the project that contains the instance. If set |
| to None or missing, the default project_id from the GCP connection is used. |
| :type project_id: str |
| :return: None |
| |
| |
| |
| |
| .. method:: _wait_for_operation_to_complete(self, project_id, operation_name) |
| |
| Waits for the named operation to complete - checks status of the |
| asynchronous call. |
| |
| :param project_id: Project ID of the project that contains the instance. |
| :type project_id: str |
| :param operation_name: Name of the operation. |
| :type operation_name: str |
| :return: None |
| |
| |
| |
| |
| .. data:: CLOUD_SQL_PROXY_DOWNLOAD_URL |
| :annotation: = https://dl.google.com/cloudsql/cloud_sql_proxy.{}.{} |
| |
| |
| |
| .. data:: CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL |
| :annotation: = https://storage.googleapis.com/cloudsql-proxy/{}/cloud_sql_proxy.{}.{} |
| |
| |
| |
| .. data:: GCP_CREDENTIALS_KEY_PATH |
| :annotation: = extra__google_cloud_platform__key_path |
| |
| |
| |
| .. data:: GCP_CREDENTIALS_KEYFILE_DICT |
| :annotation: = extra__google_cloud_platform__keyfile_dict |
| |
| |
| |
| .. py:class:: CloudSqlProxyRunner(path_prefix, instance_specification, gcp_conn_id='google_cloud_default', project_id=None, sql_proxy_version=None, sql_proxy_binary_path=None) |
| |
| Bases: :class:`airflow.LoggingMixin` |
| |
| Downloads and runs cloud-sql-proxy as subprocess of the Python process. |
| |
| The cloud-sql-proxy needs to be downloaded and started before we can connect |
| to the Google Cloud SQL instance via database connection. It establishes |
| secure tunnel connection to the database. It authorizes using the |
| GCP credentials that are passed by the configuration. |
| |
| More details about the proxy can be found here: |
| https://cloud.google.com/sql/docs/mysql/sql-proxy |
| |
| |
| .. method:: _build_command_line_parameters(self) |
| |
| |
| |
| |
| .. staticmethod:: _is_os_64bit() |
| |
| |
| |
| |
| .. method:: _download_sql_proxy_if_needed(self) |
| |
| |
| |
| |
| .. method:: _get_credential_parameters(self, session) |
| |
| |
| |
| |
| .. method:: start_proxy(self) |
| |
| Starts Cloud SQL Proxy. |
| |
| You have to remember to stop the proxy if you started it! |
| |
| |
| |
| |
| .. method:: stop_proxy(self) |
| |
| Stops running proxy. |
| |
| You should stop the proxy after you stop using it. |
| |
| |
| |
| |
| .. method:: get_proxy_version(self) |
| |
| Returns version of the Cloud SQL Proxy. |
| |
| |
| |
| |
| .. method:: get_socket_path(self) |
| |
| Retrieves UNIX socket path used by Cloud SQL Proxy. |
| |
| :return: The dynamically generated path for the socket created by the proxy. |
| :rtype: str |
| |
| |
| |
| |
| .. data:: CONNECTION_URIS |
| |
| |
| |
| |
| .. data:: CLOUD_SQL_VALID_DATABASE_TYPES |
| :annotation: = ['postgres', 'mysql'] |
| |
| |
| |
| .. py:class:: CloudSqlDatabaseHook(gcp_cloudsql_conn_id='google_cloud_sql_default', gcp_conn_id='google_cloud_default', default_gcp_project_id=None) |
| |
| Bases: :class:`airflow.hooks.base_hook.BaseHook` |
| |
| Serves DB connection configuration for Google Cloud SQL (Connections |
| of *gcpcloudsql://* type). |
| |
| The hook is a "meta" one. It does not perform an actual connection. |
| It is there to retrieve all the parameters configured in gcpcloudsql:// connection, |
| start/stop Cloud SQL Proxy if needed, dynamically generate Postgres or MySQL |
| connection in the database and return an actual Postgres or MySQL hook. |
| The returned Postgres/MySQL hooks are using direct connection or Cloud SQL |
| Proxy socket/TCP as configured. |
| |
| Main parameters of the hook are retrieved from the standard URI components: |
| |
| * **user** - User name to authenticate to the database (from login of the URI). |
| * **password** - Password to authenticate to the database (from password of the URI). |
| * **public_ip** - IP to connect to for public connection (from host of the URI). |
| * **public_port** - Port to connect to for public connection (from port of the URI). |
| * **database** - Database to connect to (from schema of the URI). |
| |
| Remaining parameters are retrieved from the extras (URI query parameters): |
| |
| * **project_id** - Optional, Google Cloud Platform project where the Cloud SQL |
| instance exists. If missing, default project id passed is used. |
| * **instance** - Name of the instance of the Cloud SQL database instance. |
| * **location** - The location of the Cloud SQL instance (for example europe-west1). |
| * **database_type** - The type of the database instance (MySQL or Postgres). |
| * **use_proxy** - (default False) Whether SQL proxy should be used to connect to Cloud |
| SQL DB. |
| * **use_ssl** - (default False) Whether SSL should be used to connect to Cloud SQL DB. |
| You cannot use proxy and SSL together. |
| * **sql_proxy_use_tcp** - (default False) If set to true, TCP is used to connect via |
| proxy, otherwise UNIX sockets are used. |
| * **sql_proxy_binary_path** - Optional path to Cloud SQL Proxy binary. If the binary |
| is not specified or the binary is not present, it is automatically downloaded. |
| * **sql_proxy_version** - Specific version of the proxy to download (for example |
| v1.13). If not specified, the latest version is downloaded. |
| * **sslcert** - Path to client certificate to authenticate when SSL is used. |
| * **sslkey** - Path to client private key to authenticate when SSL is used. |
| * **sslrootcert** - Path to server's certificate to authenticate when SSL is used. |
| |
| :param gcp_cloudsql_conn_id: URL of the connection |
| :type gcp_cloudsql_conn_id: str |
| :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform for |
| cloud-sql-proxy authentication. |
| :type gcp_conn_id: str |
| :param default_gcp_project_id: Default project id used if project_id not specified |
| in the connection URL |
| :type default_gcp_project_id: str |
| |
| .. attribute:: _conn |
| |
| |
| |
| |
| |
| .. staticmethod:: _get_bool(val) |
| |
| |
| |
| |
| .. staticmethod:: _check_ssl_file(file_to_check, name) |
| |
| |
| |
| |
| .. method:: _validate_inputs(self) |
| |
| |
| |
| |
| .. method:: validate_ssl_certs(self) |
| |
| |
| |
| |
| .. method:: validate_socket_path_length(self) |
| |
| |
| |
| |
| .. staticmethod:: _generate_unique_path() |
| |
| |
| |
| |
| .. staticmethod:: _quote(value) |
| |
| |
| |
| |
| .. method:: _generate_connection_uri(self) |
| |
| |
| |
| |
| .. method:: _get_instance_socket_name(self) |
| |
| |
| |
| |
| .. method:: _get_sqlproxy_instance_specification(self) |
| |
| |
| |
| |
| .. method:: create_connection(self, session=None) |
| |
| Create connection in the Connection table, according to whether it uses |
| proxy, TCP, UNIX sockets, SSL. Connection ID will be randomly generated. |
| |
| :param session: Session of the SQL Alchemy ORM (automatically generated with |
| decorator). |
| |
| |
| |
| |
| .. method:: retrieve_connection(self, session=None) |
| |
| Retrieves the dynamically created connection from the Connection table. |
| |
| :param session: Session of the SQL Alchemy ORM (automatically generated with |
| decorator). |
| |
| |
| |
| |
| .. method:: delete_connection(self, session=None) |
| |
| Delete the dynamically created connection from the Connection table. |
| |
| :param session: Session of the SQL Alchemy ORM (automatically generated with |
| decorator). |
| |
| |
| |
| |
| .. method:: get_sqlproxy_runner(self) |
| |
| Retrieve Cloud SQL Proxy runner. It is used to manage the proxy |
| lifecycle per task. |
| |
| :return: The Cloud SQL Proxy runner. |
| :rtype: CloudSqlProxyRunner |
| |
| |
| |
| |
| .. method:: get_database_hook(self) |
| |
| Retrieve database hook. This is the actual Postgres or MySQL database hook |
| that uses proxy or connects directly to the Google Cloud SQL database. |
| |
| |
| |
| |
| .. method:: cleanup_database_hook(self) |
| |
| Clean up database hook after it was used. |
| |
| |
| |
| |
| .. method:: reserve_free_tcp_port(self) |
| |
| Reserve free TCP port to be used by Cloud SQL Proxy |
| |
| |
| |
| |
| .. method:: free_reserved_port(self) |
| |
| Free TCP port. Makes it immediately ready to be used by Cloud SQL Proxy. |
| |
| |
| |
| |