| :py:mod:`airflow.providers.google.cloud.hooks.cloud_storage_transfer_service` |
| ============================================================================= |
| |
| .. py:module:: airflow.providers.google.cloud.hooks.cloud_storage_transfer_service |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains a Google Storage Transfer Service Hook. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GcpTransferJobsStatus |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GcpTransferOperationStatus |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceHook |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.gen_job_name |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.log |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TIME_TO_SLEEP_IN_SECONDS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ACCESS_KEY_ID |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ALREADY_EXISTING_IN_SINK |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_ACCESS_KEY |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_S3_DATA_SOURCE |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.BODY |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.BUCKET_NAME |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.COUNTERS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.DAY |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.DESCRIPTION |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER_JOB_NAMES |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER_PROJECT_ID |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GCS_DATA_SINK |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GCS_DATA_SOURCE |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.HOURS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.HTTP_DATA_SOURCE |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.JOB_NAME |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.LIST_URL |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.METADATA |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.MINUTES |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.MONTH |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.NAME |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OBJECT_CONDITIONS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OPERATIONS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.PATH |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.PROJECT_ID |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE_END_DATE |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE_START_DATE |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SECONDS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SECRET_ACCESS_KEY |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.START_TIME_OF_DAY |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.STATUS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.STATUS1 |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOB |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOBS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOB_FIELD_MASK |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_OPERATIONS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_OPTIONS |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_SPEC |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.YEAR |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ALREADY_EXIST_CODE |
| airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.NEGATIVE_STATUSES |
| |
| |
| .. py:data:: log |
| |
| |
| |
| |
| .. py:data:: TIME_TO_SLEEP_IN_SECONDS |
| :annotation: = 10 |
| |
| |
| |
| .. py:class:: GcpTransferJobsStatus |
| |
| Class with Google Cloud Transfer jobs statuses. |
| |
| .. py:attribute:: ENABLED |
| :annotation: = ENABLED |
| |
| |
| |
| .. py:attribute:: DISABLED |
| :annotation: = DISABLED |
| |
| |
| |
| .. py:attribute:: DELETED |
| :annotation: = DELETED |
| |
| |
| |
| |
| .. py:class:: GcpTransferOperationStatus |
| |
| Class with Google Cloud Transfer operations statuses. |
| |
| .. py:attribute:: IN_PROGRESS |
| :annotation: = IN_PROGRESS |
| |
| |
| |
| .. py:attribute:: PAUSED |
| :annotation: = PAUSED |
| |
| |
| |
| .. py:attribute:: SUCCESS |
| :annotation: = SUCCESS |
| |
| |
| |
| .. py:attribute:: FAILED |
| :annotation: = FAILED |
| |
| |
| |
| .. py:attribute:: ABORTED |
| :annotation: = ABORTED |
| |
| |
| |
| |
| .. py:data:: ACCESS_KEY_ID |
| :annotation: = accessKeyId |
| |
| |
| |
| .. py:data:: ALREADY_EXISTING_IN_SINK |
| :annotation: = overwriteObjectsAlreadyExistingInSink |
| |
| |
| |
| .. py:data:: AWS_ACCESS_KEY |
| :annotation: = awsAccessKey |
| |
| |
| |
| .. py:data:: AWS_S3_DATA_SOURCE |
| :annotation: = awsS3DataSource |
| |
| |
| |
| .. py:data:: BODY |
| :annotation: = body |
| |
| |
| |
| .. py:data:: BUCKET_NAME |
| :annotation: = bucketName |
| |
| |
| |
| .. py:data:: COUNTERS |
| :annotation: = counters |
| |
| |
| |
| .. py:data:: DAY |
| :annotation: = day |
| |
| |
| |
| .. py:data:: DESCRIPTION |
| :annotation: = description |
| |
| |
| |
| .. py:data:: FILTER |
| :annotation: = filter |
| |
| |
| |
| .. py:data:: FILTER_JOB_NAMES |
| :annotation: = job_names |
| |
| |
| |
| .. py:data:: FILTER_PROJECT_ID |
| :annotation: = project_id |
| |
| |
| |
| .. py:data:: GCS_DATA_SINK |
| :annotation: = gcsDataSink |
| |
| |
| |
| .. py:data:: GCS_DATA_SOURCE |
| :annotation: = gcsDataSource |
| |
| |
| |
| .. py:data:: HOURS |
| :annotation: = hours |
| |
| |
| |
| .. py:data:: HTTP_DATA_SOURCE |
| :annotation: = httpDataSource |
| |
| |
| |
| .. py:data:: JOB_NAME |
| :annotation: = name |
| |
| |
| |
| .. py:data:: LIST_URL |
| :annotation: = list_url |
| |
| |
| |
| .. py:data:: METADATA |
| :annotation: = metadata |
| |
| |
| |
| .. py:data:: MINUTES |
| :annotation: = minutes |
| |
| |
| |
| .. py:data:: MONTH |
| :annotation: = month |
| |
| |
| |
| .. py:data:: NAME |
| :annotation: = name |
| |
| |
| |
| .. py:data:: OBJECT_CONDITIONS |
| :annotation: = object_conditions |
| |
| |
| |
| .. py:data:: OPERATIONS |
| :annotation: = operations |
| |
| |
| |
| .. py:data:: PATH |
| :annotation: = path |
| |
| |
| |
| .. py:data:: PROJECT_ID |
| :annotation: = projectId |
| |
| |
| |
| .. py:data:: SCHEDULE |
| :annotation: = schedule |
| |
| |
| |
| .. py:data:: SCHEDULE_END_DATE |
| :annotation: = scheduleEndDate |
| |
| |
| |
| .. py:data:: SCHEDULE_START_DATE |
| :annotation: = scheduleStartDate |
| |
| |
| |
| .. py:data:: SECONDS |
| :annotation: = seconds |
| |
| |
| |
| .. py:data:: SECRET_ACCESS_KEY |
| :annotation: = secretAccessKey |
| |
| |
| |
| .. py:data:: START_TIME_OF_DAY |
| :annotation: = startTimeOfDay |
| |
| |
| |
| .. py:data:: STATUS |
| :annotation: = status |
| |
| |
| |
| .. py:data:: STATUS1 |
| :annotation: = status |
| |
| |
| |
| .. py:data:: TRANSFER_JOB |
| :annotation: = transfer_job |
| |
| |
| |
| .. py:data:: TRANSFER_JOBS |
| :annotation: = transferJobs |
| |
| |
| |
| .. py:data:: TRANSFER_JOB_FIELD_MASK |
| :annotation: = update_transfer_job_field_mask |
| |
| |
| |
| .. py:data:: TRANSFER_OPERATIONS |
| :annotation: = transferOperations |
| |
| |
| |
| .. py:data:: TRANSFER_OPTIONS |
| :annotation: = transfer_options |
| |
| |
| |
| .. py:data:: TRANSFER_SPEC |
| :annotation: = transferSpec |
| |
| |
| |
| .. py:data:: YEAR |
| :annotation: = year |
| |
| |
| |
| .. py:data:: ALREADY_EXIST_CODE |
| :annotation: = 409 |
| |
| |
| |
| .. py:data:: NEGATIVE_STATUSES |
| |
| |
| |
| |
| .. py:function:: gen_job_name(job_name) |
| |
| Adds unique suffix to job name. If suffix already exists, updates it. |
| Suffix — current timestamp |
| |
| :param job_name: |
| :return: job_name with suffix |
| |
| |
| .. py:class:: CloudDataTransferServiceHook(api_version = 'v1', gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None) |
| |
| Bases: :py:obj:`airflow.providers.google.common.hooks.base_google.GoogleBaseHook` |
| |
| Hook for Google Storage Transfer Service. |
| |
| All the methods in the hook where project_id is used must be called with |
| keyword arguments rather than positional. |
| |
| .. py:method:: get_conn() |
| |
| Retrieves connection to Google Storage Transfer service. |
| |
| :return: Google Storage Transfer service object |
| :rtype: dict |
| |
| |
| .. py:method:: create_transfer_job(body) |
| |
| Creates a transfer job that runs periodically. |
| |
| :param body: (Required) A request body, as described in |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body |
| :return: transfer job. |
| See: |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob |
| :rtype: dict |
| |
| |
| .. py:method:: get_transfer_job(job_name, project_id) |
| |
| Gets the latest state of a long-running operation in Google Storage |
| Transfer Service. |
| |
| :param job_name: (Required) Name of the job to be fetched |
| :param project_id: (Optional) the ID of the project that owns the Transfer |
| Job. If set to None or missing, the default project_id from the Google Cloud |
| connection is used. |
| :return: Transfer Job |
| :rtype: dict |
| |
| |
| .. py:method:: list_transfer_job(request_filter = None, **kwargs) |
| |
| Lists long-running operations in Google Storage Transfer |
| Service that match the specified filter. |
| |
| :param request_filter: (Required) A request filter, as described in |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter |
| :return: List of Transfer Jobs |
| :rtype: list[dict] |
| |
| |
| .. py:method:: enable_transfer_job(job_name, project_id) |
| |
| New transfers will be performed based on the schedule. |
| |
| :param job_name: (Required) Name of the job to be updated |
| :param project_id: (Optional) the ID of the project that owns the Transfer |
| Job. If set to None or missing, the default project_id from the Google Cloud |
| connection is used. |
| :return: If successful, TransferJob. |
| :rtype: dict |
| |
| |
| .. py:method:: update_transfer_job(job_name, body) |
| |
| Updates a transfer job that runs periodically. |
| |
| :param job_name: (Required) Name of the job to be updated |
| :param body: A request body, as described in |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body |
| :return: If successful, TransferJob. |
| :rtype: dict |
| |
| |
| .. py:method:: delete_transfer_job(job_name, project_id) |
| |
| Deletes a transfer job. This is a soft delete. After a transfer job is |
| deleted, the job and all the transfer executions are subject to garbage |
| collection. Transfer jobs become eligible for garbage collection |
| 30 days after soft delete. |
| |
| :param job_name: (Required) Name of the job to be deleted |
| :param project_id: (Optional) the ID of the project that owns the Transfer |
| Job. If set to None or missing, the default project_id from the Google Cloud |
| connection is used. |
| :rtype: None |
| |
| |
| .. py:method:: cancel_transfer_operation(operation_name) |
| |
| Cancels an transfer operation in Google Storage Transfer Service. |
| |
| :param operation_name: Name of the transfer operation. |
| :rtype: None |
| |
| |
| .. py:method:: get_transfer_operation(operation_name) |
| |
| Gets an transfer operation in Google Storage Transfer Service. |
| |
| :param operation_name: (Required) Name of the transfer operation. |
| :return: transfer operation |
| See: |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/Operation |
| :rtype: dict |
| |
| |
| .. py:method:: list_transfer_operations(request_filter = None, **kwargs) |
| |
| Gets an transfer operation in Google Storage Transfer Service. |
| |
| :param request_filter: (Required) A request filter, as described in |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter |
| With one additional improvement: |
| |
| * project_id is optional if you have a project id defined |
| in the connection |
| See: :doc:`/connections/gcp` |
| |
| :return: transfer operation |
| :rtype: list[dict] |
| |
| |
| .. py:method:: pause_transfer_operation(operation_name) |
| |
| Pauses an transfer operation in Google Storage Transfer Service. |
| |
| :param operation_name: (Required) Name of the transfer operation. |
| :rtype: None |
| |
| |
| .. py:method:: resume_transfer_operation(operation_name) |
| |
| Resumes an transfer operation in Google Storage Transfer Service. |
| |
| :param operation_name: (Required) Name of the transfer operation. |
| :rtype: None |
| |
| |
| .. py:method:: wait_for_transfer_job(job, expected_statuses = None, timeout = None) |
| |
| Waits until the job reaches the expected state. |
| |
| :param job: Transfer job |
| See: |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob |
| :param expected_statuses: State that is expected |
| See: |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status |
| :param timeout: Time in which the operation must end in seconds. If not specified, defaults to 60 |
| seconds. |
| :rtype: None |
| |
| |
| .. py:method:: operations_contain_expected_statuses(operations, expected_statuses) |
| :staticmethod: |
| |
| Checks whether the operation list has an operation with the |
| expected status, then returns true |
| If it encounters operations in FAILED or ABORTED state |
| throw :class:`airflow.exceptions.AirflowException`. |
| |
| :param operations: (Required) List of transfer operations to check. |
| :param expected_statuses: (Required) status that is expected |
| See: |
| https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status |
| :return: If there is an operation with the expected state |
| in the operation list, returns true, |
| :raises: airflow.exceptions.AirflowException If it encounters operations |
| with a state in the list, |
| :rtype: bool |
| |
| |
| |
