| :py:mod:`airflow.providers.google.cloud.hooks.datafusion` |
| ========================================================= |
| |
| .. py:module:: airflow.providers.google.cloud.hooks.datafusion |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains Google DataFusion hook. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.datafusion.PipelineStates |
| airflow.providers.google.cloud.hooks.datafusion.DataFusionHook |
| |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.datafusion.Operation |
| airflow.providers.google.cloud.hooks.datafusion.FAILURE_STATES |
| airflow.providers.google.cloud.hooks.datafusion.SUCCESS_STATES |
| |
| |
| .. py:data:: Operation |
| |
| |
| |
| |
| .. py:class:: PipelineStates |
| |
| Data Fusion pipeline states |
| |
| .. py:attribute:: PENDING |
| :annotation: = PENDING |
| |
| |
| |
| .. py:attribute:: STARTING |
| :annotation: = STARTING |
| |
| |
| |
| .. py:attribute:: RUNNING |
| :annotation: = RUNNING |
| |
| |
| |
| .. py:attribute:: SUSPENDED |
| :annotation: = SUSPENDED |
| |
| |
| |
| .. py:attribute:: RESUMING |
| :annotation: = RESUMING |
| |
| |
| |
| .. py:attribute:: COMPLETED |
| :annotation: = COMPLETED |
| |
| |
| |
| .. py:attribute:: FAILED |
| :annotation: = FAILED |
| |
| |
| |
| .. py:attribute:: KILLED |
| :annotation: = KILLED |
| |
| |
| |
| .. py:attribute:: REJECTED |
| :annotation: = REJECTED |
| |
| |
| |
| |
| .. py:data:: FAILURE_STATES |
| |
| |
| |
| |
| .. py:data:: SUCCESS_STATES |
| |
| |
| |
| |
| .. py:class:: DataFusionHook(api_version = 'v1beta1', gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None) |
| |
| Bases: :py:obj:`airflow.providers.google.common.hooks.base_google.GoogleBaseHook` |
| |
| Hook for Google DataFusion. |
| |
| .. py:method:: wait_for_operation(self, operation) |
| |
| Waits for long-lasting operation to complete. |
| |
| |
| .. py:method:: wait_for_pipeline_state(self, pipeline_name, pipeline_id, instance_url, namespace = 'default', success_states = None, failure_states = None, timeout = 5 * 60) |
| |
| Polls pipeline state and raises an exception if the state is one of |
| `failure_states` or the operation timed_out. |
| |
| |
| .. py:method:: get_conn(self) |
| |
| Retrieves connection to DataFusion. |
| |
| |
| .. py:method:: restart_instance(self, instance_name, location, project_id) |
| |
| Restart a single Data Fusion instance. |
| At the end of an operation instance is fully restarted. |
| |
| :param instance_name: The name of the instance to restart. |
| :param location: The Cloud Data Fusion location in which to handle the request. |
| :param project_id: The ID of the Google Cloud project that the instance belongs to. |
| |
| |
| .. py:method:: delete_instance(self, instance_name, location, project_id) |
| |
| Deletes a single Date Fusion instance. |
| |
| :param instance_name: The name of the instance to delete. |
| :param location: The Cloud Data Fusion location in which to handle the request. |
| :param project_id: The ID of the Google Cloud project that the instance belongs to. |
| |
| |
| .. py:method:: create_instance(self, instance_name, instance, location, project_id = PROVIDE_PROJECT_ID) |
| |
| Creates a new Data Fusion instance in the specified project and location. |
| |
| :param instance_name: The name of the instance to create. |
| :param instance: An instance of Instance. |
| https://cloud.google.com/data-fusion/docs/reference/rest/v1beta1/projects.locations.instances#Instance |
| :param location: The Cloud Data Fusion location in which to handle the request. |
| :param project_id: The ID of the Google Cloud project that the instance belongs to. |
| |
| |
| .. py:method:: get_instance(self, instance_name, location, project_id) |
| |
| Gets details of a single Data Fusion instance. |
| |
| :param instance_name: The name of the instance. |
| :param location: The Cloud Data Fusion location in which to handle the request. |
| :param project_id: The ID of the Google Cloud project that the instance belongs to. |
| |
| |
| .. py:method:: patch_instance(self, instance_name, instance, update_mask, location, project_id = PROVIDE_PROJECT_ID) |
| |
| Updates a single Data Fusion instance. |
| |
| :param instance_name: The name of the instance to create. |
| :param instance: An instance of Instance. |
| https://cloud.google.com/data-fusion/docs/reference/rest/v1beta1/projects.locations.instances#Instance |
| :param update_mask: Field mask is used to specify the fields that the update will overwrite |
| in an instance resource. The fields specified in the updateMask are relative to the resource, |
| not the full request. A field will be overwritten if it is in the mask. If the user does not |
| provide a mask, all the supported fields (labels and options currently) will be overwritten. |
| A comma-separated list of fully qualified names of fields. Example: "user.displayName,photo". |
| https://developers.google.com/protocol-buffers/docs/reference/google.protobuf?_ga=2.205612571.-968688242.1573564810#google.protobuf.FieldMask |
| :param location: The Cloud Data Fusion location in which to handle the request. |
| :param project_id: The ID of the Google Cloud project that the instance belongs to. |
| |
| |
| .. py:method:: create_pipeline(self, pipeline_name, pipeline, instance_url, namespace = 'default') |
| |
| Creates a Cloud Data Fusion pipeline. |
| |
| :param pipeline_name: Your pipeline name. |
| :param pipeline: The pipeline definition. For more information check: |
| https://docs.cdap.io/cdap/current/en/developer-manual/pipelines/developing-pipelines.html#pipeline-configuration-file-format |
| :param instance_url: Endpoint on which the REST APIs is accessible for the instance. |
| :param namespace: if your pipeline belongs to a Basic edition instance, the namespace ID |
| is always default. If your pipeline belongs to an Enterprise edition instance, you |
| can create a namespace. |
| |
| |
| .. py:method:: delete_pipeline(self, pipeline_name, instance_url, version_id = None, namespace = 'default') |
| |
| Deletes a Cloud Data Fusion pipeline. |
| |
| :param pipeline_name: Your pipeline name. |
| :param version_id: Version of pipeline to delete |
| :param instance_url: Endpoint on which the REST APIs is accessible for the instance. |
| :param namespace: f your pipeline belongs to a Basic edition instance, the namespace ID |
| is always default. If your pipeline belongs to an Enterprise edition instance, you |
| can create a namespace. |
| |
| |
| .. py:method:: list_pipelines(self, instance_url, artifact_name = None, artifact_version = None, namespace = 'default') |
| |
| Lists Cloud Data Fusion pipelines. |
| |
| :param artifact_version: Artifact version to filter instances |
| :param artifact_name: Artifact name to filter instances |
| :param instance_url: Endpoint on which the REST APIs is accessible for the instance. |
| :param namespace: f your pipeline belongs to a Basic edition instance, the namespace ID |
| is always default. If your pipeline belongs to an Enterprise edition instance, you |
| can create a namespace. |
| |
| |
| .. py:method:: get_pipeline_workflow(self, pipeline_name, instance_url, pipeline_id, namespace = 'default') |
| |
| |
| .. py:method:: start_pipeline(self, pipeline_name, instance_url, namespace = 'default', runtime_args = None) |
| |
| Starts a Cloud Data Fusion pipeline. Works for both batch and stream pipelines. |
| |
| :param pipeline_name: Your pipeline name. |
| :param instance_url: Endpoint on which the REST APIs is accessible for the instance. |
| :param runtime_args: Optional runtime JSON args to be passed to the pipeline |
| :param namespace: f your pipeline belongs to a Basic edition instance, the namespace ID |
| is always default. If your pipeline belongs to an Enterprise edition instance, you |
| can create a namespace. |
| |
| |
| .. py:method:: stop_pipeline(self, pipeline_name, instance_url, namespace = 'default') |
| |
| Stops a Cloud Data Fusion pipeline. Works for both batch and stream pipelines. |
| |
| :param pipeline_name: Your pipeline name. |
| :param instance_url: Endpoint on which the REST APIs is accessible for the instance. |
| :param namespace: f your pipeline belongs to a Basic edition instance, the namespace ID |
| is always default. If your pipeline belongs to an Enterprise edition instance, you |
| can create a namespace. |
| |
| |
| |
