| :py:mod:`airflow.providers.google.cloud.sensors.dataflow` |
| ========================================================= |
| |
| .. py:module:: airflow.providers.google.cloud.sensors.dataflow |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains a Google Cloud Dataflow sensor. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.sensors.dataflow.DataflowJobStatusSensor |
| airflow.providers.google.cloud.sensors.dataflow.DataflowJobMetricsSensor |
| airflow.providers.google.cloud.sensors.dataflow.DataflowJobMessagesSensor |
| airflow.providers.google.cloud.sensors.dataflow.DataflowJobAutoScalingEventsSensor |
| |
| |
| |
| |
| .. py:class:: DataflowJobStatusSensor(*, job_id, expected_statuses, project_id = None, location = DEFAULT_DATAFLOW_LOCATION, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Checks for the status of a job in Google Cloud Dataflow. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:DataflowJobStatusSensor` |
| |
| :param job_id: ID of the job to be checked. |
| :param expected_statuses: The expected state of the operation. |
| See: |
| https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#Job.JobState |
| :param project_id: Optional, the Google Cloud project ID in which to start a job. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| :param location: The location of the Dataflow job (for example europe-west1). See: |
| https://cloud.google.com/dataflow/docs/concepts/regional-endpoints |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. See: |
| https://developers.google.com/identity/protocols/oauth2/service-account#delegatingauthority |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['job_id'] |
| |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| |
| .. py:class:: DataflowJobMetricsSensor(*, job_id, callback, fail_on_terminal_state = True, project_id = None, location = DEFAULT_DATAFLOW_LOCATION, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Checks the metrics of a job in Google Cloud Dataflow. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:DataflowJobMetricsSensor` |
| |
| :param job_id: ID of the job to be checked. |
| :param callback: callback which is called with list of read job metrics |
| See: |
| https://cloud.google.com/dataflow/docs/reference/rest/v1b3/MetricUpdate |
| :param fail_on_terminal_state: If set to true sensor will raise Exception when |
| job is in terminal state |
| :param project_id: Optional, the Google Cloud project ID in which to start a job. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| :param location: The location of the Dataflow job (for example europe-west1). See: |
| https://cloud.google.com/dataflow/docs/concepts/regional-endpoints |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['job_id'] |
| |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| |
| .. py:class:: DataflowJobMessagesSensor(*, job_id, callback, fail_on_terminal_state = True, project_id = None, location = DEFAULT_DATAFLOW_LOCATION, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Checks for the job message in Google Cloud Dataflow. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:DataflowJobMessagesSensor` |
| |
| :param job_id: ID of the job to be checked. |
| :param callback: callback which is called with list of read job metrics |
| See: |
| https://cloud.google.com/dataflow/docs/reference/rest/v1b3/MetricUpdate |
| :param fail_on_terminal_state: If set to true sensor will raise Exception when |
| job is in terminal state |
| :param project_id: Optional, the Google Cloud project ID in which to start a job. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| :param location: Job location. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['job_id'] |
| |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| |
| .. py:class:: DataflowJobAutoScalingEventsSensor(*, job_id, callback, fail_on_terminal_state = True, project_id = None, location = DEFAULT_DATAFLOW_LOCATION, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Checks for the job autoscaling event in Google Cloud Dataflow. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:DataflowJobAutoScalingEventsSensor` |
| |
| :param job_id: ID of the job to be checked. |
| :param callback: callback which is called with list of read job metrics |
| See: |
| https://cloud.google.com/dataflow/docs/reference/rest/v1b3/MetricUpdate |
| :param fail_on_terminal_state: If set to true sensor will raise Exception when |
| job is in terminal state |
| :param project_id: Optional, the Google Cloud project ID in which to start a job. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| :param location: Job location. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['job_id'] |
| |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| |