blob: 8c6591149ccf06e32a9042cb9182555055986835 [file] [log] [blame]
:py:mod:`airflow.providers.google.cloud.triggers.bigquery`
==========================================================
.. py:module:: airflow.providers.google.cloud.triggers.bigquery
Module Contents
---------------
Classes
~~~~~~~
.. autoapisummary::
airflow.providers.google.cloud.triggers.bigquery.BigQueryInsertJobTrigger
airflow.providers.google.cloud.triggers.bigquery.BigQueryCheckTrigger
airflow.providers.google.cloud.triggers.bigquery.BigQueryGetDataTrigger
airflow.providers.google.cloud.triggers.bigquery.BigQueryIntervalCheckTrigger
airflow.providers.google.cloud.triggers.bigquery.BigQueryValueCheckTrigger
airflow.providers.google.cloud.triggers.bigquery.BigQueryTableExistenceTrigger
.. py:class:: BigQueryInsertJobTrigger(conn_id, job_id, project_id, dataset_id = None, table_id = None, poll_interval = 4.0)
Bases: :py:obj:`airflow.triggers.base.BaseTrigger`
BigQueryInsertJobTrigger run on the trigger worker to perform insert operation
:param conn_id: Reference to google cloud connection id
:param job_id: The ID of the job. It will be suffixed with hash of job configuration
:param project_id: Google Cloud Project where the job is running
:param dataset_id: The dataset ID of the requested table. (templated)
:param table_id: The table ID of the requested table. (templated)
:param poll_interval: polling period in seconds to check for the status
.. py:method:: serialize()
Serializes BigQueryInsertJobTrigger arguments and classpath.
.. py:method:: run()
:async:
Gets current job execution status and yields a TriggerEvent
.. py:class:: BigQueryCheckTrigger(conn_id, job_id, project_id, dataset_id = None, table_id = None, poll_interval = 4.0)
Bases: :py:obj:`BigQueryInsertJobTrigger`
BigQueryCheckTrigger run on the trigger worker
.. py:method:: serialize()
Serializes BigQueryCheckTrigger arguments and classpath.
.. py:method:: run()
:async:
Gets current job execution status and yields a TriggerEvent
.. py:class:: BigQueryGetDataTrigger(conn_id, job_id, project_id, dataset_id = None, table_id = None, poll_interval = 4.0)
Bases: :py:obj:`BigQueryInsertJobTrigger`
BigQueryGetDataTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class
.. py:method:: serialize()
Serializes BigQueryInsertJobTrigger arguments and classpath.
.. py:method:: run()
:async:
Gets current job execution status and yields a TriggerEvent with response data
.. py:class:: BigQueryIntervalCheckTrigger(conn_id, first_job_id, second_job_id, project_id, table, metrics_thresholds, date_filter_column = 'ds', days_back = -7, ratio_formula = 'max_over_min', ignore_zero = True, dataset_id = None, table_id = None, poll_interval = 4.0)
Bases: :py:obj:`BigQueryInsertJobTrigger`
BigQueryIntervalCheckTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class
:param conn_id: Reference to google cloud connection id
:param first_job_id: The ID of the job 1 performed
:param second_job_id: The ID of the job 2 performed
:param project_id: Google Cloud Project where the job is running
:param dataset_id: The dataset ID of the requested table. (templated)
:param table: table name
:param metrics_thresholds: dictionary of ratios indexed by metrics
:param date_filter_column: column name
:param days_back: number of days between ds and the ds we want to check
against
:param ratio_formula: ration formula
:param ignore_zero: boolean value to consider zero or not
:param table_id: The table ID of the requested table. (templated)
:param poll_interval: polling period in seconds to check for the status
.. py:method:: serialize()
Serializes BigQueryCheckTrigger arguments and classpath.
.. py:method:: run()
:async:
Gets current job execution status and yields a TriggerEvent
.. py:class:: BigQueryValueCheckTrigger(conn_id, sql, pass_value, job_id, project_id, tolerance = None, dataset_id = None, table_id = None, poll_interval = 4.0)
Bases: :py:obj:`BigQueryInsertJobTrigger`
BigQueryValueCheckTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class
:param conn_id: Reference to google cloud connection id
:param sql: the sql to be executed
:param pass_value: pass value
:param job_id: The ID of the job
:param project_id: Google Cloud Project where the job is running
:param tolerance: certain metrics for tolerance
:param dataset_id: The dataset ID of the requested table. (templated)
:param table_id: The table ID of the requested table. (templated)
:param poll_interval: polling period in seconds to check for the status
.. py:method:: serialize()
Serializes BigQueryValueCheckTrigger arguments and classpath.
.. py:method:: run()
:async:
Gets current job execution status and yields a TriggerEvent
.. py:class:: BigQueryTableExistenceTrigger(project_id, dataset_id, table_id, gcp_conn_id, hook_params, poll_interval = 4.0)
Bases: :py:obj:`airflow.triggers.base.BaseTrigger`
Initialize the BigQuery Table Existence Trigger with needed parameters
:param project_id: Google Cloud Project where the job is running
:param dataset_id: The dataset ID of the requested table.
:param table_id: The table ID of the requested table.
:param gcp_conn_id: Reference to google cloud connection id
:param hook_params: params for hook
:param poll_interval: polling period in seconds to check for the status
.. py:method:: serialize()
Serializes BigQueryTableExistenceTrigger arguments and classpath.
.. py:method:: run()
:async:
Will run until the table exists in the Google Big Query.