:py:mod:`airflow.providers.google.cloud.hooks.bigquery`
=======================================================

.. py:module:: airflow.providers.google.cloud.hooks.bigquery

.. autoapi-nested-parse::

   This module contains a BigQuery Hook, as well as a very basic PEP 249
   implementation for BigQuery.


Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

   airflow.providers.google.cloud.hooks.bigquery.BigQueryHook
   airflow.providers.google.cloud.hooks.bigquery.BigQueryConnection
   airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor
   airflow.providers.google.cloud.hooks.bigquery.BigQueryCursor


Attributes
~~~~~~~~~~

.. autoapisummary::

   airflow.providers.google.cloud.hooks.bigquery.log
   airflow.providers.google.cloud.hooks.bigquery.BigQueryJob


.. py:data:: log
   

.. py:data:: BigQueryJob
   

.. py:class:: BigQueryHook(gcp_conn_id = GoogleBaseHook.default_conn_name, delegate_to = None, use_legacy_sql = True, location = None, api_resource_configs = None, impersonation_chain = None, labels = None)

   Bases: :py:obj:`airflow.providers.google.common.hooks.base_google.GoogleBaseHook`, :py:obj:`airflow.hooks.dbapi.DbApiHook`

   Interact with BigQuery. This hook uses the Google Cloud connection.

   :param gcp_conn_id: The Airflow connection used for GCP credentials.
   :param delegate_to: This performs a task on one host with reference to other hosts.
   :param use_legacy_sql: This specifies whether to use legacy SQL dialect.
   :param location: The location of the BigQuery resource.
   :param api_resource_configs: This contains params configuration applied for Google BigQuery jobs.
   :param impersonation_chain: This is the optional service account to impersonate using short term
       credentials.
   :param labels: The BigQuery resource label.

   .. py:attribute:: conn_name_attr
      :annotation: = gcp_conn_id

      
   .. py:attribute:: default_conn_name
      :annotation: = google_cloud_bigquery_default

      
   .. py:attribute:: conn_type
      :annotation: = gcpbigquery

      
   .. py:attribute:: hook_name
      :annotation: = Google Bigquery

      
   .. py:method:: get_conn(self)

      Returns a BigQuery PEP 249 connection object.


   .. py:method:: get_service(self)

      Returns a BigQuery service object.


   .. py:method:: get_client(self, project_id = None, location = None)

      Returns authenticated BigQuery Client.

      :param project_id: Project ID for the project which the client acts on behalf of.
      :param location: Default location for jobs / datasets / tables.
      :return:


   .. py:method:: get_uri(self)

      Override DbApiHook get_uri method for get_sqlalchemy_engine()


   .. py:method:: get_sqlalchemy_engine(self, engine_kwargs=None)

      Get an sqlalchemy_engine object.

      :param engine_kwargs: Kwargs used in :func:`~sqlalchemy.create_engine`.
      :return: the created engine.


   .. py:method:: get_records(self, sql, parameters=None)

      Executes the sql and returns a set of records.

      :param sql: the sql statement to be executed (str) or a list of
          sql statements to execute
      :param parameters: The parameters to render the SQL query with.


   .. py:method:: insert_rows(self, table, rows, target_fields = None, commit_every = 1000, replace = False, **kwargs)
      :abstractmethod:

      Insertion is currently unsupported. Theoretically, you could use
      BigQuery's streaming API to insert rows into a table, but this hasn't
      been implemented.


   .. py:method:: get_pandas_df(self, sql, parameters = None, dialect = None, **kwargs)

      Returns a Pandas DataFrame for the results produced by a BigQuery
      query. The DbApiHook method must be overridden because Pandas
      doesn't support PEP 249 connections, except for SQLite. See:

      https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447
      https://github.com/pydata/pandas/issues/6900

      :param sql: The BigQuery SQL to execute.
      :param parameters: The parameters to render the SQL query with (not
          used, leave to override superclass method)
      :param dialect: Dialect of BigQuery SQL – legacy SQL or standard SQL
          defaults to use `self.use_legacy_sql` if not specified
      :param kwargs: (optional) passed into pandas_gbq.read_gbq method


   .. py:method:: table_exists(self, dataset_id, table_id, project_id)

      Checks for the existence of a table in Google BigQuery.

      :param project_id: The Google cloud project in which to look for the
          table. The connection supplied to the hook must provide access to
          the specified project.
      :param dataset_id: The name of the dataset in which to look for the
          table.
      :param table_id: The name of the table to check the existence of.


   .. py:method:: table_partition_exists(self, dataset_id, table_id, partition_id, project_id)

      Checks for the existence of a partition in a table in Google BigQuery.

      :param project_id: The Google cloud project in which to look for the
          table. The connection supplied to the hook must provide access to
          the specified project.
      :param dataset_id: The name of the dataset in which to look for the
          table.
      :param table_id: The name of the table to check the existence of.
      :param partition_id: The name of the partition to check the existence of.


   .. py:method:: create_empty_table(self, project_id = None, dataset_id = None, table_id = None, table_resource = None, schema_fields = None, time_partitioning = None, cluster_fields = None, labels = None, view = None, materialized_view = None, encryption_configuration = None, retry = DEFAULT_RETRY, location = None, exists_ok = True)

      Creates a new, empty table in the dataset.
      To create a view, which is defined by a SQL query, parse a dictionary to 'view' kwarg

      :param project_id: The project to create the table into.
      :param dataset_id: The dataset to create the table into.
      :param table_id: The Name of the table to be created.
      :param table_resource: Table resource as described in documentation:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#Table
          If provided all other parameters are ignored.
      :param schema_fields: If set, the schema field list as defined here:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema
      :param labels: a dictionary containing labels for the table, passed to BigQuery
      :param retry: Optional. How to retry the RPC.

      **Example**: ::

          schema_fields=[{"name": "emp_name", "type": "STRING", "mode": "REQUIRED"},
                         {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}]

      :param time_partitioning: configure optional time partitioning fields i.e.
          partition by field, type and expiration as per API specifications.

          .. seealso::
              https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#timePartitioning
      :param cluster_fields: [Optional] The fields used for clustering.
          BigQuery supports clustering for both partitioned and
          non-partitioned tables.
          https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#clustering.fields
      :param view: [Optional] A dictionary containing definition for the view.
          If set, it will create a view instead of a table:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#ViewDefinition

      **Example**: ::

          view = {
              "query": "SELECT * FROM `test-project-id.test_dataset_id.test_table_prefix*` LIMIT 1000",
              "useLegacySql": False
          }

      :param materialized_view: [Optional] The materialized view definition.
      :param encryption_configuration: [Optional] Custom encryption configuration (e.g., Cloud KMS keys).
          **Example**: ::

              encryption_configuration = {
                  "kmsKeyName": "projects/testp/locations/us/keyRings/test-kr/cryptoKeys/test-key"
              }
      :param num_retries: Maximum number of retries in case of connection problems.
      :param location: (Optional) The geographic location where the table should reside.
      :param exists_ok: If ``True``, ignore "already exists" errors when creating the table.
      :return: Created table


   .. py:method:: create_empty_dataset(self, dataset_id = None, project_id = None, location = None, dataset_reference = None, exists_ok = True)

      Create a new empty dataset:
      https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert

      :param project_id: The name of the project where we want to create
          an empty a dataset. Don't need to provide, if projectId in dataset_reference.
      :param dataset_id: The id of dataset. Don't need to provide, if datasetId in dataset_reference.
      :param location: (Optional) The geographic location where the dataset should reside.
          There is no default value but the dataset will be created in US if nothing is provided.
      :param dataset_reference: Dataset reference that could be provided with request body. More info:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
      :param exists_ok: If ``True``, ignore "already exists" errors when creating the dataset.


   .. py:method:: get_dataset_tables(self, dataset_id, project_id = None, max_results = None, retry = DEFAULT_RETRY)

      Get the list of tables for a given dataset.

      For more information, see:
      https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/list

      :param dataset_id: the dataset ID of the requested dataset.
      :param project_id: (Optional) the project of the requested dataset. If None,
          self.project_id will be used.
      :param max_results: (Optional) the maximum number of tables to return.
      :param retry: How to retry the RPC.
      :return: List of tables associated with the dataset.


   .. py:method:: delete_dataset(self, dataset_id, project_id = None, delete_contents = False, retry = DEFAULT_RETRY)

      Delete a dataset of Big query in your project.

      :param project_id: The name of the project where we have the dataset.
      :param dataset_id: The dataset to be delete.
      :param delete_contents: If True, delete all the tables in the dataset.
          If False and the dataset contains tables, the request will fail.
      :param retry: How to retry the RPC.


   .. py:method:: create_external_table(self, external_project_dataset_table, schema_fields, source_uris, source_format = 'CSV', autodetect = False, compression = 'NONE', ignore_unknown_values = False, max_bad_records = 0, skip_leading_rows = 0, field_delimiter = ',', quote_character = None, allow_quoted_newlines = False, allow_jagged_rows = False, encoding = 'UTF-8', src_fmt_configs = None, labels = None, description = None, encryption_configuration = None, location = None, project_id = None)

      Creates a new external table in the dataset with the data from Google
      Cloud Storage. See here:

      https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#resource

      for more details about these parameters.

      :param external_project_dataset_table:
          The dotted ``(<project>.|<project>:)<dataset>.<table>($<partition>)`` BigQuery
          table name to create external table.
          If ``<project>`` is not included, project will be the
          project defined in the connection json.
      :param schema_fields: The schema field list as defined here:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#resource
      :param source_uris: The source Google Cloud
          Storage URI (e.g. gs://some-bucket/some-file.txt). A single wild
          per-object name can be used.
      :param source_format: File format to export.
      :param autodetect: Try to detect schema and format options automatically.
          Any option specified explicitly will be honored.
      :param compression: [Optional] The compression type of the data source.
          Possible values include GZIP and NONE.
          The default value is NONE.
          This setting is ignored for Google Cloud Bigtable,
          Google Cloud Datastore backups and Avro formats.
      :param ignore_unknown_values: [Optional] Indicates if BigQuery should allow
          extra values that are not represented in the table schema.
          If true, the extra values are ignored. If false, records with extra columns
          are treated as bad records, and if there are too many bad records, an
          invalid error is returned in the job result.
      :param max_bad_records: The maximum number of bad records that BigQuery can
          ignore when running the job.
      :param skip_leading_rows: Number of rows to skip when loading from a CSV.
      :param field_delimiter: The delimiter to use when loading from a CSV.
      :param quote_character: The value that is used to quote data sections in a CSV
          file.
      :param allow_quoted_newlines: Whether to allow quoted newlines (true) or not
          (false).
      :param allow_jagged_rows: Accept rows that are missing trailing optional columns.
          The missing values are treated as nulls. If false, records with missing
          trailing columns are treated as bad records, and if there are too many bad
          records, an invalid error is returned in the job result. Only applicable when
          source_format is CSV.
      :param encoding: The character encoding of the data. See:

          .. seealso::
              https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.csvOptions.encoding
      :param src_fmt_configs: configure optional fields specific to the source format
      :param labels: A dictionary containing labels for the BiqQuery table.
      :param description: A string containing the description for the BigQuery table.
      :param encryption_configuration: [Optional] Custom encryption configuration (e.g., Cloud KMS keys).
          **Example**: ::

              encryption_configuration = {
                  "kmsKeyName": "projects/testp/locations/us/keyRings/test-kr/cryptoKeys/test-key"
              }


   .. py:method:: update_table(self, table_resource, fields = None, dataset_id = None, table_id = None, project_id = None)

      Change some fields of a table.

      Use ``fields`` to specify which fields to update. At least one field
      must be provided. If a field is listed in ``fields`` and is ``None``
      in ``table``, the field value will be deleted.

      If ``table.etag`` is not ``None``, the update will only succeed if
      the table on the server has the same ETag. Thus reading a table with
      ``get_table``, changing its fields, and then passing it to
      ``update_table`` will ensure that the changes will only be saved if
      no modifications to the table occurred since the read.

      :param project_id: The project to create the table into.
      :param dataset_id: The dataset to create the table into.
      :param table_id: The Name of the table to be created.
      :param table_resource: Table resource as described in documentation:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#Table
          The table has to contain ``tableReference`` or ``project_id``, ``dataset_id`` and ``table_id``
          have to be provided.
      :param fields: The fields of ``table`` to change, spelled as the Table
          properties (e.g. "friendly_name").


   .. py:method:: patch_table(self, dataset_id, table_id, project_id = None, description = None, expiration_time = None, external_data_configuration = None, friendly_name = None, labels = None, schema = None, time_partitioning = None, view = None, require_partition_filter = None, encryption_configuration = None)

      Patch information in an existing table.
      It only updates fields that are provided in the request object.

      Reference: https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/patch

      :param dataset_id: The dataset containing the table to be patched.
      :param table_id: The Name of the table to be patched.
      :param project_id: The project containing the table to be patched.
      :param description: [Optional] A user-friendly description of this table.
      :param expiration_time: [Optional] The time when this table expires,
          in milliseconds since the epoch.
      :param external_data_configuration: [Optional] A dictionary containing
          properties of a table stored outside of BigQuery.
      :param friendly_name: [Optional] A descriptive name for this table.
      :param labels: [Optional] A dictionary containing labels associated with this table.
      :param schema: [Optional] If set, the schema field list as defined here:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema
          The supported schema modifications and unsupported schema modification are listed here:
          https://cloud.google.com/bigquery/docs/managing-table-schemas
          **Example**: ::

              schema=[{"name": "emp_name", "type": "STRING", "mode": "REQUIRED"},
                             {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}]

      :param time_partitioning: [Optional] A dictionary containing time-based partitioning
           definition for the table.
      :param view: [Optional] A dictionary containing definition for the view.
          If set, it will patch a view instead of a table:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#ViewDefinition
          **Example**: ::

              view = {
                  "query": "SELECT * FROM `test-project-id.test_dataset_id.test_table_prefix*` LIMIT 500",
                  "useLegacySql": False
              }

      :param require_partition_filter: [Optional] If true, queries over the this table require a
          partition filter. If false, queries over the table
      :param encryption_configuration: [Optional] Custom encryption configuration (e.g., Cloud KMS keys).
          **Example**: ::

              encryption_configuration = {
                  "kmsKeyName": "projects/testp/locations/us/keyRings/test-kr/cryptoKeys/test-key"
              }


   .. py:method:: insert_all(self, project_id, dataset_id, table_id, rows, ignore_unknown_values = False, skip_invalid_rows = False, fail_on_error = False)

      Method to stream data into BigQuery one record at a time without needing
      to run a load job

      .. seealso::
          For more information, see:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll

      :param project_id: The name of the project where we have the table
      :param dataset_id: The name of the dataset where we have the table
      :param table_id: The name of the table
      :param rows: the rows to insert

      **Example or rows**:
          rows=[{"json": {"a_key": "a_value_0"}}, {"json": {"a_key": "a_value_1"}}]

      :param ignore_unknown_values: [Optional] Accept rows that contain values
          that do not match the schema. The unknown values are ignored.
          The default value  is false, which treats unknown values as errors.
      :param skip_invalid_rows: [Optional] Insert all valid rows of a request,
          even if invalid rows exist. The default value is false, which causes
          the entire request to fail if any invalid rows exist.
      :param fail_on_error: [Optional] Force the task to fail if any errors occur.
          The default value is false, which indicates the task should not fail
          even if any insertion errors occur.


   .. py:method:: update_dataset(self, fields, dataset_resource, dataset_id = None, project_id = None, retry = DEFAULT_RETRY)

      Change some fields of a dataset.

      Use ``fields`` to specify which fields to update. At least one field
      must be provided. If a field is listed in ``fields`` and is ``None`` in
      ``dataset``, it will be deleted.

      If ``dataset.etag`` is not ``None``, the update will only
      succeed if the dataset on the server has the same ETag. Thus
      reading a dataset with ``get_dataset``, changing its fields,
      and then passing it to ``update_dataset`` will ensure that the changes
      will only be saved if no modifications to the dataset occurred
      since the read.

      :param dataset_resource: Dataset resource that will be provided
          in request body.
          https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
      :param dataset_id: The id of the dataset.
      :param fields: The properties of ``dataset`` to change (e.g. "friendly_name").
      :param project_id: The Google Cloud Project ID
      :param retry: How to retry the RPC.


   .. py:method:: patch_dataset(self, dataset_id, dataset_resource, project_id = None)

      Patches information in an existing dataset.
      It only replaces fields that are provided in the submitted dataset resource.
      More info:
      https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/patch

      :param dataset_id: The BigQuery Dataset ID
      :param dataset_resource: Dataset resource that will be provided
          in request body.
          https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
      :param project_id: The Google Cloud Project ID
      :rtype: dataset
          https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource


   .. py:method:: get_dataset_tables_list(self, dataset_id, project_id = None, table_prefix = None, max_results = None)

      Method returns tables list of a BigQuery tables. If table prefix is specified,
      only tables beginning by it are returned.

      For more information, see:
      https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/list

      :param dataset_id: The BigQuery Dataset ID
      :param project_id: The Google Cloud Project ID
      :param table_prefix: Tables must begin by this prefix to be returned (case sensitive)
      :param max_results: The maximum number of results to return in a single response page.
          Leverage the page tokens to iterate through the entire collection.
      :return: List of tables associated with the dataset


   .. py:method:: get_datasets_list(self, project_id = None, include_all = False, filter_ = None, max_results = None, page_token = None, retry = DEFAULT_RETRY)

      Method returns full list of BigQuery datasets in the current project

      For more information, see:
      https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/list

      :param project_id: Google Cloud Project for which you try to get all datasets
      :param include_all: True if results include hidden datasets. Defaults to False.
      :param filter_: An expression for filtering the results by label. For syntax, see
          https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/list#filter.
      :param filter_: str
      :param max_results: Maximum number of datasets to return.
      :param max_results: int
      :param page_token: Token representing a cursor into the datasets. If not passed,
          the API will return the first page of datasets. The token marks the beginning of the
          iterator to be returned and the value of the ``page_token`` can be accessed at
          ``next_page_token`` of the :class:`~google.api_core.page_iterator.HTTPIterator`.
      :param page_token: str
      :param retry: How to retry the RPC.


   .. py:method:: get_dataset(self, dataset_id, project_id = None)

      Fetch the dataset referenced by dataset_id.

      :param dataset_id: The BigQuery Dataset ID
      :param project_id: The Google Cloud Project ID
      :return: dataset_resource

          .. seealso::
              For more information, see Dataset Resource content:
              https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource


   .. py:method:: run_grant_dataset_view_access(self, source_dataset, view_dataset, view_table, view_project = None, project_id = None)

      Grant authorized view access of a dataset to a view table.
      If this view has already been granted access to the dataset, do nothing.
      This method is not atomic.  Running it may clobber a simultaneous update.

      :param source_dataset: the source dataset
      :param view_dataset: the dataset that the view is in
      :param view_table: the table of the view
      :param project_id: the project of the source dataset. If None,
          self.project_id will be used.
      :param view_project: the project that the view is in. If None,
          self.project_id will be used.
      :return: the datasets resource of the source dataset.


   .. py:method:: run_table_upsert(self, dataset_id, table_resource, project_id = None)

      If the table already exists, update the existing table if not create new.
      Since BigQuery does not natively allow table upserts, this is not an
      atomic operation.

      :param dataset_id: the dataset to upsert the table into.
      :param table_resource: a table resource. see
          https://cloud.google.com/bigquery/docs/reference/v2/tables#resource
      :param project_id: the project to upsert the table into.  If None,
          project will be self.project_id.
      :return:


   .. py:method:: run_table_delete(self, deletion_dataset_table, ignore_if_missing = False)

      Delete an existing table from the dataset;
      If the table does not exist, return an error unless ignore_if_missing
      is set to True.

      :param deletion_dataset_table: A dotted
          ``(<project>.|<project>:)<dataset>.<table>`` that indicates which table
          will be deleted.
      :param ignore_if_missing: if True, then return success even if the
          requested table does not exist.
      :return:


   .. py:method:: delete_table(self, table_id, not_found_ok = True, project_id = None)

      Delete an existing table from the dataset. If the table does not exist, return an error
      unless not_found_ok is set to True.

      :param table_id: A dotted ``(<project>.|<project>:)<dataset>.<table>``
          that indicates which table will be deleted.
      :param not_found_ok: if True, then return success even if the
          requested table does not exist.
      :param project_id: the project used to perform the request


   .. py:method:: get_tabledata(self, dataset_id, table_id, max_results = None, selected_fields = None, page_token = None, start_index = None)

      Get the data of a given dataset.table and optionally with selected columns.
      see https://cloud.google.com/bigquery/docs/reference/v2/tabledata/list

      :param dataset_id: the dataset ID of the requested table.
      :param table_id: the table ID of the requested table.
      :param max_results: the maximum results to return.
      :param selected_fields: List of fields to return (comma-separated). If
          unspecified, all fields are returned.
      :param page_token: page token, returned from a previous call,
          identifying the result set.
      :param start_index: zero based index of the starting row to read.
      :return: list of rows


   .. py:method:: list_rows(self, dataset_id, table_id, max_results = None, selected_fields = None, page_token = None, start_index = None, project_id = None, location = None)

      List the rows of the table.
      See https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/list

      :param dataset_id: the dataset ID of the requested table.
      :param table_id: the table ID of the requested table.
      :param max_results: the maximum results to return.
      :param selected_fields: List of fields to return (comma-separated). If
          unspecified, all fields are returned.
      :param page_token: page token, returned from a previous call,
          identifying the result set.
      :param start_index: zero based index of the starting row to read.
      :param project_id: Project ID for the project which the client acts on behalf of.
      :param location: Default location for job.
      :return: list of rows


   .. py:method:: get_schema(self, dataset_id, table_id, project_id = None)

      Get the schema for a given dataset and table.
      see https://cloud.google.com/bigquery/docs/reference/v2/tables#resource

      :param dataset_id: the dataset ID of the requested table
      :param table_id: the table ID of the requested table
      :param project_id: the optional project ID of the requested table.
              If not provided, the connector's configured project will be used.
      :return: a table schema


   .. py:method:: update_table_schema(self, schema_fields_updates, include_policy_tags, dataset_id, table_id, project_id = None)

      Update fields within a schema for a given dataset and table. Note that
      some fields in schemas are immutable and trying to change them will cause
      an exception.
      If a new field is included it will be inserted which requires all required fields to be set.
      See https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableSchema

      :param include_policy_tags: If set to True policy tags will be included in
          the update request which requires special permissions even if unchanged
          see https://cloud.google.com/bigquery/docs/column-level-security#roles
      :param dataset_id: the dataset ID of the requested table to be updated
      :param table_id: the table ID of the table to be updated
      :param schema_fields_updates: a partial schema resource. see
          https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableSchema

      **Example**: ::

          schema_fields_updates=[
              {"name": "emp_name", "description": "Some New Description"},
              {"name": "salary", "description": "Some New Description"},
              {"name": "departments", "fields": [
                  {"name": "name", "description": "Some New Description"},
                  {"name": "type", "description": "Some New Description"}
              ]},
          ]

      :param project_id: The name of the project where we want to update the table.


   .. py:method:: poll_job_complete(self, job_id, project_id = None, location = None, retry = DEFAULT_RETRY)

      Check if jobs completed.

      :param job_id: id of the job.
      :param project_id: Google Cloud Project where the job is running
      :param location: location the job is running
      :param retry: How to retry the RPC.
      :rtype: bool


   .. py:method:: cancel_query(self)

      Cancel all started queries that have not yet completed


   .. py:method:: cancel_job(self, job_id, project_id = None, location = None)

      Cancels a job an wait for cancellation to complete

      :param job_id: id of the job.
      :param project_id: Google Cloud Project where the job is running
      :param location: location the job is running


   .. py:method:: get_job(self, job_id = None, project_id = None, location = None)

      Retrieves a BigQuery job. For more information see:
      https://cloud.google.com/bigquery/docs/reference/v2/jobs

      :param job_id: The ID of the job. The ID must contain only letters (a-z, A-Z),
          numbers (0-9), underscores (_), or dashes (-). The maximum length is 1,024
          characters. If not provided then uuid will be generated.
      :param project_id: Google Cloud Project where the job is running
      :param location: location the job is running


   .. py:method:: insert_job(self, configuration, job_id = None, project_id = None, location = None, nowait = False, retry = DEFAULT_RETRY, timeout = None)

      Executes a BigQuery job. Waits for the job to complete and returns job id.
      See here:

      https://cloud.google.com/bigquery/docs/reference/v2/jobs

      :param configuration: The configuration parameter maps directly to
          BigQuery's configuration field in the job object. See
          https://cloud.google.com/bigquery/docs/reference/v2/jobs for
          details.
      :param job_id: The ID of the job. The ID must contain only letters (a-z, A-Z),
          numbers (0-9), underscores (_), or dashes (-). The maximum length is 1,024
          characters. If not provided then uuid will be generated.
      :param project_id: Google Cloud Project where the job is running
      :param location: location the job is running
      :param nowait: specify whether to insert job without waiting for the result
      :param retry: How to retry the RPC.
      :param timeout: The number of seconds to wait for the underlying HTTP transport
          before using ``retry``.


   .. py:method:: run_with_configuration(self, configuration)

      Executes a BigQuery SQL query. See here:

      https://cloud.google.com/bigquery/docs/reference/v2/jobs

      For more details about the configuration parameter.

      :param configuration: The configuration parameter maps directly to
          BigQuery's configuration field in the job object. See
          https://cloud.google.com/bigquery/docs/reference/v2/jobs for
          details.


   .. py:method:: run_load(self, destination_project_dataset_table, source_uris, schema_fields = None, source_format = 'CSV', create_disposition = 'CREATE_IF_NEEDED', skip_leading_rows = 0, write_disposition = 'WRITE_EMPTY', field_delimiter = ',', max_bad_records = 0, quote_character = None, ignore_unknown_values = False, allow_quoted_newlines = False, allow_jagged_rows = False, encoding = 'UTF-8', schema_update_options = None, src_fmt_configs = None, time_partitioning = None, cluster_fields = None, autodetect = False, encryption_configuration = None, labels = None, description = None)

      Executes a BigQuery load command to load data from Google Cloud Storage
      to BigQuery. See here:

      https://cloud.google.com/bigquery/docs/reference/v2/jobs

      For more details about these parameters.

      :param destination_project_dataset_table:
          The dotted ``(<project>.|<project>:)<dataset>.<table>($<partition>)`` BigQuery
          table to load data into. If ``<project>`` is not included, project will be the
          project defined in the connection json. If a partition is specified the
          operator will automatically append the data, create a new partition or create
          a new DAY partitioned table.
      :param schema_fields: The schema field list as defined here:
          https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load
          Required if autodetect=False; optional if autodetect=True.
      :param autodetect: Attempt to autodetect the schema for CSV and JSON
          source files.
      :param source_uris: The source Google Cloud
          Storage URI (e.g. gs://some-bucket/some-file.txt). A single wild
          per-object name can be used.
      :param source_format: File format to export.
      :param create_disposition: The create disposition if the table doesn't exist.
      :param skip_leading_rows: Number of rows to skip when loading from a CSV.
      :param write_disposition: The write disposition if the table already exists.
      :param field_delimiter: The delimiter to use when loading from a CSV.
      :param max_bad_records: The maximum number of bad records that BigQuery can
          ignore when running the job.
      :param quote_character: The value that is used to quote data sections in a CSV
          file.
      :param ignore_unknown_values: [Optional] Indicates if BigQuery should allow
          extra values that are not represented in the table schema.
          If true, the extra values are ignored. If false, records with extra columns
          are treated as bad records, and if there are too many bad records, an
          invalid error is returned in the job result.
      :param allow_quoted_newlines: Whether to allow quoted newlines (true) or not
          (false).
      :param allow_jagged_rows: Accept rows that are missing trailing optional columns.
          The missing values are treated as nulls. If false, records with missing
          trailing columns are treated as bad records, and if there are too many bad
          records, an invalid error is returned in the job result. Only applicable when
          source_format is CSV.
      :param encoding: The character encoding of the data.

          .. seealso::
              https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.csvOptions.encoding
      :param schema_update_options: Allows the schema of the destination
          table to be updated as a side effect of the load job.
      :param src_fmt_configs: configure optional fields specific to the source format
      :param time_partitioning: configure optional time partitioning fields i.e.
          partition by field, type and  expiration as per API specifications.
      :param cluster_fields: Request that the result of this load be stored sorted
          by one or more columns. BigQuery supports clustering for both partitioned and
          non-partitioned tables. The order of columns given determines the sort order.
      :param encryption_configuration: [Optional] Custom encryption configuration (e.g., Cloud KMS keys).
          **Example**: ::

              encryption_configuration = {
                  "kmsKeyName": "projects/testp/locations/us/keyRings/test-kr/cryptoKeys/test-key"
              }
      :param labels: A dictionary containing labels for the BiqQuery table.
      :param description: A string containing the description for the BigQuery table.


   .. py:method:: run_copy(self, source_project_dataset_tables, destination_project_dataset_table, write_disposition = 'WRITE_EMPTY', create_disposition = 'CREATE_IF_NEEDED', labels = None, encryption_configuration = None)

      Executes a BigQuery copy command to copy data from one BigQuery table
      to another. See here:

      https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.copy

      For more details about these parameters.

      :param source_project_dataset_tables: One or more dotted
          ``(project:|project.)<dataset>.<table>``
          BigQuery tables to use as the source data. Use a list if there are
          multiple source tables.
          If ``<project>`` is not included, project will be the project defined
          in the connection json.
      :param destination_project_dataset_table: The destination BigQuery
          table. Format is: ``(project:|project.)<dataset>.<table>``
      :param write_disposition: The write disposition if the table already exists.
      :param create_disposition: The create disposition if the table doesn't exist.
      :param labels: a dictionary containing labels for the job/query,
          passed to BigQuery
      :param encryption_configuration: [Optional] Custom encryption configuration (e.g., Cloud KMS keys).
          **Example**: ::

              encryption_configuration = {
                  "kmsKeyName": "projects/testp/locations/us/keyRings/test-kr/cryptoKeys/test-key"
              }


   .. py:method:: run_extract(self, source_project_dataset_table, destination_cloud_storage_uris, compression = 'NONE', export_format = 'CSV', field_delimiter = ',', print_header = True, labels = None)

      Executes a BigQuery extract command to copy data from BigQuery to
      Google Cloud Storage. See here:

      https://cloud.google.com/bigquery/docs/reference/v2/jobs

      For more details about these parameters.

      :param source_project_dataset_table: The dotted ``<dataset>.<table>``
          BigQuery table to use as the source data.
      :param destination_cloud_storage_uris: The destination Google Cloud
          Storage URI (e.g. gs://some-bucket/some-file.txt). Follows
          convention defined here:
          https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple
      :param compression: Type of compression to use.
      :param export_format: File format to export.
      :param field_delimiter: The delimiter to use when extracting to a CSV.
      :param print_header: Whether to print a header for a CSV file extract.
      :param labels: a dictionary containing labels for the job/query,
          passed to BigQuery


   .. py:method:: run_query(self, sql, destination_dataset_table = None, write_disposition = 'WRITE_EMPTY', allow_large_results = False, flatten_results = None, udf_config = None, use_legacy_sql = None, maximum_billing_tier = None, maximum_bytes_billed = None, create_disposition = 'CREATE_IF_NEEDED', query_params = None, labels = None, schema_update_options = None, priority = 'INTERACTIVE', time_partitioning = None, api_resource_configs = None, cluster_fields = None, location = None, encryption_configuration = None)

      Executes a BigQuery SQL query. Optionally persists results in a BigQuery
      table. See here:

      https://cloud.google.com/bigquery/docs/reference/v2/jobs

      For more details about these parameters.

      :param sql: The BigQuery SQL to execute.
      :param destination_dataset_table: The dotted ``<dataset>.<table>``
          BigQuery table to save the query results.
      :param write_disposition: What to do if the table already exists in
          BigQuery.
      :param allow_large_results: Whether to allow large results.
      :param flatten_results: If true and query uses legacy SQL dialect, flattens
          all nested and repeated fields in the query results. ``allowLargeResults``
          must be true if this is set to false. For standard SQL queries, this
          flag is ignored and results are never flattened.
      :param udf_config: The User Defined Function configuration for the query.
          See https://cloud.google.com/bigquery/user-defined-functions for details.
      :param use_legacy_sql: Whether to use legacy SQL (true) or standard SQL (false).
          If `None`, defaults to `self.use_legacy_sql`.
      :param api_resource_configs: a dictionary that contain params
          'configuration' applied for Google BigQuery Jobs API:
          https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
          for example, {'query': {'useQueryCache': False}}. You could use it
          if you need to provide some params that are not supported by the
          BigQueryHook like args.
      :param maximum_billing_tier: Positive integer that serves as a
          multiplier of the basic price.
      :param maximum_bytes_billed: Limits the bytes billed for this job.
          Queries that will have bytes billed beyond this limit will fail
          (without incurring a charge). If unspecified, this will be
          set to your project default.
      :param create_disposition: Specifies whether the job is allowed to
          create new tables.
      :param query_params: a list of dictionary containing query parameter types and
          values, passed to BigQuery
      :param labels: a dictionary containing labels for the job/query,
          passed to BigQuery
      :param schema_update_options: Allows the schema of the destination
          table to be updated as a side effect of the query job.
      :param priority: Specifies a priority for the query.
          Possible values include INTERACTIVE and BATCH.
          The default value is INTERACTIVE.
      :param time_partitioning: configure optional time partitioning fields i.e.
          partition by field, type and expiration as per API specifications.
      :param cluster_fields: Request that the result of this query be stored sorted
          by one or more columns. BigQuery supports clustering for both partitioned and
          non-partitioned tables. The order of columns given determines the sort order.
      :param location: The geographic location of the job. Required except for
          US and EU. See details at
          https://cloud.google.com/bigquery/docs/locations#specifying_your_location
      :param encryption_configuration: [Optional] Custom encryption configuration (e.g., Cloud KMS keys).
          **Example**: ::

              encryption_configuration = {
                  "kmsKeyName": "projects/testp/locations/us/keyRings/test-kr/cryptoKeys/test-key"
              }


.. py:class:: BigQueryConnection(*args, **kwargs)

   BigQuery does not have a notion of a persistent connection. Thus, these
   objects are small stateless factories for cursors, which do all the real
   work.

   .. py:method:: close(self)

      The BigQueryConnection does not have anything to close


   .. py:method:: commit(self)

      The BigQueryConnection does not support transactions


   .. py:method:: cursor(self)

      Return a new :py:class:`Cursor` object using the connection


   .. py:method:: rollback(self)
      :abstractmethod:

      The BigQueryConnection does not have transactions


.. py:class:: BigQueryBaseCursor(service, project_id, hook, use_legacy_sql = True, api_resource_configs = None, location = None, num_retries = 5, labels = None)

   Bases: :py:obj:`airflow.utils.log.logging_mixin.LoggingMixin`

   The BigQuery base cursor contains helper methods to execute queries against
   BigQuery. The methods can be used directly by operators, in cases where a
   PEP 249 cursor isn't needed.

   .. py:method:: create_empty_table(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_empty_table`


   .. py:method:: create_empty_dataset(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_empty_dataset`


   .. py:method:: get_dataset_tables(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset_tables`


   .. py:method:: delete_dataset(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.delete_dataset`


   .. py:method:: create_external_table(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_external_table`


   .. py:method:: patch_table(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.patch_table`


   .. py:method:: insert_all(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.insert_all`


   .. py:method:: update_dataset(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.update_dataset`


   .. py:method:: patch_dataset(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.patch_dataset`


   .. py:method:: get_dataset_tables_list(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset_tables_list`


   .. py:method:: get_datasets_list(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_datasets_list`


   .. py:method:: get_dataset(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset`


   .. py:method:: run_grant_dataset_view_access(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_grant_dataset_view_access`


   .. py:method:: run_table_upsert(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_table_upsert`


   .. py:method:: run_table_delete(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_table_delete`


   .. py:method:: get_tabledata(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_tabledata`


   .. py:method:: get_schema(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_schema`


   .. py:method:: poll_job_complete(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.poll_job_complete`


   .. py:method:: cancel_query(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.cancel_query`


   .. py:method:: run_with_configuration(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_with_configuration`


   .. py:method:: run_load(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_load`


   .. py:method:: run_copy(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_copy`


   .. py:method:: run_extract(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_extract`


   .. py:method:: run_query(self, *args, **kwargs)

      This method is deprecated.
      Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_query`


.. py:class:: BigQueryCursor(service, project_id, hook, use_legacy_sql = True, location = None, num_retries = 5)

   Bases: :py:obj:`BigQueryBaseCursor`

   A very basic BigQuery PEP 249 cursor implementation. The PyHive PEP 249
   implementation was used as a reference:

   https://github.com/dropbox/PyHive/blob/master/pyhive/presto.py
   https://github.com/dropbox/PyHive/blob/master/pyhive/common.py

   .. py:attribute:: arraysize
      

   .. py:method:: description(self)
      :property:

      The schema description method is not currently implemented


   .. py:method:: close(self)

      By default, do nothing


   .. py:method:: rowcount(self)
      :property:

      By default, return -1 to indicate that this is not supported


   .. py:method:: execute(self, operation, parameters = None)

      Executes a BigQuery query, and returns the job ID.

      :param operation: The query to execute.
      :param parameters: Parameters to substitute into the query.


   .. py:method:: executemany(self, operation, seq_of_parameters)

      Execute a BigQuery query multiple times with different parameters.

      :param operation: The query to execute.
      :param seq_of_parameters: List of dictionary parameters to substitute into the
          query.


   .. py:method:: flush_results(self)

      Flush results related cursor attributes


   .. py:method:: fetchone(self)

      Fetch the next row of a query result set


   .. py:method:: next(self)

      Helper method for fetchone, which returns the next row from a buffer.
      If the buffer is empty, attempts to paginate through the result set for
      the next page, and load it into the buffer.


   .. py:method:: fetchmany(self, size = None)

      Fetch the next set of rows of a query result, returning a sequence of sequences
      (e.g. a list of tuples). An empty sequence is returned when no more rows are
      available. The number of rows to fetch per call is specified by the parameter.
      If it is not given, the cursor's arraysize determines the number of rows to be
      fetched. The method should try to fetch as many rows as indicated by the size
      parameter. If this is not possible due to the specified number of rows not being
      available, fewer rows may be returned. An :py:class:`~pyhive.exc.Error`
      (or subclass) exception is raised if the previous call to
      :py:meth:`execute` did not produce any result set or no call was issued yet.


   .. py:method:: fetchall(self)

      Fetch all (remaining) rows of a query result, returning them as a sequence of
      sequences (e.g. a list of tuples).


   .. py:method:: get_arraysize(self)

      Specifies the number of rows to fetch at a time with .fetchmany()


   .. py:method:: set_arraysize(self, arraysize)

      Specifies the number of rows to fetch at a time with .fetchmany()


   .. py:method:: setinputsizes(self, sizes)

      Does nothing by default


   .. py:method:: setoutputsize(self, size, column = None)

      Does nothing by default