blob: 170e082a01f436c75ed5fa01b359e34f09c5ca46 [file] [log] [blame]
:py:mod:`airflow.providers.amazon.aws.sensors.glue_catalog_partition`
=====================================================================
.. py:module:: airflow.providers.amazon.aws.sensors.glue_catalog_partition
Module Contents
---------------
Classes
~~~~~~~
.. autoapisummary::
airflow.providers.amazon.aws.sensors.glue_catalog_partition.GlueCatalogPartitionSensor
airflow.providers.amazon.aws.sensors.glue_catalog_partition.AwsGlueCatalogPartitionSensor
.. py:class:: GlueCatalogPartitionSensor(*, table_name, expression = "ds='{{ ds }}'", aws_conn_id = 'aws_default', region_name = None, database_name = 'default', poke_interval = 60 * 3, **kwargs)
Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator`
Waits for a partition to show up in AWS Glue Catalog.
:param table_name: The name of the table to wait for, supports the dot
notation (my_database.my_table)
:param expression: The partition clause to wait for. This is passed as
is to the AWS Glue Catalog API's get_partitions function,
and supports SQL like notation as in ``ds='2015-01-01'
AND type='value'`` and comparison operators as in ``"ds>=2015-01-01"``.
See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html
#aws-glue-api-catalog-partitions-GetPartitions
:param aws_conn_id: ID of the Airflow connection where
credentials and extra configuration are stored
:param region_name: Optional aws region name (example: us-east-1). Uses region from connection
if not specified.
:param database_name: The name of the catalog database where the partitions reside.
:param poke_interval: Time in seconds that the job should wait in
between each tries
.. py:attribute:: template_fields
:annotation: :Sequence[str] = ['database_name', 'table_name', 'expression']
.. py:attribute:: ui_color
:annotation: = #C5CAE9
.. py:method:: poke(self, context)
Checks for existence of the partition in the AWS Glue Catalog table
.. py:method:: get_hook(self)
Gets the GlueCatalogHook
.. py:class:: AwsGlueCatalogPartitionSensor(*args, **kwargs)
Bases: :py:obj:`GlueCatalogPartitionSensor`
This sensor is deprecated. Please use
:class:`airflow.providers.amazon.aws.sensors.glue_catalog_partition.GlueCatalogPartitionSensor`.