| :py:mod:`airflow.providers.amazon.aws.sensors.glue_catalog_partition` |
| ===================================================================== |
| |
| .. py:module:: airflow.providers.amazon.aws.sensors.glue_catalog_partition |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.sensors.glue_catalog_partition.GlueCatalogPartitionSensor |
| airflow.providers.amazon.aws.sensors.glue_catalog_partition.AwsGlueCatalogPartitionSensor |
| |
| |
| |
| |
| .. py:class:: GlueCatalogPartitionSensor(*, table_name, expression = "ds='{{ ds }}'", aws_conn_id = 'aws_default', region_name = None, database_name = 'default', poke_interval = 60 * 3, **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.base.BaseSensorOperator` |
| |
| Waits for a partition to show up in AWS Glue Catalog. |
| |
| :param table_name: The name of the table to wait for, supports the dot |
| notation (my_database.my_table) |
| :param expression: The partition clause to wait for. This is passed as |
| is to the AWS Glue Catalog API's get_partitions function, |
| and supports SQL like notation as in ``ds='2015-01-01' |
| AND type='value'`` and comparison operators as in ``"ds>=2015-01-01"``. |
| See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html |
| #aws-glue-api-catalog-partitions-GetPartitions |
| :param aws_conn_id: ID of the Airflow connection where |
| credentials and extra configuration are stored |
| :param region_name: Optional aws region name (example: us-east-1). Uses region from connection |
| if not specified. |
| :param database_name: The name of the catalog database where the partitions reside. |
| :param poke_interval: Time in seconds that the job should wait in |
| between each tries |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['database_name', 'table_name', 'expression'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #C5CAE9 |
| |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Checks for existence of the partition in the AWS Glue Catalog table |
| |
| |
| .. py:method:: get_hook(self) |
| |
| Gets the GlueCatalogHook |
| |
| |
| |
| .. py:class:: AwsGlueCatalogPartitionSensor(*args, **kwargs) |
| |
| Bases: :py:obj:`GlueCatalogPartitionSensor` |
| |
| This sensor is deprecated. Please use |
| :class:`airflow.providers.amazon.aws.sensors.glue_catalog_partition.GlueCatalogPartitionSensor`. |
| |
| |