| :py:mod:`airflow.providers.apache.hive.sensors.metastore_partition` |
| =================================================================== |
| |
| .. py:module:: airflow.providers.apache.hive.sensors.metastore_partition |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.apache.hive.sensors.metastore_partition.MetastorePartitionSensor |
| |
| |
| |
| |
| .. py:class:: MetastorePartitionSensor(*, table, partition_name, schema = 'default', mysql_conn_id = 'metastore_mysql', **kwargs) |
| |
| |
| Bases: :py:obj:`airflow.providers.common.sql.sensors.sql.SqlSensor` |
| |
| An alternative to the HivePartitionSensor that talk directly to the MySQL db. |
| |
| This was created as a result of observing sub optimal queries generated by the |
| Metastore thrift service when hitting subpartitioned tables. The Thrift service's |
| queries were written in a way that would not leverage the indexes. |
| |
| :param schema: the schema |
| :param table: the table |
| :param partition_name: the partition name, as defined in the PARTITIONS |
| table of the Metastore. Order of the fields does matter. |
| Examples: ``ds=2016-01-01`` or |
| ``ds=2016-01-01/sub=foo`` for a sub partitioned table |
| :param mysql_conn_id: a reference to the MySQL conn_id for the metastore |
| |
| .. py:attribute:: template_fields |
| :type: Sequence[str] |
| :value: ('partition_name', 'table', 'schema') |
| |
| |
| |
| .. py:attribute:: ui_color |
| :value: '#8da7be' |
| |
| |
| |
| .. py:method:: poke(context) |
| |
| Override when deriving this class. |
| |
| |
| |