| :py:mod:`airflow.providers.apache.hive.sensors.metastore_partition` |
| =================================================================== |
| |
| .. py:module:: airflow.providers.apache.hive.sensors.metastore_partition |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.apache.hive.sensors.metastore_partition.MetastorePartitionSensor |
| |
| |
| |
| |
| .. py:class:: MetastorePartitionSensor(*, table, partition_name, schema = 'default', mysql_conn_id = 'metastore_mysql', **kwargs) |
| |
| Bases: :py:obj:`airflow.sensors.sql.SqlSensor` |
| |
| An alternative to the HivePartitionSensor that talk directly to the |
| MySQL db. This was created as a result of observing sub optimal |
| queries generated by the Metastore thrift service when hitting |
| subpartitioned tables. The Thrift service's queries were written in a |
| way that wouldn't leverage the indexes. |
| |
| :param schema: the schema |
| :param table: the table |
| :param partition_name: the partition name, as defined in the PARTITIONS |
| table of the Metastore. Order of the fields does matter. |
| Examples: ``ds=2016-01-01`` or |
| ``ds=2016-01-01/sub=foo`` for a sub partitioned table |
| :param mysql_conn_id: a reference to the MySQL conn_id for the metastore |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['partition_name', 'table', 'schema'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #8da7be |
| |
| |
| |
| .. py:attribute:: poke_context_fields |
| :annotation: = ['partition_name', 'table', 'schema', 'mysql_conn_id'] |
| |
| |
| |
| .. py:method:: poke(self, context) |
| |
| Function that the sensors defined while deriving this class should |
| override. |
| |
| |
| |
