| :py:mod:`airflow.providers.amazon.aws.hooks.glue_catalog` |
| ========================================================= |
| |
| .. py:module:: airflow.providers.amazon.aws.hooks.glue_catalog |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains AWS Glue Catalog Hook |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.hooks.glue_catalog.GlueCatalogHook |
| airflow.providers.amazon.aws.hooks.glue_catalog.AwsGlueCatalogHook |
| |
| |
| |
| |
| .. py:class:: GlueCatalogHook(*args, **kwargs) |
| |
| Bases: :py:obj:`airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook` |
| |
| Interact with AWS Glue Catalog |
| |
| Additional arguments (such as ``aws_conn_id``) may be specified and |
| are passed down to the underlying AwsBaseHook. |
| |
| .. seealso:: |
| :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook` |
| |
| .. py:method:: get_partitions(self, database_name, table_name, expression = '', page_size = None, max_items = None) |
| |
| Retrieves the partition values for a table. |
| |
| :param database_name: The name of the catalog database where the partitions reside. |
| :param table_name: The name of the partitions' table. |
| :param expression: An expression filtering the partitions to be returned. |
| Please see official AWS documentation for further information. |
| https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-GetPartitions |
| :param page_size: pagination size |
| :param max_items: maximum items to return |
| :return: set of partition values where each value is a tuple since |
| a partition may be composed of multiple columns. For example: |
| ``{('2018-01-01','1'), ('2018-01-01','2')}`` |
| |
| |
| .. py:method:: check_for_partition(self, database_name, table_name, expression) |
| |
| Checks whether a partition exists |
| |
| :param database_name: Name of hive database (schema) @table belongs to |
| :param table_name: Name of hive table @partition belongs to |
| :expression: Expression that matches the partitions to check for |
| (eg `a = 'b' AND c = 'd'`) |
| :rtype: bool |
| |
| >>> hook = GlueCatalogHook() |
| >>> t = 'static_babynames_partitioned' |
| >>> hook.check_for_partition('airflow', t, "ds='2015-01-01'") |
| True |
| |
| |
| .. py:method:: get_table(self, database_name, table_name) |
| |
| Get the information of the table |
| |
| :param database_name: Name of hive database (schema) @table belongs to |
| :param table_name: Name of hive table |
| :rtype: dict |
| |
| >>> hook = GlueCatalogHook() |
| >>> r = hook.get_table('db', 'table_foo') |
| >>> r['Name'] = 'table_foo' |
| |
| |
| .. py:method:: get_table_location(self, database_name, table_name) |
| |
| Get the physical location of the table |
| |
| :param database_name: Name of hive database (schema) @table belongs to |
| :param table_name: Name of hive table |
| :return: str |
| |
| |
| .. py:method:: get_partition(self, database_name, table_name, partition_values) |
| |
| Gets a Partition |
| |
| :param database_name: Database name |
| :param table_name: Database's Table name |
| :param partition_values: List of utf-8 strings that define the partition |
| Please see official AWS documentation for further information. |
| https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-GetPartition |
| |
| :rtype: dict |
| |
| :raises: AirflowException |
| |
| >>> hook = GlueCatalogHook() |
| >>> partition = hook.get_partition('db', 'table', ['string']) |
| >>> partition['Values'] |
| |
| |
| .. py:method:: create_partition(self, database_name, table_name, partition_input) |
| |
| Creates a new Partition |
| |
| :param database_name: Database name |
| :param table_name: Database's Table name |
| :param partition_input: Definition of how the partition is created |
| Please see official AWS documentation for further information. |
| https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-CreatePartition |
| |
| :rtype: dict |
| |
| :raises: AirflowException |
| |
| >>> hook = GlueCatalogHook() |
| >>> partition_input = {"Values": []} |
| >>> hook.create_partition(database_name="db", table_name="table", partition_input=partition_input) |
| |
| |
| |
| .. py:class:: AwsGlueCatalogHook(*args, **kwargs) |
| |
| Bases: :py:obj:`GlueCatalogHook` |
| |
| This hook is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.hooks.glue_catalog.GlueCatalogHook`. |
| |
| |