docs/hive_metastore.adoc - kudu - Git at Google

 // Licensed to the Apache Software Foundation (ASF) under one
 // or more contributor license agreements.  See the NOTICE file
 // distributed with this work for additional information
 // regarding copyright ownership.  The ASF licenses this file
 // to you under the Apache License, Version 2.0 (the
 // "License"); you may not use this file except in compliance
 // with the License.  You may obtain a copy of the License at
 //
 //   http://www.apache.org/licenses/LICENSE-2.0
 //
 // Unless required by applicable law or agreed to in writing,
 // software distributed under the License is distributed on an
 // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 // KIND, either express or implied.  See the License for the
 // specific language governing permissions and limitations
 // under the License.

 = Using the Hive Metastore with Kudu

 :author: Kudu Team
 :imagesdir: ./images
 :icons: font
 :toc: left
 :toclevels: 3
 :doctype: book
 :backend: html5
 :sectlinks:
 :experimental:

 [[hive_metastore]]
 ## Overview

 Kudu has an optional feature which allows it to integrate its own catalog with
 the Hive Metastore (HMS). The HMS is the de-facto standard catalog and metadata
 provider in the Hadoop ecosystem. When the HMS integration is enabled, Kudu
 tables can be discovered and used by external HMS-aware tools, even if they are
 not otherwise aware of or integrated with Kudu. Additionally, these components
 can use the HMS to discover necessary information to connect to the Kudu
 cluster which owns the table, such as the Kudu master addresses.

 ## Databases and Table Names

 With the Hive Metastore integration disabled, Kudu presents tables as a single
 flat namespace, with no hierarchy or concept of a database. Additionally,
 Kudu's only restriction on table names is that they be a valid UTF-8 encoded
 string. When the HMS integration is enabled in Kudu, both of these properties
 change in order to match the HMS model: the table name must indicate the
 table's membership of a Hive database, and table name identifiers (i.e. the
 table name and database name) are subject to the Hive table name identifier
 constraints.

 ### Databases

 Hive has the concept of a database, which is a collection of individual tables.
 Each database forms its own independent namespace of table names. In order to
 fit into this model, Kudu tables must be assigned a database when the HMS
 integration is enabled. No new APIs have been added to create or delete
 databases, nor are there APIs to assign an existing Kudu table to a database.
 Instead, a new convention has been introduced that Kudu table names must be in
 the format `<hive-database-name>.<hive-table-name>`. Thus, databases are an
 implicit part of the Kudu table name. By including databases as an implicit
 part of the Kudu table name, existing applications that use Kudu tables can
 operate on non-HMS-integrated and HMS-integrated table names with minimal or no
 changes.

 Kudu provides no additional tooling to create or drop Hive databases.
 Administrators or users should use existing Hive tools such as the Beeline
 Shell or Impala to do so.

 ### Naming Constraints

 When the Hive Metastore integration is enabled, the database and table names of
 Kudu tables must follow the Hive Metastore naming constraints. Namely, the
 database and table name must contain only alphanumeric ASCII characters and
 underscores (`_`).

 NOTE: When the `hive.support.special.characters.tablename` Hive configuration
 is `true`, the forward-slash (`/`) character in table name identifiers (i.e. the
 table name and database name) is also supported.

 Additionally, the Hive Metastore does not enforce case sensitivity for table
 name identifiers. As such, when enabled, Kudu will follow suit and disallow
 tables from being created when one already exists whose table name identifier
 differs only by case. Operations that open, alter, or drop tables will also be
 case-insensitive for the table name identifiers.

 WARNING: Given the case insensitivity upon enabling the integration, if
 multiple Kudu tables exist whose names only differ by case, the Kudu master(s)
 will fail to start up. Be sure to rename such conflicting tables before
 enabling the Hive Metastore integration.

 [[metadata_sync]]
 ### Metadata Synchronization
 When the Hive Metastore integration is enabled, Kudu will automatically
 synchronize metadata changes to Kudu tables between Kudu and the HMS. As such,
 it is important to always ensure that the Kudu and HMS have a consistent view
 of existing tables, using the administrative tools described in the below
 section. Failure to do so may result in issues like Kudu tables not being
 discoverable or usable by external, HMS-aware components (e.g. Apache Sentry,
 Apache Impala).

 NOTE: the Hive Metastore automatically creates directories for Kudu tables.
 These directories are benign and can safely be ignored.

 Impala has notions of internal and external Kudu tables. When dropping an
 internal table from Impala, the table's data is dropped in Kudu; in contrast
 when dropping an external table, the table's data is not dropped in Kudu.
 External tables may refer to tables by names that are different from the names
 of the underlying Kudu tables, while internal tables must use the same names as
 those stored in Kudu. Additionally, multiple external tables may refer to the
 same underlying Kudu table. Thus, since external tables may not map one-to-one
 with Kudu tables, the Hive Metastore integration and tooling will only
 automatically synchronize metadata for internal tables. See the
 <<kudu_impala_integration.adoc#kudu_impala,Kudu Impala integration documentation>>
 for more information about table types in Impala

 [[enabling-the-hive-metastore-integration]]
 ## Enabling the Hive Metastore Integration

 WARNING: Before enabling the Hive Metastore integration on an existing cluster,
 make sure to upgrade any tables that may exist in Kudu's or in the HMS's
 catalog. See <<upgrading-tables>> for more details.

 * When the Hive Metastore is configured with fine-grained authorization, the
 Kudu admin needs to be able to access and modify directories that are created
 for Kudu by the HMS. This can be done by adding the Kudu admin user to the group
 of the Hive service users, e.g.  by running `usermod -aG hive kudu` on the HMS
 nodes.

 * Configure the Hive Metastore to include the notification event listener and
 the Kudu HMS plugin, to allow altering and dropping columns, and to add full
 Thrift objects in notifications. Add the following values to the HMS
 configuration in `hive-site.xml`:

 ```xml
 <property>
   <name>hive.metastore.transactional.event.listeners</name>
   <value>
     org.apache.hive.hcatalog.listener.DbNotificationListener,
     org.apache.kudu.hive.metastore.KuduMetastorePlugin
   </value>
 </property>

 <property>
   <name>hive.metastore.disallow.incompatible.col.type.changes</name>
   <value>false</value>
 </property>

 <property>
   <name>hive.metastore.notifications.add.thrift.objects</name>
   <value>true</value>
 </property>
 ```

 * After building Kudu from source, add the `hms-plugin.jar` found under the build
 directory (e.g. `build/release/bin`) to the HMS classpath.

 * Restart the HMS.

 * Enable the Hive Metastore integration in Kudu with the following
 configuration properties for the Kudu master(s):

 ```
 --hive_metastore_uris=<HMS Thrift URI(s)>
 --hive_metastore_sasl_enabled=<value of the Hive Metastore's hive.metastore.sasl.enabled configuration>
 ```
 NOTE: In a secured cluster, in which `--hive_metastore_sasl_enabled` is set to
 true, `--hive_metastore_kerberos_principal` must match the primary portion of
 `hive.metastore.kerberos.principal` in the Hive Metastore configuration.

 * Restart the Kudu master(s).

 ## Administrative Tools

 Kudu provides the command line tools `kudu hms list`, `kudu hms precheck`,
 `kudu hms check`, and `kudu hms fix` to allow administrators to find and fix
 metadata inconsistencies between the internal Kudu catalog and the Hive
 Metastore catalog, during the upgrade process described below or during the
 normal operation of a Kudu cluster.

 `kudu hms` tools should be run from the command line as the Kudu admin user.
 They require the full list of master addresses to be specified:

 [source,bash]
 ----
 $ sudo -u kudu kudu hms check master-name-1:7051,master-name-2:7051,master-name-3:7051
 ----

 To see a full list of the options available with the `kudu hms` tool, use the
 `--help` flag.

 NOTE: When fine-grained authorization is enabled, the Kudu admin user, commonly
 "kudu", needs to have access to all the Kudu tables to be
 able to run the `kudu hms` tools. This can be done by configuring the user as a
 trusted user via the `--trusted_user_acl` master configuration. See
 <<security.adoc#trusted-users,here>> for more information about trusted users.

 NOTE: If the Hive Metastore is configured with fine-grained authorization, the
 Kudu admin user needs to have read and write privileges on HMS table entries.
 Configured this in the Hive Metastore. For Apache Sentry this can be configured
 using the `sentry.metastore.service.users` property.

 ### `kudu hms list`

 The `kudu hms list` tool scans the Hive Metastore catalog, and lists the HMS
 entries (including table name and type) for Kudu tables, as indicated by their
 HMS storage handler.

 ### `kudu hms precheck`

 The `kudu hms precheck` tool scans the Kudu catalog and validates that if there
 are multiple Kudu tables whose names only differ by case and logs the conflicted
 table names.

 ### `kudu hms check`

 The `kudu hms check` tool scans the Kudu and Hive Metastore catalogs, and
 validates that the two catalogs agree on what Kudu tables exist. The tool will
 make suggestions on how to fix any inconsistencies that are found. Typically,
 the suggestion will be to run the `kudu hms fix` tool, however some certain
 inconsistencies require using Impala Shell for fixing.

 ### `kudu hms fix`

 The `kudu hms fix` tool analyzes the Kudu and HMS catalogs and attempts to fix
 any automatically-fixable issues, for instance, by creating a table entry in
 the HMS for each Kudu table that doesn't already have one. The `--dryrun` option
 shows the proposed fix instead of actually executing it. When no automatic fix
 is available, it will make suggestions on how a manual fix can help.

 NOTE: The `kudu hms fix` tool will not automatically fix Impala external tables
 for the reasons described above. It is instead recommended to fix issues with
 external tables by dropping and recreating them.

 ### `kudu hms downgrade`

 The `kudu hms downgrade` downgrades the metadata to legacy format for Kudu and
 the Hive Metastores. It is discouraged to use unless necessary, since the legacy
 format can be deprecated in future releases.

 [[upgrading-tables]]
 ## Upgrading Existing Tables

 Before enabling the Kudu-HMS integration, it is important to ensure that the
 Kudu and HMS start with a consistent view of existing tables. This may entail
 renaming Kudu tables to conform to the Hive naming constraints. This detailed
 workflow describes how to upgrade existing tables before enabling the Hive
 Metastore integration.

 ### Prepare for the Upgrade

 . Establish a maintenance window. During this time the Kudu cluster will still be
   available, but tables in Kudu and the Hive Metastore may be altered or
   renamed as a part of the upgrade process.

 . Make note of all external tables using the following command and drop them. This reduces
   the chance of having naming conflicts with Kudu tables which can lead to errors during
   upgrading process. It also helps in cases where a catalog upgrade breaks
   external tables, due to the underlying Kudu tables being renamed. The
   external tables can be recreated after upgrade is complete.
 +
 [source,bash]
 ----
 $ sudo -u kudu kudu hms list master-name-1:7051,master-name-2:7051,master-name-3:7051
 ----

 ### Perform the Upgrade

 . Run the `kudu hms precheck` tool to ensure no Kudu tables only differ by
   case. If the tool does not report any warnings, you can skip the next step.
 +
 [source,bash]
 ----
 $ sudo -u kudu kudu hms precheck master-name-1:7051,master-name-2:7051,master-name-3:7051
 ----

 . If the `kudu hms precheck` tool reports conflicting tables, rename these to
   case-insensitive unique names using the following command:
 +
 [source,bash]
 ----
 $ sudo -u kudu kudu table rename_table master-name-1:7051,master-name-2:7051,master-name-3:7051 <conflicting_table_name> <new_table_name>
 ----
 . Run the `kudu hms check` tool using the following command. If the tool does
   not report any catalog inconsistencies, skip to Step 7 below.
 +
 [source,bash]
 ----
 $ sudo -u kudu kudu hms check master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=<hive_metastore_uris> [--ignore_other_clusters=<ignores_other_clusters>]
 ----
 +
 WARNING: By default, the `kudu hms` tools will ignore metadata in the HMS that
 refer to a different Kudu cluster than that being operated on, as indicated by
 having different masters specified. The tools compare the value of the
 `kudu.master_addresses` table property (either supplied at table creation or as
 `--kudu_master_hosts` on impalad daemons) in each HMS metadata entry against
 the RPC endpoints (including the ports) of the Kudu masters. To have the
 tooling account for and fix metadata entries with different master RPC
 endpoints specified (e.g. if ports are not specified in the HMS), supply
 `--ignore_other_clusters=false` as an argument to the `kud hms check` and `fix`
 tools.
 +
 Example::
 +
 ----
 $ sudo -u kudu kudu hms check master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=thrift://hive-metastore:9083 --ignore_other_clusters=false
 ----
 +
 . If the `kudu hms check` tool reports an inconsistent catalog, perform a
   dry-run of the `kudu hms fix` tool to understand how the tool will attempt to
   address the automatically-fixable issues.
 +
 [source,bash]
 ----
 $ sudo -u kudu kudu hms fix master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=<hive_metastore_uris> --dryrun=true [--ignore_other_clusters=<ignore_other_clusters>]
 ----
 Example::
 +
 ----
 $ sudo -u kudu kudu hms check master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=thrift://hive-metastore:9083 --dryrun=true --ignore_other_clusters=false
 ----
 +
 . Manually fix any issues that are reported by the check tool that cannot
   be automatically fixed. For example, rename any tables with names that are not
   Hive-conformant.
 . Run `kudu hms fix` tool to automatically fix all the remaining issues.
 +
 [source,bash]
 ----
 $ sudo -u kudu kudu hms fix master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=<hive_metastore_uris> [--drop_orphan_hms_tables=<drops_orphan_hms_tables>] [--ignore_other_clusters=<ignore_other_clusters>]
 ----
 +
 Example::
 +
 ----
 $ sudo -u kudu kudu hms fix master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=thrift://hive-metastore:9083 --ignore_other_clusters=false
 ----
 +
 NOTE: The `--drop_orphan_hms_tables` argument indicates whether to drop orphan
 Hive Metastore tables that refer to non-existent Kudu tables. Due to
 link:https://issues.apache.org/jira/browse/KUDU-2883[KUDU-2883] this option may
 fail to drop HMS entries that have no table ID. A workaround to this is to drop
 the table via Impala Shell.

 . Recreate any external tables that were dropped when preparing for the upgrade
   by using Impala Shell.

 . Enable the Hive Metastore Integration as described
 <<enabling-the-hive-metastore-integration>>.
	// Licensed to the Apache Software Foundation (ASF) under one
	// or more contributor license agreements. See the NOTICE file
	// distributed with this work for additional information
	// regarding copyright ownership. The ASF licenses this file
	// to you under the Apache License, Version 2.0 (the
	// "License"); you may not use this file except in compliance
	// with the License. You may obtain a copy of the License at
	//
	// http://www.apache.org/licenses/LICENSE-2.0
	//
	// Unless required by applicable law or agreed to in writing,
	// software distributed under the License is distributed on an
	// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	// KIND, either express or implied. See the License for the
	// specific language governing permissions and limitations
	// under the License.

	= Using the Hive Metastore with Kudu

	:author: Kudu Team
	:imagesdir: ./images
	:icons: font
	:toc: left
	:toclevels: 3
	:doctype: book
	:backend: html5
	:sectlinks:
	:experimental:

	[[hive_metastore]]
	## Overview

	Kudu has an optional feature which allows it to integrate its own catalog with
	the Hive Metastore (HMS). The HMS is the de-facto standard catalog and metadata
	provider in the Hadoop ecosystem. When the HMS integration is enabled, Kudu
	tables can be discovered and used by external HMS-aware tools, even if they are
	not otherwise aware of or integrated with Kudu. Additionally, these components
	can use the HMS to discover necessary information to connect to the Kudu
	cluster which owns the table, such as the Kudu master addresses.

	## Databases and Table Names

	With the Hive Metastore integration disabled, Kudu presents tables as a single
	flat namespace, with no hierarchy or concept of a database. Additionally,
	Kudu's only restriction on table names is that they be a valid UTF-8 encoded
	string. When the HMS integration is enabled in Kudu, both of these properties
	change in order to match the HMS model: the table name must indicate the
	table's membership of a Hive database, and table name identifiers (i.e. the
	table name and database name) are subject to the Hive table name identifier
	constraints.

	### Databases

	Hive has the concept of a database, which is a collection of individual tables.
	Each database forms its own independent namespace of table names. In order to
	fit into this model, Kudu tables must be assigned a database when the HMS
	integration is enabled. No new APIs have been added to create or delete
	databases, nor are there APIs to assign an existing Kudu table to a database.
	Instead, a new convention has been introduced that Kudu table names must be in
	the format `<hive-database-name>.<hive-table-name>`. Thus, databases are an
	implicit part of the Kudu table name. By including databases as an implicit
	part of the Kudu table name, existing applications that use Kudu tables can
	operate on non-HMS-integrated and HMS-integrated table names with minimal or no
	changes.

	Kudu provides no additional tooling to create or drop Hive databases.
	Administrators or users should use existing Hive tools such as the Beeline
	Shell or Impala to do so.

	### Naming Constraints

	When the Hive Metastore integration is enabled, the database and table names of
	Kudu tables must follow the Hive Metastore naming constraints. Namely, the
	database and table name must contain only alphanumeric ASCII characters and
	underscores (`_`).

	NOTE: When the `hive.support.special.characters.tablename` Hive configuration
	is `true`, the forward-slash (`/`) character in table name identifiers (i.e. the
	table name and database name) is also supported.

	Additionally, the Hive Metastore does not enforce case sensitivity for table
	name identifiers. As such, when enabled, Kudu will follow suit and disallow
	tables from being created when one already exists whose table name identifier
	differs only by case. Operations that open, alter, or drop tables will also be
	case-insensitive for the table name identifiers.

	WARNING: Given the case insensitivity upon enabling the integration, if
	multiple Kudu tables exist whose names only differ by case, the Kudu master(s)
	will fail to start up. Be sure to rename such conflicting tables before
	enabling the Hive Metastore integration.

	[[metadata_sync]]
	### Metadata Synchronization
	When the Hive Metastore integration is enabled, Kudu will automatically
	synchronize metadata changes to Kudu tables between Kudu and the HMS. As such,
	it is important to always ensure that the Kudu and HMS have a consistent view
	of existing tables, using the administrative tools described in the below
	section. Failure to do so may result in issues like Kudu tables not being
	discoverable or usable by external, HMS-aware components (e.g. Apache Sentry,
	Apache Impala).

	NOTE: the Hive Metastore automatically creates directories for Kudu tables.
	These directories are benign and can safely be ignored.

	Impala has notions of internal and external Kudu tables. When dropping an
	internal table from Impala, the table's data is dropped in Kudu; in contrast
	when dropping an external table, the table's data is not dropped in Kudu.
	External tables may refer to tables by names that are different from the names
	of the underlying Kudu tables, while internal tables must use the same names as
	those stored in Kudu. Additionally, multiple external tables may refer to the
	same underlying Kudu table. Thus, since external tables may not map one-to-one
	with Kudu tables, the Hive Metastore integration and tooling will only
	automatically synchronize metadata for internal tables. See the
	<<kudu_impala_integration.adoc#kudu_impala,Kudu Impala integration documentation>>
	for more information about table types in Impala

	[[enabling-the-hive-metastore-integration]]
	## Enabling the Hive Metastore Integration

	WARNING: Before enabling the Hive Metastore integration on an existing cluster,
	make sure to upgrade any tables that may exist in Kudu's or in the HMS's
	catalog. See <<upgrading-tables>> for more details.

	* When the Hive Metastore is configured with fine-grained authorization, the
	Kudu admin needs to be able to access and modify directories that are created
	for Kudu by the HMS. This can be done by adding the Kudu admin user to the group
	of the Hive service users, e.g. by running `usermod -aG hive kudu` on the HMS
	nodes.

	* Configure the Hive Metastore to include the notification event listener and
	the Kudu HMS plugin, to allow altering and dropping columns, and to add full
	Thrift objects in notifications. Add the following values to the HMS
	configuration in `hive-site.xml`:

	```xml
	<property>
	<name>hive.metastore.transactional.event.listeners</name>
	<value>
	org.apache.hive.hcatalog.listener.DbNotificationListener,
	org.apache.kudu.hive.metastore.KuduMetastorePlugin
	</value>
	</property>

	<property>
	<name>hive.metastore.disallow.incompatible.col.type.changes</name>
	<value>false</value>
	</property>

	<property>
	<name>hive.metastore.notifications.add.thrift.objects</name>
	<value>true</value>
	</property>
	```

	* After building Kudu from source, add the `hms-plugin.jar` found under the build
	directory (e.g. `build/release/bin`) to the HMS classpath.

	* Restart the HMS.

	* Enable the Hive Metastore integration in Kudu with the following
	configuration properties for the Kudu master(s):

	```
	--hive_metastore_uris=<HMS Thrift URI(s)>
	--hive_metastore_sasl_enabled=<value of the Hive Metastore's hive.metastore.sasl.enabled configuration>
	```
	NOTE: In a secured cluster, in which `--hive_metastore_sasl_enabled` is set to
	true, `--hive_metastore_kerberos_principal` must match the primary portion of
	`hive.metastore.kerberos.principal` in the Hive Metastore configuration.

	* Restart the Kudu master(s).

	## Administrative Tools

	Kudu provides the command line tools `kudu hms list`, `kudu hms precheck`,
	`kudu hms check`, and `kudu hms fix` to allow administrators to find and fix
	metadata inconsistencies between the internal Kudu catalog and the Hive
	Metastore catalog, during the upgrade process described below or during the
	normal operation of a Kudu cluster.

	`kudu hms` tools should be run from the command line as the Kudu admin user.
	They require the full list of master addresses to be specified:

	[source,bash]
	----
	$ sudo -u kudu kudu hms check master-name-1:7051,master-name-2:7051,master-name-3:7051
	----

	To see a full list of the options available with the `kudu hms` tool, use the
	`--help` flag.

	NOTE: When fine-grained authorization is enabled, the Kudu admin user, commonly
	"kudu", needs to have access to all the Kudu tables to be
	able to run the `kudu hms` tools. This can be done by configuring the user as a
	trusted user via the `--trusted_user_acl` master configuration. See
	<<security.adoc#trusted-users,here>> for more information about trusted users.

	NOTE: If the Hive Metastore is configured with fine-grained authorization, the
	Kudu admin user needs to have read and write privileges on HMS table entries.
	Configured this in the Hive Metastore. For Apache Sentry this can be configured
	using the `sentry.metastore.service.users` property.

	### `kudu hms list`

	The `kudu hms list` tool scans the Hive Metastore catalog, and lists the HMS
	entries (including table name and type) for Kudu tables, as indicated by their
	HMS storage handler.

	### `kudu hms precheck`

	The `kudu hms precheck` tool scans the Kudu catalog and validates that if there
	are multiple Kudu tables whose names only differ by case and logs the conflicted
	table names.

	### `kudu hms check`

	The `kudu hms check` tool scans the Kudu and Hive Metastore catalogs, and
	validates that the two catalogs agree on what Kudu tables exist. The tool will
	make suggestions on how to fix any inconsistencies that are found. Typically,
	the suggestion will be to run the `kudu hms fix` tool, however some certain
	inconsistencies require using Impala Shell for fixing.

	### `kudu hms fix`

	The `kudu hms fix` tool analyzes the Kudu and HMS catalogs and attempts to fix
	any automatically-fixable issues, for instance, by creating a table entry in
	the HMS for each Kudu table that doesn't already have one. The `--dryrun` option
	shows the proposed fix instead of actually executing it. When no automatic fix
	is available, it will make suggestions on how a manual fix can help.

	NOTE: The `kudu hms fix` tool will not automatically fix Impala external tables
	for the reasons described above. It is instead recommended to fix issues with
	external tables by dropping and recreating them.

	### `kudu hms downgrade`

	The `kudu hms downgrade` downgrades the metadata to legacy format for Kudu and
	the Hive Metastores. It is discouraged to use unless necessary, since the legacy
	format can be deprecated in future releases.

	[[upgrading-tables]]
	## Upgrading Existing Tables

	Before enabling the Kudu-HMS integration, it is important to ensure that the
	Kudu and HMS start with a consistent view of existing tables. This may entail
	renaming Kudu tables to conform to the Hive naming constraints. This detailed
	workflow describes how to upgrade existing tables before enabling the Hive
	Metastore integration.

	### Prepare for the Upgrade

	. Establish a maintenance window. During this time the Kudu cluster will still be
	available, but tables in Kudu and the Hive Metastore may be altered or
	renamed as a part of the upgrade process.

	. Make note of all external tables using the following command and drop them. This reduces
	the chance of having naming conflicts with Kudu tables which can lead to errors during
	upgrading process. It also helps in cases where a catalog upgrade breaks
	external tables, due to the underlying Kudu tables being renamed. The
	external tables can be recreated after upgrade is complete.
	+
	[source,bash]
	----
	$ sudo -u kudu kudu hms list master-name-1:7051,master-name-2:7051,master-name-3:7051
	----

	### Perform the Upgrade

	. Run the `kudu hms precheck` tool to ensure no Kudu tables only differ by
	case. If the tool does not report any warnings, you can skip the next step.
	+
	[source,bash]
	----
	$ sudo -u kudu kudu hms precheck master-name-1:7051,master-name-2:7051,master-name-3:7051
	----

	. If the `kudu hms precheck` tool reports conflicting tables, rename these to
	case-insensitive unique names using the following command:
	+
	[source,bash]
	----
	$ sudo -u kudu kudu table rename_table master-name-1:7051,master-name-2:7051,master-name-3:7051 <conflicting_table_name> <new_table_name>
	----
	. Run the `kudu hms check` tool using the following command. If the tool does
	not report any catalog inconsistencies, skip to Step 7 below.
	+
	[source,bash]
	----
	$ sudo -u kudu kudu hms check master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=<hive_metastore_uris> [--ignore_other_clusters=<ignores_other_clusters>]
	----
	+
	WARNING: By default, the `kudu hms` tools will ignore metadata in the HMS that
	refer to a different Kudu cluster than that being operated on, as indicated by
	having different masters specified. The tools compare the value of the
	`kudu.master_addresses` table property (either supplied at table creation or as
	`--kudu_master_hosts` on impalad daemons) in each HMS metadata entry against
	the RPC endpoints (including the ports) of the Kudu masters. To have the
	tooling account for and fix metadata entries with different master RPC
	endpoints specified (e.g. if ports are not specified in the HMS), supply
	`--ignore_other_clusters=false` as an argument to the `kud hms check` and `fix`
	tools.
	+
	Example::
	+
	----
	$ sudo -u kudu kudu hms check master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=thrift://hive-metastore:9083 --ignore_other_clusters=false
	----
	+
	. If the `kudu hms check` tool reports an inconsistent catalog, perform a
	dry-run of the `kudu hms fix` tool to understand how the tool will attempt to
	address the automatically-fixable issues.
	+
	[source,bash]
	----
	$ sudo -u kudu kudu hms fix master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=<hive_metastore_uris> --dryrun=true [--ignore_other_clusters=<ignore_other_clusters>]
	----
	Example::
	+
	----
	$ sudo -u kudu kudu hms check master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=thrift://hive-metastore:9083 --dryrun=true --ignore_other_clusters=false
	----
	+
	. Manually fix any issues that are reported by the check tool that cannot
	be automatically fixed. For example, rename any tables with names that are not
	Hive-conformant.
	. Run `kudu hms fix` tool to automatically fix all the remaining issues.
	+
	[source,bash]
	----
	$ sudo -u kudu kudu hms fix master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=<hive_metastore_uris> [--drop_orphan_hms_tables=<drops_orphan_hms_tables>] [--ignore_other_clusters=<ignore_other_clusters>]
	----
	+
	Example::
	+
	----
	$ sudo -u kudu kudu hms fix master-name-1:7051,master-name-2:7051,master-name-3:7051 --hive_metastore_uris=thrift://hive-metastore:9083 --ignore_other_clusters=false
	----
	+
	NOTE: The `--drop_orphan_hms_tables` argument indicates whether to drop orphan
	Hive Metastore tables that refer to non-existent Kudu tables. Due to
	link:https://issues.apache.org/jira/browse/KUDU-2883[KUDU-2883] this option may
	fail to drop HMS entries that have no table ID. A workaround to this is to drop
	the table via Impala Shell.

	. Recreate any external tables that were dropped when preparing for the upgrade
	by using Impala Shell.

	. Enable the Hive Metastore Integration as described
	<<enabling-the-hive-metastore-integration>>.