| </a></div><button type="button" class="clean-btn close announcementBarClose_iXyO" aria-label="Close"><svg viewBox="0 0 15 15" width="14" height="14" style="color:white"><g stroke="currentColor" stroke-width="3.1"><path d="M.75.75l13.5 13.5M14.25.75L.75 14.25"></path></g></svg></button></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner" style="padding:"><div class="navbar__items"><div class="navbar-left"><div class="navbar-logo-wrapper flex items-center"><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="https://cdnd.selectdb.com/images/logo.svg" alt="Apache Doris" class="themedImage_ToTc themedImage--light_HNdA"><img src="https://cdnd.selectdb.com/images/logo.svg" alt="Apache Doris" class="themedImage_ToTc themedImage--dark_i4oU"></div><b class="navbar__title text--truncate"></b></a></div><a class="navbar__item navbar__link" style="text-align:center" href="/docs/get-starting/quick-start">Docs</a><a class="navbar__item navbar__link" style="text-align:center" href="/blog">Blog</a><a class="navbar__item navbar__link" style="text-align:center" href="/users">Users</a><a href="https://github.com/apache/doris/discussions" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link" style="text-align:center">Discussions</a><a class="navbar__item navbar__link" style="text-align:center" href="/ecosystem/cluster-management">Ecosystem</a><a class="navbar__item navbar__link" style="text-align:center" href="/community/join-community">Community</a></div></div><div class="navbar__items navbar__items--right"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><div class="docs-search searchBox_H2mL"><div class="navbar__search searchBarContainer_PzyC"><input placeholder="Search" aria-label="Search" class="navbar__search-input navbarSearchInput_tb6T"><div class="loadingRing__K5d searchBarLoadingRing_e2f0"><div></div><div></div><div></div><div></div></div><div class="searchHintContainer_m7ml"><kbd class="searchHint_zuPL">ctrl</kbd><kbd class="searchHint_zuPL">K</kbd></div></div></div><div class="custom-navbar-item navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link"><svg class="icon-language" xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16" fill="none"><path d="M7.75756 14.3L10.5816 6.91667H11.8759L14.7 14.3H13.4057L12.7501 12.4167H9.74113L9.06873 14.3H7.75756ZM10.1109 11.35H12.3467L11.254 8.3H11.2036L10.1109 11.35ZM2.84908 12.45L1.97498 11.5833L5.11841 8.48333C4.72618 8.05 4.38439 7.60267 4.09302 7.14133C3.80165 6.68044 3.54389 6.19444 3.31976 5.68333H4.61412C4.80463 6.06111 5.00635 6.39711 5.21927 6.69133C5.43219 6.986 5.68434 7.29444 5.97571 7.61667C6.43519 7.12778 6.81621 6.62511 7.11879 6.10867C7.42137 5.59178 7.67352 5.03889 7.87523 4.45H1V3.23333H5.33694V2H6.58087V3.23333H10.9178V4.45H9.11916C8.89503 5.18333 8.59805 5.89155 8.22824 6.57467C7.85842 7.25822 7.39895 7.90555 6.84983 8.51667L8.3459 10.0167L7.87523 11.2833L5.95891 9.38333L2.84908 12.45Z" fill="#4C576C"></path></svg></a><ul class="dropdown__menu"><li><a href="/docs/dev/lakehouse/datalake-analytics/hive" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" style="text-align:center">English</a></li><li><a href="/zh-CN/docs/dev/lakehouse/datalake-analytics/hive" target="_self" rel="noopener noreferrer" class="dropdown__link" style="text-align:center">中文</a></li></ul></div><div class="custom-navbar-item navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link" aria-haspopup="true" aria-expanded="false" role="button" href="/docs/dev/get-starting/what-is-apache-doris"><span class="text-sm">Versions: <!-- -->Dev</span></a><ul class="dropdown__menu"><li><a aria-current="page" class="dropdown__link dropdown__link--active" style="text-align:center" href="/docs/dev/lakehouse/datalake-analytics/hive">Dev</a></li><li><a class="dropdown__link" style="text-align:center" href="/docs/lakehouse/datalake-analytics/hive">2.1</a></li><li><a class="dropdown__link" style="text-align:center" href="/docs/2.0/get-starting/what-is-apache-doris">2.0</a></li><li><a class="dropdown__link" style="text-align:center" href="/docs/1.2/get-starting/">1.2</a></li></ul></div><a class="navbar__item navbar__link header-right-button-primary navbar-download-mobile" style="text-align:center" href="/download">Download</a><a href="https://github.com/apache/doris" target="_blank" rel="noopener noreferrer" class="github-btn desktop header-right-button-github"></a><a href="https://join.slack.com/t/apachedoriscommunity/shared_invite/zt-2kl08hzc0-SPJe4VWmL_qzrFd2u2XYQA" target="_blank" rel="noopener noreferrer" class="slack-btn desktop header-right-button-slack"></a><a class="header-right-button-primary navbar-download-desktop" href="/download">Download</a></div></div><div class="navbar__bottom"></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="__docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#__docusaurus_skipToContent_fallback">Skip to main content</a></div><div class="main-wrapper docsWrapper_BCFX"><button aria-label="Scroll back to top" class="clean-btn theme-back-to-top-button backToTopButton_sjWU" type="button"></button><div class="docPage__5DB"><aside class="theme-doc-sidebar-container docSidebarContainer_b6E3"><div class="sidebarViewport_Xe31"><div class="sidebar"><nav aria-label="Docs sidebar" class="menu custom-scrollbar menu_Y1UP menuWithAnnouncementBar_fPny"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/get-starting/what-is-apache-doris">Getting Started</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/install/source-install/compilation-with-docker">Installation and Deployment</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/db-connect/database-connect">Database Connection</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/table-design/data-type">Data Table Design</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/data-operate/import/load-manual">Data Manipulation</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/query/query-data/mysql-compatibility">Data Queries</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret menu__link--active" aria-expanded="true" href="/docs/dev/lakehouse/lakehouse-overview">Lakehouse</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list menu__list_level_2"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/lakehouse-overview">Lakehouse Overview</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link_level_2 menu__link--sublist menu__link--sublist-caret menu__link--active" aria-expanded="true" tabindex="0" href="/docs/dev/lakehouse/datalake-analytics/hive">Analyzing Data Lake</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/dev/lakehouse/datalake-analytics/hive">Hive</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/datalake-analytics/hudi">Hudi</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/datalake-analytics/iceberg">Iceberg</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/datalake-analytics/paimon">Paimon</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/datalake-analytics/dlf">Alibaba Cloud DLF</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/datalake-analytics/tpch">TPCH</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/datalake-analytics/tpcds">TPCDS</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link_level_2 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" tabindex="0" href="/docs/dev/lakehouse/datalake-building/hive-build">Building Data Lake</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link_level_2 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" tabindex="0" href="/docs/dev/lakehouse/database/jdbc">Analyzing Database</a></div></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/file">Analyzing File</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/filecache">File Caches</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/external-statistics">External Table Statistics</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/cloud-auth">Cloud Service Authentication</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/dev/lakehouse/sql-dialect">SQL Dialect Convertor</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/admin-manual/cluster-management/upgrade">Managing Doris</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/ecosystem/spark-doris-connector">Ecosystem</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/sql-manual/sql-functions/array-functions/array">SQL Reference</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/separation-of-storage-and-compute/overview">Separation of Storage and Compute</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/benchmark/ssb">Benchmark</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/faq/install-faq">FAQ</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a id="menu__group" class="menu__link menu__link_level_1 menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/dev/releasenotes/release-2.1.3">Release notes</a></div></li></ul></nav></div></div></aside><main class="docMainContainer_gTbr"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_z5aJ"><div class="theme-doc-version-banner alert alert--warning margin-bottom--md" role="alert"><div>This is unreleased documentation for <!-- -->Apache Doris<!-- --> <b>Dev</b> version.</div><div class="margin-top--md">For up-to-date documentation, see the <b><a href="/docs/lakehouse/datalake-analytics/hive">latest version</a></b> (<!-- -->2.1<!-- -->).</div></div><div class="docItemContainer_c0TR"><article><nav class="theme-doc-breadcrumbs breadcrumbsContainer_Z_bl" aria-label="Breadcrumbs"><ul class="breadcrumbs" itemscope="" itemtype="https://schema.org/BreadcrumbList"><li class="breadcrumbs__item"><a aria-label="Home page" class="breadcrumbs__link" href="/"><svg viewBox="0 0 24 24" class="breadcrumbHomeIcon_YNFT"><path d="M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z" fill="currentColor"></path></svg></a></li><li class="breadcrumbs__item"><span class="breadcrumbs__link">Lakehouse</span><meta itemprop="position" content="1"></li><li class="breadcrumbs__item"><span class="breadcrumbs__link">Analyzing Data Lake</span><meta itemprop="position" content="2"></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item breadcrumbs__item--active"><span class="breadcrumbs__link" itemprop="name">Hive</span><meta itemprop="position" content="3"></li></ul></nav><div class="tocCollapsible_ETCw theme-doc-toc-mobile tocMobile_ITEo"><button type="button" class="clean-btn tocCollapsibleButton_TO0P">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Hive</h1></header><h1>Hive</h1><p>By connecting to Hive Metastore, or a metadata service compatible with Hive Metatore, Doris can automatically obtain Hive database table information and perform data queries.</p><p>In addition to Hive, many other systems also use the Hive Metastore to store metadata. So through Hive Catalog, we can not only access Hive, but also access systems that use Hive Metastore as metadata storage. Such as Iceberg, Hudi, etc.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="terms-and-conditions">Terms and Conditions<a href="#terms-and-conditions" class="hash-link" aria-label="Direct link to Terms and Conditions" title="Direct link to Terms and Conditions"></a></h2><ol><li>Need to put core-site.xml, hdfs-site.xml and hive-site.xml in the conf directory of FE and BE. First read the hadoop configuration file in the conf directory, and then read the related to the environment variable <code>HADOOP_CONF_DIR</code> configuration file.</li><li>hive supports version 1/2/3.</li><li>Support Managed Table and External Table and part of Hive View.</li><li>Can identify hive, iceberg, hudi metadata stored in Hive Metastore.</li><li>If the Hadoop node is configured with hostname, please ensure to add the corresponding mapping relationship to the /etc/hosts file.</li></ol><h2 class="anchor anchorWithStickyNavbar_LWe7" id="create-catalog">Create Catalog<a href="#create-catalog" class="hash-link" aria-label="Direct link to Create Catalog" title="Direct link to Create Catalog"></a></h2><h3 class="anchor anchorWithStickyNavbar_LWe7" id="hive-on-hdfs">Hive On HDFS<a href="#hive-on-hdfs" class="hash-link" aria-label="Direct link to Hive On HDFS" title="Direct link to Hive On HDFS"></a></h3><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'type'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'hms'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.uris'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'thrift://172.0.0.1:9083'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.username'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>In addition to the two required parameters of <code>type</code> and <code>hive.metastore.uris</code>, more parameters can be passed to pass the information required for the connection.</p><p>Most of this properties can be found in <code>core-site.xml</code>, <code>hdfs-site.xml</code> and <code>hive-site.xml</code>.</p><p>If HDFS HA information is provided, the example is as follows:</p><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'type'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'hms'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.uris'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'thrift://172.0.0.1:9083'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.username'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.nameservices'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'your-nameservice'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.ha.namenodes.your-nameservice'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'nn1,nn2'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.namenode.rpc-address.your-nameservice.nn1'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'172.21.0.2:8088'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.namenode.rpc-address.your-nameservice.nn2'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'172.21.0.3:8088'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.client.failover.proxy.provider.your-nameservice'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><blockquote><p>About Kerberos, please see <code>Connect to Kerberos enabled Hive</code>.</p></blockquote><h3 class="anchor anchorWithStickyNavbar_LWe7" id="hive-on-viewfs">Hive On ViewFS<a href="#hive-on-viewfs" class="hash-link" aria-label="Direct link to Hive On ViewFS" title="Direct link to Hive On ViewFS"></a></h3><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'type'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'hms'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.uris'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'thrift://172.0.0.1:9083'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.username'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.nameservices'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'your-nameservice'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.ha.namenodes.your-nameservice'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'nn1,nn2'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.namenode.rpc-address.your-nameservice.nn1'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'172.21.0.2:8088'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.namenode.rpc-address.your-nameservice.nn2'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'172.21.0.3:8088'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.client.failover.proxy.provider.your-nameservice'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'fs.defaultFS'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'viewfs://your-cluster'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'fs.viewfs.mounttable.your-cluster.link./ns1'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hdfs://your-nameservice/'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'fs.viewfs.mounttable.your-cluster.homedir'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'/ns1'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>ViewFs related parameters can be added to the catalog configuration as above, or added to <code>conf/core-site.xml</code>.</p><p>How ViewFs works and parameter configuration, please refer to relevant hadoop documents, for example, <a href="https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ViewFs.html" target="_blank" rel="noopener noreferrer">https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ViewFs.html</a></p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="hive-on-juicefs">Hive On JuiceFS<a href="#hive-on-juicefs" class="hash-link" aria-label="Direct link to Hive On JuiceFS" title="Direct link to Hive On JuiceFS"></a></h3><p>Data is stored in JuiceFS, examples are as follows:</p><p>(Need to put <code>juicefs-hadoop-x.x.x.jar</code> under <code>fe/lib/</code> and <code>apache_hdfs_broker/lib/</code>)</p><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'type'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'hms'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.uris'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'thrift://172.0.0.1:9083'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.username'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'root'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'fs.jfs.impl'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'io.juicefs.JuiceFileSystem'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'fs.AbstractFileSystem.jfs.impl'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'io.juicefs.JuiceFS'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'juicefs.meta'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'xxx'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="hive-on-s3">Hive On S3<a href="#hive-on-s3" class="hash-link" aria-label="Direct link to Hive On S3" title="Direct link to Hive On S3"></a></h3><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"type"</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"hms"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"hive.metastore.uris"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"thrift://172.0.0.1:9083"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"s3.endpoint"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"s3.us-east-1.amazonaws.com"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"s3.region"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"us-east-1"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"s3.access_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"ak"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"s3.secret_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"sk"</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"use_path_style"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"true"</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Options:</p><ul><li>s3.connection.maximum: s3 maximum connection number, default 50</li><li>s3.connection.request.timeout: s3 request timeout, default 3000ms</li><li>s3.connection.timeout: s3 connection timeout, default 1000ms</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="hive-on-oss">Hive On OSS<a href="#hive-on-oss" class="hash-link" aria-label="Direct link to Hive On OSS" title="Direct link to Hive On OSS"></a></h3><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"type"</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"hms"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"hive.metastore.uris"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"thrift://172.0.0.1:9083"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"oss.endpoint"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"oss.oss-cn-beijing.aliyuncs.com"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"oss.access_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"ak"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"oss.secret_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"sk"</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="hive-on-obs">Hive On OBS<a href="#hive-on-obs" class="hash-link" aria-label="Direct link to Hive On OBS" title="Direct link to Hive On OBS"></a></h3><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"type"</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"hms"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"hive.metastore.uris"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"thrift://172.0.0.1:9083"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"obs.endpoint"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"obs.cn-north-4.myhuaweicloud.com"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"obs.access_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"ak"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"obs.secret_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"sk"</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="hive-on-cos">Hive On COS<a href="#hive-on-cos" class="hash-link" aria-label="Direct link to Hive On COS" title="Direct link to Hive On COS"></a></h3><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"type"</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"hms"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"hive.metastore.uris"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"thrift://172.0.0.1:9083"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"cos.endpoint"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"cos.ap-beijing.myqcloud.com"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"cos.access_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"ak"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"cos.secret_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"sk"</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="hive-with-glue">Hive With Glue<a href="#hive-with-glue" class="hash-link" aria-label="Direct link to Hive With Glue" title="Direct link to Hive With Glue"></a></h3><blockquote><p>When connecting Glue, if it's not on the EC2 environment, need copy the <code>~/.aws</code> from the EC2 environment to the current environment. And can also download and configure the <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" target="_blank" rel="noopener noreferrer">AWS Cli tools</a>, which also creates the <code>.aws</code> directory under the current user directory.</p></blockquote><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"type"</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"hms"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"hive.metastore.type"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"glue"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"glue.endpoint"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"https://glue.us-east-1.amazonaws.com"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"glue.access_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"ak"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"glue.secret_key"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"sk"</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="metadata-cache--refresh">Metadata Cache & Refresh<a href="#metadata-cache--refresh" class="hash-link" aria-label="Direct link to Metadata Cache & Refresh" title="Direct link to Metadata Cache & Refresh"></a></h2><p>For Hive Catalog, 4 types of metadata are cached in Doris:</p><ol><li>Table structure: cache table column information, etc.</li><li>Partition value: Cache the partition value information of all partitions of a table.</li><li>Partition information: Cache the information of each partition, such as partition data format, partition storage location, partition value, etc.</li><li>File information: Cache the file information corresponding to each partition, such as file path location, etc.</li></ol><p>The above cache information will not be persisted to Doris, so operations such as restarting Doris's FE node, switching masters, etc. may cause the cache to become invalid. After the cache expires, Doris will directly access the Hive MetaStore to obtain information and refill the cache.</p><p>Metadata cache can be updated automatically, manually, or configured with TTL (Time-to-Live) according to user needs.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="default-behavior-and-ttl">Default behavior and TTL<a href="#default-behavior-and-ttl" class="hash-link" aria-label="Direct link to Default behavior and TTL" title="Direct link to Default behavior and TTL"></a></h3><p>By default, the metadata cache expires 10 minutes after it is first accessed. This time is determined by the configuration parameter <code>external_cache_expire_time_minutes_after_access</code> in fe.conf. (Note that in versions 2.0.1 and earlier, the default value for this parameter was 1 day).</p><p>For example, if the user accesses the metadata of table A for the first time at 10:00, then the metadata will be cached and will automatically expire after 10:10. If the user accesses the same metadata again at 10:11, Doris will directly access the Hive MetaStore to obtain information and refill the cache.</p><p><code>external_cache_expire_time_minutes_after_access</code> affects all 4 caches under Catalog.</p><p>For the <code>INSERT INTO OVERWRITE PARTITION</code> operation commonly used in Hive, you can also timely update the <code>File Information Cache</code> by configuring the TTL of the <code>File Information Cache</code>:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">CREATE CATALOG hive PROPERTIES (</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> 'type'='hms',</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> 'hive.metastore.uris' = 'thrift://172.0.0.1:9083',</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> 'file.meta.cache.ttl-second' = '60'</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">);</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>In the above example, <code>file.meta.cache.ttl-second</code> is set to 60 seconds, and the cache will expire after 60 seconds. This parameter will only affect the <code>file information cache</code>.</p><p>You can also set this value to 0 to disable file caching, which will fetch file information directly from the Hive MetaStore every time.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="manual-refresh">Manual refresh<a href="#manual-refresh" class="hash-link" aria-label="Direct link to Manual refresh" title="Direct link to Manual refresh"></a></h3><p>Users need to manually refresh the metadata through the <a href="/docs/dev/sql-manual/sql-statements/Utility-Statements/REFRESH">REFRESH</a> command.</p><ol><li><p>REFRESH CATALOG: Refresh the specified Catalog.</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">REFRESH CATALOG ctl1 PROPERTIES("invalid_cache" = "true");</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p> This command will refresh the database list, table list, and all cache information of the specified Catalog.</p><p> <code>invalid_cache</code> indicates whether to flush the cache. Defaults to true. If it is false, only the database and table list of the catalog will be refreshed, but the cache information will not be refreshed. This parameter is applicable when the user only wants to synchronize newly added or deleted database/table information.</p></li><li><p>REFRESH DATABASE: Refresh the specified Database.</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">REFRESH DATABASE [ctl.]db1 PROPERTIES("invalid_cache" = "true");</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p> This command will refresh the table list of the specified Database and all cached information under the Database.</p><p> The meaning of the <code>invalid_cache</code> attribute is the same as above. Defaults to true. If false, only the Database's table list will be refreshed, not cached information. This parameter is suitable for users who only want to synchronize newly added or deleted table information.</p></li><li><p>REFRESH TABLE: Refresh the specified Table.</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">REFRESH TABLE [ctl.][db.]tbl1;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p> This command will refresh all cache information under the specified Table.</p></li></ol><h3 class="anchor anchorWithStickyNavbar_LWe7" id="regular-refresh">Regular refresh<a href="#regular-refresh" class="hash-link" aria-label="Direct link to Regular refresh" title="Direct link to Regular refresh"></a></h3><p>Users can set the scheduled refresh of the Catalog when creating the Catalog.</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">CREATE CATALOG hive PROPERTIES (</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> 'type'='hms',</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> 'hive.metastore.uris' = 'thrift://172.0.0.1:9083',</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> 'metadata_refresh_interval_sec' = '600'</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">);</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>In the above example, <code>metadata_refresh_interval_sec</code> means refreshing the Catalog every 600 seconds. Equivalent to automatically executing every 600 seconds:</p><p><code>REFRESH CATALOG ctl1 PROPERTIES("invalid_cache" = "true");</code></p><p>The scheduled refresh interval must not be less than 5 seconds.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="auto-refresh">Auto Refresh<a href="#auto-refresh" class="hash-link" aria-label="Direct link to Auto Refresh" title="Direct link to Auto Refresh"></a></h3><p>Currently, Doris only supports automatic update of metadata in Hive Metastore (HMS). It perceives changes in metadata by the FE node which regularly reads the notification events from HMS. The supported events are as follows:</p><table><thead><tr><th align="left">Event</th><th align="left">Corresponding Update Operation</th></tr></thead><tbody><tr><td align="left">CREATE DATABASE</td><td align="left">Create a database in the corresponding catalog.</td></tr><tr><td align="left">DROP DATABASE</td><td align="left">Delete a database in the corresponding catalog.</td></tr><tr><td align="left">ALTER DATABASE</td><td align="left">Such alterations mainly include changes in properties, comments, or storage location of databases. They do not affect Doris' queries in External Catalogs so they will not be synchronized.</td></tr><tr><td align="left">CREATE TABLE</td><td align="left">Create a table in the corresponding database.</td></tr><tr><td align="left">DROP TABLE</td><td align="left">Delete a table in the corresponding database, and invalidate the cache of that table.</td></tr><tr><td align="left">ALTER TABLE</td><td align="left">If it is a renaming, delete the table of the old name, and then create a new table with the new name; otherwise, invalidate the cache of that table.</td></tr><tr><td align="left">ADD PARTITION</td><td align="left">Add a partition to the cached partition list of the corresponding table.</td></tr><tr><td align="left">DROP PARTITION</td><td align="left">Delete a partition from the cached partition list of the corresponding table, and invalidate the cache of that partition.</td></tr><tr><td align="left">ALTER PARTITION</td><td align="left">If it is a renaming, delete the partition of the old name, and then create a new partition with the new name; otherwise, invalidate the cache of that partition.</td></tr></tbody></table><blockquote><p>After data ingestion, changes in partition tables will follow the <code>ALTER PARTITION</code> logic, while those in non-partition tables will follow the <code>ALTER TABLE</code> logic.</p><p>If changes are conducted on the file system directly instead of through the HMS, the HMS will not generate an event. As a result, such changes will not be perceived by Doris.</p></blockquote><p>The automatic update feature involves the following parameters in fe.conf:</p><ol><li><code>enable_hms_events_incremental_sync</code>: This specifies whether to enable automatic incremental synchronization for metadata, which is disabled by default. </li><li><code>hms_events_polling_interval_ms</code>: This specifies the interval between two readings, which is set to 10000 by default. (Unit: millisecond) </li><li><code>hms_events_batch_size_per_rpc</code>: This specifies the maximum number of events that are read at a time, which is set to 500 by default.</li></ol><p>To enable automatic update(Excluding Huawei MRS), you need to modify the hive-site.xml of HMS and then restart HMS and HiveServer2:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain"><property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <name>hive.metastore.event.db.notification.api.auth</name></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <value>false</value></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"><property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <name>hive.metastore.dml.events</name></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <value>true</value></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"><property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <name>hive.metastore.transactional.event.listeners</name></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Huawei's MRS needs to change hivemetastore-site.xml and restart HMS and HiveServer2:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain"><property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <name>metastore.transactional.event.listeners</name></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></property></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Note: Value is appended with commas separated from the original value, not overwritten.For example, the default configuration for MRS 3.1.0 is</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain"><property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <name>metastore.transactional.event.listeners</name></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <value>com.huawei.bigdata.hive.listener.TableKeyFileManagerListener,org.apache.hadoop.hive.metastore.listener.FileAclListener</value></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></property></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We need to change to</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain"><property></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <name>metastore.transactional.event.listeners</name></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> <value>com.huawei.bigdata.hive.listener.TableKeyFileManagerListener,org.apache.hadoop.hive.metastore.listener.FileAclListener,org.apache.hive.hcatalog.listener.DbNotificationListener</value></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></property></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><blockquote><p>Note: To enable automatic update, whether for existing Catalogs or newly created Catalogs, all you need is to set <code>enable_hms_events_incremental_sync</code> to <code>true</code>, and then restart the FE node. You don't need to manually update the metadata before or after the restart.</p></blockquote><h2 class="anchor anchorWithStickyNavbar_LWe7" id="hive-version">Hive Version<a href="#hive-version" class="hash-link" aria-label="Direct link to Hive Version" title="Direct link to Hive Version"></a></h2><p>Doris can correctly access the Hive Metastore in different Hive versions. By default, Doris will access the Hive Metastore with a Hive 2.3 compatible interface.</p><p>If you meet error message like <code>Invalid method name: 'get_table_req'</code>, which means the hive version is mismatch.</p><p>You can specify the hive version when creating the Catalog. If accessing Hive 1.1.0 version:</p><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'type'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'hms'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.uris'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'thrift://172.0.0.1:9083'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.version'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'1.1.0'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="column-type-mapping">Column type mapping<a href="#column-type-mapping" class="hash-link" aria-label="Direct link to Column type mapping" title="Direct link to Column type mapping"></a></h2><p>For Hive/Iceberge/Hudi</p><table><thead><tr><th>HMS Type</th><th>Doris Type</th><th>Comment</th></tr></thead><tbody><tr><td>boolean</td><td>boolean</td><td></td></tr><tr><td>tinyint</td><td>tinyint</td><td></td></tr><tr><td>smallint</td><td>smallint</td><td></td></tr><tr><td>int</td><td>int</td><td></td></tr><tr><td>bigint</td><td>bigint</td><td></td></tr><tr><td>date</td><td>date</td><td></td></tr><tr><td>timestamp</td><td>datetime</td><td></td></tr><tr><td>float</td><td>float</td><td></td></tr><tr><td>double</td><td>double</td><td></td></tr><tr><td>char</td><td>char</td><td></td></tr><tr><td>varchar</td><td>varchar</td><td></td></tr><tr><td>decimal</td><td>decimal</td><td></td></tr><tr><td><code>array<type></code></td><td><code>array<type></code></td><td>support nested type, for example <code>array<array<int>></code></td></tr><tr><td><code>map<KeyType, ValueType></code></td><td><code>map<KeyType, ValueType></code></td><td>support nested type, for example <code>map<string, array<int>></code></td></tr><tr><td><code>struct<col1: Type1, col2: Type2, ...></code></td><td><code>struct<col1: Type1, col2: Type2, ...></code></td><td>support nested type, for example <code>struct<col1: array<int>, col2: map<int, date>></code></td></tr><tr><td>other</td><td>unsupported</td><td></td></tr></tbody></table><blockquote><p>Note: Whether to truncate char or varchar columns according to the schema of the hive table</p></blockquote><blockquote><p>If the session variable <code>truncate_char_or_varchar_columns</code> is enabled, when the maximum length of the char or varchar column in the schema of the hive table is inconsistent with the schema in the underlying parquet or orc file, it will be truncated according to the maximum length of the hive table column.</p></blockquote><blockquote><p>The variable default is false.</p></blockquote><h2 class="anchor anchorWithStickyNavbar_LWe7" id="access-hms-with-broker">Access HMS with broker<a href="#access-hms-with-broker" class="hash-link" aria-label="Direct link to Access HMS with broker" title="Direct link to Access HMS with broker"></a></h2><p>Add following setting when creating an HMS catalog, file splitting and scanning for Hive external table will be completed by broker named <code>test_broker</code></p><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token string" style="color:rgb(255, 121, 198)">"broker.name"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"test_broker"</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Doris has implemented Broker query support for HMS Catalog Iceberg based on the Iceberg <code>FileIO</code> interface. If needed, the following configuration can be added when creating the HMS Catalog.</p><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token string" style="color:rgb(255, 121, 198)">"io-impl"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"org.apache.doris.datasource.iceberg.broker.IcebergBrokerIO"</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="integrate-with-apache-ranger">Integrate with Apache Ranger<a href="#integrate-with-apache-ranger" class="hash-link" aria-label="Direct link to Integrate with Apache Ranger" title="Direct link to Integrate with Apache Ranger"></a></h2><p>Apache Ranger is a security framework for monitoring, enabling services, and comprehensive data security access management on the Hadoop platform.</p><p>Doris supports using Apache Ranger for authentication for a specified External Hive Catalog.</p><p>Currently, authentication of database, table, and column is supported. Functions such as encryption, row policy, and data masks are not currently supported.</p><p>To use Apache Ranger to authenticate the entire Doris cluster service, please refer to <a href="/docs/dev/admin-manual/auth/ranger">Apache Ranger</a></p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="settings">Settings<a href="#settings" class="hash-link" aria-label="Direct link to Settings" title="Direct link to Settings"></a></h3><p>To connect to the Hive Metastore with Ranger permission verification enabled, you need to add configuration & configuration environment:</p><ol><li><p>When creating a Catalog, add:</p><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token string" style="color:rgb(255, 121, 198)">"access_controller.properties.ranger.service.name"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"hive"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token string" style="color:rgb(255, 121, 198)">"access_controller.class"</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessControllerFactory"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><blockquote><p>Note:</p><p><code>access_controller.properties.ranger.service.name</code> refers to the type of service, such as <code>hive</code>, <code>hdfs</code>, etc. It is not the value of <code>ranger.plugin.hive.service.name</code> in the configuration file.</p></blockquote></li><li><p>Configure all FE environments:</p><ol><li><p>Copy the configuration files ranger-hive-audit.xml, ranger-hive-security.xml, and ranger-policymgr-ssl.xml under the HMS conf directory to the FE conf directory.</p></li><li><p>Modify the properties of ranger-hive-security.xml, the reference configuration is as follows:</p><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token operator"><</span><span class="token plain">?xml version</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"1.0"</span><span class="token plain"> encoding</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"UTF-8"</span><span class="token plain">?</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token operator"><</span><span class="token plain">?xml</span><span class="token operator">-</span><span class="token plain">stylesheet </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">type</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"text/xsl"</span><span class="token plain"> href</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"configuration.xsl"</span><span class="token plain">?</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token operator"><</span><span class="token plain">configuration</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)">#The directory for caching permission data, needs to be writable</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">plugin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">policy</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">cache</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">dir</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token operator">/</span><span class="token plain">mnt</span><span class="token operator">/</span><span class="token plain">datadisk0</span><span class="token operator">/</span><span class="token plain">zhangdong</span><span class="token operator">/</span><span class="token plain">rangerdata</span><span class="token operator"><</span><span class="token operator">/</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)">#The time interval for periodically pulling permission data</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">plugin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">policy</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">pollIntervalMs</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token number">30000</span><span class="token operator"><</span><span class="token operator">/</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">plugin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">policy</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">rest</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">client</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">connection</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">timeoutMs</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token number">60000</span><span class="token operator"><</span><span class="token operator">/</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">plugin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">policy</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">rest</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">client</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">read</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">timeoutMs</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token number">60000</span><span class="token operator"><</span><span class="token operator">/</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">plugin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">policy</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">rest</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">ssl</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">config</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">file</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token operator"><</span><span class="token operator">/</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">plugin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">policy</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">rest</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">url</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain">http:</span><span class="token comment" style="color:rgb(98, 114, 164)">//172.21.0.32:6080</value></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">plugin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">policy</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">source</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">impl</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain">org</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">apache</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">admin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">client</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">RangerAdminRESTClient</span><span class="token operator"><</span><span class="token operator">/</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">ranger</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">plugin</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">service</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">name</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain">hive</span><span class="token operator"><</span><span class="token operator">/</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain">xasecure</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">hive</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">update</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain">xapolicies</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">on</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">grant</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">revoke</span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">name</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token boolean">true</span><span class="token operator"><</span><span class="token operator">/</span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">value</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">property</span><span class="token operator">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token operator"><</span><span class="token operator">/</span><span class="token plain">configuration</span><span class="token operator">></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div></li><li><p>In order to obtain the log of Ranger authentication itself, add the configuration file log4j.properties in the <code><doris_home>/conf</code> directory.</p></li><li><p>Restart FE.</p></li></ol></li></ol><h3 class="anchor anchorWithStickyNavbar_LWe7" id="best-practices">Best Practices<a href="#best-practices" class="hash-link" aria-label="Direct link to Best Practices" title="Direct link to Best Practices"></a></h3><ol><li><p>Create user user1 on the ranger side and authorize the query permission of db1.table1.col1</p></li><li><p>Create role role1 on the ranger side and authorize the query permission of db1.table1.col2</p></li><li><p>Create a user user1 with the same name in doris, user1 will directly have the query authority of db1.table1.col1</p></li><li><p>Create role1 with the same name in doris, and assign role1 to user1, user1 will have the query authority of db1.table1.col1 and col2 at the same time</p></li><li><p>The permissions of Admin and Root users are not controlled by Apache Ranger.</p></li></ol><h2 class="anchor anchorWithStickyNavbar_LWe7" id="connect-to-kerberos-enabled-hive">Connect to Kerberos enabled Hive<a href="#connect-to-kerberos-enabled-hive" class="hash-link" aria-label="Direct link to Connect to Kerberos enabled Hive" title="Direct link to Connect to Kerberos enabled Hive"></a></h2><p>This section mainly introduces how to connect to a Hive + HDFS cluster with Kerberos authentication enabled.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="environment-preparation">Environment preparation<a href="#environment-preparation" class="hash-link" aria-label="Direct link to Environment preparation" title="Direct link to Environment preparation"></a></h3><ul><li><p><code>krb5.conf</code></p><p><code>krb5.conf</code> is the configuration file for the Kerberos authentication protocol. This file needs to be deployed on all FE and BE nodes. And ensure that the Doris cluster can connect to the KDC service recorded in this file.</p><p>By default, this file is located in the <code>/etc</code> directory of the Hadoop cluster. But please contact the Hadoop cluster administrator to obtain the correct <code>krb5.conf</code> file and deploy it to the <code>/etc</code> directory of all FE and BE nodes.</p><p>Note that in some cases the file location of <code>krb5.conf</code> may depend on the environment variable <code>KRB5_CONFIG</code> or the <code>-Djava.security.krb5.conf</code> in the JVM parameters. Please check these properties to determine the exact location of <code>krb5.conf</code>.</p></li><li><p>JVM parameters</p><p>Please add the following options to the JVM of FE and BE (located in <code>fe.conf</code> and <code>be.conf</code>):</p><ul><li><code>-Djavax.security.auth.useSubjectCredsOnly=false</code></li><li><code>-Dsun.security.krb5.debug=true</code></li></ul><p>And restart the FE and BE nodes to ensure it takes effect.</p></li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="catalog-configuration">Catalog configuration<a href="#catalog-configuration" class="hash-link" aria-label="Direct link to Catalog configuration" title="Direct link to Catalog configuration"></a></h3><p>Normally, to connect to a Kerberos enabled Hive cluster, you need to add the following attributes to the Catalog:</p><ul><li><code>"hadoop.security.authentication" = "kerberos"</code>: Enable kerberos authentication method.</li><li><code>"hadoop.kerberos.principal" = "your_principal"</code>: The principal of the HDFS namenode. Typically the <code>dfs.namenode.kerberos.principal</code> configuration of <code>hdfs-site.xml</code>.</li><li><code>"hadoop.kerberos.keytab" = "/path/to/your_keytab"</code>: keytab file of HDFS namenode. Typically the <code>dfs.namenode.keytab.file</code> configuration of <code>hdfs-site.xml</code>. Note that this file needs to be deployed to the same directory of all FE and BE nodes (can be customized).</li><li><code>"yarn.resourcemanager.principal" = "your_principal"</code>: The principal of Yarn Resource Manager, which can be found in <code>yarn-site.xml</code>.</li><li><code>"hive.metastore.kerberos.principal" = "your_principal"</code>: The principal of the Hive metastore. Can be found in <code>hive-site.xml</code>.</li></ul><blockquote><p>Note: Suggest to use <code>kinit -kt your_principal /path/to/your_keytab</code> 以及 <code>klist -k /path/to/your_keytab</code> to get the ticket or check its validation.</p></blockquote><p>Examples are as follows:</p><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive_krb PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'type'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'hms'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.uris'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'thrift://172.0.0.1:9083'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.sasl.enabled'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'true'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.kerberos.principal'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'your-hms-principal'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.security.authentication'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'kerberos'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.kerberos.keytab'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'/your-keytab-filepath/your.keytab'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.kerberos.principal'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'your-principal@YOUR.COM'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'yarn.resourcemanager.principal'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'your-rm-principal'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">CREATE</span><span class="token plain"> CATALOG hive_krb_ha PROPERTIES </span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'type'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'hms'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.uris'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'thrift://172.0.0.1:9083'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.sasl.enabled'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'true'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hive.metastore.kerberos.principal'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'your-hms-principal'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.security.authentication'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'kerberos'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.kerberos.keytab'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'/your-keytab-filepath/your.keytab'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'hadoop.kerberos.principal'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'your-principal@YOUR.COM'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'yarn.resourcemanager.principal'</span><span class="token plain"> </span><span class="token operator">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'your-rm-principal'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.nameservices'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'your-nameservice'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.ha.namenodes.your-nameservice'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'nn1,nn2'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.namenode.rpc-address.your-nameservice.nn1'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'172.21.0.2:8088'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.namenode.rpc-address.your-nameservice.nn2'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'172.21.0.3:8088'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'dfs.client.failover.proxy.provider.your-nameservice'</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">'org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="troubleshooting">Troubleshooting<a href="#troubleshooting" class="hash-link" aria-label="Direct link to Troubleshooting" title="Direct link to Troubleshooting"></a></h3><p>In case of Kerberos authentication problems, after setting the JVM parameter <code>-Dsun.security.krb5.debug=true</code>, Kerberos authentication related information will be printed in <code>fe.out</code> or <code>be.out</code>. You can refer to the related errors in <a href="/docs/dev/faq/lakehouse-faq">FAQ</a> for troubleshooting.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="hive-transactional-tables">Hive Transactional Tables<a href="#hive-transactional-tables" class="hash-link" aria-label="Direct link to Hive Transactional Tables" title="Direct link to Hive Transactional Tables"></a></h2><p>Hive transactional tables are tables in Hive that support ACID (Atomicity, Consistency, Isolation, Durability) semantics. For more details, you can refer to: <a href="https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions" target="_blank" rel="noopener noreferrer">Hive Transactions</a>.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="supported-operations-for-hive-transactional-tables">Supported Operations for Hive Transactional Tables:<a href="#supported-operations-for-hive-transactional-tables" class="hash-link" aria-label="Direct link to Supported Operations for Hive Transactional Tables:" title="Direct link to Supported Operations for Hive Transactional Tables:"></a></h3><table><thead><tr><th>Transactional Table Type</th><th>Supported Operations in Hive</th><th>Hive Table Properties</th><th>Supported Hive Versions</th></tr></thead><tbody><tr><td>Full-ACID Transactional Table</td><td>Supports insert, update, delete operations</td><td>'transactional'='true', 'transactional_properties'='insert_only'</td><td>3.x, 2.x (requires major compaction in Hive before loading)</td></tr><tr><td>Insert-Only Transactional Table</td><td>Supports only Insert operations</td><td>'transactional'='true'</td><td>3.x, 2.x</td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="current-limitations">Current Limitations:<a href="#current-limitations" class="hash-link" aria-label="Direct link to Current Limitations:" title="Direct link to Current Limitations:"></a></h3><p>Currently, it does not support scenarios involving Original Files. |