| <link rel="stylesheet" href="https://at-ui.github.io/feather-font/css/iconfont.css"><title data-react-helmet="true">Basic Configurations | Apache Hudi</title><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" property="og:url" content="https://hudi.apache.org/docs/0.14.0/basic_configurations"><meta data-react-helmet="true" name="docsearch:language" content="en"><meta data-react-helmet="true" name="docsearch:version" content="0.14.0"><meta data-react-helmet="true" name="docsearch:docusaurus_tag" content="docs-default-0.14.0"><meta data-react-helmet="true" name="keywords" content="apache hudi, data lake, lakehouse, big data, apache spark, apache flink, presto, trino, analytics, data engineering"><meta data-react-helmet="true" property="og:title" content="Basic Configurations | Apache Hudi"><meta data-react-helmet="true" name="description" content="This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the All Configurations page."><meta data-react-helmet="true" property="og:description" content="This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the All Configurations page."><link data-react-helmet="true" rel="icon" href="/assets/images/favicon.ico"><link data-react-helmet="true" rel="canonical" href="https://hudi.apache.org/docs/0.14.0/basic_configurations"><link data-react-helmet="true" rel="alternate" href="https://hudi.apache.org/docs/0.14.0/basic_configurations" hreflang="en"><link data-react-helmet="true" rel="alternate" href="https://hudi.apache.org/cn/docs/0.14.0/basic_configurations" hreflang="cn"><link data-react-helmet="true" rel="alternate" href="https://hudi.apache.org/docs/0.14.0/basic_configurations" hreflang="x-default"><link data-react-helmet="true" rel="preconnect" href="https://BH4D9OD16A-dsn.algolia.net" crossorigin="anonymous"><link rel="stylesheet" href="/assets/css/styles.ea681a30.css"> |
| <div><a href="#" class="skipToContent_OuoZ">Skip to main content</a></div><div class="announcementBar_axC9" role="banner"><div class="announcementBarPlaceholder_xYHE"></div><div class="announcementBarContent_6uhP">⭐️ If you like Apache Hudi, give it a star on <a target="_blank" rel="noopener noreferrer" href="https://github.com/apache/hudi">GitHub</a>! ⭐</div><button type="button" class="clean-btn close announcementBarClose_A3A1" aria-label="Close"><svg viewBox="0 0 15 15" width="14" height="14"><g stroke="currentColor" stroke-width="3.1"><path d="M.75.75l13.5 13.5M14.25.75L.75 14.25"></path></g></svg></button></div><nav class="navbar navbar--fixed-top navbarWrapper_UIa0"><div class="navbar__inner"><img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=8f594acf-9b77-44fb-9475-3e82ead1910c" width="0" height="0" alt=""><img referrerpolicy="no-referrer-when-downgrade" src="https://analytics.apache.org/matomo.php?idsite=47&rec=1" width="0" height="0" alt=""><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo navbarLogo_Bz6n"><img src="/assets/images/hudi.png" alt="Apache Hudi" class="themedImage_TMUO themedImage--light_4Vu1"><img src="/assets/images/hudi.png" alt="Apache Hudi" class="themedImage_TMUO themedImage--dark_uzRr"></div></a><a class="navbar__item navbar__link" href="/docs/overview"><div class="labelWrapperDropdown_Mqbj">Docs</div></a><div class="navbar__item dropdown dropdown--hoverable"><a href="#" class="navbar__link downloadLinkDropdownHide_aDP3"><div class="labelWrapperDropdown_Mqbj">Learn<svg width="10" height="6" viewBox="0 0 10 6" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M8.5 1.25 5 4.75l-3.5-3.5" stroke="#1C1E21" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path></svg></div></a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/talks"><div class="labelWrapperDropdown_Mqbj">Talks</div></a></li><li><a class="dropdown__link" href="/videos"><div class="labelWrapperDropdown_Mqbj">Video Guides</div></a></li><li><a class="dropdown__link" href="/docs/faq"><div class="labelWrapperDropdown_Mqbj">FAQ</div></a></li><li><a class="dropdown__link" href="/tech-specs"><div class="labelWrapperDropdown_Mqbj">Tech Specs</div></a></li><li><a class="dropdown__link" href="/tech-specs-1point0"><div class="labelWrapperDropdown_Mqbj">Tech Specs 1.0</div></a></li><li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank" rel="noopener noreferrer" class="dropdown__link"><span class="externalLink_AE3f">Technical Wiki<svg width="20" height="20" viewBox="0 0 26 26" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M16.965 8.745 9.01 16.7M10.561 8.758l6.403-.013-.013 6.403" stroke="#0DB1F9" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><rect x="4.5" y="4.5" width="17" height="17" rx="2.5" stroke="#0DB1F9"></rect></svg></span></a></li></ul></div><div class="navbar__item dropdown dropdown--hoverable"><a href="#" class="navbar__link downloadLinkDropdownHide_aDP3"><div class="labelWrapperDropdown_Mqbj">Contribute<svg width="10" height="6" viewBox="0 0 10 6" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M8.5 1.25 5 4.75l-3.5-3.5" stroke="#1C1E21" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path></svg></div></a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/contribute/how-to-contribute"><div class="labelWrapperDropdown_Mqbj">How to Contribute</div></a></li><li><a class="dropdown__link" href="/contribute/developer-setup"><div class="labelWrapperDropdown_Mqbj">Developer Setup</div></a></li><li><a class="dropdown__link" href="/contribute/rfc-process"><div class="labelWrapperDropdown_Mqbj">RFC Process</div></a></li><li><a class="dropdown__link" href="/contribute/report-security-issues"><div class="labelWrapperDropdown_Mqbj">Report Security Issues</div></a></li><li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank" rel="noopener noreferrer" class="dropdown__link"><span class="externalLink_AE3f">Report Issues<svg width="20" height="20" viewBox="0 0 26 26" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M16.965 8.745 9.01 16.7M10.561 8.758l6.403-.013-.013 6.403" stroke="#0DB1F9" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path><rect x="4.5" y="4.5" width="17" height="17" rx="2.5" stroke="#0DB1F9"></rect></svg></span></a></li></ul></div><div class="navbar__item dropdown dropdown--hoverable"><a href="#" class="navbar__link downloadLinkDropdownHide_aDP3"><div class="labelWrapperDropdown_Mqbj">Community<svg width="10" height="6" viewBox="0 0 10 6" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M8.5 1.25 5 4.75l-3.5-3.5" stroke="#1C1E21" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path></svg></div></a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/community/get-involved"><div class="labelWrapperDropdown_Mqbj">Get Involved</div></a></li><li><a class="dropdown__link" href="/community/syncs"><div class="labelWrapperDropdown_Mqbj">Community Syncs</div></a></li><li><a class="dropdown__link" href="/community/office_hours"><div class="labelWrapperDropdown_Mqbj">Office Hours</div></a></li><li><a class="dropdown__link" href="/community/team"><div class="labelWrapperDropdown_Mqbj">Team</div></a></li></ul></div><a class="navbar__item navbar__link" href="/blog"><div class="labelWrapperDropdown_Mqbj">Blog</div></a><a class="navbar__item navbar__link" href="/powered-by"><div class="labelWrapperDropdown_Mqbj">Who's Using</div></a><a class="navbar__item navbar__link" href="/roadmap"><div class="labelWrapperDropdown_Mqbj">Roadmap</div></a><a class="navbar__item navbar__link" href="/releases/download"><div class="labelWrapperDropdown_Mqbj">Download</div></a></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link downloadLinkDropdownHide_aDP3" href="/docs/0.14.0/overview"><div class="labelWrapperDropdown_Mqbj">0.14.0<svg width="10" height="6" viewBox="0 0 10 6" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M8.5 1.25 5 4.75l-3.5-3.5" stroke="#1C1E21" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path></svg></div></a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/next/basic_configurations"><div class="labelWrapperDropdown_Mqbj">Current</div></a></li><li><a class="dropdown__link" href="/docs/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.14.1</div></a></li><li><a aria-current="page" class="dropdown__link dropdown__link--active" href="/docs/0.14.0/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.14.0</div></a></li><li><a class="dropdown__link" href="/docs/0.13.1/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.13.1</div></a></li><li><a class="dropdown__link" href="/docs/0.13.0/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.13.0</div></a></li><li><a class="dropdown__link" href="/docs/0.12.3/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.12.3</div></a></li><li><a class="dropdown__link" href="/docs/0.12.2/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.12.2</div></a></li><li><a class="dropdown__link" href="/docs/0.12.1/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.12.1</div></a></li><li><a class="dropdown__link" href="/docs/0.12.0/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.12.0</div></a></li><li><a class="dropdown__link" href="/docs/0.11.1/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.11.1</div></a></li><li><a class="dropdown__link" href="/docs/0.11.0/overview"><div class="labelWrapperDropdown_Mqbj">0.11.0</div></a></li><li><a class="dropdown__link" href="/docs/0.10.1/overview"><div class="labelWrapperDropdown_Mqbj">0.10.1</div></a></li><li><a class="dropdown__link" href="/docs/0.10.0/overview"><div class="labelWrapperDropdown_Mqbj">0.10.0</div></a></li><li><a class="dropdown__link" href="/docs/0.9.0/overview"><div class="labelWrapperDropdown_Mqbj">0.9.0</div></a></li><li><a class="dropdown__link" href="/docs/0.8.0/overview"><div class="labelWrapperDropdown_Mqbj">0.8.0</div></a></li><li><a class="dropdown__link" href="/docs/0.7.0/overview"><div class="labelWrapperDropdown_Mqbj">0.7.0</div></a></li><li><a class="dropdown__link" href="/docs/0.6.0/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.6.0</div></a></li><li><a class="dropdown__link" href="/docs/0.5.3/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.5.3</div></a></li><li><a class="dropdown__link" href="/docs/0.5.2/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.5.2</div></a></li><li><a class="dropdown__link" href="/docs/0.5.1/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.5.1</div></a></li><li><a class="dropdown__link" href="/docs/0.5.0/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.5.0</div></a></li></ul></div><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" class="navbar__link downloadLinkDropdownHide_aDP3"><div class="labelWrapperDropdown_Mqbj"><span><svg viewBox="0 0 20 20" width="20" height="20" aria-hidden="true" class="iconLanguage_zID8"><path fill="currentColor" d="M19.753 10.909c-.624-1.707-2.366-2.726-4.661-2.726-.09 0-.176.002-.262.006l-.016-2.063 3.525-.607c.115-.019.133-.119.109-.231-.023-.111-.167-.883-.188-.976-.027-.131-.102-.127-.207-.109-.104.018-3.25.461-3.25.461l-.013-2.078c-.001-.125-.069-.158-.194-.156l-1.025.016c-.105.002-.164.049-.162.148l.033 2.307s-3.061.527-3.144.543c-.084.014-.17.053-.151.143.019.09.19 1.094.208 1.172.018.08.072.129.188.107l2.924-.504.035 2.018c-1.077.281-1.801.824-2.256 1.303-.768.807-1.207 1.887-1.207 2.963 0 1.586.971 2.529 2.328 2.695 3.162.387 5.119-3.06 5.769-4.715 1.097 1.506.256 4.354-2.094 5.98-.043.029-.098.129-.033.207l.619.756c.08.096.206.059.256.023 2.51-1.73 3.661-4.515 2.869-6.683zm-7.386 3.188c-.966-.121-.944-.914-.944-1.453 0-.773.327-1.58.876-2.156a3.21 3.21 0 011.229-.799l.082 4.277a2.773 2.773 0 01-1.243.131zm2.427-.553l.046-4.109c.084-.004.166-.01.252-.01.773 0 1.494.145 1.885.361.391.217-1.023 2.713-2.183 3.758zm-8.95-7.668a.196.196 0 00-.196-.145h-1.95a.194.194 0 00-.194.144L.008 16.916c-.017.051-.011.076.062.076h1.733c.075 0 .099-.023.114-.072l1.008-3.318h3.496l1.008 3.318c.016.049.039.072.113.072h1.734c.072 0 .078-.025.062-.076-.014-.05-3.083-9.741-3.494-11.04zm-2.618 6.318l1.447-5.25 1.447 5.25H3.226z"></path></svg><span>English</span></span><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg"><g clip-path="url(#a)"><path d="M14 6.457a6.842 6.842 0 0 0-7-6.02 6.843 6.843 0 0 0-7 6.02v1.085a6.843 6.843 0 0 0 7 6.02 6.843 6.843 0 0 0 7-6.02V6.457Zm-1.094 0h-2.625a9.92 9.92 0 0 0-.376-2.222 6.65 6.65 0 0 0 1.531-.875 5.25 5.25 0 0 1 1.444 3.097h.026Zm-8.032 0a8.479 8.479 0 0 1 .324-1.872 7.376 7.376 0 0 0 3.63 0c.175.61.284 1.239.325 1.872h-4.28Zm4.305 1.085a8.391 8.391 0 0 1-.324 1.873 7.464 7.464 0 0 0-3.658 0 8.479 8.479 0 0 1-.323-1.873h4.305Zm.35-4.375A10.342 10.342 0 0 0 8.75 1.75c.627.194 1.218.49 1.75.875a5.748 5.748 0 0 1-.998.577l.027-.035ZM7.254 1.54A8.75 8.75 0 0 1 8.46 3.552c-.48.11-.97.165-1.461.167-.492-.001-.982-.057-1.461-.167.308-.722.715-1.4 1.207-2.012h.508ZM4.498 3.202a5.748 5.748 0 0 1-.998-.577 6.029 6.029 0 0 1 1.75-.875c-.294.46-.546.947-.753 1.452Zm-1.873.15c.47.358.984.652 1.531.874A9.625 9.625 0 0 0 3.78 6.45H1.155a5.25 5.25 0 0 1 1.47-3.098ZM1.12 7.541h2.625c.038.753.164 1.5.376 2.223a6.649 6.649 0 0 0-1.531.875 5.25 5.25 0 0 1-1.47-3.098Zm3.377 3.255c.207.506.459.992.753 1.453a6.03 6.03 0 0 1-1.75-.875c.312-.226.646-.419.997-.578Zm2.25 1.663a8.594 8.594 0 0 1-1.208-2.013 6.501 6.501 0 0 1 2.922 0 8.54 8.54 0 0 1-1.207 2.013h-.508Zm2.755-1.663c.367.156.716.35 1.042.578a6.338 6.338 0 0 1-1.75.875c.275-.464.512-.95.708-1.453Zm1.873-.148a6.647 6.647 0 0 0-1.531-.875 9.45 9.45 0 0 0 .376-2.223h2.625a5.25 5.25 0 0 1-1.47 3.098Z" fill="#1C1E21"></path></g><defs><clipPath id="a"><path fill="#fff" d="M0 0h14v14H0z"></path></clipPath></defs></svg></div></a><ul class="dropdown__menu"><li><a href="/docs/0.14.0/basic_configurations" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active"><div class="labelWrapperDropdown_Mqbj">English</div></a></li><li><a href="/cn/docs/0.14.0/basic_configurations" target="_self" rel="noopener noreferrer" class="dropdown__link"><div class="labelWrapperDropdown_Mqbj">Chinese</div></a></li></ul></div><a href="https://github.com/apache/hudi" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-github-link" aria-label="GitHub repository"><div class="labelWrapperDropdown_Mqbj"></div></a><a href="https://twitter.com/ApacheHudi" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-twitter-link" aria-label="Hudi Twitter Handle"><div class="labelWrapperDropdown_Mqbj"></div></a><a href="https://join.slack.com/t/apache-hudi/shared_invite/zt-2ggm1fub8-_yt4Reu9djwqqVRFC7X49g" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-slack-link" aria-label="Hudi Slack Channel"><div class="labelWrapperDropdown_Mqbj"></div></a><a href="https://www.youtube.com/channel/UCs7AhE0BWaEPZSChrBR-Muw" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-youtube-link" aria-label="Hudi YouTube Channel"><div class="labelWrapperDropdown_Mqbj"></div></a><a href="https://www.linkedin.com/company/apache-hudi/?viewAsMember=true" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-linkedin-link" aria-label="Hudi Linkedin Page"><div class="labelWrapperDropdown_Mqbj"></div></a><div class="searchBox_fBfG"><div role="button" class="searchButton_g9-U" aria-label="Search"><span class="searchText_RI6l">Search</span><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg"><circle cx="6.864" cy="6.864" r="5.243" stroke="#1C1E21" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></circle><path d="m10.51 10.783 2.056 2.05" stroke="#1C1E21" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path></svg></div></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div><div class="navbar-sidebar"><div class="navbar-sidebar__brand"><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/assets/images/hudi.png" alt="Apache Hudi" class="themedImage_TMUO themedImage--light_4Vu1"><img src="/assets/images/hudi.png" alt="Apache Hudi" class="themedImage_TMUO themedImage--dark_uzRr"></div></a><button type="button" class="clean-btn navbar-sidebar__close"><svg viewBox="0 0 15 15" width="21" height="21"><g stroke="var(--ifm-color-emphasis-600)" stroke-width="1.2"><path d="M.75.75l13.5 13.5M14.25.75L.75 14.25"></path></g></svg></button></div><div class="navbar-sidebar__items"><div class="navbar-sidebar__item menu"><ul class="menu__list"><li class="menu__list-item"><a class="menu__link" href="/docs/overview"><div class="labelWrapperDropdown_Mqbj">Docs</div></a></li><li class="menu__list-item menu__list-item--collapsed"><a role="button" class="menu__link menu__link--sublist"><div class="labelWrapperDropdown_Mqbj">Learn</div></a></li><li class="menu__list-item menu__list-item--collapsed"><a role="button" class="menu__link menu__link--sublist"><div class="labelWrapperDropdown_Mqbj">Contribute</div></a></li><li class="menu__list-item menu__list-item--collapsed"><a role="button" class="menu__link menu__link--sublist"><div class="labelWrapperDropdown_Mqbj">Community</div></a></li><li class="menu__list-item"><a class="menu__link" href="/blog"><div class="labelWrapperDropdown_Mqbj">Blog</div></a></li><li class="menu__list-item"><a class="menu__link" href="/powered-by"><div class="labelWrapperDropdown_Mqbj">Who's Using</div></a></li><li class="menu__list-item"><a class="menu__link" href="/roadmap"><div class="labelWrapperDropdown_Mqbj">Roadmap</div></a></li><li class="menu__list-item"><a class="menu__link" href="/releases/download"><div class="labelWrapperDropdown_Mqbj">Download</div></a></li><li class="menu__list-item"><a role="button" class="menu__link menu__link--sublist"><div class="labelWrapperDropdown_Mqbj">Versions</div></a><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="menu__list-item"><a class="menu__link" href="/docs/next/basic_configurations"><div class="labelWrapperDropdown_Mqbj">Current</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.14.1</div></a></li><li class="menu__list-item"><a aria-current="page" class="menu__link menu__link--active" href="/docs/0.14.0/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.14.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.13.1/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.13.1</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.13.0/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.13.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.12.3/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.12.3</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.12.2/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.12.2</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.12.1/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.12.1</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.12.0/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.12.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.11.1/basic_configurations"><div class="labelWrapperDropdown_Mqbj">0.11.1</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.11.0/overview"><div class="labelWrapperDropdown_Mqbj">0.11.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.10.1/overview"><div class="labelWrapperDropdown_Mqbj">0.10.1</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.10.0/overview"><div class="labelWrapperDropdown_Mqbj">0.10.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.9.0/overview"><div class="labelWrapperDropdown_Mqbj">0.9.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.8.0/overview"><div class="labelWrapperDropdown_Mqbj">0.8.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.7.0/overview"><div class="labelWrapperDropdown_Mqbj">0.7.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.6.0/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.6.0</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.5.3/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.5.3</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.5.2/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.5.2</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.5.1/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.5.1</div></a></li><li class="menu__list-item"><a class="menu__link" href="/docs/0.5.0/quick-start-guide"><div class="labelWrapperDropdown_Mqbj">0.5.0</div></a></li></ul></li><li class="menu__list-item menu__list-item--collapsed"><a role="button" class="menu__link menu__link--sublist"><div class="labelWrapperDropdown_Mqbj"><span><svg viewBox="0 0 20 20" width="20" height="20" aria-hidden="true" class="iconLanguage_zID8"><path fill="currentColor" d="M19.753 10.909c-.624-1.707-2.366-2.726-4.661-2.726-.09 0-.176.002-.262.006l-.016-2.063 3.525-.607c.115-.019.133-.119.109-.231-.023-.111-.167-.883-.188-.976-.027-.131-.102-.127-.207-.109-.104.018-3.25.461-3.25.461l-.013-2.078c-.001-.125-.069-.158-.194-.156l-1.025.016c-.105.002-.164.049-.162.148l.033 2.307s-3.061.527-3.144.543c-.084.014-.17.053-.151.143.019.09.19 1.094.208 1.172.018.08.072.129.188.107l2.924-.504.035 2.018c-1.077.281-1.801.824-2.256 1.303-.768.807-1.207 1.887-1.207 2.963 0 1.586.971 2.529 2.328 2.695 3.162.387 5.119-3.06 5.769-4.715 1.097 1.506.256 4.354-2.094 5.98-.043.029-.098.129-.033.207l.619.756c.08.096.206.059.256.023 2.51-1.73 3.661-4.515 2.869-6.683zm-7.386 3.188c-.966-.121-.944-.914-.944-1.453 0-.773.327-1.58.876-2.156a3.21 3.21 0 011.229-.799l.082 4.277a2.773 2.773 0 01-1.243.131zm2.427-.553l.046-4.109c.084-.004.166-.01.252-.01.773 0 1.494.145 1.885.361.391.217-1.023 2.713-2.183 3.758zm-8.95-7.668a.196.196 0 00-.196-.145h-1.95a.194.194 0 00-.194.144L.008 16.916c-.017.051-.011.076.062.076h1.733c.075 0 .099-.023.114-.072l1.008-3.318h3.496l1.008 3.318c.016.049.039.072.113.072h1.734c.072 0 .078-.025.062-.076-.014-.05-3.083-9.741-3.494-11.04zm-2.618 6.318l1.447-5.25 1.447 5.25H3.226z"></path></svg><span>Languages</span></span></div></a></li><li class="menu__list-item"><a href="https://github.com/apache/hudi" target="_blank" rel="noopener noreferrer" class="menu__link header-github-link" aria-label="GitHub repository"><div class="labelWrapperDropdown_Mqbj"></div></a></li><li class="menu__list-item"><a href="https://twitter.com/ApacheHudi" target="_blank" rel="noopener noreferrer" class="menu__link header-twitter-link" aria-label="Hudi Twitter Handle"><div class="labelWrapperDropdown_Mqbj"></div></a></li><li class="menu__list-item"><a href="https://join.slack.com/t/apache-hudi/shared_invite/zt-2ggm1fub8-_yt4Reu9djwqqVRFC7X49g" target="_blank" rel="noopener noreferrer" class="menu__link header-slack-link" aria-label="Hudi Slack Channel"><div class="labelWrapperDropdown_Mqbj"></div></a></li><li class="menu__list-item"><a href="https://www.youtube.com/channel/UCs7AhE0BWaEPZSChrBR-Muw" target="_blank" rel="noopener noreferrer" class="menu__link header-youtube-link" aria-label="Hudi YouTube Channel"><div class="labelWrapperDropdown_Mqbj"></div></a></li><li class="menu__list-item"><a href="https://www.linkedin.com/company/apache-hudi/?viewAsMember=true" target="_blank" rel="noopener noreferrer" class="menu__link header-linkedin-link" aria-label="Hudi Linkedin Page"><div class="labelWrapperDropdown_Mqbj"></div></a></li></ul></div><div class="navbar-sidebar__item menu"><button type="button" class="clean-btn navbar-sidebar__back">← Back to main menu</button></div></div></div></nav><div class="main-wrapper docs-wrapper docs-doc-page"><div class="docPage_GMj9"><button aria-label="Scroll back to top" class="clean-btn theme-back-to-top-button backToTopButton_i9tI" type="button"></button><aside class="docSidebarContainer_k0Pq"><div class="sidebar_a3j0"><nav class="menu thin-scrollbar menu_cyFh menuWithAnnouncementBar_+O1J"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/docs/0.14.0/overview">Overview</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist hasHref_TwRn" href="/docs/0.14.0/quick-start-guide">Quick Start</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/0.14.0/quick-start-guide">Spark Guide</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/0.14.0/flink-quick-start-guide">Flink Guide</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/0.14.0/docker_demo">Docker Demo</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist hasHref_TwRn" href="/docs/0.14.0/timeline">Concepts</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist hasHref_TwRn" href="/docs/0.14.0/sql_ddl">How To</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist hasHref_TwRn" href="/docs/0.14.0/migration_guide">Services</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--active hasHref_TwRn" href="/docs/0.14.0/basic_configurations">Configurations</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/0.14.0/basic_configurations">Basic Configurations</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/0.14.0/configurations">All Configurations</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist hasHref_TwRn" href="/docs/0.14.0/performance">Guides</a></div></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/docs/0.14.0/use_cases">Use Cases</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/docs/0.14.0/faq">FAQs</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/docs/0.14.0/privacy">Privacy Policy</a></li></ul></nav></div></aside><main class="docMainContainer_Q970"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_zHA2"><div class="theme-doc-version-banner alert alert--warning margin-bottom--md" role="alert"><div>This is documentation for <!-- -->Apache Hudi<!-- --> <b>0.14.0</b>, which is no longer actively maintained.</div><div class="margin-top--md">For up-to-date documentation, see the <b><a href="/docs/basic_configurations">latest version</a></b> (<!-- -->0.14.1<!-- -->).</div></div><div class="docItemContainer_oiyr"><article><span class="theme-doc-version-badge badge badge--secondary">Version: <!-- -->0.14.0</span><div class="tocCollapsible_aw-L theme-doc-toc-mobile tocMobile_Tx6Y"><button type="button" class="clean-btn tocCollapsibleButton_zr6a">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Basic Configurations</h1></header><p>This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the <a href="/docs/configurations">All Configurations</a> page.</p><ul><li><a href="#SPARK_DATASOURCE"><strong>Spark Datasource Configs</strong></a>: These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.</li><li><a href="#FLINK_SQL"><strong>Flink Sql Configs</strong></a>: These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.</li><li><a href="#WRITE_CLIENT"><strong>Write Client Configs</strong></a>: Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.</li><li><a href="#META_SYNC"><strong>Metastore and Catalog Sync Configs</strong></a>: Configurations used by the Hudi to sync metadata to external metastores and catalogs.</li><li><a href="#METRICS"><strong>Metrics Configs</strong></a>: These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.</li><li><a href="#KAFKA_CONNECT"><strong>Kafka Connect Configs</strong></a>: These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables</li><li><a href="#HUDI_STREAMER"><strong>Hudi Streamer Configs</strong></a>: These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.</li></ul><div class="admonition admonition-note alert alert--secondary"><div class="admonition-heading"><h5><span class="admonition-icon"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="16" viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>note</h5></div><div class="admonition-content"><p>In the tables below <strong>(N/A)</strong> means there is no default value set</p></div></div><h2 class="anchor anchorWithStickyNavbar_y2LR" id="SPARK_DATASOURCE">Spark Datasource Configs<a class="hash-link" href="#SPARK_DATASOURCE" title="Direct link to heading"></a></h2><p>These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Read-Options">Read Options<a class="hash-link" href="#Read-Options" title="Direct link to heading"></a></h3><p>Options useful for reading tables via <code>read.format.option(...)</code></p><p><a href="#Read-Options-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatasourcereadbegininstanttime">hoodie.datasource.read.begin.instanttime</a></td><td>(N/A)</td><td>Required when <code>hoodie.datasource.query.type</code> is set to <code>incremental</code>. Represents the instant time to start incrementally pulling data from. The instanttime here need not necessarily correspond to an instant on the timeline. New data written with an instant_time <!-- -->><!-- --> BEGIN_INSTANTTIME are fetched out. For e.g: ‘20170901080000’ will get all new data written after Sep 1, 2017 08:00AM. Note that if <code>hoodie.read.timeline.holes.resolution.policy</code> set to USE_TRANSITION_TIME, will use instant's <code>stateTransitionTime</code> to perform comparison.<br><code>Config Param: BEGIN_INSTANTTIME</code></td></tr><tr><td><a href="#hoodiedatasourcereadendinstanttime">hoodie.datasource.read.end.instanttime</a></td><td>(N/A)</td><td>Used when <code>hoodie.datasource.query.type</code> is set to <code>incremental</code>. Represents the instant time to limit incrementally fetched data to. When not specified latest commit time from timeline is assumed by default. When specified, new data written with an instant_time <!-- --><<!-- -->= END_INSTANTTIME are fetched out. Point in time type queries make more sense with begin and end instant times specified. Note that if <code>hoodie.read.timeline.holes.resolution.policy</code> set to <code>USE_TRANSITION_TIME</code>, will use instant's <code>stateTransitionTime</code> to perform comparison.<br><code>Config Param: END_INSTANTTIME</code></td></tr><tr><td><a href="#hoodiedatasourcequerytype">hoodie.datasource.query.type</a></td><td>snapshot</td><td>Whether data needs to be read, in <code>incremental</code> mode (new data since an instantTime) (or) <code>read_optimized</code> mode (obtain latest view, based on base files) (or) <code>snapshot</code> mode (obtain latest view, by merging base and (if any) log files)<br><code>Config Param: QUERY_TYPE</code></td></tr><tr><td><a href="#hoodiedatasourcewriteprecombinefield">hoodie.datasource.write.precombine.field</a></td><td>ts</td><td>Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br><code>Config Param: READ_PRE_COMBINE_FIELD</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Write-Options">Write Options<a class="hash-link" href="#Write-Options" title="Direct link to heading"></a></h3><p>You can pass down any of the WriteClient level configs directly using <code>options()</code> or <code>option(k,v)</code> methods.</p><div class="codeBlockContainer_J+bg language-java theme-code-block"><div class="codeBlockContent_csEI java"><pre tabindex="0" class="prism-code language-java codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">inputDF.write()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.format("org.apache.hudi")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.options(clientOpts) // any of the Hudi client opts can be passed in as well</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.option(HoodieWriteConfig.TABLE_NAME, tableName)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.mode(SaveMode.Append)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.save(basePath);</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Options useful for writing tables via <code>write.format.option(...)</code></p><p><a href="#Write-Options-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatasourcehive_syncmode">hoodie.datasource.hive_sync.mode</a></td><td>(N/A)</td><td>Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br><code>Config Param: HIVE_SYNC_MODE</code></td></tr><tr><td><a href="#hoodiedatasourcewritepartitionpathfield">hoodie.datasource.write.partitionpath.field</a></td><td>(N/A)</td><td>Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br><code>Config Param: PARTITIONPATH_FIELD</code></td></tr><tr><td><a href="#hoodiedatasourcewriterecordkeyfield">hoodie.datasource.write.recordkey.field</a></td><td>(N/A)</td><td>Record key field. Value to be used as the <code>recordKey</code> component of <code>HoodieKey</code>. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: <code>a.b.c</code><br><code>Config Param: RECORDKEY_FIELD</code></td></tr><tr><td><a href="#hoodieclusteringasyncenabled">hoodie.clustering.async.enabled</a></td><td>false</td><td>Enable running of clustering service, asynchronously as inserts happen on the table.<br><code>Config Param: ASYNC_CLUSTERING_ENABLE</code><br><code>Since Version: 0.7.0</code></td></tr><tr><td><a href="#hoodieclusteringinline">hoodie.clustering.inline</a></td><td>false</td><td>Turn on inline clustering - clustering will be run after each write operation is complete<br><code>Config Param: INLINE_CLUSTERING_ENABLE</code><br><code>Since Version: 0.7.0</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncenable">hoodie.datasource.hive_sync.enable</a></td><td>false</td><td>When set to true, register/sync the table to Apache Hive metastore.<br><code>Config Param: HIVE_SYNC_ENABLED</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncjdbcurl">hoodie.datasource.hive_sync.jdbcurl</a></td><td>jdbc:hive2://localhost:10000</td><td>Hive metastore url<br><code>Config Param: HIVE_URL</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncmetastoreuris">hoodie.datasource.hive_sync.metastore.uris</a></td><td>thrift://localhost:9083</td><td>Hive metastore url<br><code>Config Param: METASTORE_URIS</code></td></tr><tr><td><a href="#hoodiedatasourcemetasyncenable">hoodie.datasource.meta.sync.enable</a></td><td>false</td><td>Enable Syncing the Hudi Table with an external meta store or data catalog.<br><code>Config Param: META_SYNC_ENABLED</code></td></tr><tr><td><a href="#hoodiedatasourcewritehive_style_partitioning">hoodie.datasource.write.hive_style_partitioning</a></td><td>false</td><td>Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <!-- --><<!-- -->partition_column_name<!-- -->><!-- -->=<!-- --><<!-- -->partition_value<!-- -->><!-- --> format. By default false (the names of partition folders are only partition values)<br><code>Config Param: HIVE_STYLE_PARTITIONING</code></td></tr><tr><td><a href="#hoodiedatasourcewriteoperation">hoodie.datasource.write.operation</a></td><td>upsert</td><td>Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br><code>Config Param: OPERATION</code></td></tr><tr><td><a href="#hoodiedatasourcewriteprecombinefield">hoodie.datasource.write.precombine.field</a></td><td>ts</td><td>Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br><code>Config Param: PRECOMBINE_FIELD</code></td></tr><tr><td><a href="#hoodiedatasourcewritetabletype">hoodie.datasource.write.table.type</a></td><td>COPY_ON_WRITE</td><td>The table type for the underlying data, for this write. This can’t change between writes.<br><code>Config Param: TABLE_TYPE</code></td></tr></tbody></table><hr><h2 class="anchor anchorWithStickyNavbar_y2LR" id="FLINK_SQL">Flink Sql Configs<a class="hash-link" href="#FLINK_SQL" title="Direct link to heading"></a></h2><p>These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Flink-Options">Flink Options<a class="hash-link" href="#Flink-Options" title="Direct link to heading"></a></h3><p>Flink jobs using the SQL can be configured through the options in WITH clause. The actual datasource level configs are listed below.</p><p><a href="#Flink-Options-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatabasename">hoodie.database.name</a></td><td>(N/A)</td><td>Database name to register to Hive metastore<br> <code>Config Param: DATABASE_NAME</code></td></tr><tr><td><a href="#hoodietablename">hoodie.table.name</a></td><td>(N/A)</td><td>Table name to register to Hive metastore<br> <code>Config Param: TABLE_NAME</code></td></tr><tr><td><a href="#path">path</a></td><td>(N/A)</td><td>Base path for the target hoodie table. The path would be created if it does not exist, otherwise a Hoodie table expects to be initialized successfully<br> <code>Config Param: PATH</code></td></tr><tr><td><a href="#readend-commit">read.end-commit</a></td><td>(N/A)</td><td>End commit instant for reading, the commit time format should be 'yyyyMMddHHmmss'<br> <code>Config Param: READ_END_COMMIT</code></td></tr><tr><td><a href="#readstart-commit">read.start-commit</a></td><td>(N/A)</td><td>Start commit instant for reading, the commit time format should be 'yyyyMMddHHmmss', by default reading from the latest instant for streaming read<br> <code>Config Param: READ_START_COMMIT</code></td></tr><tr><td><a href="#archivemax_commits">archive.max_commits</a></td><td>50</td><td>Max number of commits to keep before archiving older commits into a sequential log, default 50<br> <code>Config Param: ARCHIVE_MAX_COMMITS</code></td></tr><tr><td><a href="#archivemin_commits">archive.min_commits</a></td><td>40</td><td>Min number of commits to keep before archiving older commits into a sequential log, default 40<br> <code>Config Param: ARCHIVE_MIN_COMMITS</code></td></tr><tr><td><a href="#cdcenabled">cdc.enabled</a></td><td>false</td><td>When enable, persist the change data if necessary, and can be queried as a CDC query mode<br> <code>Config Param: CDC_ENABLED</code></td></tr><tr><td><a href="#cdcsupplementalloggingmode">cdc.supplemental.logging.mode</a></td><td>DATA_BEFORE_AFTER</td><td>Setting 'op_key_only' persists the 'op' and the record key only, setting 'data_before' persists the additional 'before' image, and setting 'data_before_after' persists the additional 'before' and 'after' images.<br> <code>Config Param: SUPPLEMENTAL_LOGGING_MODE</code></td></tr><tr><td><a href="#changelogenabled">changelog.enabled</a></td><td>false</td><td>Whether to keep all the intermediate changes, we try to keep all the changes of a record when enabled: 1). The sink accept the UPDATE_BEFORE message; 2). The source try to emit every changes of a record. The semantics is best effort because the compaction job would finally merge all changes of a record into one. default false to have UPSERT semantics<br> <code>Config Param: CHANGELOG_ENABLED</code></td></tr><tr><td><a href="#cleanasyncenabled">clean.async.enabled</a></td><td>true</td><td>Whether to cleanup the old commits immediately on new commits, enabled by default<br> <code>Config Param: CLEAN_ASYNC_ENABLED</code></td></tr><tr><td><a href="#cleanretain_commits">clean.retain_commits</a></td><td>30</td><td>Number of commits to retain. So data will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much you can incrementally pull on this table, default 30<br> <code>Config Param: CLEAN_RETAIN_COMMITS</code></td></tr><tr><td><a href="#clusteringasyncenabled">clustering.async.enabled</a></td><td>false</td><td>Async Clustering, default false<br> <code>Config Param: CLUSTERING_ASYNC_ENABLED</code></td></tr><tr><td><a href="#clusteringplanstrategysmallfilelimit">clustering.plan.strategy.small.file.limit</a></td><td>600</td><td>Files smaller than the size specified here are candidates for clustering, default 600 MB<br> <code>Config Param: CLUSTERING_PLAN_STRATEGY_SMALL_FILE_LIMIT</code></td></tr><tr><td><a href="#clusteringplanstrategytargetfilemaxbytes">clustering.plan.strategy.target.file.max.bytes</a></td><td>1073741824</td><td>Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups, default 1 GB<br> <code>Config Param: CLUSTERING_PLAN_STRATEGY_TARGET_FILE_MAX_BYTES</code></td></tr><tr><td><a href="#compactionasyncenabled">compaction.async.enabled</a></td><td>true</td><td>Async Compaction, enabled by default for MOR<br> <code>Config Param: COMPACTION_ASYNC_ENABLED</code></td></tr><tr><td><a href="#compactiondelta_commits">compaction.delta_commits</a></td><td>5</td><td>Max delta commits needed to trigger compaction, default 5 commits<br> <code>Config Param: COMPACTION_DELTA_COMMITS</code></td></tr><tr><td><a href="#hive_syncenabled">hive_sync.enabled</a></td><td>false</td><td>Asynchronously sync Hive meta to HMS, default false<br> <code>Config Param: HIVE_SYNC_ENABLED</code></td></tr><tr><td><a href="#hive_syncjdbc_url">hive_sync.jdbc_url</a></td><td>jdbc:hive2://localhost:10000</td><td>Jdbc URL for hive sync, default 'jdbc:hive2://localhost:10000'<br> <code>Config Param: HIVE_SYNC_JDBC_URL</code></td></tr><tr><td><a href="#hive_syncmetastoreuris">hive_sync.metastore.uris</a></td><td></td><td>Metastore uris for hive sync, default ''<br> <code>Config Param: HIVE_SYNC_METASTORE_URIS</code></td></tr><tr><td><a href="#hive_syncmode">hive_sync.mode</a></td><td>HMS</td><td>Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql, default 'hms'<br> <code>Config Param: HIVE_SYNC_MODE</code></td></tr><tr><td><a href="#hoodiedatasourcequerytype">hoodie.datasource.query.type</a></td><td>snapshot</td><td>Decides how data files need to be read, in 1) Snapshot mode (obtain latest view, based on row <!-- -->&<!-- --> columnar data); 2) incremental mode (new data since an instantTime); 3) Read Optimized mode (obtain latest view, based on columnar data) .Default: snapshot<br> <code>Config Param: QUERY_TYPE</code></td></tr><tr><td><a href="#hoodiedatasourcewritehive_style_partitioning">hoodie.datasource.write.hive_style_partitioning</a></td><td>false</td><td>Whether to use Hive style partitioning. If set true, the names of partition folders follow <!-- --><<!-- -->partition_column_name<!-- -->><!-- -->=<!-- --><<!-- -->partition_value<!-- -->><!-- --> format. By default false (the names of partition folders are only partition values)<br> <code>Config Param: HIVE_STYLE_PARTITIONING</code></td></tr><tr><td><a href="#hoodiedatasourcewritepartitionpathfield">hoodie.datasource.write.partitionpath.field</a></td><td></td><td>Partition path field. Value to be used at the <code>partitionPath</code> component of <code>HoodieKey</code>. Actual value obtained by invoking .toString(), default ''<br> <code>Config Param: PARTITION_PATH_FIELD</code></td></tr><tr><td><a href="#hoodiedatasourcewriterecordkeyfield">hoodie.datasource.write.recordkey.field</a></td><td>uuid</td><td>Record key field. Value to be used as the <code>recordKey</code> component of <code>HoodieKey</code>. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: <code>a.b.c</code><br> <code>Config Param: RECORD_KEY_FIELD</code></td></tr><tr><td><a href="#indextype">index.type</a></td><td>FLINK_STATE</td><td>Index type of Flink write job, default is using state backed index.<br> <code>Config Param: INDEX_TYPE</code></td></tr><tr><td><a href="#metadatacompactiondelta_commits">metadata.compaction.delta_commits</a></td><td>10</td><td>Max delta commits for metadata table to trigger compaction, default 10<br> <code>Config Param: METADATA_COMPACTION_DELTA_COMMITS</code></td></tr><tr><td><a href="#metadataenabled">metadata.enabled</a></td><td>false</td><td>Enable the internal metadata table which serves table metadata like level file listings, default disabled<br> <code>Config Param: METADATA_ENABLED</code></td></tr><tr><td><a href="#precombinefield">precombine.field</a></td><td>ts</td><td>Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br> <code>Config Param: PRECOMBINE_FIELD</code></td></tr><tr><td><a href="#readstreamingenabled">read.streaming.enabled</a></td><td>false</td><td>Whether to read as streaming source, default false<br> <code>Config Param: READ_AS_STREAMING</code></td></tr><tr><td><a href="#tabletype">table.type</a></td><td>COPY_ON_WRITE</td><td>Type of table to write. COPY_ON_WRITE (or) MERGE_ON_READ<br> <code>Config Param: TABLE_TYPE</code></td></tr><tr><td><a href="#writeoperation">write.operation</a></td><td>upsert</td><td>The write operation, that this write should do<br> <code>Config Param: OPERATION</code></td></tr><tr><td><a href="#writeparquetmaxfilesize">write.parquet.max.file.size</a></td><td>120</td><td>Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br> <code>Config Param: WRITE_PARQUET_MAX_FILE_SIZE</code></td></tr></tbody></table><hr><h2 class="anchor anchorWithStickyNavbar_y2LR" id="WRITE_CLIENT">Write Client Configs<a class="hash-link" href="#WRITE_CLIENT" title="Direct link to heading"></a></h2><p>Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Metadata-Configs">Metadata Configs<a class="hash-link" href="#Metadata-Configs" title="Direct link to heading"></a></h3><p>Configurations used by the Hudi Metadata Table. This table maintains the metadata about a given Hudi table (e.g file listings) to avoid overhead of accessing cloud storage, during queries.</p><p><a href="#Metadata-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiemetadataenable">hoodie.metadata.enable</a></td><td>true</td><td>Enable the internal metadata table which serves table metadata like level file listings<br><code>Config Param: ENABLE</code><br><code>Since Version: 0.7.0</code></td></tr><tr><td><a href="#hoodiemetadataindexbloomfilterenable">hoodie.metadata.index.bloom.filter.enable</a></td><td>false</td><td>Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br><code>Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER</code><br><code>Since Version: 0.11.0</code></td></tr><tr><td><a href="#hoodiemetadataindexcolumnstatsenable">hoodie.metadata.index.column.stats.enable</a></td><td>false</td><td>Enable indexing column ranges of user data files under metadata table key lookups. When enabled, metadata table will have a partition to store the column ranges and will be used for pruning files during the index lookups.<br><code>Config Param: ENABLE_METADATA_INDEX_COLUMN_STATS</code><br><code>Since Version: 0.11.0</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Storage-Configs">Storage Configs<a class="hash-link" href="#Storage-Configs" title="Direct link to heading"></a></h3><p>Configurations that control aspects around writing, sizing, reading base and log files.</p><p><a href="#Storage-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodieparquetcompressioncodec">hoodie.parquet.compression.codec</a></td><td>gzip</td><td>Compression Codec for parquet files<br><code>Config Param: PARQUET_COMPRESSION_CODEC_NAME</code></td></tr><tr><td><a href="#hoodieparquetmaxfilesize">hoodie.parquet.max.file.size</a></td><td>125829120</td><td>Target size in bytes for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br><code>Config Param: PARQUET_MAX_FILE_SIZE</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Archival-Configs">Archival Configs<a class="hash-link" href="#Archival-Configs" title="Direct link to heading"></a></h3><p>Configurations that control archival.</p><p><a href="#Archival-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiekeepmaxcommits">hoodie.keep.max.commits</a></td><td>30</td><td>Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. <br><code>Config Param: MAX_COMMITS_TO_KEEP</code></td></tr><tr><td><a href="#hoodiekeepmincommits">hoodie.keep.min.commits</a></td><td>20</td><td>Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline.<br><code>Config Param: MIN_COMMITS_TO_KEEP</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Bootstrap-Configs">Bootstrap Configs<a class="hash-link" href="#Bootstrap-Configs" title="Direct link to heading"></a></h3><p>Configurations that control how you want to bootstrap your existing tables for the first time into hudi. The bootstrap operation can flexibly avoid copying data over before you can use Hudi and support running the existing writers and new hudi writers in parallel, to validate the migration.</p><p><a href="#Bootstrap-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiebootstrapbasepath">hoodie.bootstrap.base.path</a></td><td>(N/A)</td><td>Base path of the dataset that needs to be bootstrapped as a Hudi table<br><code>Config Param: BASE_PATH</code><br><code>Since Version: 0.6.0</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Clean-Configs">Clean Configs<a class="hash-link" href="#Clean-Configs" title="Direct link to heading"></a></h3><p>Cleaning (reclamation of older/unused file groups/slices).</p><p><a href="#Clean-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiecleanasync">hoodie.clean.async</a></td><td>false</td><td>Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br><code>Config Param: ASYNC_CLEAN</code></td></tr><tr><td><a href="#hoodiecleanercommitsretained">hoodie.cleaner.commits.retained</a></td><td>10</td><td>Number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br><code>Config Param: CLEANER_COMMITS_RETAINED</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Clustering-Configs">Clustering Configs<a class="hash-link" href="#Clustering-Configs" title="Direct link to heading"></a></h3><p>Configurations that control the clustering table service in hudi, which optimizes the storage layout for better query performance by sorting and sizing data files.</p><p><a href="#Clustering-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodieclusteringasyncenabled">hoodie.clustering.async.enabled</a></td><td>false</td><td>Enable running of clustering service, asynchronously as inserts happen on the table.<br><code>Config Param: ASYNC_CLUSTERING_ENABLE</code><br><code>Since Version: 0.7.0</code></td></tr><tr><td><a href="#hoodieclusteringinline">hoodie.clustering.inline</a></td><td>false</td><td>Turn on inline clustering - clustering will be run after each write operation is complete<br><code>Config Param: INLINE_CLUSTERING</code><br><code>Since Version: 0.7.0</code></td></tr><tr><td><a href="#hoodieclusteringplanstrategysmallfilelimit">hoodie.clustering.plan.strategy.small.file.limit</a></td><td>314572800</td><td>Files smaller than the size in bytes specified here are candidates for clustering<br><code>Config Param: PLAN_STRATEGY_SMALL_FILE_LIMIT</code><br><code>Since Version: 0.7.0</code></td></tr><tr><td><a href="#hoodieclusteringplanstrategytargetfilemaxbytes">hoodie.clustering.plan.strategy.target.file.max.bytes</a></td><td>1073741824</td><td>Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br><code>Config Param: PLAN_STRATEGY_TARGET_FILE_MAX_BYTES</code><br><code>Since Version: 0.7.0</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Compaction-Configs">Compaction Configs<a class="hash-link" href="#Compaction-Configs" title="Direct link to heading"></a></h3><p>Configurations that control compaction (merging of log files onto a new base files).</p><p><a href="#Compaction-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiecompactinline">hoodie.compact.inline</a></td><td>false</td><td>When set to true, compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br><code>Config Param: INLINE_COMPACT</code></td></tr><tr><td><a href="#hoodiecompactinlinemaxdeltacommits">hoodie.compact.inline.max.delta.commits</a></td><td>5</td><td>Number of delta commits after the last compaction, before scheduling of a new compaction is attempted. This config takes effect only for the compaction triggering strategy based on the number of commits, i.e., NUM_COMMITS, NUM_COMMITS_AFTER_LAST_REQUEST, NUM_AND_TIME, and NUM_OR_TIME.<br><code>Config Param: INLINE_COMPACT_NUM_DELTA_COMMITS</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Write-Configurations">Write Configurations<a class="hash-link" href="#Write-Configurations" title="Direct link to heading"></a></h3><p>Configurations that control write behavior on Hudi tables. These can be directly passed down from even higher level frameworks (e.g Spark datasources, Flink sink) and utilities (e.g Hudi Streamer).</p><p><a href="#Write-Configurations-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiebasepath">hoodie.base.path</a></td><td>(N/A)</td><td>Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br><code>Config Param: BASE_PATH</code></td></tr><tr><td><a href="#hoodietablename">hoodie.table.name</a></td><td>(N/A)</td><td>Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br><code>Config Param: TBL_NAME</code></td></tr><tr><td><a href="#hoodiedatasourcewriteprecombinefield">hoodie.datasource.write.precombine.field</a></td><td>ts</td><td>Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br><code>Config Param: PRECOMBINE_FIELD_NAME</code></td></tr><tr><td><a href="#hoodiewriteconcurrencymode">hoodie.write.concurrency.mode</a></td><td>SINGLE_WRITER</td><td>org.apache.hudi.common.model.WriteConcurrencyMode: Concurrency modes for write operations. SINGLE_WRITER(default): Only one active writer to the table. Maximizes throughput. OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table with lazy conflict resolution using locks. This means that only one writer succeeds if multiple writers write to the same file group.<br><code>Config Param: WRITE_CONCURRENCY_MODE</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="KEY_GENERATOR">Key Generator Configs<a class="hash-link" href="#KEY_GENERATOR" title="Direct link to heading"></a></h3><p>Hudi maintains keys (record key + partition path) for uniquely identifying a particular record. These configs allow developers to setup the Key generator class that extracts these out of incoming records.</p><h4 class="anchor anchorWithStickyNavbar_y2LR" id="Key-Generator-Options">Key Generator Options<a class="hash-link" href="#Key-Generator-Options" title="Direct link to heading"></a></h4><p><a href="#Key-Generator-Options-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatasourcewritepartitionpathfield">hoodie.datasource.write.partitionpath.field</a></td><td>(N/A)</td><td>Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br><code>Config Param: PARTITIONPATH_FIELD_NAME</code></td></tr><tr><td><a href="#hoodiedatasourcewriterecordkeyfield">hoodie.datasource.write.recordkey.field</a></td><td>(N/A)</td><td>Record key field. Value to be used as the <code>recordKey</code> component of <code>HoodieKey</code>. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: <code>a.b.c</code><br><code>Config Param: RECORDKEY_FIELD_NAME</code></td></tr><tr><td><a href="#hoodiedatasourcewritehive_style_partitioning">hoodie.datasource.write.hive_style_partitioning</a></td><td>false</td><td>Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <!-- --><<!-- -->partition_column_name<!-- -->><!-- -->=<!-- --><<!-- -->partition_value<!-- -->><!-- --> format. By default false (the names of partition folders are only partition values)<br><code>Config Param: HIVE_STYLE_PARTITIONING_ENABLE</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="INDEX">Index Configs<a class="hash-link" href="#INDEX" title="Direct link to heading"></a></h3><p>Configurations that control indexing behavior, which tags incoming records as either inserts or updates to older records.</p><h4 class="anchor anchorWithStickyNavbar_y2LR" id="Common-Index-Configs">Common Index Configs<a class="hash-link" href="#Common-Index-Configs" title="Direct link to heading"></a></h4><p><a href="#Common-Index-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodieindextype">hoodie.index.type</a></td><td>(N/A)</td><td>org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. HBASE: uses an external managed Apache HBase table to store record key to location mapping. HBase index is a global index, enforcing key uniqueness across all partitions in the table. INMEMORY: Uses in-memory hashmap in Spark and Java engine and Flink in-memory state in Flink for indexing. BLOOM: Employs bloom filters built out of the record keys, optionally also pruning candidate files using record key ranges. Key uniqueness is enforced inside partitions. GLOBAL_BLOOM: Employs bloom filters built out of the record keys, optionally also pruning candidate files using record key ranges. Key uniqueness is enforced across all partitions in the table. SIMPLE: Performs a lean join of the incoming update/delete records against keys extracted from the table on storage.Key uniqueness is enforced inside partitions. GLOBAL_SIMPLE: Performs a lean join of the incoming update/delete records against keys extracted from the table on storage.Key uniqueness is enforced across all partitions in the table. BUCKET: locates the file group containing the record fast by using bucket hashing, particularly beneficial in large scale. Use <code>hoodie.index.bucket.engine</code> to choose bucket engine type, i.e., how buckets are generated. FLINK_STATE: Internal Config for indexing based on Flink state. RECORD_INDEX: Index which saves the record key to location mappings in the HUDI Metadata Table. Record index is a global index, enforcing key uniqueness across all partitions in the table. Supports sharding to achieve very high scale.<br><code>Config Param: INDEX_TYPE</code></td></tr></tbody></table><hr><h2 class="anchor anchorWithStickyNavbar_y2LR" id="META_SYNC">Metastore and Catalog Sync Configs<a class="hash-link" href="#META_SYNC" title="Direct link to heading"></a></h2><p>Configurations used by the Hudi to sync metadata to external metastores and catalogs.</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Common-Metadata-Sync-Configs">Common Metadata Sync Configs<a class="hash-link" href="#Common-Metadata-Sync-Configs" title="Direct link to heading"></a></h3><p><a href="#Common-Metadata-Sync-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatasourcemetasyncenable">hoodie.datasource.meta.sync.enable</a></td><td>false</td><td>Enable Syncing the Hudi Table with an external meta store or data catalog.<br><code>Config Param: META_SYNC_ENABLED</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="BigQuery-Sync-Configs">BigQuery Sync Configs<a class="hash-link" href="#BigQuery-Sync-Configs" title="Direct link to heading"></a></h3><p>Configurations used by the Hudi to sync metadata to Google BigQuery.</p><p><a href="#BigQuery-Sync-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatasourcemetasyncenable">hoodie.datasource.meta.sync.enable</a></td><td>false</td><td>Enable Syncing the Hudi Table with an external meta store or data catalog.<br><code>Config Param: META_SYNC_ENABLED</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Hive-Sync-Configs">Hive Sync Configs<a class="hash-link" href="#Hive-Sync-Configs" title="Direct link to heading"></a></h3><p>Configurations used by the Hudi to sync metadata to Hive Metastore.</p><p><a href="#Hive-Sync-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatasourcehive_syncmode">hoodie.datasource.hive_sync.mode</a></td><td>(N/A)</td><td>Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br><code>Config Param: HIVE_SYNC_MODE</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncenable">hoodie.datasource.hive_sync.enable</a></td><td>false</td><td>When set to true, register/sync the table to Apache Hive metastore.<br><code>Config Param: HIVE_SYNC_ENABLED</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncjdbcurl">hoodie.datasource.hive_sync.jdbcurl</a></td><td>jdbc:hive2://localhost:10000</td><td>Hive metastore url<br><code>Config Param: HIVE_URL</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncmetastoreuris">hoodie.datasource.hive_sync.metastore.uris</a></td><td>thrift://localhost:9083</td><td>Hive metastore url<br><code>Config Param: METASTORE_URIS</code></td></tr><tr><td><a href="#hoodiedatasourcemetasyncenable">hoodie.datasource.meta.sync.enable</a></td><td>false</td><td>Enable Syncing the Hudi Table with an external meta store or data catalog.<br><code>Config Param: META_SYNC_ENABLED</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Global-Hive-Sync-Configs">Global Hive Sync Configs<a class="hash-link" href="#Global-Hive-Sync-Configs" title="Direct link to heading"></a></h3><p>Global replication configurations used by the Hudi to sync metadata to Hive Metastore.</p><p><a href="#Global-Hive-Sync-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatasourcehive_syncmode">hoodie.datasource.hive_sync.mode</a></td><td>(N/A)</td><td>Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br><code>Config Param: HIVE_SYNC_MODE</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncenable">hoodie.datasource.hive_sync.enable</a></td><td>false</td><td>When set to true, register/sync the table to Apache Hive metastore.<br><code>Config Param: HIVE_SYNC_ENABLED</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncjdbcurl">hoodie.datasource.hive_sync.jdbcurl</a></td><td>jdbc:hive2://localhost:10000</td><td>Hive metastore url<br><code>Config Param: HIVE_URL</code></td></tr><tr><td><a href="#hoodiedatasourcehive_syncmetastoreuris">hoodie.datasource.hive_sync.metastore.uris</a></td><td>thrift://localhost:9083</td><td>Hive metastore url<br><code>Config Param: METASTORE_URIS</code></td></tr><tr><td><a href="#hoodiedatasourcemetasyncenable">hoodie.datasource.meta.sync.enable</a></td><td>false</td><td>Enable Syncing the Hudi Table with an external meta store or data catalog.<br><code>Config Param: META_SYNC_ENABLED</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="DataHub-Sync-Configs">DataHub Sync Configs<a class="hash-link" href="#DataHub-Sync-Configs" title="Direct link to heading"></a></h3><p>Configurations used by the Hudi to sync metadata to DataHub.</p><p><a href="#DataHub-Sync-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiedatasourcemetasyncenable">hoodie.datasource.meta.sync.enable</a></td><td>false</td><td>Enable Syncing the Hudi Table with an external meta store or data catalog.<br><code>Config Param: META_SYNC_ENABLED</code></td></tr></tbody></table><hr><h2 class="anchor anchorWithStickyNavbar_y2LR" id="METRICS">Metrics Configs<a class="hash-link" href="#METRICS" title="Direct link to heading"></a></h2><p>These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Metrics-Configurations">Metrics Configurations<a class="hash-link" href="#Metrics-Configurations" title="Direct link to heading"></a></h3><p>Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean, rollback etc. The following sections list the supported reporters.</p><p><a href="#Metrics-Configurations-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiemetricson">hoodie.metrics.on</a></td><td>false</td><td>Turn on/off metrics reporting. off by default.<br><code>Config Param: TURN_METRICS_ON</code><br><code>Since Version: 0.5.0</code></td></tr><tr><td><a href="#hoodiemetricsreportertype">hoodie.metrics.reporter.type</a></td><td>GRAPHITE</td><td>Type of metrics reporter.<br><code>Config Param: METRICS_REPORTER_TYPE_VALUE</code><br><code>Since Version: 0.5.0</code></td></tr><tr><td><a href="#hoodiemetricscompactionlogblockson">hoodie.metricscompaction.log.blocks.on</a></td><td>false</td><td>Turn on/off metrics reporting for log blocks with compaction commit. off by default.<br><code>Config Param: TURN_METRICS_COMPACTION_LOG_BLOCKS_ON</code><br><code>Since Version: 0.14.0</code></td></tr></tbody></table><hr><h2 class="anchor anchorWithStickyNavbar_y2LR" id="KAFKA_CONNECT">Kafka Connect Configs<a class="hash-link" href="#KAFKA_CONNECT" title="Direct link to heading"></a></h2><p>These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Kafka-Sink-Connect-Configurations">Kafka Sink Connect Configurations<a class="hash-link" href="#Kafka-Sink-Connect-Configurations" title="Direct link to heading"></a></h3><p>Configurations for Kafka Connect Sink Connector for Hudi.</p><p><a href="#Kafka-Sink-Connect-Configurations-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#bootstrapservers">bootstrap.servers</a></td><td>localhost:9092</td><td>The bootstrap servers for the Kafka Cluster.<br><code>Config Param: KAFKA_BOOTSTRAP_SERVERS</code></td></tr></tbody></table><hr><h2 class="anchor anchorWithStickyNavbar_y2LR" id="HUDI_STREAMER">Hudi Streamer Configs<a class="hash-link" href="#HUDI_STREAMER" title="Direct link to heading"></a></h2><p>These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Hudi-Streamer-Configs">Hudi Streamer Configs<a class="hash-link" href="#Hudi-Streamer-Configs" title="Direct link to heading"></a></h3><p><a href="#Hudi-Streamer-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamersourcekafkatopic">hoodie.streamer.source.kafka.topic</a></td><td>(N/A)</td><td>Kafka topic name. The config is specific to HoodieMultiTableStreamer<br><code>Config Param: KAFKA_TOPIC</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="Hudi-Streamer-SQL-Transformer-Configs">Hudi Streamer SQL Transformer Configs<a class="hash-link" href="#Hudi-Streamer-SQL-Transformer-Configs" title="Direct link to heading"></a></h3><p>Configurations controlling the behavior of SQL transformer in Hudi Streamer.</p><p><a href="#Hudi-Streamer-SQL-Transformer-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamertransformersql">hoodie.streamer.transformer.sql</a></td><td>(N/A)</td><td>SQL Query to be executed during write<br><code>Config Param: TRANSFORMER_SQL</code></td></tr><tr><td><a href="#hoodiestreamertransformersqlfile">hoodie.streamer.transformer.sql.file</a></td><td>(N/A)</td><td>File with a SQL script to be executed during write<br><code>Config Param: TRANSFORMER_SQL_FILE</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="DELTA_STREAMER_SOURCE">Hudi Streamer Source Configs<a class="hash-link" href="#DELTA_STREAMER_SOURCE" title="Direct link to heading"></a></h3><p>Configurations controlling the behavior of reading source data.</p><h4 class="anchor anchorWithStickyNavbar_y2LR" id="DFS-Path-Selector-Configs">DFS Path Selector Configs<a class="hash-link" href="#DFS-Path-Selector-Configs" title="Direct link to heading"></a></h4><p>Configurations controlling the behavior of path selector for DFS source in Hudi Streamer.</p><p><a href="#DFS-Path-Selector-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamersourcedfsroot">hoodie.streamer.source.dfs.root</a></td><td>(N/A)</td><td>Root path of the source on DFS<br><code>Config Param: ROOT_INPUT_PATH</code></td></tr></tbody></table><hr><h4 class="anchor anchorWithStickyNavbar_y2LR" id="Hudi-Incremental-Source-Configs">Hudi Incremental Source Configs<a class="hash-link" href="#Hudi-Incremental-Source-Configs" title="Direct link to heading"></a></h4><p>Configurations controlling the behavior of incremental pulling from a Hudi table as a source in Hudi Streamer.</p><p><a href="#Hudi-Incremental-Source-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamersourcehoodieincrpath">hoodie.streamer.source.hoodieincr.path</a></td><td>(N/A)</td><td>Base-path for the source Hudi table<br><code>Config Param: HOODIE_SRC_BASE_PATH</code></td></tr></tbody></table><hr><h4 class="anchor anchorWithStickyNavbar_y2LR" id="Kafka-Source-Configs">Kafka Source Configs<a class="hash-link" href="#Kafka-Source-Configs" title="Direct link to heading"></a></h4><p>Configurations controlling the behavior of Kafka source in Hudi Streamer.</p><p><a href="#Kafka-Source-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamersourcekafkatopic">hoodie.streamer.source.kafka.topic</a></td><td>(N/A)</td><td>Kafka topic name.<br><code>Config Param: KAFKA_TOPIC_NAME</code></td></tr></tbody></table><hr><h4 class="anchor anchorWithStickyNavbar_y2LR" id="Pulsar-Source-Configs">Pulsar Source Configs<a class="hash-link" href="#Pulsar-Source-Configs" title="Direct link to heading"></a></h4><p>Configurations controlling the behavior of Pulsar source in Hudi Streamer.</p><p><a href="#Pulsar-Source-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamersourcepulsartopic">hoodie.streamer.source.pulsar.topic</a></td><td>(N/A)</td><td>Name of the target Pulsar topic to source data from<br><code>Config Param: PULSAR_SOURCE_TOPIC_NAME</code></td></tr><tr><td><a href="#hoodiestreamersourcepulsarendpointadminurl">hoodie.streamer.source.pulsar.endpoint.admin.url</a></td><td>http://localhost:8080</td><td>URL of the target Pulsar endpoint (of the form 'pulsar://host:port'<br><code>Config Param: PULSAR_SOURCE_ADMIN_ENDPOINT_URL</code></td></tr><tr><td><a href="#hoodiestreamersourcepulsarendpointserviceurl">hoodie.streamer.source.pulsar.endpoint.service.url</a></td><td>pulsar://localhost:6650</td><td>URL of the target Pulsar endpoint (of the form 'pulsar://host:port'<br><code>Config Param: PULSAR_SOURCE_SERVICE_ENDPOINT_URL</code></td></tr></tbody></table><hr><h4 class="anchor anchorWithStickyNavbar_y2LR" id="S3-Source-Configs">S3 Source Configs<a class="hash-link" href="#S3-Source-Configs" title="Direct link to heading"></a></h4><p>Configurations controlling the behavior of S3 source in Hudi Streamer.</p><p><a href="#S3-Source-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamers3sourcequeueurl">hoodie.streamer.s3.source.queue.url</a></td><td>(N/A)</td><td>Queue url for cloud object events<br><code>Config Param: S3_SOURCE_QUEUE_URL</code></td></tr></tbody></table><hr><h4 class="anchor anchorWithStickyNavbar_y2LR" id="SQL-Source-Configs">SQL Source Configs<a class="hash-link" href="#SQL-Source-Configs" title="Direct link to heading"></a></h4><p>Configurations controlling the behavior of SQL source in Hudi Streamer.</p><p><a href="#SQL-Source-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamersourcesqlsqlquery">hoodie.streamer.source.sql.sql.query</a></td><td>(N/A)</td><td>SQL query for fetching source data.<br><code>Config Param: SOURCE_SQL</code></td></tr></tbody></table><hr><h3 class="anchor anchorWithStickyNavbar_y2LR" id="SCHEMA_PROVIDER">Hudi Streamer Schema Provider Configs<a class="hash-link" href="#SCHEMA_PROVIDER" title="Direct link to heading"></a></h3><p>Configurations that control the schema provider for Hudi Streamer.</p><h4 class="anchor anchorWithStickyNavbar_y2LR" id="Hudi-Streamer-Schema-Provider-Configs">Hudi Streamer Schema Provider Configs<a class="hash-link" href="#Hudi-Streamer-Schema-Provider-Configs" title="Direct link to heading"></a></h4><p><a href="#Hudi-Streamer-Schema-Provider-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamerschemaproviderregistrytargetUrl">hoodie.streamer.schemaprovider.registry.targetUrl</a></td><td>(N/A)</td><td>The schema of the target you are writing to e.g. https://foo:<a href="mailto:bar@schemaregistry.org" target="_blank" rel="noopener noreferrer">bar@schemaregistry.org</a><br><code>Config Param: TARGET_SCHEMA_REGISTRY_URL</code></td></tr><tr><td><a href="#hoodiestreamerschemaproviderregistryurl">hoodie.streamer.schemaprovider.registry.url</a></td><td>(N/A)</td><td>The schema of the source you are reading from e.g. https://foo:<a href="mailto:bar@schemaregistry.org" target="_blank" rel="noopener noreferrer">bar@schemaregistry.org</a><br><code>Config Param: SRC_SCHEMA_REGISTRY_URL</code></td></tr></tbody></table><hr><h4 class="anchor anchorWithStickyNavbar_y2LR" id="File-based-Schema-Provider-Configs">File-based Schema Provider Configs<a class="hash-link" href="#File-based-Schema-Provider-Configs" title="Direct link to heading"></a></h4><p>Configurations for file-based schema provider.</p><p><a href="#File-based-Schema-Provider-Configs-basic-configs"><strong>Basic Configs</strong></a></p><table><thead><tr><th>Config Name</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td><a href="#hoodiestreamerschemaprovidersourceschemafile">hoodie.streamer.schemaprovider.source.schema.file</a></td><td>(N/A)</td><td>The schema of the source you are reading from<br><code>Config Param: SOURCE_SCHEMA_FILE</code></td></tr><tr><td><a href="#hoodiestreamerschemaprovidertargetschemafile">hoodie.streamer.schemaprovider.target.schema.file</a></td><td>(N/A)</td><td>The schema of the target you are writing to<br><code>Config Param: TARGET_SCHEMA_FILE</code></td></tr></tbody></table><hr></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/hudi/tree/asf-site/website/versioned_docs/version-0.14.0/basic_configurations.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_mS5F" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_mt2f"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/docs/0.14.0/precommit_validator"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">Data Quality</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/docs/0.14.0/configurations"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">All Configurations</div></a></div></nav></div></div><div class="col col--3"><div class="tableOfContents_vrFS thin-scrollbar theme-doc-toc-desktop"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#SPARK_DATASOURCE" class="table-of-contents__link toc-highlight">Spark Datasource Configs</a><ul><li><a href="#Read-Options" class="table-of-contents__link toc-highlight">Read Options</a></li><li><a href="#Write-Options" class="table-of-contents__link toc-highlight">Write Options</a></li></ul></li><li><a href="#FLINK_SQL" class="table-of-contents__link toc-highlight">Flink Sql Configs</a><ul><li><a href="#Flink-Options" class="table-of-contents__link toc-highlight">Flink Options</a></li></ul></li><li><a href="#WRITE_CLIENT" class="table-of-contents__link toc-highlight">Write Client Configs</a><ul><li><a href="#Metadata-Configs" class="table-of-contents__link toc-highlight">Metadata Configs</a></li><li><a href="#Storage-Configs" class="table-of-contents__link toc-highlight">Storage Configs</a></li><li><a href="#Archival-Configs" class="table-of-contents__link toc-highlight">Archival Configs</a></li><li><a href="#Bootstrap-Configs" class="table-of-contents__link toc-highlight">Bootstrap Configs</a></li><li><a href="#Clean-Configs" class="table-of-contents__link toc-highlight">Clean Configs</a></li><li><a href="#Clustering-Configs" class="table-of-contents__link toc-highlight">Clustering Configs</a></li><li><a href="#Compaction-Configs" class="table-of-contents__link toc-highlight">Compaction Configs</a></li><li><a href="#Write-Configurations" class="table-of-contents__link toc-highlight">Write Configurations</a></li><li><a href="#KEY_GENERATOR" class="table-of-contents__link toc-highlight">Key Generator Configs</a></li><li><a href="#INDEX" class="table-of-contents__link toc-highlight">Index Configs</a></li></ul></li><li><a href="#META_SYNC" class="table-of-contents__link toc-highlight">Metastore and Catalog Sync Configs</a><ul><li><a href="#Common-Metadata-Sync-Configs" class="table-of-contents__link toc-highlight">Common Metadata Sync Configs</a></li><li><a href="#BigQuery-Sync-Configs" class="table-of-contents__link toc-highlight">BigQuery Sync Configs</a></li><li><a href="#Hive-Sync-Configs" class="table-of-contents__link toc-highlight">Hive Sync Configs</a></li><li><a href="#Global-Hive-Sync-Configs" class="table-of-contents__link toc-highlight">Global Hive Sync Configs</a></li><li><a href="#DataHub-Sync-Configs" class="table-of-contents__link toc-highlight">DataHub Sync Configs</a></li></ul></li><li><a href="#METRICS" class="table-of-contents__link toc-highlight">Metrics Configs</a><ul><li><a href="#Metrics-Configurations" class="table-of-contents__link toc-highlight">Metrics Configurations</a></li></ul></li><li><a href="#KAFKA_CONNECT" class="table-of-contents__link toc-highlight">Kafka Connect Configs</a><ul><li><a href="#Kafka-Sink-Connect-Configurations" class="table-of-contents__link toc-highlight">Kafka Sink Connect Configurations</a></li></ul></li><li><a href="#HUDI_STREAMER" class="table-of-contents__link toc-highlight">Hudi Streamer Configs</a><ul><li><a href="#Hudi-Streamer-Configs" class="table-of-contents__link toc-highlight">Hudi Streamer Configs</a></li><li><a href="#Hudi-Streamer-SQL-Transformer-Configs" class="table-of-contents__link toc-highlight">Hudi Streamer SQL Transformer Configs</a></li><li><a href="#DELTA_STREAMER_SOURCE" class="table-of-contents__link toc-highlight">Hudi Streamer Source Configs</a></li><li><a href="#SCHEMA_PROVIDER" class="table-of-contents__link toc-highlight">Hudi Streamer Schema Provider Configs</a></li></ul></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">About</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/blog/2021/07/21/streaming-data-lake-platform">Our Vision</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/concepts">Concepts</a></li><li class="footer__item"><a class="footer__link-item" href="/community/team">Team</a></li><li class="footer__item"><a class="footer__link-item" href="/releases/release-0.14.1">Releases</a></li><li class="footer__item"><a class="footer__link-item" href="/releases/download">Download</a></li><li class="footer__item"><a class="footer__link-item" href="/powered-by">Who's Using</a></li></ul></div><div class="col footer__col"><div class="footer__title">Learn</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/quick-start-guide">Quick Start</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/docker_demo">Docker Demo</a></li><li class="footer__item"><a class="footer__link-item" href="/blog">Blog</a></li><li class="footer__item"><a class="footer__link-item" href="/talks">Talks</a></li><li class="footer__item"><a class="footer__link-item" href="/videos">Video Guides</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/faq">FAQ</a></li><li class="footer__item"><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Technical Wiki<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Hudi On Cloud</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/s3_hoodie">AWS</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/gcs_hoodie">Google Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/oss_hoodie">Alibaba Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/azure_hoodie">Microsoft Azure</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/cos_hoodie">Tencent Cloud</a></li><li class="footer__item"><a class="footer__link-item" href="/docs/ibm_cos_hoodie">IBM Cloud</a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/community/get-involved">Get Involved</a></li><li class="footer__item"><a href="https://join.slack.com/t/apache-hudi/shared_invite/zt-2ggm1fub8-_yt4Reu9djwqqVRFC7X49g" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Slack<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/hudi" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://twitter.com/ApacheHudi" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Twitter<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://www.youtube.com/channel/UCs7AhE0BWaEPZSChrBR-Muw" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>YouTube<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://www.linkedin.com/company/apache-hudi/?viewAsMember=true" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Linkedin<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="mailto:dev-subscribe@hudi.apache.org?Subject=SubscribeToHudi" target="_blank" rel="noopener noreferrer" class="footer__link-item">Mailing List</a></li></ul></div><div class="col footer__col"><div class="footer__title">Apache</div><ul class="footer__items"><li class="footer__item"><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="footer__link-item">Events</a></li><li class="footer__item"><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Thanks</a></li><li class="footer__item"><a href="https://www.apache.org/licenses" target="_blank" rel="noopener noreferrer" class="footer__link-item">License</a></li><li class="footer__item"><a href="https://www.apache.org/security" target="_blank" rel="noopener noreferrer" class="footer__link-item">Security</a></li><li class="footer__item"><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Sponsorship</a></li><li class="footer__item"><a href="https://www.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item">Foundation</a></li></ul></div></div><div class="footer__bottom text--center"><div class="margin-bottom--sm"><a href="https://hudi.apache.org/" target="_blank" rel="noopener noreferrer" class="footerLogoLink_SRtH"><img src="/assets/images/logo-big.png" alt="Apache Hudi™" class="themedImage_TMUO themedImage--light_4Vu1 footer__logo"><img src="/assets/images/logo-big.png" alt="Apache Hudi™" class="themedImage_TMUO themedImage--dark_uzRr footer__logo"></a></div><div class="footer__copyright">Copyright © 2021 <a href="https://apache.org">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0"> Apache License, Version 2.0</a>. <br>Hudi, Apache and the Apache feather logo are trademarks of The Apache Software Foundation.</div></div></div></footer></div> |
