| <title data-rh="true">Release 1.10.0 | Apache InLong</title><meta data-rh="true" name="viewport" content="width=device-width,initial-scale=1"><meta data-rh="true" name="twitter:card" content="summary_large_image"><meta data-rh="true" property="og:url" content="https://inlong.apache.org/blog/2023/12/13/release-1.10.0"><meta data-rh="true" name="docusaurus_locale" content="en"><meta data-rh="true" name="docusaurus_tag" content="default"><meta data-rh="true" name="docsearch:language" content="en"><meta data-rh="true" name="docsearch:docusaurus_tag" content="default"><meta data-rh="true" property="og:title" content="Release 1.10.0 | Apache InLong"><meta data-rh="true" name="description" content="Apache InLong recently released version 1.10.0, which closed about 200+ issues, including 6+ major features and 30+ optimizations. The main features include Manager supporting viewing operation logs, supporting Group migration between clusters, Agent supporting periodic collection and task supplementation, Sort real-time synchronization supporting Transform, supporting MySQL to Iceberg whole database synchronization, and supporting automatic table creation. After the release of 1.10.0, Apache InLong has enriched and optimized Agent function scenarios, added the ability of whole database synchronization and automatic table creation, supported viewing operation logs, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance."><meta data-rh="true" property="og:description" content="Apache InLong recently released version 1.10.0, which closed about 200+ issues, including 6+ major features and 30+ optimizations. The main features include Manager supporting viewing operation logs, supporting Group migration between clusters, Agent supporting periodic collection and task supplementation, Sort real-time synchronization supporting Transform, supporting MySQL to Iceberg whole database synchronization, and supporting automatic table creation. After the release of 1.10.0, Apache InLong has enriched and optimized Agent function scenarios, added the ability of whole database synchronization and automatic table creation, supported viewing operation logs, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance."><meta data-rh="true" property="og:type" content="article"><meta data-rh="true" property="article:published_time" content="2023-12-13T00:00:00.000Z"><meta data-rh="true" property="article:author" content="https://github.com/vernedeng"><meta data-rh="true" property="article:tag" content="Apache InLong,Version"><link data-rh="true" rel="icon" href="/img/logo.svg"><link data-rh="true" rel="canonical" href="https://inlong.apache.org/blog/2023/12/13/release-1.10.0"><link data-rh="true" rel="alternate" href="https://inlong.apache.org/blog/2023/12/13/release-1.10.0" hreflang="en"><link data-rh="true" rel="alternate" href="https://inlong.apache.org/zh-CN/blog/2023/12/13/release-1.10.0" hreflang="zh-CN"><link data-rh="true" rel="alternate" href="https://inlong.apache.org/blog/2023/12/13/release-1.10.0" hreflang="x-default"><link data-rh="true" rel="preconnect" href="https://YUW9QEL53E-dsn.algolia.net" crossorigin="anonymous"><link rel="alternate" type="application/rss+xml" href="/blog/rss.xml" title="Apache InLong RSS Feed"> |
| <div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/logo.svg" alt="Apache" class="themedImage_ToTc themedImage--light_HNdA"><img src="/img/logo.svg" alt="Apache" class="themedImage_ToTc themedImage--dark_i4oU"></div><b class="navbar__title text--truncate">Apache InLong</b></a></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link" aria-haspopup="true" aria-expanded="false" role="button" href="/docs/introduction">Docs</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/next/introduction">Next</a></li><li><a class="dropdown__link" href="/docs/introduction">1.11.0</a></li><li><a class="dropdown__link" href="/docs/1.10.0/introduction">1.10.0</a></li><li><a class="dropdown__link" href="/docs/1.9.0/introduction">1.9.0</a></li><li><a class="dropdown__link" href="/docs/1.8.0/introduction">1.8.0</a></li><li><a class="dropdown__link" href="/versions/">All versions</a></li></ul></div><a class="navbar__item navbar__link" href="/downloads">Download</a><a class="navbar__item navbar__link" href="/community/how-to-contribute">Community</a><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><a class="navbar__item navbar__link" href="/team">Team</a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">ASF</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Apache Software Foundation</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship</a></li><li><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks</a></li></ul></div><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link"><svg viewBox="0 0 24 24" width="20" height="20" aria-hidden="true" class="iconLanguage_nlXk"><path fill="currentColor" d="M12.87 15.07l-2.54-2.51.03-.03c1.74-1.94 2.98-4.17 3.71-6.53H17V4h-7V2H8v2H1v1.99h11.17C11.5 7.92 10.44 9.75 9 11.35 8.07 10.32 7.3 9.19 6.69 8h-2c.73 1.63 1.73 3.17 2.98 4.56l-5.09 5.02L4 19l5-5 3.11 3.11.76-2.04zM18.5 10h-2L12 22h2l1.12-3h4.75L21 22h2l-4.5-12zm-2.62 7l1.62-4.33L19.12 17h-3.24z"></path></svg>English</a><ul class="dropdown__menu"><li><a href="/blog/2023/12/13/release-1.10.0" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" lang="en">English</a></li><li><a href="/zh-CN/blog/2023/12/13/release-1.10.0" target="_self" rel="noopener noreferrer" class="dropdown__link" lang="zh-CN">简体中文</a></li></ul></div><a href="https://github.com/apache/inlong" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link">GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a><div class="toggle_vylO colorModeToggle_DEke"><button class="clean-btn toggleButton_gllP toggleButtonDisabled_aARS" type="button" disabled="" title="Switch between dark and light mode (currently light mode)" aria-label="Switch between dark and light mode (currently light mode)" aria-live="polite"><svg viewBox="0 0 24 24" width="24" height="24" class="lightToggleIcon_pyhR"><path fill="currentColor" d="M12,9c1.65,0,3,1.35,3,3s-1.35,3-3,3s-3-1.35-3-3S10.35,9,12,9 M12,7c-2.76,0-5,2.24-5,5s2.24,5,5,5s5-2.24,5-5 S14.76,7,12,7L12,7z M2,13l2,0c0.55,0,1-0.45,1-1s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S1.45,13,2,13z M20,13l2,0c0.55,0,1-0.45,1-1 s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S19.45,13,20,13z M11,2v2c0,0.55,0.45,1,1,1s1-0.45,1-1V2c0-0.55-0.45-1-1-1S11,1.45,11,2z M11,20v2c0,0.55,0.45,1,1,1s1-0.45,1-1v-2c0-0.55-0.45-1-1-1C11.45,19,11,19.45,11,20z M5.99,4.58c-0.39-0.39-1.03-0.39-1.41,0 c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0s0.39-1.03,0-1.41L5.99,4.58z M18.36,16.95 c-0.39-0.39-1.03-0.39-1.41,0c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0c0.39-0.39,0.39-1.03,0-1.41 L18.36,16.95z M19.42,5.99c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06c-0.39,0.39-0.39,1.03,0,1.41 s1.03,0.39,1.41,0L19.42,5.99z M7.05,18.36c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06 c-0.39,0.39-0.39,1.03,0,1.41s1.03,0.39,1.41,0L7.05,18.36z"></path></svg><svg viewBox="0 0 24 24" width="24" height="24" class="darkToggleIcon_wfgR"><path fill="currentColor" d="M9.37,5.51C9.19,6.15,9.1,6.82,9.1,7.5c0,4.08,3.32,7.4,7.4,7.4c0.68,0,1.35-0.09,1.99-0.27C17.45,17.19,14.93,19,12,19 c-3.86,0-7-3.14-7-7C5,9.07,6.81,6.55,9.37,5.51z M12,3c-4.97,0-9,4.03-9,9s4.03,9,9,9s9-4.03,9-9c0-0.46-0.04-0.92-0.1-1.36 c-0.98,1.37-2.58,2.26-4.4,2.26c-2.98,0-5.4-2.42-5.4-5.4c0-1.81,0.89-3.42,2.26-4.4C12.92,3.04,12.46,3,12,3L12,3z"></path></svg></button></div><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a aria-current="page" class="sidebarItemLink_mo7H sidebarItemLinkActive_I1ZP" href="/blog/2023/12/13/release-1.10.0">Release 1.10.0</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/09/25/release-1.9.0">Release 1.9.0</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/07/24/release-1.8.0">Release 1.8.0</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/05/19/release-1.7.0">Release 1.7.0</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/03/23/release-1.6.0">Release 1.6.0</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><article itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h1 class="title_f1Hy" itemprop="headline">Release 1.10.0</h1><div class="container_mt6G margin-vert--md"><time datetime="2023-12-13T00:00:00.000Z" itemprop="datePublished">December 13, 2023</time> · <!-- -->6 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/vernedeng" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://avatars.githubusercontent.com/u/45282474?v=4" alt="Verne Deng"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/vernedeng" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Verne Deng</span></a></div></div></div></div></div></header><div id="post-content" class="markdown" itemprop="articleBody"><p>Apache InLong recently released version 1.10.0, which closed about 200+ issues, including 6+ major features and 30+ optimizations. The main features include Manager supporting viewing operation logs, supporting Group migration between clusters, Agent supporting periodic collection and task supplementation, Sort real-time synchronization supporting Transform, supporting MySQL to Iceberg whole database synchronization, and supporting automatic table creation. After the release of 1.10.0, Apache InLong has enriched and optimized Agent function scenarios, added the ability of whole database synchronization and automatic table creation, supported viewing operation logs, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="about-apache-inlong">About Apache InLong<a href="#about-apache-inlong" class="hash-link" aria-label="Direct link to About Apache InLong" title="Direct link to About Apache InLong"></a></h2><p>As the industry's first one-stop, full-scenario, open-source massive data integration framework, Apache InLong provides automatic, safe, reliable, and high-performance data transmission capabilities to facilitate businesses to build stream-based data analysis, modeling, and applications quickly. At present, InLong is widely used in various industries such as advertising, payment, social networking, games, artificial intelligence, etc., serving thousands of businesses, among which the scale of high-performance scene data exceeds 1 trillion lines per day, and the scale of high-reliability scene data exceeds 10 trillion lines per day.</p><p>The core keywords of InLong project positioning are "one-stop" and "massive data". For "one-stop", we hope to shield technical details, provide complete data integration and support services, and implement out-of-the-box; With its advantages, such as multi-cluster management, it can stably support larger-scale data volumes based on trillions of lines per day.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="1100-version-overview">1.10.0 Version Overview<a href="#1100-version-overview" class="hash-link" aria-label="Direct link to 1.10.0 Version Overview" title="Direct link to 1.10.0 Version Overview"></a></h2><p>Apache InLong recently released version 1.10.0, which closed about 200+ issues, including 6+ major features and 30+ optimizations. The main features include Manager supporting viewing operation logs, supporting Group migration between clusters, Agent supporting periodic collection and task supplementation, Sort real-time synchronization supporting Transform, supporting MySQL to Iceberg whole database synchronization, and supporting automatic table creation. After the release of 1.10.0, Apache InLong has enriched and optimized Agent function scenarios, added the ability of whole database synchronization and automatic table creation, supported viewing operation logs, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance. In Apache InLong 1.10.0 version, a large number of other features have also been completed, mainly including:</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="agent-module">Agent Module<a href="#agent-module" class="hash-link" aria-label="Direct link to Agent Module" title="Direct link to Agent Module"></a></h3><ul><li>Support periodic collection and task supplementation</li><li>Global memory control to avoid OOM caused by large business data</li><li>Achieve final consistency between Agent and Manager tasks</li><li>Enrich Agent audit dimensions</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="sort-module">Sort Module<a href="#sort-module" class="hash-link" aria-label="Direct link to Sort Module" title="Direct link to Sort Module"></a></h3><ul><li>Support MySQL to Iceberg whole database synchronization</li><li>Support richer Flink 1.15 Connectors: MongoDB, Iceberg, SQLServer, HBase</li><li>Support task status management, able to view real-time synchronization task status</li><li>Real-time tasks support automatic table creation</li><li>Support configuration of Transform</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="manager-module">Manager Module<a href="#manager-module" class="hash-link" aria-label="Direct link to Manager Module" title="Direct link to Manager Module"></a></h3><ul><li>Support Sort Standalone management</li><li>Separate transmission protocol and data protocol types</li><li>Support Group cluster switching</li><li>Support query audit data size</li><li>Support viewing operation logs</li><li>Apache Iceberg supports automatic Schema information fetching</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="dashboard-module">Dashboard Module<a href="#dashboard-module" class="hash-link" aria-label="Direct link to Dashboard Module" title="Direct link to Dashboard Module"></a></h3><ul><li>Support displaying resource creators and recent modifiers simultaneously</li><li>Support viewing operation logs</li><li>Approval directory displays Group corresponding to consumer groups</li><li>Support batch parsing of source fields</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="audit-module">Audit Module<a href="#audit-module" class="hash-link" aria-label="Direct link to Audit Module" title="Direct link to Audit Module"></a></h3><ul><li>Support automatic creation of required Kafka Topics when the service starts</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="sdk-module">SDK Module<a href="#sdk-module" class="hash-link" aria-label="Direct link to SDK Module" title="Direct link to SDK Module"></a></h3><ul><li>Resolve dependency version conflicts in DataProxy Java SDK</li><li>Update dependency version in DataProxy Go SDK</li><li>Use UUID instead of snowflake algorithm to generate data ID in DataProxy Go SDK</li><li>DataProxy C++ SDK supports dynamic load balancing</li><li>DataProxy C++ SDK supports multi-dimensional resource isolation</li><li>Optimize the ability of DataProxy C++ SDK to access nearby resources in multi-location deployment</li><li>DataProxy C++ SDK supports local disaster recovery</li><li>DataProxy C++ SDK supports dynamic update of DataProxy node information</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="others">Others<a href="#others" class="hash-link" aria-label="Direct link to Others" title="Direct link to Others"></a></h3><ul><li>Strengthen protection against request forgery attacks</li><li>Update Snappy version</li><li>Add Master branch protection strategy</li></ul><h2 class="anchor anchorWithStickyNavbar_LWe7" id="1100-version-feature-introduction">1.10.0 Version Feature Introduction<a href="#1100-version-feature-introduction" class="hash-link" aria-label="Direct link to 1.10.0 Version Feature Introduction" title="Direct link to 1.10.0 Version Feature Introduction"></a></h2><h3 class="anchor anchorWithStickyNavbar_LWe7" id="agent-supports-periodic-collection">Agent Supports Periodic Collection<a href="#agent-supports-periodic-collection" class="hash-link" aria-label="Direct link to Agent Supports Periodic Collection" title="Direct link to Agent Supports Periodic Collection"></a></h3><p>In version 1.10.0, InLong adds the ability to perform periodic file collection tasks. The periodic strategy includes daily, hourly, and real-time, and users can specify this strategy when creating a new file data source. At the same time, users can also configure time offsets to delay or advance collection. Thanks to @Justinhuang, @Blue |
