| <title data-rh="true">Release 1.8.0 | Apache InLong</title><meta data-rh="true" name="viewport" content="width=device-width,initial-scale=1"><meta data-rh="true" name="twitter:card" content="summary_large_image"><meta data-rh="true" property="og:url" content="https://inlong.apache.org/blog/2023/07/24/release-1.8.0"><meta data-rh="true" name="docusaurus_locale" content="en"><meta data-rh="true" name="docusaurus_tag" content="default"><meta data-rh="true" name="docsearch:language" content="en"><meta data-rh="true" name="docsearch:docusaurus_tag" content="default"><meta data-rh="true" property="og:title" content="Release 1.8.0 | Apache InLong"><meta data-rh="true" name="description" content="Apache InLong recently released version 1.8.0, which closed about 190+ issues, including 6+ major features and 30+ optimizations. The main improvements include multi-tenant management, support for multiple Apache Flink versions, data synchronization in Dashboard, data preview support, and optimization of ultra-long log processing logic. After the release of 1.8.0, Apache InLong has completed the layout of full-scenario data integration around data access, data synchronization, and data subscription, combined with multi-tenant management, multi-cluster management, approval flow management, and full-link audit/metrics."><meta data-rh="true" property="og:description" content="Apache InLong recently released version 1.8.0, which closed about 190+ issues, including 6+ major features and 30+ optimizations. The main improvements include multi-tenant management, support for multiple Apache Flink versions, data synchronization in Dashboard, data preview support, and optimization of ultra-long log processing logic. After the release of 1.8.0, Apache InLong has completed the layout of full-scenario data integration around data access, data synchronization, and data subscription, combined with multi-tenant management, multi-cluster management, approval flow management, and full-link audit/metrics."><meta data-rh="true" property="og:type" content="article"><meta data-rh="true" property="article:published_time" content="2023-07-24T00:00:00.000Z"><meta data-rh="true" property="article:author" content="https://github.com/vernedeng"><meta data-rh="true" property="article:tag" content="Apache InLong,Version"><link data-rh="true" rel="icon" href="/img/logo.svg"><link data-rh="true" rel="canonical" href="https://inlong.apache.org/blog/2023/07/24/release-1.8.0"><link data-rh="true" rel="alternate" href="https://inlong.apache.org/blog/2023/07/24/release-1.8.0" hreflang="en"><link data-rh="true" rel="alternate" href="https://inlong.apache.org/zh-CN/blog/2023/07/24/release-1.8.0" hreflang="zh-CN"><link data-rh="true" rel="alternate" href="https://inlong.apache.org/blog/2023/07/24/release-1.8.0" hreflang="x-default"><link data-rh="true" rel="preconnect" href="https://YUW9QEL53E-dsn.algolia.net" crossorigin="anonymous"><link rel="alternate" type="application/rss+xml" href="/blog/rss.xml" title="Apache InLong RSS Feed"> |
| <div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/logo.svg" alt="Apache" class="themedImage_ToTc themedImage--light_HNdA"><img src="/img/logo.svg" alt="Apache" class="themedImage_ToTc themedImage--dark_i4oU"></div><b class="navbar__title text--truncate">Apache InLong</b></a></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link" aria-haspopup="true" aria-expanded="false" role="button" href="/docs/introduction">Docs</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/next/introduction">Next</a></li><li><a class="dropdown__link" href="/docs/introduction">1.11.0</a></li><li><a class="dropdown__link" href="/docs/1.10.0/introduction">1.10.0</a></li><li><a class="dropdown__link" href="/docs/1.9.0/introduction">1.9.0</a></li><li><a class="dropdown__link" href="/docs/1.8.0/introduction">1.8.0</a></li><li><a class="dropdown__link" href="/versions/">All versions</a></li></ul></div><a class="navbar__item navbar__link" href="/downloads">Download</a><a class="navbar__item navbar__link" href="/community/how-to-contribute">Community</a><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><a class="navbar__item navbar__link" href="/team">Team</a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">ASF</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Apache Software Foundation</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship</a></li><li><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks</a></li></ul></div><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link"><svg viewBox="0 0 24 24" width="20" height="20" aria-hidden="true" class="iconLanguage_nlXk"><path fill="currentColor" d="M12.87 15.07l-2.54-2.51.03-.03c1.74-1.94 2.98-4.17 3.71-6.53H17V4h-7V2H8v2H1v1.99h11.17C11.5 7.92 10.44 9.75 9 11.35 8.07 10.32 7.3 9.19 6.69 8h-2c.73 1.63 1.73 3.17 2.98 4.56l-5.09 5.02L4 19l5-5 3.11 3.11.76-2.04zM18.5 10h-2L12 22h2l1.12-3h4.75L21 22h2l-4.5-12zm-2.62 7l1.62-4.33L19.12 17h-3.24z"></path></svg>English</a><ul class="dropdown__menu"><li><a href="/blog/2023/07/24/release-1.8.0" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" lang="en">English</a></li><li><a href="/zh-CN/blog/2023/07/24/release-1.8.0" target="_self" rel="noopener noreferrer" class="dropdown__link" lang="zh-CN">简体中文</a></li></ul></div><a href="https://github.com/apache/inlong" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link">GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a><div class="toggle_vylO colorModeToggle_DEke"><button class="clean-btn toggleButton_gllP toggleButtonDisabled_aARS" type="button" disabled="" title="Switch between dark and light mode (currently light mode)" aria-label="Switch between dark and light mode (currently light mode)" aria-live="polite"><svg viewBox="0 0 24 24" width="24" height="24" class="lightToggleIcon_pyhR"><path fill="currentColor" d="M12,9c1.65,0,3,1.35,3,3s-1.35,3-3,3s-3-1.35-3-3S10.35,9,12,9 M12,7c-2.76,0-5,2.24-5,5s2.24,5,5,5s5-2.24,5-5 S14.76,7,12,7L12,7z M2,13l2,0c0.55,0,1-0.45,1-1s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S1.45,13,2,13z M20,13l2,0c0.55,0,1-0.45,1-1 s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S19.45,13,20,13z M11,2v2c0,0.55,0.45,1,1,1s1-0.45,1-1V2c0-0.55-0.45-1-1-1S11,1.45,11,2z M11,20v2c0,0.55,0.45,1,1,1s1-0.45,1-1v-2c0-0.55-0.45-1-1-1C11.45,19,11,19.45,11,20z M5.99,4.58c-0.39-0.39-1.03-0.39-1.41,0 c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0s0.39-1.03,0-1.41L5.99,4.58z M18.36,16.95 c-0.39-0.39-1.03-0.39-1.41,0c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0c0.39-0.39,0.39-1.03,0-1.41 L18.36,16.95z M19.42,5.99c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06c-0.39,0.39-0.39,1.03,0,1.41 s1.03,0.39,1.41,0L19.42,5.99z M7.05,18.36c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06 c-0.39,0.39-0.39,1.03,0,1.41s1.03,0.39,1.41,0L7.05,18.36z"></path></svg><svg viewBox="0 0 24 24" width="24" height="24" class="darkToggleIcon_wfgR"><path fill="currentColor" d="M9.37,5.51C9.19,6.15,9.1,6.82,9.1,7.5c0,4.08,3.32,7.4,7.4,7.4c0.68,0,1.35-0.09,1.99-0.27C17.45,17.19,14.93,19,12,19 c-3.86,0-7-3.14-7-7C5,9.07,6.81,6.55,9.37,5.51z M12,3c-4.97,0-9,4.03-9,9s4.03,9,9,9s9-4.03,9-9c0-0.46-0.04-0.92-0.1-1.36 c-0.98,1.37-2.58,2.26-4.4,2.26c-2.98,0-5.4-2.42-5.4-5.4c0-1.81,0.89-3.42,2.26-4.4C12.92,3.04,12.46,3,12,3L12,3z"></path></svg></button></div><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/12/13/release-1.10.0">Release 1.10.0</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/09/25/release-1.9.0">Release 1.9.0</a></li><li class="sidebarItem__DBe"><a aria-current="page" class="sidebarItemLink_mo7H sidebarItemLinkActive_I1ZP" href="/blog/2023/07/24/release-1.8.0">Release 1.8.0</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/05/19/release-1.7.0">Release 1.7.0</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/03/23/release-1.6.0">Release 1.6.0</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><article itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h1 class="title_f1Hy" itemprop="headline">Release 1.8.0</h1><div class="container_mt6G margin-vert--md"><time datetime="2023-07-24T00:00:00.000Z" itemprop="datePublished">July 24, 2023</time> · <!-- -->8 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/vernedeng" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://avatars.githubusercontent.com/u/45282474?v=4" alt="Verne Deng"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/vernedeng" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Verne Deng</span></a></div></div></div></div></div></header><div id="post-content" class="markdown" itemprop="articleBody"><p>Apache InLong recently released version 1.8.0, which closed about 190+ issues, including 6+ major features and 30+ optimizations. The main improvements include multi-tenant management, support for multiple Apache Flink versions, data synchronization in Dashboard, data preview support, and optimization of ultra-long log processing logic. After the release of 1.8.0, Apache InLong has completed the layout of full-scenario data integration around data access, data synchronization, and data subscription, combined with multi-tenant management, multi-cluster management, approval flow management, and full-link audit/metrics.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="about-apache-inlong">About Apache InLong<a href="#about-apache-inlong" class="hash-link" aria-label="Direct link to About Apache InLong" title="Direct link to About Apache InLong"></a></h2><p>As the industry's first one-stop, full-scenario, open-source massive data integration framework, Apache InLong provides automatic, safe, reliable, and high-performance data transmission capabilities to facilitate businesses to build stream-based data analysis, modeling, and applications quickly. At present, InLong is widely used in various industries such as advertising, payment, social networking, games, artificial intelligence, etc., serving thousands of businesses, among which the scale of high-performance scene data exceeds 1 trillion lines per day, and the scale of high-reliability scene data exceeds 10 trillion lines per day.</p><p>The core keywords of InLong project positioning are "one-stop" and "massive data". For "one-stop", we hope to shield technical details, provide complete data integration and support services, and implement out-of-the-box; With its advantages, such as multi-cluster management, it can stably support larger-scale data volumes based on trillions of lines per day.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="overview-of-version-180">Overview of version 1.8.0<a href="#overview-of-version-180" class="hash-link" aria-label="Direct link to Overview of version 1.8.0" title="Direct link to Overview of version 1.8.0"></a></h2><p>Apache InLong recently released version 1.8.0, which closed about 190+ issues, including 6+ major features and 30+ optimizations, mainly completing multi-tenant management, support for multiple Apache Flink versions, data synchronization in Dashboard, data preview support, and optimization of ultra-long log processing logic. After the release of 1.8.0, Apache InLong has completed the layout of full-scenario data integration around data access, data synchronization, and data subscription, combined with multi-tenant management, multi-cluster management, approval flow management, and full-link audit/metrics. Apache InLong has built a comprehensive data integration solution, achieving out-of-the-box usability:</p><ul><li>Data access: Data access is the process of aggregating data from data sources to the same storage service for further data querying and analysis;</li><li>Data synchronization: Data synchronization is the process of establishing consistency between data sources and target data storage, which can coordinate data over time;</li><li>Data subscription: Data subscription provides subscribers with the data they are authorized to access;</li></ul><p>In version 1.8.0 of Apache InLong, many other features have also been completed, mainly including:</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="agent-module">Agent module<a href="#agent-module" class="hash-link" aria-label="Direct link to Agent module" title="Direct link to Agent module"></a></h3><ul><li>Optimized ultra-long log processing logic, improving file collection efficiency and stability</li><li>Fixed the issue of thread leakage caused by task termination</li><li>Adopted flow control to solve the OOM problem caused by file number growth</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="dataproxy-module">DataProxy module<a href="#dataproxy-module" class="hash-link" aria-label="Direct link to DataProxy module" title="Direct link to DataProxy module"></a></h3><ul><li>Support for Golang SDK</li><li>Support for configuring black and white lists based on full IP or CIDR format IP segments</li><li>Support for configuring the maximum number of write retries</li><li>Support for sending data to the default Topic when write fails</li><li>Code refactoring, unified configuration acquisition method</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="sort-module">Sort module<a href="#sort-module" class="hash-link" aria-label="Direct link to Sort module" title="Direct link to Sort module"></a></h3><ul><li>Enhanced DDL parsing capability, improving the stability of DDL-aware scenarios</li><li>Support for multiple Flink versions</li><li>Support for Decimal precision recognition in whole-library scenarios</li><li>Hive supports whole-library migration, with the implementation consistent with MySQL whole-library migration</li><li>Iceberg supports automatic column updates and column deletions, greatly enriching Schema change capabilities</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="manager-module">Manager module<a href="#manager-module" class="hash-link" aria-label="Direct link to Manager module" title="Direct link to Manager module"></a></h3><ul><li>Support for Pulsar, TubeMQ data preview</li><li>Support for dynamic configuration of audit data sources</li><li>Support for querying audit delay information</li><li>Support for multi-tenant management</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="dashboard-module">Dashboard module<a href="#dashboard-module" class="hash-link" aria-label="Direct link to Dashboard module" title="Direct link to Dashboard module"></a></h3><ul><li>Support for data flow preview</li><li>Support for viewing InLongGroup resource details</li><li>Support for tenant management and tenant switching</li><li>Support for data synchronization</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="others">Others<a href="#others" class="hash-link" aria-label="Direct link to Others" title="Direct link to Others"></a></h3><ul><li>Remove conflicting Jsqlparser versions</li><li>Upgrade Spring-Boot-Autoconfigure version to 2.6.15</li><li>Upgrade the Snappy-Java version to 1.1.10.1</li><li>Fix syntax errors in Workflow configuration files</li></ul><h2 class="anchor anchorWithStickyNavbar_LWe7" id="180-feature-introduction">1.8.0 Feature Introduction<a href="#180-feature-introduction" class="hash-link" aria-label="Direct link to 1.8.0 Feature Introduction" title="Direct link to 1.8.0 Feature Introduction"></a></h2><h3 class="anchor anchorWithStickyNavbar_LWe7" id="agent-optimizes-ultra-long-log-processing-logic-improving-file-collection-efficiency-and-stability">Agent optimizes ultra-long log processing logic, improving file collection efficiency and stability<a href="#agent-optimizes-ultra-long-log-processing-logic-improving-file-collection-efficiency-and-stability" class="hash-link" aria-label="Direct link to Agent optimizes ultra-long log processing logic, improving file collection efficiency and stability" title="Direct link to Agent optimizes ultra-long log processing logic, improving file collection efficiency and stability"></a></h3><p>In actual use, due to improper use by users or bugs in data production programs, occasionally, a single data length reaches MB or even GB level. For Agents deployed in low-profile environments, this type of data greatly affects the performance of sending. The lower version Agent reads this type of data directly into memory based on the newline character and then discards it, but is limited by the hardware configuration of the Agent deployment environment, a single ultra-long data is extremely likely to cause OOM exceptions. In version 1.8.0, Agent optimized the processing logic of ultra-long logs, ensuring that data loading does not exceed memory limits through segmented collection and segmented discarding. Thanks to @justinhuang's contribution, see INLONG-8180 for details. |
| <img loading="lazy" alt="1.8.0-agent-flow-controll.png" src="/assets/images/1.8.0-agent-flow-controll-586153f2b9ee526b805dd01af8db56f1.png" width="3356" height="1213" class="img_ev3q"></p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="support-multiple-flink-versions">Support multiple Flink versions<a href="#support-multiple-flink-versions" class="hash-link" aria-label="Direct link to Support multiple Flink versions" title="Direct link to Support multiple Flink versions"></a></h3><p>As community users go deeper into using InLong, the scenarios InLong faces become more diverse and complex. To support the needs of users in different Flink environments, InLong has added support for multiple Flink versions in the current version. Users can choose the Flink version to start in the plugins/flink-sort-plugin.properties configuration file in InLong-Manager.</p><p>When changing the Flink version required to run the Sort component, you also need to replace the connectors in the InLong-Sort/connector directory with the corresponding version of the jar package. For details, see the InLong official website documentation. Thanks to @Emsnap, @GanfengTan, and @haifxu for their contributions to this capability.</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain"># inLong-manager/plugins/flink-sort-plugin.properties</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"># Flink version, support [1.13|1.15]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">flink.version=1.13</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="support-multi-tenant-management">Support multi-tenant management<a href="#support-multi-tenant-management" class="hash-link" aria-label="Direct link to Support multi-tenant management" title="Direct link to Support multi-tenant management"></a></h3><p>To address the requirements for permission and resource isolation in multi-user scenarios, InLong introduces a multi-tenant architecture in the current version. The multi-tenant architecture ensures that data and permissions do not interfere with each other among different users within the same group of services. Thanks to @vernedeng and @bluewang for their contributions to this feature, see INLONG-7914 for feature details. The following is the core process: |
