| <div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_ToTc themedImage--light_HNdA"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_ToTc themedImage--dark_i4oU"></div><b class="navbar__title text--truncate">Apache Linkis</b></a><a class="navbar__item navbar__link" href="/download/main">Download</a><a class="navbar__item navbar__link" href="/community/how-to-subscribe">Community</a><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><a class="navbar__item navbar__link" href="/team">Team</a><a class="navbar__item navbar__link" href="/user">Users</a><a href="https://docs.qq.com/doc/DUkdTTGhVSlZ0VXVt" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link">FAQ</a><div class="navbar__item dropdown dropdown--hoverable"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">ASF</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Foundation</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship</a></li><li><a href="https://privacy.apache.org/policies/privacy-policy-public.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks</a></li></ul></div></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Doc</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/latest/about/introduction">1.5.0</a></li><li><a class="dropdown__link" href="/docs/1.4.0/about/introduction">1.4.0</a></li><li><a class="dropdown__link" href="/docs/1.3.2/about/introduction">1.3.2</a></li><li><a class="dropdown__link" href="/docs/1.6.0/about/introduction">Next(1.6.0)</a></li><li><a class="dropdown__link" href="/versions">All Version</a></li></ul></div><a href="https://github.com/apache/linkis" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-github-link" aria-label="GitHub"></a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link"><svg viewBox="0 0 24 24" width="20" height="20" aria-hidden="true" class="iconLanguage_nlXk"><path fill="currentColor" d="M12.87 15.07l-2.54-2.51.03-.03c1.74-1.94 2.98-4.17 3.71-6.53H17V4h-7V2H8v2H1v1.99h11.17C11.5 7.92 10.44 9.75 9 11.35 8.07 10.32 7.3 9.19 6.69 8h-2c.73 1.63 1.73 3.17 2.98 4.56l-5.09 5.02L4 19l5-5 3.11 3.11.76-2.04zM18.5 10h-2L12 22h2l1.12-3h4.75L21 22h2l-4.5-12zm-2.62 7l1.62-4.33L19.12 17h-3.24z"></path></svg>English</a><ul class="dropdown__menu"><li><a href="/blog/tags/spark-2-4-8" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" lang="en">English</a></li><li><a href="/zh-CN/blog/tags/spark-2-4-8" target="_self" rel="noopener noreferrer" class="dropdown__link" lang="zh-CN">简体中文</a></li></ul></div><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2024/01/26/linkis130-adaptation-Huawei-MRS-share">Linkis 1.3.0 adapts to Huawei MRS+SCRIPTIS actual combat sharing</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2023/03/08/linkis-integration-with-oceanbase">Linkis 1.3.2 Integration with OceanBase</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/12/02/material-manage">Engine Material Management</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/10/09/linkis-service-merge">Apache Linkis 1.3.0 PES(Public Enhancement Services) Some Service Merge</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/09/27/linkis111-deploy">Deploy Apache Linkis1.1.1 and DSS1.1.0 based on CDH6.3.2</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/08/08/linkis111-compile-integration">Linkis1.1.1 adapts Hadoop 3.1.1 and deploys other services</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/07/16/deploy-linkis-with-kubernetes">Deploy Linkis with Kubernetes</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/07/04/how-to-add-auto-bot">How to add a GitHub Action for the GitHub repository</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/06/09/meetup-content-review">Apache Linkis Meet up</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/04/15/how-to-download-engineconn-plugin">How to Download Engine Plugins Not Included in the Installation Package By Default</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/03/20/openlookeng">Implementation of OpenLookEng Engine</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/02/21/linkis-deploy">Linkis Deployment Troubleshooting</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/2022/02/08/how-to-user-blog">How to Write a Blog</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "spark2.4.8"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/2022/09/27/linkis111-deploy">Deploy Apache Linkis1.1.1 and DSS1.1.0 based on CDH6.3.2</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-09-27T00:00:00.000Z" itemprop="datePublished">September 27, 2022</time> · <!-- -->5 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/kongslove" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://avatars.githubusercontent.com/u/42604208?v=4" alt="kevinWdong"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/kongslove" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">kevinWdong</span></a></div><small class="avatar__subtitle" itemprop="description">contributors</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><p>With the development of business and the update and iteration of community products, we found that Linkis1. X has greatly improved its performance in terms of resource management and engine management, which can better meet the requirements of the construction of data middle stations. Compared with version 0.9.3 and the platform we used before, the user experience has also been greatly improved, and the problems such as the inability to view details on the task failure page have also been improved. Therefore, we decided to upgrade Linkis and the WDS suite. The following are the specific practical operations, which we hope will give you a reference.</p><h1>1.Environment</h1><h2 class="anchor anchorWithStickyNavbar_LWe7" id="cdh632-component-versions">CDH6.3.2 Component versions<a class="hash-link" href="#cdh632-component-versions" title="Direct link to heading"></a></h2><ul><li>hadoop:3.0.0-cdh6.3.2</li><li>hive:2.1.1-cdh6.3.2</li><li>spark：2.4.8</li></ul><h2 class="anchor anchorWithStickyNavbar_LWe7" id="hardware-environment">hardware environment <a class="hash-link" href="#hardware-environment" title="Direct link to heading"></a></h2><p>128G cloud physical machine*2</p><h1>2. Linkis installation and deployment</h1><h2 class="anchor anchorWithStickyNavbar_LWe7" id="21-compile-code-or-release-installation-package">2.1 Compile code or release installation package?<a class="hash-link" href="#21-compile-code-or-release-installation-package" title="Direct link to heading"></a></h2><p>This installation deployment adopts the release installation package method. In order to adapt to the company's CDH6.3.2 version, the dependency packages of hadoop and hive need to be replaced with the CDH6.3.2 version. Here, the installation package is directly replaced. The dependent packages and modules to be replaced are shown in the following list.</p><div class="language-plain codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-plain codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">// Modules involved </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">linkis-engineconn-plugins/spark</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">linkis-engineconn-plugins/hive</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">/linkis-commons/public-module</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">/linkis-computation-governance/</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><div class="language-plain codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-plain codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">// List of cdh packages that need to be replaced</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/spark/dist/v2.4.8/lib/hive-shims-0.23-2.1.1-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/spark/dist/v2.4.8/lib/hive-shims-scheduler-2.1.1-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/spark/dist/v2.4.8/lib/hadoop-annotations-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/spark/dist/v2.4.8/lib/hadoop-auth-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/spark/dist/v2.4.8/lib/hadoop-common-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/spark/dist/v2.4.8/lib/hadoop-hdfs-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/spark/dist/v2.4.8/lib/hadoop-hdfs-client-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-client-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-mapreduce-client-common-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-mapreduce-client-jobclient-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-yarn-api-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-yarn-client-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-yarn-server-common-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-hdfs-client-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-mapreduce-client-core-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-mapreduce-client-shuffle-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/hive/dist/v2.1.1/lib/hadoop-yarn-common-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/flink/dist/v1.12.2/lib/hadoop-annotations-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/flink/dist/v1.12.2/lib/hadoop-auth-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/flink/dist/v1.12.2/lib/hadoop-mapreduce-client-core-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/flink/dist/v1.12.2/lib/hadoop-yarn-api-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/flink/dist/v1.12.2/lib/hadoop-yarn-client-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-engineconn-plugins/flink/dist/v1.12.2/lib/hadoop-yarn-common-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-commons/public-module/hadoop-annotations-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-commons/public-module/hadoop-auth-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-commons/public-module/hadoop-common-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-commons/public-module/hadoop-hdfs-client-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-computation-governance/linkis-cg-linkismanager/hadoop-annotations-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-computation-governance/linkis-cg-linkismanager/hadoop-auth-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-computation-governance/linkis-cg-linkismanager/hadoop-yarn-api-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-computation-governance/linkis-cg-linkismanager/hadoop-yarn-client-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">./lib/linkis-computation-governance/linkis-cg-linkismanager/hadoop-yarn-common-3.0.0-cdh6.3.2.jar</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="22-problems-encountered-during-deployment">2.2 Problems encountered during deployment<a class="hash-link" href="#22-problems-encountered-during-deployment" title="Direct link to heading"></a></h2><h3 class="anchor anchorWithStickyNavbar_LWe7" id="221--kerberos-configuration">2.2.1 Kerberos configuration<a class="hash-link" href="#221--kerberos-configuration" title="Direct link to heading"></a></h3><p>It needs to be added in the linkis.properties public configuration</p><p>Each engine conf also needs to be added</p><div class="language-plain codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-plain codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.keytab.enable=true</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.keytab.file=/hadoop/bigdata/kerberos/keytab</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.keytab.host.enabled=false</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.keytab.host=your_host</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="222--error-is-reported-after-hadoop-dependency-package-is-replaced">2.2.2 Error is reported after Hadoop dependency package is replaced<a class="hash-link" href="#222--error-is-reported-after-hadoop-dependency-package-is-replaced" title="Direct link to heading"></a></h3><p>java.lang.NoClassDefFoundError:org/apache/commons/configuration2/Configuration</p><p><img loading="lazy" alt="image" src="/assets/images/config-err-0b7267a47e539866aefbe8be9db6c197.png" width="750" height="216" class="img_ev3q"></p><p>Cause: Configuration class conflict. Add a commons-configuration2-2.1.1.jar under the linkis commons module to resolve the conflict</p><h3></h3><h3 class="anchor anchorWithStickyNavbar_LWe7" id="223-running-spark-python-etc-in-script-reports-no-plugin-for-xxx">2.2.3 Running spark, python, etc. in script reports no plugin for XXX<a class="hash-link" href="#223-running-spark-python-etc-in-script-reports-no-plugin-for-xxx" title="Direct link to heading"></a></h3><p>Phenomenon: After modifying the version of Spark/Python in the configuration file, the startup engine reports no plugin for XXX</p><p><img loading="lazy" alt="image" src="/assets/images/pugin-error-0b7267a47e539866aefbe8be9db6c197.png" width="750" height="216" class="img_ev3q"></p><p>Reason: LabelCommonConfig.java and GovernanceCommonConf In scala, the version of the engine is written dead, the corresponding version is modified, and all jars containing these two classes (linkis computation governance common-1.1.1. jar and linkis label common-1.1.1. jar) in linkis and other components (including scheduleris) are replaced after compilation</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="224-python-engine-execution-error-initialization-failed">2.2.4 Python engine execution error, initialization failed<a class="hash-link" href="#224-python-engine-execution-error-initialization-failed" title="Direct link to heading"></a></h3><ul><li>Modify python. py and remove the imported pandas module</li><li>Configure the python loading directory and modify the python engine's linkis-enginecon.properties</li></ul><div class="language-plain codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-plain codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">pythonVersion=/usr/local/bin/python3.6</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="225-failed-to-run-the-pyspark-task-and-reported-an-error">2.2.5 Failed to run the pyspark task and reported an error<a class="hash-link" href="#225-failed-to-run-the-pyspark-task-and-reported-an-error" title="Direct link to heading"></a></h3><p><img loading="lazy" alt="image" src="/assets/images/pyspark-error-59e263d21e848c5bda995cdc900e0ca9.png" width="1542" height="341" class="img_ev3q"></p><p>Reason: PYSPARK is not set_ VERSION</p><p>resolvent:</p><p>Set two parameters in/etc/profile</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">export PYSPARK_ PYTHON=/usr/local/bin/python3.6</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">export PYSPARK_ DRIVER_PYTHON=/usr/local/bin/python3.6</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="226-error-occurs-when-executing-the-pyspark-task">2.2.6 Error occurs when executing the pyspark task<a class="hash-link" href="#226-error-occurs-when-executing-the-pyspark-task" title="Direct link to heading"></a></h3><p>java.lang.NoSuchFieldError: HIVE<em> STATS</em> JDBC_ TIMEOUT</p><p><img loading="lazy" alt="image" src="/assets/images/spark-hive-verion-error-73f07dc42df540b1aeb4ea048a131fa8.png" width="1549" height="361" class="img_ev3q"></p><p>Reason: Spark 2.4.8 uses the hive1.2.1 package, but our hive has been upgraded to version 2.1.1. This parameter has been removed from hive2. Then the code in spark sql still calls the hive parameter, and then an error is reported,</p><p>Therefore, HIVE is deleted from the spark sql/hive code<em> STATS</em> JDBC<em> TIMEOUT This parameter is recompiled and packaged to replace the spark hive in spark 2.4.8</em> 2.11-2.4.8.jar</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="227-proxy-user-exception-during-jdbc-engine-execution">2.2.7 Proxy user exception during jdbc engine execution<a class="hash-link" href="#227-proxy-user-exception-during-jdbc-engine-execution" title="Direct link to heading"></a></h3><p>Phenomenon: User A is used to execute a jdbc task 1. The engine chooses to reuse it. Then I also use User B to execute a jdbc task 2. It is found that the submitter of task 2 is A</p><p>Analysis reason:</p><p>ConnectionManager::getConnection</p><p><img loading="lazy" alt="image" src="/assets/images/jdbc-engine-analyze-e765a7057a729bb49816e9e7ebffc787.png" width="1027" height="439" class="img_ev3q"></p><p>When creating a datasource, we judge whether to create it according to the key. The key is a jdbc url, but this granularity may be a bit large, because different users may access the same datasource, such as hive. Their urls are the same, but their account passwords are different. So when the first user creates a datasource, the username has been specified. When the second user comes in, If the data source is found to exist, it will be used directly instead of creating a new data source. Therefore, the code submitted by user B will be executed by user A.</p><p>Solution: Reduce the key granularity of the data source cache map, and change it to jdbc. url+jdbc. user.</p><ol start="3"><li>DSS deployment |
| The installation process refers to the official website documents for installation configuration. The following describes some issues encountered in the installation and debugging process.</li></ol><h2 class="anchor anchorWithStickyNavbar_LWe7" id="31-the-database-list-displayed-on-the-left-side-of-the-dss-is-incomplete">3.1 The database list displayed on the left side of the DSS is incomplete<a class="hash-link" href="#31-the-database-list-displayed-on-the-left-side-of-the-dss-is-incomplete" title="Direct link to heading"></a></h2><p>Analysis: The database information displayed in the DSS data source module is from the hive metabase. However, because of the permission control through the Sentry in CDH6, most of the hive table metadata information does not exist in the hive metastore, so the displayed data is missing.</p><p>resolvent:</p><p>The original logic is transformed into the way of using jdbc to link hive and obtain table data display from jdbc.</p><p>Simple logic description:</p><p>The properties information of jdbc is obtained through the IDE jdbc configuration information configured on the linkis console.</p><p>DBS: Get the schema through connection. getMetaData()</p><p>TBS: connection. getMetaData(). getTables() Get the tables under the corresponding db</p><p>COLUMNS: Get the columns information of the table by executing describe table</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="32-error-jdbc-is-reported-when-executing-jdbc-script-in-dss-workflow-name-is-empty">3.2 Error jdbc is reported when executing jdbc script in DSS workflow name is empty<a class="hash-link" href="#32-error-jdbc-is-reported-when-executing-jdbc-script-in-dss-workflow-name-is-empty" title="Direct link to heading"></a></h2><p>Analysis: The default creator in the dss workflow is Schedulis. Because the related engine parameters of Schedulis are not configured in the management console, the parameters read are all empty.</p><p>Adding a category of Schedulis to the console gives an error, ”The Schedulis directory already exists. Because the creator in the scheduling system is schedulis, the Schedulis Category cannot be added. In order to better identify each system, the default creator in the dss workflow is changed to nod_exception. This parameter can add wds. linkis. flow. job. creator. v1=nod_execution in the dss flow execution server. properties.</p></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/blog">blog</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/linki-1-1-1">linki1.1.1</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/hadoop-3-0-0-cdh-6-3-2">hadoop3.0.0-cdh6.3.2</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/spark-2-4-8">spark2.4.8</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/hive-2-1-1">hive2.1.1</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">Linkis</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/docs/latest/about/introduction">Document</a></li><li class="footer__item"><a class="footer__link-item" href="/faq/main">FAQ</a></li><li class="footer__item"><a href="https://github.com/apache/linkis/releases" target="_blank" rel="noopener noreferrer" class="footer__link-item">Releases</a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/apache/linkis" target="_blank" rel="noopener noreferrer" class="footer__link-item">GitHub</a></li><li class="footer__item"><a href="https://github.com/apache/linkis/issues" target="_blank" rel="noopener noreferrer" class="footer__link-item">Issue Tracker</a></li><li class="footer__item"><a href="https://github.com/apache/linkis/pulls" target="_blank" rel="noopener noreferrer" class="footer__link-item">Pull Requests</a></li></ul></div><div class="col footer__col"><div class="footer__title">Subscribe Mailing List</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/community/how-to-subscribe">How to Subscribe</a></li><li class="footer__item"><a href="mailto:dev-subscribe@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item">Subscribe Mail</a></li><li class="footer__item"><a href="https://lists.apache.org/list.html?dev@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item">Mail Archive</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright"><div><img style="height:50px" alt="Apache Software Foundation" src="/img/incubator-logo.svg"><p style="color: #999999; padding: 0 20px 30px;font-weight:400;text-align:left">Apache Linkis is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p><p></p> |
