| <!doctype html> |
| <html lang="zh-CN" dir="ltr"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width,initial-scale=1"> |
| <meta name="generator" content="Docusaurus v2.0.0-beta.14"> |
| <link rel="alternate" type="application/rss+xml" href="/zh-CN/blog/rss.xml" title="Apache Linkis Blog RSS Feed"> |
| <link rel="alternate" type="application/atom+xml" href="/zh-CN/blog/atom.xml" title="Apache Linkis Blog Atom Feed"> |
| <link rel="search" type="application/opensearchdescription+xml" title="Apache Linkis" href="/zh-CN/opensearch.xml"><title data-react-helmet="true">Linkis 部署排障 | Apache Linkis</title><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" property="og:url" content="https://linkis.incubator.apache.org/zh-CN/blog/2022/02/21/linkis-deploy"><meta data-react-helmet="true" name="docsearch:language" content="zh-CN"><meta data-react-helmet="true" name="docsearch:docusaurus_tag" content="default"><meta data-react-helmet="true" property="og:title" content="Linkis 部署排障 | Apache Linkis"><meta data-react-helmet="true" name="description" content="linkis的部署说明和注意点"><meta data-react-helmet="true" property="og:description" content="linkis的部署说明和注意点"><meta data-react-helmet="true" property="og:type" content="article"><meta data-react-helmet="true" property="article:published_time" content="2022-02-21T00:00:00.000Z"><link data-react-helmet="true" rel="shortcut icon" href="/zh-CN/img/favicon.ico"><link data-react-helmet="true" rel="canonical" href="https://linkis.incubator.apache.org/zh-CN/blog/2022/02/21/linkis-deploy"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/blog/2022/02/21/linkis-deploy" hreflang="en"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/zh-CN/blog/2022/02/21/linkis-deploy" hreflang="zh-CN"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/blog/2022/02/21/linkis-deploy" hreflang="x-default"><link data-react-helmet="true" rel="preconnect" href="https://AE29KQB3IA-dsn.algolia.net" crossorigin="anonymous"><link rel="stylesheet" href="/zh-CN/assets/css/styles.e9faf9bb.css"> |
| <link rel="preload" href="/zh-CN/assets/js/runtime~main.14d5df0c.js" as="script"> |
| <link rel="preload" href="/zh-CN/assets/js/main.5a9c4743.js" as="script"> |
| </head> |
| <body> |
| <script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus"> |
| <div><a href="#" class="skipToContent_OuoZ">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/zh-CN/"><img src="/zh-CN/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--light_4Vu1 navbar__logo"><img src="/zh-CN/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--dark_uzRr navbar__logo"><b class="navbar__title">Apache Linkis(Incubating)</b></a><a class="navbar__item navbar__link" href="/zh-CN/">首页</a><a class="navbar__item navbar__link" href="/zh-CN/faq/main">FAQ</a><a class="navbar__item navbar__link" href="/zh-CN/download/main">下载</a><a class="navbar__item navbar__link" href="/zh-CN/community/how-to-subscribe">社区</a><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/zh-CN/blog">博客</a><a class="navbar__item navbar__link" href="/zh-CN/team">团队</a><a class="navbar__item navbar__link" href="/zh-CN/user">用户</a><div class="navbar__item dropdown dropdown--hoverable"><a class="navbar__link">Apache</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">基金会</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">证书</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">事件</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">安全</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">赞助</a></li><li><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">致谢</a></li></ul></div></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link">文档</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/zh-CN/docs/1.1.3/introduction">Next(1.1.3)</a></li><li><a class="dropdown__link" href="/zh-CN/docs/latest/introduction">1.1.2</a></li><li><a class="dropdown__link" href="/zh-CN/docs/1.1.1/introduction">1.1.1</a></li><li><a class="dropdown__link" href="/zh-CN/docs/1.1.0/introduction">1.1.0</a></li><li><a class="dropdown__link" href="/zh-CN/docs/1.0.3/introduction">1.0.3</a></li><li><a class="dropdown__link" href="/zh-CN/docs/1.0.2/introduction">1.0.2</a></li><li><a class="dropdown__link" href="/zh-CN/docs/0.11.0/introduction">0.11.0</a></li><li><a class="dropdown__link" href="/zh-CN/versions">All Version</a></li></ul></div><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-github-link" aria-label="GitHub"></a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" class="navbar__link"><span><svg t="1631348384596" class="iconLanguage_EbrZ" viewBox="0 0 1024 1024" version="1.1" p-id="557" width="20" height="20"><path d="M547.797333 638.208l-104.405333-103.168 1.237333-1.28a720.170667 720.170667 0 0 0 152.490667-268.373333h120.448V183.082667h-287.744V100.906667H347.605333v82.218666H59.818667V265.386667h459.178666a648.234667 648.234667 0 0 1-130.304 219.946666 643.242667 643.242667 0 0 1-94.976-137.728H211.541333a722.048 722.048 0 0 0 122.453334 187.434667l-209.194667 206.378667 58.368 58.368 205.525333-205.525334 127.872 127.829334 31.232-83.84m231.424-208.426667h-82.218666l-184.96 493.312h82.218666l46.037334-123.306667h195.242666l46.464 123.306667h82.218667l-185.002667-493.312m-107.690666 287.744l66.56-178.005333 66.602666 178.005333z" fill="currentColor" p-id="558"></path></svg><span>简体中文</span></span></a><ul class="dropdown__menu"><li><a href="/blog/2022/02/21/linkis-deploy" target="_self" rel="noopener noreferrer" class="dropdown__link" style="text-transform:capitalize">English</a></li><li><a href="/zh-CN/blog/2022/02/21/linkis-deploy" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" style="text-transform:capitalize">简体中文</a></li></ul></div><div class="searchBox_Bc3W"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div class="main-wrapper blog-wrapper blog-post-page"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_q+wC thin-scrollbar" aria-label="最新博文导航"><div class="sidebarItemTitle_9G5K margin-bottom--md">最新博文</div><ul class="sidebarItemList_6T4b"><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/zh-CN/blog/2022/07/16/deploy-linkis-with-kubernetes">部署Linkis到Kubernetes</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/zh-CN/blog/2022/07/04/how-to-add-auto-bot">如何为github仓库添加一个github action</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/zh-CN/blog/2022/06/09/meetup-content-review">Apache Linkis(Incubating) Meep Up</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/zh-CN/blog/2022/04/15/how-to-download-engineconn-plugin">如何下载安装包中默认没有的引擎插件</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/zh-CN/blog/2022/03/20/openlookeng">OpenLooKeng的引擎的实现</a></li><li class="sidebarItem_cjdF"><a aria-current="page" class="sidebarItemLink_zyXk sidebarItemLinkActive_wcJs" href="/zh-CN/blog/2022/02/21/linkis-deploy">Linkis 部署排障</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/zh-CN/blog/2022/02/08/how-to-user-blog">如何编写博客</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><article itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h1 class="blogPostTitle_d4p0" itemprop="headline">Linkis 部署排障</h1><div class="blogPostData_-Im+ margin-vert--md"><time datetime="2022-02-21T00:00:00.000Z" itemprop="datePublished">2022年2月21日</time> · <!-- -->22 分钟阅读</div></header><div class="markdown" itemprop="articleBody"><blockquote><p>linkis的部署说明和注意点</p></blockquote><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="1-前置准备注意事项"></a>1. 前置准备注意事项<a class="hash-link" href="#1-前置准备注意事项" title="Direct link to heading">#</a></h2><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="11-linux服务器"></a>1.1 linux服务器<a class="hash-link" href="#11-linux服务器" title="Direct link to heading">#</a></h3><p><strong>硬件要求</strong><br> |
| <!-- -->安装linkis 微服务近10个，每个微服务默认配置启动的jvm -Xmx 内存大小为 512M(内存不够的情况下，可以尝试调小至256/128M，内存足够情况下也可以调大)</p><p><strong>软件要求</strong><br> |
| <!-- -->基本的软件环境<br> |
| <!-- -->使用下面命令检查是否已安装对应的软件 ，若未安装，请先进行安装</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">#java环境 推荐>=1.0.8</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">command</span><span class="token plain"> -v java</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#yum 包管理工具，主要是在执行linkis web管理台安装脚本时,会通过yum安装nginx/policycoreutils-python软件</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">command</span><span class="token plain"> -v yum</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#linkis的数据库初始化时，会使用到mysql客户端连接数据库，执行sql语句</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">command</span><span class="token plain"> -v mysql</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">command</span><span class="token plain"> -v telnet</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#用于解压linkis的相关安装包</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">command</span><span class="token plain"> -v </span><span class="token function" style="color:rgb(80, 250, 123)">tar</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#初始的服务配置文件linkis-*.properties是执行安装脚本install.sh时候，根据`deploy-config/*sh`配置的数据 通过sed 替换到properties文件中的</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">command</span><span class="token plain"> -v </span><span class="token function" style="color:rgb(80, 250, 123)">sed</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="12-添加部署用户"></a>1.2 添加部署用户<a class="hash-link" href="#12-添加部署用户" title="Direct link to heading">#</a></h3><p>部署用户: linkis核心进程的启动用户，同时此用户会默认作为管理员权限，<font color="red">部署过程中会生成对应的管理员登录密码，位于conf/linkis-mg-gateway.properties文件中</font></p><p>linkis支持指定提交、执行的用户。linkis主要进程服务会通过sudo -u ${linkis-user} 切换到对应用户下，然后执行对应的引擎启动命令，所以引擎linkis-engine进程归属的用户是任务的执行者（所以部署用户需要有sudo权限，而且是免密的）</p><p>以hadoop用户为例:</p><p>先查看系统中是否已经有 hadoop 用户，若已经存在，则直接授权即可；若不存在，先创建用户，再授权。</p><p>查看是否已存在 hadoop 用户</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token function" style="color:rgb(80, 250, 123)">id</span><span class="token plain"> hadoop</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">uid</span><span class="token operator">=</span><span class="token number">2001</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">hadoop</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">gid</span><span class="token operator">=</span><span class="token number">2001</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">hadoop</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">groups</span><span class="token operator">=</span><span class="token number">2001</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">hadoop</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>若不存在，则需要创建 hadoop 用户，并加入 hadoop 用户组</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token function" style="color:rgb(80, 250, 123)">sudo</span><span class="token plain"> </span><span class="token function" style="color:rgb(80, 250, 123)">useradd</span><span class="token plain"> hadoop -g hadoop</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token function" style="color:rgb(80, 250, 123)">vi</span><span class="token plain"> /etc/sudoers</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#加上配置</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">hadoop </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">ALL</span><span class="token operator">=</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">ALL</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"> NOPASSWD: NOPASSWD: ALL</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>修改安装用户的环境变量,<code>vim /home/hadoop/.bash_rc</code>配置环境变量，环境变量如下：</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">JAVA_HOME</span><span class="token operator">=</span><span class="token plain">/opt/jdk1.8</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HADOOP_HOME</span><span class="token operator">=</span><span class="token plain">/opt/install/hadoop</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HADOOP_CONF_DIR</span><span class="token operator">=</span><span class="token plain">/etc/conf/hadoop</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HIVE_HOME</span><span class="token operator">=</span><span class="token plain">/opt/install/hive</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HIVE_CONF_DIR</span><span class="token operator">=</span><span class="token plain">/etc/conf/conf</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">SPARK_HOME</span><span class="token operator">=</span><span class="token plain">/opt/install/spark</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">SPARK_CONF_DIR</span><span class="token operator">=</span><span class="token plain">/etc/spark/conf</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">PYSPARK_ALLOW_INSECURE_GATEWAY</span><span class="token operator">=</span><span class="token number">1</span><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)"># 如果是Pyspark，则必须加此参数</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>刷新配置 </p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">source</span><span class="token plain"> /home/hadoop/.bash_rc</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#如果.bashrc文件不存，可以配置到.bash_profile中</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>查看是否生效</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token function" style="color:rgb(80, 250, 123)">sudo</span><span class="token plain"> -su hadoop</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">echo</span><span class="token plain"> </span><span class="token variable" style="color:rgb(189, 147, 249);font-style:italic">$JAVA_HOME</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">echo</span><span class="token plain"> </span><span class="token variable" style="color:rgb(189, 147, 249);font-style:italic">$HADOOP_HOME</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><font color="red">以下操作都是在hadoop用户下进行</font><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="13-安装包准备"></a>1.3 安装包准备<a class="hash-link" href="#13-安装包准备" title="Direct link to heading">#</a></h3><p>linkis安装包，推荐使用1.X及上版本 |
| 0.X 和1.X的版本差异化比较大，1.0.3前是<code>com.webank.wedatasphere.linkis</code>的包名，linkis>=1.0.3为<code>org.apache.linkis</code>的包名</p><p><a href="https://linkis.apache.org/zh-CN/download/main" target="_blank" rel="noopener noreferrer">下载地址</a>:<a href="https://linkis.apache.org/zh-CN/download/main" target="_blank" rel="noopener noreferrer">https://linkis.apache.org/zh-CN/download/main</a></p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="14-底层依赖检查"></a>1.4 底层依赖检查<a class="hash-link" href="#14-底层依赖检查" title="Direct link to heading">#</a></h3><p>可以执行相应的命令，查看是否支持</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">spark/hive/hdfs/python/</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ spark-submit --version //spark 任务会通过这个命令提交到YARN上执行</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ python --version</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ hdfs version</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ hive --version</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="15-资源依赖"></a>1.5 资源依赖<a class="hash-link" href="#15-资源依赖" title="Direct link to heading">#</a></h3><p>可以访问的mysql数据库资源 用来存储linkis自身的业务数据的数据库<br> |
| <!-- -->可以访问的yarn资源队列 spark/hive/flink引擎的执行都需要yarn队列资源<br> |
| <!-- -->可以访问的hive的matedata数据库资源(mysql为例) hive引擎执行时需要 </p><div class="admonition admonition-caution alert alert--warning"><div class="admonition-heading"><h5><span class="admonition-icon"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>注意</h5></div><div class="admonition-content"><p>注意：hive spark的版本,如果和默认版本区别比较大，最好修改linkis maven依赖相关的hive/spark版本，然后重新进行编译</p></div></div><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="2-安裝"></a>2. 安裝<a class="hash-link" href="#2-安裝" title="Direct link to heading">#</a></h2><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="21-安装包解压"></a>2.1 安装包解压<a class="hash-link" href="#21-安装包解压" title="Direct link to heading">#</a></h3><p>上传安装包<code>apache-linkis-1.0.3-incubating-bin.tar.gz</code>后，进行解压安装包 </p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token function" style="color:rgb(80, 250, 123)">tar</span><span class="token plain"> -xvf apache-linkis-1.0.3-incubating-bin.tar.gz</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">pwd</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">/data/Install/1.0.3</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>解压后的目录结构如下</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">-rw-r--r-- </span><span class="token number">1</span><span class="token plain"> hadoop hadoop </span><span class="token number">531847342</span><span class="token plain"> Feb </span><span class="token number">21</span><span class="token plain"> </span><span class="token number">10</span><span class="token plain">:10 apache-linkis-1.0.3-incubating-bin.tar.gz</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">drwxrwxr-x </span><span class="token number">2</span><span class="token plain"> hadoop hadoop </span><span class="token number">4096</span><span class="token plain"> Feb </span><span class="token number">21</span><span class="token plain"> </span><span class="token number">10</span><span class="token plain">:13 bin //执行环境检查和安装的脚本</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">drwxrwxr-x </span><span class="token number">2</span><span class="token plain"> hadoop hadoop </span><span class="token number">4096</span><span class="token plain"> Feb </span><span class="token number">21</span><span class="token plain"> </span><span class="token number">10</span><span class="token plain">:13 deploy-config // 部署时依赖的DB等环境配置信息</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">-rw-r--r-- </span><span class="token number">1</span><span class="token plain"> hadoop hadoop </span><span class="token number">1707</span><span class="token plain"> Jan </span><span class="token number">22</span><span class="token plain"> </span><span class="token number">2020</span><span class="token plain"> DISCLAIMER-WIP</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">-rw-r--r-- </span><span class="token number">1</span><span class="token plain"> hadoop hadoop </span><span class="token number">66058</span><span class="token plain"> Jan </span><span class="token number">22</span><span class="token plain"> </span><span class="token number">2020</span><span class="token plain"> LICENSE</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">drwxrwxr-x </span><span class="token number">2</span><span class="token plain"> hadoop hadoop </span><span class="token number">16384</span><span class="token plain"> Feb </span><span class="token number">21</span><span class="token plain"> </span><span class="token number">10</span><span class="token plain">:13 licenses</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">drwxrwxr-x </span><span class="token number">7</span><span class="token plain"> hadoop hadoop </span><span class="token number">4096</span><span class="token plain"> Feb </span><span class="token number">21</span><span class="token plain"> </span><span class="token number">10</span><span class="token plain">:13 linkis-package // 实际的软件包，包括lib/服务启动脚本工具/db的初始化脚本/微服务的配置文件等</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">-rw-r--r-- </span><span class="token number">1</span><span class="token plain"> hadoop hadoop </span><span class="token number">83126</span><span class="token plain"> Jan </span><span class="token number">22</span><span class="token plain"> </span><span class="token number">2020</span><span class="token plain"> NOTICE</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">-rw-r--r-- </span><span class="token number">1</span><span class="token plain"> hadoop hadoop </span><span class="token number">7900</span><span class="token plain"> Jan </span><span class="token number">22</span><span class="token plain"> </span><span class="token number">2020</span><span class="token plain"> README_CN.md</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">-rw-r--r-- </span><span class="token number">1</span><span class="token plain"> hadoop hadoop </span><span class="token number">8184</span><span class="token plain"> Jan </span><span class="token number">22</span><span class="token plain"> </span><span class="token number">2020</span><span class="token plain"> README.md</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="22-配置linkis自身数据库信息"></a>2.2 配置linkis自身数据库信息<a class="hash-link" href="#22-配置linkis自身数据库信息" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">vim</span><span class="token plain"> deploy-config/db.sh</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">示例:</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">MYSQL_HOST</span><span class="token operator">=</span><span class="token plain">xx.xx.xx.xx</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">MYSQL_PORT</span><span class="token operator">=</span><span class="token number">3306</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">MYSQL_DB</span><span class="token operator">=</span><span class="token plain">linkis_test</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">MYSQL_USER</span><span class="token operator">=</span><span class="token plain">test</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">MYSQL_PASSWORD</span><span class="token operator">=</span><span class="token plain">xxxxx</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="23-配置基础环境变量"></a>2.3 配置基础环境变量<a class="hash-link" href="#23-配置基础环境变量" title="Direct link to heading">#</a></h3><p>文件位于<code>deploy-config/linkis-env.sh</code></p><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="基础目录配置"></a>基础目录配置<a class="hash-link" href="#基础目录配置" title="Direct link to heading">#</a></h4><blockquote><p>请确认部署用户deployUser，拥有这些配置目录的读写权限</p></blockquote><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">deployUser</span><span class="token operator">=</span><span class="token plain">hadoop </span><span class="token comment" style="color:rgb(98, 114, 164)">#执行部署的用户，之前创建的用户hadoop</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">WORKSPACE_USER_ROOT_PATH</span><span class="token operator">=</span><span class="token plain">file:///tmp/linkis </span><span class="token comment" style="color:rgb(98, 114, 164)"># 指定用户使用的目录路径，一般用于存储用户的脚本文件和日志文件等，是用户的工作空间。 对应的配置文件配置项为 wds.linkis.filesystem.root.path(linkis.properties)</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">RESULT_SET_ROOT_PATH</span><span class="token operator">=</span><span class="token plain">file:///tmp/linkis </span><span class="token comment" style="color:rgb(98, 114, 164)"># 结果集日志等文件路径，用于存储Job的结果集文件 wds.linkis.resultSet.store.path(linkis-cg-entrance.properties) //如果未配置 使用HDFS_USER_ROOT_PATH的配置</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HDFS_USER_ROOT_PATH</span><span class="token operator">=</span><span class="token plain">hdfs:///tmp/linkis </span><span class="token comment" style="color:rgb(98, 114, 164)"># 结果集日志等文件路径，用于存储Job的结果集文件 wds.linkis.filesystem.hdfs.root.path(linkis.properties)</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">ENGINECONN_ROOT_PATH</span><span class="token operator">=</span><span class="token plain">/appcom/tmp </span><span class="token comment" style="color:rgb(98, 114, 164)">#存放执行引擎的工作路径，需要部署用户有写权限的本地目录 wds.linkis.engineconn.root.dir(linkis-cg-engineconnmanager.properties)</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>注意：确认部署用户是否有对应文件目录的读写的权限</p><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="hive的meta配置"></a>HIVE的META配置<a class="hash-link" href="#hive的meta配置" title="Direct link to heading">#</a></h4><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HIVE_META_URL</span><span class="token operator">=</span><span class="token plain">jdbc:mysql://10.10.10.10:3306/hive_meta_demo?useUnicode</span><span class="token operator">=</span><span class="token plain">true</span><span class="token operator">&</span><span class="token plain">amp</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">characterEncoding</span><span class="token operator">=</span><span class="token plain">UTF-8 </span><span class="token comment" style="color:rgb(98, 114, 164)"># HiveMeta元数据库的URL</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HIVE_META_USER</span><span class="token operator">=</span><span class="token plain">demo </span><span class="token comment" style="color:rgb(98, 114, 164)"># HiveMeta元数据库的用户</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HIVE_META_PASSWORD</span><span class="token operator">=</span><span class="token plain">demo123 </span><span class="token comment" style="color:rgb(98, 114, 164)"># HiveMeta元数据库的密码</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="yarn的resourcemanager的地址"></a>Yarn的ResourceManager的地址<a class="hash-link" href="#yarn的resourcemanager的地址" title="Direct link to heading">#</a></h4><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#可以通过访问http://xx.xx.xx.xx:8088/ws/v1/cluster/scheduler 接口确认是否正常</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">YARN_RESTFUL_URL</span><span class="token operator">=</span><span class="token plain">http://xx.xx.xx.xx:8088 </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>执行spark任务时，需要使用到yarn的ResourceManager，linkis默认它是未开启权限验证的，如果ResourceManager开启了密码权限验证，请安装部署后，修改<code>linkis_cg_engine_conn_plugin_bml_resources</code>表数据(或则参见<a href="#6-%E6%A3%80%E6%9F%A5%E5%BC%95%E6%93%8E%E7%89%A9%E6%96%99%E8%B5%84%E6%BA%90%E6%98%AF%E5%90%A6%E4%B8%8A%E4%BC%A0%E6%88%90%E5%8A%9F">检查引擎物料资源是否上传成功</a>)</p><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="ldap-登录验证"></a>LDAP 登录验证<a class="hash-link" href="#ldap-登录验证" title="Direct link to heading">#</a></h4><blockquote><p>linkis默认是使用静态用户和密码,静态用户即部署用户，静态密码会在执行部署是随机生成一个密码串，存储于{InstallPath}/conf/linkis-mg-gateway.properties(>=1.0.3版本)。</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">#LDAP配置，默认Linkis只支持部署用户登录，如果需要支持多用户登录可以使用LDAP，需要配置以下参数：</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#LDAP_URL=ldap://localhost:1389/</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#LDAP_BASEDN=dc=webank,dc=com</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div></blockquote><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="基础组件环境信息"></a>基础组件环境信息<a class="hash-link" href="#基础组件环境信息" title="Direct link to heading">#</a></h4><blockquote><p>最好通过用户的系统环境变量配置(步骤 1.2 添加部署用户已说明), deploy-config/linkis-env.sh配置文件中可以不进行配置 直接注释掉</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">###HADOOP CONF DIR</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#HADOOP_CONF_DIR=/appcom/config/hadoop-config</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">###HIVE CONF DIR</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#HIVE_CONF_DIR=/appcom/config/hive-config</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">###SPARK CONF DIR</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#SPARK_CONF_DIR=/appcom/config/spark-config</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div></blockquote><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="引擎版本信息"></a>引擎版本信息<a class="hash-link" href="#引擎版本信息" title="Direct link to heading">#</a></h4><div class="admonition admonition-caution alert alert--warning"><div class="admonition-heading"><h5><span class="admonition-icon"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>注意</h5></div><div class="admonition-content"><p>如果使用的官方发布包 无需修改，如果是有自行修改Hive/Spark引擎版本编译的，需要修改。</p></div></div><p>如果spark不是2.4.3的版本，需要修改参数：</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## Engine version conf</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#SPARK_VERSION，如果安装的Spark版本不是2.4.3，则需要修改为相应的版本，如3.1.1</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">SPARK_VERSION</span><span class="token operator">=</span><span class="token number">3.1</span><span class="token plain">.1</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>如果hive不是2.3.3的版本，需要修改参数：</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## Engine version conf</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">##HIVE_VERSION，如果安装的Hive版本不是2.3.3，则需要修改为相应的版本，如2.3.4</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">HIVE_VERSION</span><span class="token operator">=</span><span class="token number">2.3</span><span class="token plain">.4</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>如果配置了，执行安装部署后，实际会在<code>{linkisInstallPath}/conf/linkis.properties</code>文件中被更新</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">#wds.linkis.spark.engine.version=</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#wds.linkis.hive.engine.version=</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#wds.linkis.python.engine.version=</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="jvm内存配置"></a>JVM内存配置<a class="hash-link" href="#jvm内存配置" title="Direct link to heading">#</a></h4><blockquote><p>微服务启动jvm内存配置，可以根据机器实际情况进行调整，如果机器内存资源较少，可以尝试调小至256/128M</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## java application default jvm memory</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">SERVER_HEAP_SIZE</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"512M"</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div></blockquote><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="安装目录配置"></a>安装目录配置<a class="hash-link" href="#安装目录配置" title="Direct link to heading">#</a></h4><blockquote><p>linkis最终会被安装到此目录下，不配置默认是与当前安装包同一级目录下</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">##The decompression directory and the installation directory need to be inconsistent</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">LINKIS_HOME</span><span class="token operator">=</span><span class="token plain">/appcom/Install/LinkisInstall</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div></blockquote><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="3-部署流程"></a>3. 部署流程<a class="hash-link" href="#3-部署流程" title="Direct link to heading">#</a></h2><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="31-执行部署脚本"></a>3.1 执行部署脚本<a class="hash-link" href="#31-执行部署脚本" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token plain"> bin/install.sh</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><div class="admonition admonition-tip alert alert--success"><div class="admonition-heading"><h5><span class="admonition-icon"><svg xmlns="http://www.w3.org/2000/svg" width="12" height="16" viewBox="0 0 12 16"><path fill-rule="evenodd" d="M6.5 0C3.48 0 1 2.19 1 5c0 .92.55 2.25 1 3 1.34 2.25 1.78 2.78 2 4v1h5v-1c.22-1.22.66-1.75 2-4 .45-.75 1-2.08 1-3 0-2.81-2.48-5-5.5-5zm3.64 7.48c-.25.44-.47.8-.67 1.11-.86 1.41-1.25 2.06-1.45 3.23-.02.05-.02.11-.02.17H5c0-.06 0-.13-.02-.17-.2-1.17-.59-1.83-1.45-3.23-.2-.31-.42-.67-.67-1.11C2.44 6.78 2 5.65 2 5c0-2.2 2.02-4 4.5-4 1.22 0 2.36.42 3.22 1.19C10.55 2.94 11 3.94 11 5c0 .66-.44 1.78-.86 2.48zM4 14h5c-.23 1.14-1.3 2-2.5 2s-2.27-.86-2.5-2z"></path></svg></span>注意</h5></div><div class="admonition-content"><ul><li>如果出现报错，又不清楚具体是执行什么命令报错，可以加 -v 参数<code>sh -v bin/install.sh</code>，将shell脚本执行过程日志打印出来，方便定位问题</li><li>权限问题:mkdir: cannot create directory ‘xxxx’: Permission denied,请确认部署用户是否拥有该路径的读写权限 </li></ul></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="32-可能遇到的问题"></a>3.2 可能遇到的问题<a class="hash-link" href="#32-可能遇到的问题" title="Direct link to heading">#</a></h3><p>执行成功提示如下: |
| <code>Congratulations! You have installed Linkis 1.0.3 successfully, please use sh /data/Install/linkis/sbin/linkis-start-all.sh to start it! |
| Your default account password is [hadoop/5e8e312b4]</code></p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="33-配置的修改"></a>3.3 配置的修改<a class="hash-link" href="#33-配置的修改" title="Direct link to heading">#</a></h3><p>安装完成后，如果需要修改配置，可以重新执行安装，或则修改对应${InstallPath}/conf/*properties文件，重启对应的服务</p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="34-添加mysql驱动103版本"></a>3.4 添加mysql驱动(>=1.0.3)版本<a class="hash-link" href="#34-添加mysql驱动103版本" title="Direct link to heading">#</a></h3><p>因为license原因，linkis官方发布包中(dss集成的全家桶会包含，无需手动添加)移除了mysql-connector-java，需要手动添加<br> |
| <!-- -->具体参见<a href="/zh-CN/docs/latest/deployment/quick_deploy#-44-%E6%B7%BB%E5%8A%A0mysql%E9%A9%B1%E5%8A%A8%E5%8C%85"> 添加mysql驱动包</a></p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="35-启动服务"></a>3.5 启动服务<a class="hash-link" href="#35-启动服务" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token plain"> sbin/linkis-start-all.sh</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="36-检查服务是否正常启动"></a>3.6 检查服务是否正常启动<a class="hash-link" href="#36-检查服务是否正常启动" title="Direct link to heading">#</a></h3><p>访问eureka服务页面(http://eurekaip:20303)，1.0.x版本，以下服务是必须正常启动</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">LINKIS-CG-ENGINECONNMANAGER</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">LINKIS-CG-ENGINEPLUGIN</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">LINKIS-CG-ENTRANCE</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">LINKIS-CG-LINKISMANAGER </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">LINKIS-MG-EUREKA </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">LINKIS-MG-GATEWAY</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">LINKIS-PS-CS</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">LINKIS-PS-PUBLICSERVICE</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>如果有服务未启动，可以在对应的log/${服务名}.log文件中查看详细异常日志。</p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="4-安装web前端"></a>4. 安装web前端<a class="hash-link" href="#4-安装web前端" title="Direct link to heading">#</a></h2><p>web端是使用nginx作为静态资源服务器的，访问请求流程是: |
| linkis管理台请求->nginx ip:port->linkis-gateway ip:port-> 其他服务</p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="41-下载前端安装包并解压"></a>4.1 下载前端安装包并解压<a class="hash-link" href="#41-下载前端安装包并解压" title="Direct link to heading">#</a></h3><p>tar -xvf apache-linkis-1.0.x-incubating-web-bin.tar.gz</p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="42-修改配置configsh"></a>4.2 修改配置config.sh<a class="hash-link" href="#42-修改配置configsh" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">#管理台访问的端口 http://localhost:8088</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">linkis_port</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"8088"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#linkis-mg-gateway服务地址</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">linkis_url</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"http://localhost:9020"</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="43-执行前端部署"></a>4.3 执行前端部署<a class="hash-link" href="#43-执行前端部署" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">#nginx 需要sudo权限进行安装</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token function" style="color:rgb(80, 250, 123)">sudo</span><span class="token plain"> </span><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token plain"> install.sh</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>安装后，linkis的nginx配置文件默认是 在/etc/nginx/conf.d/linkis.conf |
| nginx的日志文件在 /var/log/nginx/access.log 和/var/log/nginx/error.log</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI nginx"><pre tabindex="0" class="prism-code language-nginx codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> server {</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> listen 8188;# 访问端口 如果端口被占用，则需要修改</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> server_name localhost;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> #charset koi8-r;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> #access_log /var/log/nginx/host.access.log main;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> location /linkis/visualis {</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> root /appcom/Install/linkis-web/linkis/visualis; # 静态文件目录 </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> autoindex on;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> location / {</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> root /appcom/Install/linkis-web/dist; # 静态文件目录 </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> index index.html index.html;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> location /ws {</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_pass http://localhost:9020;#后端Linkis的地址</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_http_version 1.1;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header Upgrade $http_upgrade;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header Connection upgrade;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> location /api {</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_pass http://localhost:9020; #后端Linkis的地址</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header Host $host;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header X-Real-IP $remote_addr;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header x_real_ipP $remote_addr;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header remote_addr $remote_addr;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_http_version 1.1;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_connect_timeout 4s;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_read_timeout 600s;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_send_timeout 12s;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header Upgrade $http_upgrade;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> proxy_set_header Connection upgrade;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> #error_page 404 /404.html;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> # redirect server error pages to the static page /50x.html</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> #</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> error_page 500 502 503 504 /50x.html;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> location = /50x.html {</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> root /usr/share/nginx/html;</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>如果需要修改端口或则静态资源目录等，请修改/etc/nginx/conf.d/linkis.conf 文件后执行 <code>sudo nginx -s reload</code> 命令</p><div class="admonition admonition-caution alert alert--warning"><div class="admonition-heading"><h5><span class="admonition-icon"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>注意</h5></div><div class="admonition-content"><ul><li>目前暂未集成visualis功能，安装过程中如果提示安装linkis/visualis失败，可以忽略 </li><li>查看nginx是否正常启动：检查nginx进程是否存在 ps -ef |grep nginx </li><li>检查nginx的配置是否正确 sudo nginx -T </li><li>如果端口被占用，可以修改nginx启动的服务端口<code>/etc/nginx/conf.d/linkis.conf</code>listen端口值，保存后重新启动</li><li>如果访问管理台出现接口502，或则<code>Unexpected token < in JSON at position 0</code>异常，请确认linkis-mg-gateway是否正常启动，如果正常启动，查看nginx配置文件中配置的linkis-mg-gateway服务地址是否正确</li></ul></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="44-登录web端查看信息"></a>4.4 登录web端查看信息<a class="hash-link" href="#44-登录web端查看信息" title="Direct link to heading">#</a></h3><p><a href="http://xx.xx.xx.xx:8188/#/login" target="_blank" rel="noopener noreferrer">http://xx.xx.xx.xx:8188/#/login</a> |
| 用户名/密码在{InstallPath}/conf/linkis-mg-gateway.properties中查看</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">#未使用LDAP配置时</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.admin.user</span><span class="token operator">=</span><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)">#用户</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.admin.password</span><span class="token operator">=</span><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)">#密码</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="5-yarn队列检查"></a>5 Yarn队列检查<a class="hash-link" href="#5-yarn队列检查" title="Direct link to heading">#</a></h2><blockquote><p>如果需要使用到spark/hive/flink引擎 </p></blockquote><p>登录后查看能否正常显示yarn队列资源(点击页面右下角按钮)<br> |
| <!-- -->正常如下图所示:<br> |
| <img src="https://user-images.githubusercontent.com/7869972/159955494-2f305a38-a3d6-4798-83aa-58cde23bc436.png" alt="yarn-normal"></p><p>若如果无法显示：可以按以下指引调整 </p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="51-查看yarn地址是否配置正确"></a>5.1 查看yarn地址是否配置正确<a class="hash-link" href="#51-查看yarn地址是否配置正确" title="Direct link to heading">#</a></h3><p>数据库表 linkis_cg_rm_external_resource_provider<br> |
| <!-- -->插入yarn数据信息 </p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">INSERT</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">INTO</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token plain">linkis_cg_rm_external_resource_provider</span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token plain">resource_type</span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token plain">name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token plain">labels</span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token plain">config</span><span class="token punctuation" style="color:rgb(248, 248, 242)">`</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">VALUES</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token string" style="color:rgb(255, 121, 198)">'Yarn'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'sit'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token boolean">NULL</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token string" style="color:rgb(255, 121, 198)">'{\r\n"rmWebAddress": "http://xx.xx.xx.xx:8088",\r\n"hadoopVersion": "2.7.2",\r\n"authorEnable":false,\r\n"user":"hadoop",\r\n"pwd":"123456"\r\n}'</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">config字段属性</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token string" style="color:rgb(255, 121, 198)">"rmWebAddress"</span><span class="token plain">: </span><span class="token string" style="color:rgb(255, 121, 198)">"http://xx.xx.xx.xx:8088"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)">#需要带上http以及端口</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token string" style="color:rgb(255, 121, 198)">"hadoopVersion"</span><span class="token plain">: </span><span class="token string" style="color:rgb(255, 121, 198)">"2.7.2"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token string" style="color:rgb(255, 121, 198)">"authorEnable"</span><span class="token plain">:</span><span class="token boolean">true</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)">//是否需要认证 可以在浏览器中通过访问http://xx.xx.xx.xx:8088验证用户名和密码</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token string" style="color:rgb(255, 121, 198)">"user"</span><span class="token plain">:</span><span class="token string" style="color:rgb(255, 121, 198)">"user"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token comment" style="color:rgb(98, 114, 164)">//用户名</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token string" style="color:rgb(255, 121, 198)">"pwd"</span><span class="token plain">:</span><span class="token string" style="color:rgb(255, 121, 198)">"pwd"</span><span class="token comment" style="color:rgb(98, 114, 164)">//密码</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>更新后，因为程序中有使用到缓存，想要立即生效，需要重启linkis-cg-linkismanager服务</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token plain"> sbin/linkis-daemon.sh restart cg-linkismanager</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="52-查看yarn队列是否正确"></a>5.2 查看yarn队列是否正确<a class="hash-link" href="#52-查看yarn队列是否正确" title="Direct link to heading">#</a></h4><p>异常信息:<code>desc: queue ide is not exists in YARN.</code> |
| 表明配置的yarn队列不存在，需要进行调整</p><p>修改方式:linkis管理台/参数配置>全局设置>yarn队列名<!-- -->[wds.linkis.rm.yarnqueue]<!-- -->,修改一个可以使用的yarn队列,可以使用的yarn 队列可以在 rmWebAddress:<a href="http://xx.xx.xx.xx:8088/cluster/scheduler" target="_blank" rel="noopener noreferrer">http://xx.xx.xx.xx:8088/cluster/scheduler</a> 上查看到 </p><h4><a aria-hidden="true" tabindex="-1" class="anchor anchor__h4 anchorWithStickyNavbar_y2LR" id="53-查看yarn队列"></a>5.3 查看yarn队列<a class="hash-link" href="#53-查看yarn队列" title="Direct link to heading">#</a></h4><ul><li>查看hadoop集群地址: http://ip:8088/cluster </li><li>查看yarn队列地址：http://ip:8888/cluster/scheduler </li></ul><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="6-检查引擎物料资源是否上传成功"></a>6 检查引擎物料资源是否上传成功<a class="hash-link" href="#6-检查引擎物料资源是否上传成功" title="Direct link to heading">#</a></h2><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI sql"><pre tabindex="0" class="prism-code language-sql codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">#登陆到linkis的数据库 </span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">select</span><span class="token plain"> </span><span class="token operator">*</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">from</span><span class="token plain"> linkis_cg_engine_conn_plugin_bml_resources</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>正常如下： |
| <img src="https://user-images.githubusercontent.com/29391030/156343249-9f6dca8f-4e0d-438b-995f-4f469270a22d.png" alt="bml"></p><p>查看引擎的物料记录是否存在(如果有更新,查看更新时间是否正确)。</p><p>如果不存在或则未更新，先尝试手动刷新物料资源(详细见<a href="/zh-CN/docs/latest/deployment/engine_conn_plugin_installation#23-%E5%BC%95%E6%93%8E%E5%88%B7%E6%96%B0">引擎物料资源刷新</a>)。通过<code>log/linkis-cg-engineplugin.log</code>日志，查看物料失败的具体原因，很多时候可能是hdfs目录没有权限导致，检查gateway地址配置是否正确<code>conf/linkis.properties:wds.linkis.gateway.url</code> </p><p>引擎的物料资源默认上传到hdfs目录为 <code>/apps-data/${deployUser}/bml</code> </p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">hdfs dfs -ls /apps-data/hadoop/bml</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">#如果没有该目录 请手动创建目录并授予${deployUser}读写权限</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">hdfs dfs -mkdir /apps-data</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">hdfs dfs -chown hadoop:hadoop /apps-data</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="7-验证基础功能"></a>7 验证基础功能<a class="hash-link" href="#7-验证基础功能" title="Direct link to heading">#</a></h2><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI"><pre tabindex="0" class="prism-code language-undefined codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">#引擎的engineType 拼接的版本号，一定要与实际的相匹配</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">sh bin/linkis-cli -submitUser hadoop -engineType shell-1 -codeType shell -code "whoami"</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">sh bin/linkis-cli -submitUser hadoop -engineType hive-2.3.3 -codeType hql -code "show tables"</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">sh bin/linkis-cli -submitUser hadoop -engineType spark-2.4.3 -codeType sql -code "show tables"</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">sh bin/linkis-cli -submitUser hadoop -engineType python-python2 -codeType python -code 'print("hello, world!")'</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="8-查看支持的各个引擎的版本"></a>8 查看支持的各个引擎的版本<a class="hash-link" href="#8-查看支持的各个引擎的版本" title="Direct link to heading">#</a></h2><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="81-方式1查看引擎打包的目录"></a>8.1 方式1:查看引擎打包的目录<a class="hash-link" href="#81-方式1查看引擎打包的目录" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI"><pre tabindex="0" class="prism-code language-undefined codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ tree linkis-package/lib/linkis-engineconn-plugins/ -L 3</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">linkis-package/lib/linkis-engineconn-plugins/</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">├── hive</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── dist</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ │ └── v2.3.3 #版本为2.3.3 engineType 为hive-2.3.3</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ └── plugin</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ └── 2.3.3</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">├── python</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── dist</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ │ └── vpython2</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ └── plugin</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ └── python2 #版本为python2 engineType 为python-python2</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">├── shell</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── dist</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ │ └── v1</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ └── plugin</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ └── 1</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">└── spark</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ├── dist</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> │ └── v2.4.3</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> └── plugin</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> └── 2.4.3</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="82-方式2查看linkis的数据库表"></a>8.2 方式2:查看linkis的数据库表<a class="hash-link" href="#82-方式2查看linkis的数据库表" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">select</span><span class="token plain"> * from linkis_cg_engine_conn_plugin_bml_resources</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="9-安装部署常见问题的排障"></a><font color="red">9. 安装部署常见问题的排障</font><a class="hash-link" href="#9-安装部署常见问题的排障" title="Direct link to heading">#</a></h2><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="90-登陆密码问题"></a>9.0 登陆密码问题<a class="hash-link" href="#90-登陆密码问题" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI"><pre tabindex="0" class="prism-code language-undefined codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain"> linkis默认是使用静态用户和密码,静态用户即部署用户，静态密码会在执行部署是随机生成一个密码串，存储于{InstallPath}/conf/linkis-mg-gateway.properties(>=1.0.3版本)</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="91-版本兼容性问题"></a>9.1 版本兼容性问题<a class="hash-link" href="#91-版本兼容性问题" title="Direct link to heading">#</a></h3><p>linkis默认支持的引擎，与dss兼容关系可以查看此文档 <a href="https://github.com/apache/incubator-linkis/blob/master/README.md" target="_blank" rel="noopener noreferrer">https://github.com/apache/incubator-linkis/blob/master/README.md</a> </p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="92-如何定位服务端异常日志"></a>9.2 如何定位服务端异常日志<a class="hash-link" href="#92-如何定位服务端异常日志" title="Direct link to heading">#</a></h3><p>linkis的微服务比较多，若对系统不熟悉，有时候无法定位到具体哪个模块出现了异常，可以通过全局日志搜索方式 </p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">tail</span><span class="token plain"> -f log/* </span><span class="token operator">|</span><span class="token function" style="color:rgb(80, 250, 123)">grep</span><span class="token plain"> -5n exception</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">或则tail -f log/* </span><span class="token operator">|</span><span class="token function" style="color:rgb(80, 250, 123)">grep</span><span class="token plain"> -5n ERROR</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"> </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token function" style="color:rgb(80, 250, 123)">less</span><span class="token plain"> log/* </span><span class="token operator">|</span><span class="token function" style="color:rgb(80, 250, 123)">grep</span><span class="token plain"> -5n exception</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">或则less log/* </span><span class="token operator">|</span><span class="token function" style="color:rgb(80, 250, 123)">grep</span><span class="token plain"> -5n ERROR</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"> </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="93-执行引擎任务的异常排查"></a>9.3 执行引擎任务的异常排查<a class="hash-link" href="#93-执行引擎任务的异常排查" title="Direct link to heading">#</a></h3><p><strong> step1:找到引擎的启动部署目录 </strong><br> |
| <!-- -->方式1：如果执行日志中有显示，可以在管理台上查看到 如下图:<br> |
| <img src="https://user-images.githubusercontent.com/29391030/156343802-9d47fa98-dc70-4206-b07f-df439b291028.png" alt="engine-log"> |
| 方式2:如果方式1中没有找到，可以通过找到<code>conf/linkis-cg-engineconnmanager.properties</code>配置的<code>wds.linkis.engineconn.root.dir</code>的参数，该值就是引擎启动部署的目录，子目录按执行引擎的用户进行了隔离。</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">如果不清楚taskid，可以按时间排序后进行选择 ll -rt /appcom/tmp/</span><span class="token variable" style="color:rgb(189, 147, 249);font-style:italic">${执行的用户}</span><span class="token plain">/workDir </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">cd</span><span class="token plain"> /appcom/tmp/</span><span class="token variable" style="color:rgb(189, 147, 249);font-style:italic">${执行的用户}</span><span class="token plain">/workDir/</span><span class="token variable" style="color:rgb(189, 147, 249);font-style:italic">${taskId}</span><span class="token plain"> </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">目录大体如下 </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">conf -</span><span class="token operator">></span><span class="token plain"> /appcom/tmp/engineConnPublickDir/6a09d5fb-81dd-41af-a58b-9cb5d5d81b5a/v000002/conf </span><span class="token comment" style="color:rgb(98, 114, 164)">#引擎的配置文件 </span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">engineConnExec.sh </span><span class="token comment" style="color:rgb(98, 114, 164)">#生成的引擎的启动脚本 </span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">lib -</span><span class="token operator">></span><span class="token plain"> /appcom/tmp/engineConnPublickDir/45bf0e6b-0fa5-47da-9532-c2a9f3ec764d/v000003/lib </span><span class="token comment" style="color:rgb(98, 114, 164)">#引擎依赖的包 </span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">logs </span><span class="token comment" style="color:rgb(98, 114, 164)">#引擎启动执行的相关日志 </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong> step2：查看引擎的日志 </strong> </p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">less</span><span class="token plain"> logs/stdout </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong> step3：尝试手动执行脚本(如果需要) </strong><br> |
| <!-- -->可以通过尝试手动执行脚本，进行调试 </p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI"><pre tabindex="0" class="prism-code language-undefined codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">sh -v engineConnExec.sh </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="94-cdh适配版本的注意事项"></a>9.4 CDH适配版本的注意事项<a class="hash-link" href="#94-cdh适配版本的注意事项" title="Direct link to heading">#</a></h3><p>CDH本身不是使用的官方标准的hive/spark包,进行适配时，最好修改linkis的源码中的hive/spark版本的依赖，进行重新编译部署。<br> |
| <!-- -->具体可以参考CDH适配博文<br> |
| <a href="https://mp.weixin.qq.com/s/__QxC1NoLQFwme1yljy-Nw" target="_blank" rel="noopener noreferrer">【Linkis1.0——CDH5环境中的安装与踩坑】</a><br> |
| <a href="https://mp.weixin.qq.com/s/9Pl9P0hizDWbbTBf1yzGJA" target="_blank" rel="noopener noreferrer">【DSS1.0.0+Linkis1.0.2——CDH5环境中的试用记录】</a><br> |
| <a href="https://mp.weixin.qq.com/s/vcFge4BNiEuW-7OC3P-yaw" target="_blank" rel="noopener noreferrer">【DSS1.0.0与Linkis1.0.2——JDBC引擎相关问题汇总】</a><br> |
| <a href="https://mp.weixin.qq.com/s/VxZ16IPMd1CvcrvHFuU4RQ" target="_blank" rel="noopener noreferrer">【DSS1.0.0与Linkis1.0.2——Flink引擎相关问题汇总】</a> </p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="95-http接口的调试"></a>9.5 Http接口的调试<a class="hash-link" href="#95-http接口的调试" title="Direct link to heading">#</a></h3><ul><li>方式1 可以开启<a href="/zh-CN/docs/latest/api/login_api/#2%E5%85%8D%E7%99%BB%E5%BD%95%E9%85%8D%E7%BD%AE">免登陆模式指引</a> </li><li>方式2 postman中的，请求头带上登陆成功的cookie值 |
| cookie值可以在浏览器端登陆成功后，获取 |
| <img src="https://user-images.githubusercontent.com/7869972/157619718-3afb480f-6087-4d5c-9a77-5e75c8cb4a3c.png" alt="bml"></li></ul><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">Cookie: bdp-user-ticket-id</span><span class="token operator">=</span><span class="token plain">xxxxxxxxxxxxxxxxxxxxxxxxxxx</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><ul><li>方式3 http请求头添加静态的Token令牌<br>Token在conf/token.properties进行配置 |
| 如:TEST-AUTH=hadoop,root,user01</li></ul><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">Token-Code:TEST-AUTH</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Token-User:hadoop</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="96-异常问题的排查流程"></a>9.6 异常问题的排查流程<a class="hash-link" href="#96-异常问题的排查流程" title="Direct link to heading">#</a></h3><p>首先要按上述步骤检查服务/环境等是否都正常启动<br> |
| <!-- -->按上述罗列的一些场景的方式进行基础问题的排查<br> |
| <a href="https://docs.qq.com/doc/DSGZhdnpMV3lTUUxq" target="_blank" rel="noopener noreferrer">QA文档</a>中查找是否有解决方案，链接：<a href="https://docs.qq.com/doc/DSGZhdnpMV3lTUUxq" target="_blank" rel="noopener noreferrer">https://docs.qq.com/doc/DSGZhdnpMV3lTUUxq</a><br> |
| <!-- -->通过搜索issue中的内容,看是否能找到解决方案<br> |
| <img src="https://user-images.githubusercontent.com/29391030/156343419-81cc25e0-aa94-4c06-871c-bb036eb6d4ff.png" alt="issues"><br> |
| <!-- -->通过官网文档搜索，对于某些问题，可以通过官网搜索关键字进行查询，比如搜索"部署"相关。(如果出现404,请浏览器中刷新一下)<br> |
| <img src="https://user-images.githubusercontent.com/29391030/156343459-7911bd05-4d8d-4a7b-b9f8-35c152d52c41.png" alt="search"> </p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="10-相关的资料如何获取"></a>10. 相关的资料如何获取<a class="hash-link" href="#10-相关的资料如何获取" title="Direct link to heading">#</a></h2><p>linkis官网文档正在不断的完善,可以在本官网查看/关键字搜索相关文档。<br> |
| <!-- -->相关博文链接 </p><ul><li>Linkis的技术博文集 <a href="https://github.com/apache/incubator-linkis/issues/1233" target="_blank" rel="noopener noreferrer">https://github.com/apache/incubator-linkis/issues/1233</a> </li><li>公众号技术博文<a href="https://mp.weixin.qq.com/mp/homepage?__biz=MzI4MDkxNzUxMg==&hid=1&sn=088cbf2bbed1c80d003c5865bc92ace8&scene=18" target="_blank" rel="noopener noreferrer">https://mp.weixin.qq.com/mp/homepage?__biz=MzI4MDkxNzUxMg==&hid=1&sn=088cbf2bbed1c80d003c5865bc92ace8&scene=18</a> </li><li>官网文档 <a href="https://linkis.apache.org/zh-CN/docs/latest/introduction" target="_blank" rel="noopener noreferrer">https://linkis.apache.org/zh-CN/docs/latest/introduction</a> </li><li>bili技术分享视频 <a href="https://space.bilibili.com/598542776?spm_id_from=333.788.b_765f7570696e666f.2" target="_blank" rel="noopener noreferrer">https://space.bilibili.com/598542776?spm_id_from=333.788.b_765f7570696e666f.2</a></li></ul></div></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="博文分页导航"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/zh-CN/blog/2022/03/20/openlookeng"><div class="pagination-nav__sublabel">上一篇</div><div class="pagination-nav__label">« <!-- -->OpenLooKeng的引擎的实现</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/zh-CN/blog/2022/02/08/how-to-user-blog"><div class="pagination-nav__sublabel">下一篇</div><div class="pagination-nav__label">如何编写博客<!-- --> »</div></a></div></nav></main><div class="col col--2"><div class="tableOfContents_vrFS thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#1-前置准备注意事项" class="table-of-contents__link">1. 前置准备注意事项</a><ul><li><a href="#11-linux服务器" class="table-of-contents__link">1.1 linux服务器</a></li><li><a href="#12-添加部署用户" class="table-of-contents__link">1.2 添加部署用户</a></li><li><a href="#13-安装包准备" class="table-of-contents__link">1.3 安装包准备</a></li><li><a href="#14-底层依赖检查" class="table-of-contents__link">1.4 底层依赖检查</a></li><li><a href="#15-资源依赖" class="table-of-contents__link">1.5 资源依赖</a></li></ul></li><li><a href="#2-安裝" class="table-of-contents__link">2. 安裝</a><ul><li><a href="#21-安装包解压" class="table-of-contents__link">2.1 安装包解压</a></li><li><a href="#22-配置linkis自身数据库信息" class="table-of-contents__link">2.2 配置linkis自身数据库信息</a></li><li><a href="#23-配置基础环境变量" class="table-of-contents__link">2.3 配置基础环境变量</a></li></ul></li><li><a href="#3-部署流程" class="table-of-contents__link">3. 部署流程</a><ul><li><a href="#31-执行部署脚本" class="table-of-contents__link">3.1 执行部署脚本</a></li><li><a href="#32-可能遇到的问题" class="table-of-contents__link">3.2 可能遇到的问题</a></li><li><a href="#33-配置的修改" class="table-of-contents__link">3.3 配置的修改</a></li><li><a href="#34-添加mysql驱动103版本" class="table-of-contents__link">3.4 添加mysql驱动(>=1.0.3)版本</a></li><li><a href="#35-启动服务" class="table-of-contents__link">3.5 启动服务</a></li><li><a href="#36-检查服务是否正常启动" class="table-of-contents__link">3.6 检查服务是否正常启动</a></li></ul></li><li><a href="#4-安装web前端" class="table-of-contents__link">4. 安装web前端</a><ul><li><a href="#41-下载前端安装包并解压" class="table-of-contents__link">4.1 下载前端安装包并解压</a></li><li><a href="#42-修改配置configsh" class="table-of-contents__link">4.2 修改配置config.sh</a></li><li><a href="#43-执行前端部署" class="table-of-contents__link">4.3 执行前端部署</a></li><li><a href="#44-登录web端查看信息" class="table-of-contents__link">4.4 登录web端查看信息</a></li></ul></li><li><a href="#5-yarn队列检查" class="table-of-contents__link">5 Yarn队列检查</a><ul><li><a href="#51-查看yarn地址是否配置正确" class="table-of-contents__link">5.1 查看yarn地址是否配置正确</a></li></ul></li><li><a href="#6-检查引擎物料资源是否上传成功" class="table-of-contents__link">6 检查引擎物料资源是否上传成功</a></li><li><a href="#7-验证基础功能" class="table-of-contents__link">7 验证基础功能</a></li><li><a href="#8-查看支持的各个引擎的版本" class="table-of-contents__link">8 查看支持的各个引擎的版本</a><ul><li><a href="#81-方式1查看引擎打包的目录" class="table-of-contents__link">8.1 方式1:查看引擎打包的目录</a></li><li><a href="#82-方式2查看linkis的数据库表" class="table-of-contents__link">8.2 方式2:查看linkis的数据库表</a></li></ul></li><li><a href="#9-安装部署常见问题的排障" class="table-of-contents__link"><font color="red">9. 安装部署常见问题的排障</font></a><ul><li><a href="#90-登陆密码问题" class="table-of-contents__link">9.0 登陆密码问题</a></li><li><a href="#91-版本兼容性问题" class="table-of-contents__link">9.1 版本兼容性问题</a></li><li><a href="#92-如何定位服务端异常日志" class="table-of-contents__link">9.2 如何定位服务端异常日志</a></li><li><a href="#93-执行引擎任务的异常排查" class="table-of-contents__link">9.3 执行引擎任务的异常排查</a></li><li><a href="#94-cdh适配版本的注意事项" class="table-of-contents__link">9.4 CDH适配版本的注意事项</a></li><li><a href="#95-http接口的调试" class="table-of-contents__link">9.5 Http接口的调试</a></li><li><a href="#96-异常问题的排查流程" class="table-of-contents__link">9.6 异常问题的排查流程</a></li></ul></li><li><a href="#10-相关的资料如何获取" class="table-of-contents__link">10. 相关的资料如何获取</a></li></ul></div></div></div></div></div><footer class="footer"><div class="container"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">Linkis</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/zh-CN/docs/latest/introduction">文档</a></li><li class="footer__item"><a class="footer__link-item" href="/zh-CN/faq/main">FAQ</a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/releases" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>版本<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">社区</div><ul class="footer__items"><li class="footer__item"><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/issues" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Issue Tracker<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/pulls" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Pull Requests<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">订阅邮件组</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/zh-CN/community/how-to-subscribe">如何订阅</a></li><li class="footer__item"><a href="mailto:dev-subscribe@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>订阅邮件<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://lists.apache.org/list.html?dev@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>邮件归档<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright"><div style="text-align: left;"> |
| <div> |
| <p style="font-family: Avenir-Medium;font-size: 14px;color: #999;line-height: 20px;">Apache Linkis (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p> |
| </div> |
| <div style="border-top: 1px solid #ccc;min-height: 60px;line-height: 20px;text-align: center;font-family: Avenir-Medium;font-size: 14px;color: #999;display: flex;align-items: center;"><span>Copyright © 2019-2020 The Apache Software Foundation. Apache Linkis, Linkis, and its feather logo are trademarks of The Apache Software Foundation.</span></div> |
| </div></div></div></div></footer></div> |
| <script src="/zh-CN/assets/js/runtime~main.14d5df0c.js"></script> |
| <script src="/zh-CN/assets/js/main.5a9c4743.js"></script> |
| </body> |
| </html> |
