| <!doctype html> |
| <html class="docs-version-1.1.2" lang="en" dir="ltr"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width,initial-scale=1"> |
| <meta name="generator" content="Docusaurus v2.0.0-beta.14"> |
| <link rel="alternate" type="application/rss+xml" href="/blog/rss.xml" title="Apache Linkis Blog RSS Feed"> |
| <link rel="alternate" type="application/atom+xml" href="/blog/atom.xml" title="Apache Linkis Blog Atom Feed"> |
| <link rel="search" type="application/opensearchdescription+xml" title="Apache Linkis" href="/opensearch.xml"><title data-react-helmet="true">Introduction to Linkis Configuration Parameters | Apache Linkis</title><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" property="og:url" content="https://linkis.incubator.apache.org/docs/latest/development/linkis_config"><meta data-react-helmet="true" name="docsearch:language" content="en"><meta data-react-helmet="true" name="docsearch:version" content="1.1.2"><meta data-react-helmet="true" name="docsearch:docusaurus_tag" content="docs-default-1.1.2"><meta data-react-helmet="true" property="og:title" content="Introduction to Linkis Configuration Parameters | Apache Linkis"><meta data-react-helmet="true" name="description" content="1. Parameter classification"><meta data-react-helmet="true" property="og:description" content="1. Parameter classification"><link data-react-helmet="true" rel="shortcut icon" href="/img/favicon.ico"><link data-react-helmet="true" rel="canonical" href="https://linkis.incubator.apache.org/docs/latest/development/linkis_config"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/docs/latest/development/linkis_config" hreflang="en"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/zh-CN/docs/latest/development/linkis_config" hreflang="zh-CN"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/docs/latest/development/linkis_config" hreflang="x-default"><link data-react-helmet="true" rel="preconnect" href="https://AE29KQB3IA-dsn.algolia.net" crossorigin="anonymous"><link rel="stylesheet" href="/assets/css/styles.92d87943.css"> |
| <link rel="preload" href="/assets/js/runtime~main.a4c60f50.js" as="script"> |
| <link rel="preload" href="/assets/js/main.42471b13.js" as="script"> |
| </head> |
| <body> |
| <script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus"> |
| <div><a href="#" class="skipToContent_OuoZ">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--light_4Vu1 navbar__logo"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--dark_uzRr navbar__logo"><b class="navbar__title">Apache Linkis(Incubating)</b></a><a class="navbar__item navbar__link" href="/">Home</a><a class="navbar__item navbar__link" href="/faq/main">FAQ</a><a class="navbar__item navbar__link" href="/download/main">Download</a><a class="navbar__item navbar__link" href="/community/how-to-subscribe">Community</a><a class="navbar__item navbar__link" href="/blog">Blog</a><a class="navbar__item navbar__link" href="/team">Team</a><a class="navbar__item navbar__link" href="/user">Users</a><div class="navbar__item dropdown dropdown--hoverable"><a class="navbar__link">ASF</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Foundation</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship</a></li><li><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks</a></li></ul></div></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link">Doc</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/1.1.3/introduction">Next(1.1.3)</a></li><li><a class="dropdown__link" href="/docs/latest/introduction">1.1.2</a></li><li><a class="dropdown__link" href="/docs/1.1.1/introduction">1.1.1</a></li><li><a class="dropdown__link" href="/docs/1.1.0/introduction">1.1.0</a></li><li><a class="dropdown__link" href="/docs/1.0.3/introduction">1.0.3</a></li><li><a class="dropdown__link" href="/docs/1.0.2/introduction">1.0.2</a></li><li><a class="dropdown__link" href="/docs/0.11.0/introduction">0.11.0</a></li><li><a class="dropdown__link" href="/versions">All Version</a></li></ul></div><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-github-link" aria-label="GitHub"></a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" class="navbar__link"><span><svg t="1631348384596" class="iconLanguage_EbrZ" viewBox="0 0 1024 1024" version="1.1" p-id="557" width="20" height="20"><path d="M547.797333 638.208l-104.405333-103.168 1.237333-1.28a720.170667 720.170667 0 0 0 152.490667-268.373333h120.448V183.082667h-287.744V100.906667H347.605333v82.218666H59.818667V265.386667h459.178666a648.234667 648.234667 0 0 1-130.304 219.946666 643.242667 643.242667 0 0 1-94.976-137.728H211.541333a722.048 722.048 0 0 0 122.453334 187.434667l-209.194667 206.378667 58.368 58.368 205.525333-205.525334 127.872 127.829334 31.232-83.84m231.424-208.426667h-82.218666l-184.96 493.312h82.218666l46.037334-123.306667h195.242666l46.464 123.306667h82.218667l-185.002667-493.312m-107.690666 287.744l66.56-178.005333 66.602666 178.005333z" fill="currentColor" p-id="558"></path></svg><span>English</span></span></a><ul class="dropdown__menu"><li><a href="/docs/latest/development/linkis_config" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" style="text-transform:capitalize">English</a></li><li><a href="/zh-CN/docs/latest/development/linkis_config" target="_self" rel="noopener noreferrer" class="dropdown__link" style="text-transform:capitalize">简体中文</a></li></ul></div><div class="searchBox_Bc3W"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div class="main-wrapper docs-wrapper docs-doc-page"><div class="docPage_lDyR"><button class="clean-btn backToTopButton_i9tI" type="button"><svg viewBox="0 0 24 24" width="28"><path d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z" fill="currentColor"></path></svg></button><aside class="docSidebarContainer_0YBq"><div class="sidebar_a3j0"><nav class="menu thin-scrollbar menu_cyFh menuWithAnnouncementBar_+O1J"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" href="/docs/latest/introduction">Introduction</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" href="/docs/latest/release">Version Overview</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Deployment</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">User Guide</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Engine Usage</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">API Docs</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Table Structure</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Architecture</a></li><li class="theme-doc-sidebar-item-category menu__list-item"><a class="menu__link menu__link--sublist menu__link--active" href="#">Development Doc</a><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/development/linkis_compile_and_package">Compile And Package</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/latest/development/linkis_config">Introduction to Linkis Configuration Parameters</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/development/linkis_debug">Linkis Debug</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/development/linkis_debug_in_mac">Linkis Debug In Mac</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/development/new_engine_conn">How To Quickly Implement A New Engine</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/development/web_build">Linkis Console Compile</a></li></ul></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Upgrade Guide</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Tuning And Troubleshooting</a></li></ul></nav></div></aside><main class="docMainContainer_r8cw"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_zHA2"><div class="docItemContainer_oiyr"><article><span class="theme-doc-version-badge badge badge--secondary">Version: <!-- -->1.1.2</span><div class="tocCollapsible_aw-L theme-doc-toc-mobile tocMobile_Tx6Y"><button type="button" class="clean-btn tocCollapsibleButton_zr6a">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Introduction to Linkis Configuration Parameters</h1></header><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="1-parameter-classification"></a>1. Parameter classification<a class="hash-link" href="#1-parameter-classification" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Linkis parameters are mainly divided into the following three parts:</p><ol><li>Linkis server parameters, mainly including the parameters of Linkis itself and the parameters of Spring</li><li>Parameters submitted by client calls such as Linkis SDK and Restful</li><li>Linkis console parameters</li></ol><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="2-linkis-server-parameters"></a>2. Linkis server parameters<a class="hash-link" href="#2-linkis-server-parameters" title="Direct link to heading">#</a></h2><ol><li>Parameters of Linkis itself |
| <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->The parameters of linkis itself can be set in the configuration file, or through environment variables and system properties. It is recommended to use the configuration file to set. |
| The Linkis configuration file format is as follows:</li></ol><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">├── conf configuration directory</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── application-eureka.yml</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── application-linkis.yml</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-cg-engineconnmanager-io.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-cg-engineconnmanager.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-cg-engineplugin.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-cg-entrance.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-cg-linkismanager.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis.properties ── linkis global properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-ps-bml.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-ps-cs.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-ps-datasource.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── linkis-ps-publicservice.properties</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ ├── log4j2.xml</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Each service loads two property configuration files, a common main configuration file linkis.properties, and a service configuration file linkis-serviceName.properties. The priority of settings is that the service profile is higher than the main profile |
| It is recommended that common parameters be placed in the main configuration file, and individual configuration files are placed in the service configuration file</p><ol start="2"><li>Spring parameters |
| <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Linkis service is based on SpringBoot application, Spring related parameters can be set in application-linkis.yml or in linkis configuration file. The configuration in the linkis configuration file needs to be prefixed with spring. as follows:</li></ol><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)"># spring port default</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">server.port</span><span class="token operator">=</span><span class="token number">9102</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># in linkis conf need spring prefix</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">spring.server.port</span><span class="token operator">=</span><span class="token number">9102</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="3-linkis-client-parameters"></a>3. Linkis client parameters<a class="hash-link" href="#3-linkis-client-parameters" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Linkis client parameters mainly refer to the parameters when the task is submitted, mainly the parameters specified in the submission interface.</p><ol><li>How restful sets parameters:</li></ol><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"executionContent"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token string" style="color:rgb(255, 121, 198)">"code"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"show tables"</span><span class="token plain">, </span><span class="token string" style="color:rgb(255, 121, 198)">"runType"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"sql"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">,</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"params"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"> // submit parameters</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"variable"</span><span class="token plain">:</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"> //Custom variables needed </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">in</span><span class="token plain"> the code</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"k1"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token string" style="color:rgb(255, 121, 198)">"v1"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">,</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"configuration"</span><span class="token plain">:</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"special"</span><span class="token plain">:</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"> //Special configuration parameters such as log path, result </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">set</span><span class="token plain"> path, etc.</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"k2"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token string" style="color:rgb(255, 121, 198)">"v2"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">,</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"runtime"</span><span class="token plain">:</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"> //Runtime parameters, execution configuration parameters, such as database connection parameters of JDBC engine, data </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">source</span><span class="token plain"> parameters of presto engine</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"k3"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token string" style="color:rgb(255, 121, 198)">"v3"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">,</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"startup"</span><span class="token plain">:</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"> //Startup parameters, such as memory parameters </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">for</span><span class="token plain"> starting EC, spark engine parameters, hive engine parameters, etc.</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"k4"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token string" style="color:rgb(255, 121, 198)">"v4"</span><span class="token plain"> For example: spark.executor.memory:5G Set the Spark executor memory, the underlying Spark, hive and other engine parameters keyName are consistent with the native parameters</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">,</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"labels"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"> //Label parameters, support setting engine version, user and application</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"engineType"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"spark-2.4.3"</span><span class="token plain">,</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"userCreator"</span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"hadoop-IDE"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><ol start="2"><li>How to set parameters in SDK:</li></ol><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI java"><pre tabindex="0" class="prism-code language-java codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">JobSubmitAction jobSubmitAction = JobSubmitAction.builder()</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addExecuteCode(code)</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .setStartupParams(startupMap) //Startup parameters, such as memory parameters for starting EC, spark engine parameters, hive engine parameters, etc., such as: spark.executor.memory:5G Set the Spark executor memory, the underlying Spark, hive and other engine parameters keyName is the same as the original parameter</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .setRuntimeParams(runTimeMap) //Engine, execute configuration parameters, such as database connection parameters of JDBC engine, data source parameters of presto engine</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .setVariableMap(varMap) //Custom variables needed in the code</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .setLabels(labels) //Label parameters, support setting engine version, user and application, etc.</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .setUser(user) //submit user</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addExecuteUser(user) // execute user</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .build();</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><ol start="3"><li>How linkis-cli sets parameters</li></ol><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">linkis-cli -runtieMap </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">key1</span><span class="token operator">=</span><span class="token plain">value -runtieMap </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">key2</span><span class="token operator">=</span><span class="token plain">value</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -labelMap </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">key1</span><span class="token operator">=</span><span class="token plain">value</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -varMap </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">key1</span><span class="token operator">=</span><span class="token plain">value</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -startUpMap </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">key1</span><span class="token operator">=</span><span class="token plain">value</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Note: When submitting client parameters, only engine-related parameters, tag parameters, and Yarn queue settings can take effect. Other Linkis server-side parameters and resource limit parameters, such as task and engine concurrency parameters wds.linkis.rm.instances do not support task settings</p><ol start="4"><li>Common label parameters:</li></ol><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain"> Map</span><span class="token operator"><</span><span class="token plain">String, Object</span><span class="token operator">></span><span class="token plain"> labels </span><span class="token operator">=</span><span class="token plain"> new HashMap</span><span class="token operator"><</span><span class="token plain">String, Object</span><span class="token operator">></span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> labels.put</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">LabelKeyConstant.ENGINE_TYPE_KEY, </span><span class="token string" style="color:rgb(255, 121, 198)">"spark-2.4.3"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"> // Specify engine </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">type</span><span class="token plain"> and version</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> labels.put</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">LabelKeyConstant.USER_CREATOR_TYPE_KEY, user + </span><span class="token string" style="color:rgb(255, 121, 198)">"-IDE"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain">// Specify the running user and your APPName</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> labels.put</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">LabelKeyConstant.CODE_TYPE_KEY, </span><span class="token string" style="color:rgb(255, 121, 198)">"sql"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"> // Specify the </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">type</span><span class="token plain"> of script to run: spark supports: sql, scala, py</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"> Hive: hql</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"> shell: </span><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"> python: python</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"> presto: psql</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> labels.put</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">LabelKeyConstant.JOB_RUNNING_TIMEOUT_KEY, </span><span class="token string" style="color:rgb(255, 121, 198)">"10000"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain">//The job runs </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">for</span><span class="token plain"> 10s and automatically initiates Kill, the unit is s</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> labels.put</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">LabelKeyConstant.JOB_QUEUING_TIMEOUT_KEY, </span><span class="token string" style="color:rgb(255, 121, 198)">"10000"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain">//The job is queued </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">for</span><span class="token plain"> </span><span class="token function" style="color:rgb(80, 250, 123)">more</span><span class="token plain"> than 10s and automatically initiates Kill, the unit is s</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> labels.put</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">LabelKeyConstant.RETRY_TIMEOUT_KEY, </span><span class="token string" style="color:rgb(255, 121, 198)">"10000"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain">//The waiting </span><span class="token function" style="color:rgb(80, 250, 123)">time</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">for</span><span class="token plain"> the job to retry due to resources and other reasons, the unit is ms. If it fails due to insufficient queue resources, it will initiate </span><span class="token number">10</span><span class="token plain"> retries at intervals by default</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> labels.put</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">LabelKeyConstant.TENANT_KEY,</span><span class="token string" style="color:rgb(255, 121, 198)">"hduser02"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain">//Tenant label, </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">if</span><span class="token plain"> the tenant parameter is specified </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">for</span><span class="token plain"> the task, the task will be routed to a separate ECM machine</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> labels.put</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">LabelKeyConstant.EXECUTE_ONCE_KEY,</span><span class="token string" style="color:rgb(255, 121, 198)">""</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain">//Execute the label once, this parameter is not recommended to be set. After setting, the engine will not reuse the task and the engine will end after running. Only a certain task parameter can be specialized. </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">set</span><span class="token plain"> up</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="4-linkis-console-parameters"></a>4. Linkis console parameters<a class="hash-link" href="#4-linkis-console-parameters" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Linkis management console parameters are convenient for users to specify resource limit parameters and default task parameters. The web interface provided is as follows: |
| Global configuration parameters: |
| <img src="/assets/images/linkis_global_conf-039e3e727f7e331beaa2298d9f69b488.png"> |
| It mainly includes the global queue parameter <!-- -->[wds.linkis.rm.yarnqueue]<!-- -->, the Yarn queue used by the task by default, which can be specified in the client StartUPMap. |
| Resource limit parameters, these parameters do not support task settings, but can be adjusted by the management console.</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI shell"><pre tabindex="0" class="prism-code language-shell codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">Queue CPU usage upper limit </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token plain">wds.linkis.rm.yarnqueue.cores.max</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain">, currently only supports limit the usage of total queue resources </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">for</span><span class="token plain"> Spark </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">type</span><span class="token plain"> tasks</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Queue memory usage limit </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token plain">wds.linkis.rm.yarnqueue.memory.max</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">The upper limit of the global memory usage of each engine </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token plain">wds.linkis.rm.client.memory.max</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"> This parameter does not refer to the total memory that can only be used, but specifies the total memory usage of a specific engine of a Creator, such as limiting the IDE-SPARK task to only Can use 10G memory</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">The maximum number of global engine cores </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token plain">wds.linkis.rm.client.core.max</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"> This parameter does not refer to the total number of CPUs that can only be used, but specifies the total memory usage of a specific engine of a Creator, such as limiting IDE-SPARK tasks Can only use 10Cores</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">The maximum concurrent number of each engine </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">in</span><span class="token plain"> the world </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token plain">wds.linkis.rm.instance</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain">, this parameter has two meanings, one is to limit how many a Creator-specific engine can start </span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">in</span><span class="token plain"> total, and to limit the tasks that a Creator-specific engine task can run at the same </span><span class="token function" style="color:rgb(80, 250, 123)">time</span><span class="token plain"> number</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Engine configuration parameters: |
| <img src="/assets/images/linkis_creator_ec_conf-a6fd39de358f4bd72c3335f319d9bdd9.png"> |
| It mainly specifies the startup parameters and runtime parameters of the engine. These parameters can be set on the client side. It is recommended to use the client side for personalized submission settings. Only the default values are set on the page.</p></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/incubator-linkis-website/edit/dev/versioned_docs/version-1.1.2/development/linkis_config.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_mS5F" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_mt2f"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/docs/latest/development/linkis_compile_and_package"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">« <!-- -->Compile And Package</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/docs/latest/development/linkis_debug"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Linkis Debug<!-- --> »</div></a></div></nav></div></div><div class="col col--3"><div class="tableOfContents_vrFS thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#1-parameter-classification" class="table-of-contents__link">1. Parameter classification</a></li><li><a href="#2-linkis-server-parameters" class="table-of-contents__link">2. Linkis server parameters</a></li><li><a href="#3-linkis-client-parameters" class="table-of-contents__link">3. Linkis client parameters</a></li><li><a href="#4-linkis-console-parameters" class="table-of-contents__link">4. Linkis console parameters</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">Linkis</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/latest/introduction">Document</a></li><li class="footer__item"><a class="footer__link-item" href="/faq/main">FAQ</a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/releases" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Releases<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items"><li class="footer__item"><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/issues" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Issue Tracker<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/pulls" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Pull Requests<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Subscribe Mailing List</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/community/how-to-subscribe">How to Subscribe</a></li><li class="footer__item"><a href="mailto:dev-subscribe@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Subscribe Mail<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://lists.apache.org/list.html?dev@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Mail Archive<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright"><div><img style="height:50px" alt="Apache Software Foundation" src="/img/incubator-logo.svg"><p style="color: #999999; padding: 0 20px 30px;font-weight:400;text-align:left">Apache Linkis is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p><p></p> |
| <p style="padding: 0 20px 30px;color: #999999;font-weight: 400;"> Copyright © 2022 The Apache Software Foundation. Licensed under the Apache License, Version 2.0. Apache Linkis, Apache Incubator, Apache, the Apache feather logo, the Apache Linkis logo and the Apache Incubator project logo are trademarks of The Apache Software Foundation.</p> |
| <div></div></div></div></div></div></footer></div> |
| <script src="/assets/js/runtime~main.a4c60f50.js"></script> |
| <script src="/assets/js/main.42471b13.js"></script> |
| </body> |
| </html> |
