| <!doctype html> |
| <html class="docs-version-0.11.0" lang="en" dir="ltr"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width,initial-scale=1"> |
| <meta name="generator" content="Docusaurus v2.0.0-beta.14"> |
| <link rel="alternate" type="application/rss+xml" href="/blog/rss.xml" title="Apache Linkis Blog RSS Feed"> |
| <link rel="alternate" type="application/atom+xml" href="/blog/atom.xml" title="Apache Linkis Blog Atom Feed"> |
| <link rel="search" type="application/opensearchdescription+xml" title="Apache Linkis" href="/opensearch.xml"><title data-react-helmet="true">Spark Engine | Apache Linkis</title><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" property="og:url" content="https://linkis.incubator.apache.org/docs/0.11.0/engine_usage/spark"><meta data-react-helmet="true" name="docsearch:language" content="en"><meta data-react-helmet="true" name="docsearch:version" content="0.11.0"><meta data-react-helmet="true" name="docsearch:docusaurus_tag" content="docs-default-0.11.0"><meta data-react-helmet="true" property="og:title" content="Spark Engine | Apache Linkis"><meta data-react-helmet="true" name="description" content="1 Use of Spark Engine"><meta data-react-helmet="true" property="og:description" content="1 Use of Spark Engine"><link data-react-helmet="true" rel="shortcut icon" href="/img/favicon.ico"><link data-react-helmet="true" rel="canonical" href="https://linkis.incubator.apache.org/docs/0.11.0/engine_usage/spark"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/docs/0.11.0/engine_usage/spark" hreflang="en"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/zh-CN/docs/0.11.0/engine_usage/spark" hreflang="zh-CN"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/docs/0.11.0/engine_usage/spark" hreflang="x-default"><link data-react-helmet="true" rel="preconnect" href="https://AE29KQB3IA-dsn.algolia.net" crossorigin="anonymous"><link rel="stylesheet" href="/assets/css/styles.92d87943.css"> |
| <link rel="preload" href="/assets/js/runtime~main.a4c60f50.js" as="script"> |
| <link rel="preload" href="/assets/js/main.42471b13.js" as="script"> |
| </head> |
| <body> |
| <script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus"> |
| <div><a href="#" class="skipToContent_OuoZ">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--light_4Vu1 navbar__logo"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--dark_uzRr navbar__logo"><b class="navbar__title">Apache Linkis(Incubating)</b></a><a class="navbar__item navbar__link" href="/">Home</a><a class="navbar__item navbar__link" href="/faq/main">FAQ</a><a class="navbar__item navbar__link" href="/download/main">Download</a><a class="navbar__item navbar__link" href="/community/how-to-subscribe">Community</a><a class="navbar__item navbar__link" href="/blog">Blog</a><a class="navbar__item navbar__link" href="/team">Team</a><a class="navbar__item navbar__link" href="/user">Users</a><div class="navbar__item dropdown dropdown--hoverable"><a class="navbar__link">ASF</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Foundation</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship</a></li><li><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks</a></li></ul></div></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link">Doc</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/1.1.3/introduction">Next(1.1.3)</a></li><li><a class="dropdown__link" href="/docs/latest/introduction">1.1.2</a></li><li><a class="dropdown__link" href="/docs/1.1.1/introduction">1.1.1</a></li><li><a class="dropdown__link" href="/docs/1.1.0/introduction">1.1.0</a></li><li><a class="dropdown__link" href="/docs/1.0.3/introduction">1.0.3</a></li><li><a class="dropdown__link" href="/docs/1.0.2/introduction">1.0.2</a></li><li><a class="dropdown__link" href="/docs/0.11.0/introduction">0.11.0</a></li><li><a class="dropdown__link" href="/versions">All Version</a></li></ul></div><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-github-link" aria-label="GitHub"></a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" class="navbar__link"><span><svg t="1631348384596" class="iconLanguage_EbrZ" viewBox="0 0 1024 1024" version="1.1" p-id="557" width="20" height="20"><path d="M547.797333 638.208l-104.405333-103.168 1.237333-1.28a720.170667 720.170667 0 0 0 152.490667-268.373333h120.448V183.082667h-287.744V100.906667H347.605333v82.218666H59.818667V265.386667h459.178666a648.234667 648.234667 0 0 1-130.304 219.946666 643.242667 643.242667 0 0 1-94.976-137.728H211.541333a722.048 722.048 0 0 0 122.453334 187.434667l-209.194667 206.378667 58.368 58.368 205.525333-205.525334 127.872 127.829334 31.232-83.84m231.424-208.426667h-82.218666l-184.96 493.312h82.218666l46.037334-123.306667h195.242666l46.464 123.306667h82.218667l-185.002667-493.312m-107.690666 287.744l66.56-178.005333 66.602666 178.005333z" fill="currentColor" p-id="558"></path></svg><span>English</span></span></a><ul class="dropdown__menu"><li><a href="/docs/0.11.0/engine_usage/spark" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" style="text-transform:capitalize">English</a></li><li><a href="/zh-CN/docs/0.11.0/engine_usage/spark" target="_self" rel="noopener noreferrer" class="dropdown__link" style="text-transform:capitalize">简体中文</a></li></ul></div><div class="searchBox_Bc3W"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div class="main-wrapper docs-wrapper docs-doc-page"><div class="docPage_lDyR"><button class="clean-btn backToTopButton_i9tI" type="button"><svg viewBox="0 0 24 24" width="28"><path d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z" fill="currentColor"></path></svg></button><aside class="docSidebarContainer_0YBq"><div class="sidebar_a3j0"><nav class="menu thin-scrollbar menu_cyFh menuWithAnnouncementBar_+O1J"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" href="/docs/0.11.0/introduction">Introduction</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Deployment</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">User Guide</a></li><li class="theme-doc-sidebar-item-category menu__list-item"><a class="menu__link menu__link--sublist menu__link--active" href="#">Engine Usage</a><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/0.11.0/engine_usage/hive">Hive Engine</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/0.11.0/engine_usage/python">Python Engine</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/0.11.0/engine_usage/spark">Spark Engine</a></li></ul></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">API Docs</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Architecture</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Development Doc</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Upgrade Guide</a></li></ul></nav></div></aside><main class="docMainContainer_r8cw"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_zHA2"><div class="theme-doc-version-banner alert alert--warning margin-bottom--md" role="alert"><div>This is documentation for <!-- -->Apache Linkis<!-- --> <b>0.11.0</b>, which is no longer actively maintained.</div><div class="margin-top--md">For up-to-date documentation, see the <b><a href="/docs/latest/engine_usage/spark">latest version</a></b> (<!-- -->1.1.2<!-- -->).</div></div><div class="docItemContainer_oiyr"><article><span class="theme-doc-version-badge badge badge--secondary">Version: <!-- -->0.11.0</span><div class="tocCollapsible_aw-L theme-doc-toc-mobile tocMobile_Tx6Y"><button type="button" class="clean-btn tocCollapsibleButton_zr6a">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Spark Engine</h1></header><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="1-use-of-spark-engine"></a>1 Use of Spark Engine<a class="hash-link" href="#1-use-of-spark-engine" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Linkis's Spark execution engine provides users with the ability to submit spark jobs to the Yarn cluster and feedback logs, progress, status, and result sets.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Spark execution engine supports users to submit three types of jobs: sparksql, pyspark, and scala. By default, the job is submitted by yarn-client, and the engine is used to maintain users One or more spark sessions.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->After the user downloads the Linkis release package and unzips and installs it, a number of specific services need to be started correctly before it can be used to execute spark operations.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Here are the specific steps.</p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="11-environment-variable-configuration"></a>1.1 Environment variable configuration<a class="hash-link" href="#11-environment-variable-configuration" title="Direct link to heading">#</a></h3><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Environmental variables that the Spark engine depends on: HADOOP_HOME, HADOOP_CONF_DIR, HIVE_CONF_DIR, SPARK_HOME, and SPARK_CONF_DIR.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Before starting the SparkEngineManager microservice, please make sure that the above environment variables have been set.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->If you have not set it, please first download linkis.properties in the /home/${USER}/.bash_rc or linkis-ujes-spark-enginemanager/conf directory Set in the configuration file. As shown below</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI properties"><pre tabindex="0" class="prism-code language-properties codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain"> HADOOP_HOME=${real hadoop home}</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> HADOOP_CONF_DIR=${Real hadoop configuration directory}</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> HIVE_CONF_DIR=${Real hive configuration directory}</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> SPARK_CONF_DIR=${Real hive configuration directory}</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> SPARK_HOME=${Real spark installation directory}</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="12-start-dependent-services"></a>1.2 Start dependent services<a class="hash-link" href="#12-start-dependent-services" title="Direct link to heading">#</a></h3><p>The startup of the Spark engine requires the following Linkis microservices:</p><p>-1), Eureka: Used for service registration and discovery. |
| -2), Linkis-gateway: used for user request forwarding. |
| -3) Linkis-publicService: Provides basic functions such as persistence and udf. |
| -4) Linkis-ResourceManager: Provides Linkis resource management functions.</p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="13-custom-parameter-configuration"></a>1.3 Custom parameter configuration<a class="hash-link" href="#13-custom-parameter-configuration" title="Direct link to heading">#</a></h3><p>Before starting spark related microservices, users can set related configuration parameters about the spark engine.</p><p>Considering that users want to be able to set parameters more freely, Linkis provides many configuration parameters.</p><p>The following table has some commonly used parameters. The Spark engine supports configuring more parameters for better performance. If you have tuning needs, welcome to read the tuning manual.</p><p>Users can configure these parameters in linkis.properties.</p><table><thead><tr><th>Parameter name</th><th>Reference value</th><th>Description</th></tr></thead><tbody><tr><td>wds.linkis.enginemanager.memory.max</td><td>40G</td><td>Used to specify the total memory of the client of all engines started by sparkEM</td></tr><tr><td>wds.linkis.enginemanager.cores.max</td><td>20</td><td>Used to specify the total number of CPU cores of the client for all engines started by sparkEM</td></tr><tr><td>wds.linkis.enginemanager.engine.instances.max</td><td>10</td><td>Used to specify the number of engines that sparkEM can start</td></tr></tbody></table><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="14-front-end-deployment"></a>1.4 Front-end deployment<a class="hash-link" href="#14-front-end-deployment" title="Direct link to heading">#</a></h3><p>After the above microservice deployment is successfully started, if users need to submit their own sparkSQL, pyspark or Scala code through a web browser, they can deploy another open source front-end product of Weizhong <a href="https://github.com" target="_blank" rel="noopener noreferrer">Scriptis</a> /WeBankFinTech/Scriptis/blob/master/docs/zh_CN/ch1/%E5%89%8D%E5%8F%B0%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1% A3.md), this product allows users to edit and submit codes on web pages, and receive information from the background in real time.</p><p>In addition, Scriptis also has a management console function for configuring the startup parameters of the spark engine.</p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="15-spark-startup-parameter-configuration"></a>1.5 spark startup parameter configuration<a class="hash-link" href="#15-spark-startup-parameter-configuration" title="Direct link to heading">#</a></h3><p>The Scriptis page provides us with a configuration page where we can set startup parameters. The memory size of the Driver, the number of executors, and the number of memory and CPU cores can be set. These parameters will be read and used to start a spark engine.</p><p><img alt="Spark startup parameter configuration diagram" src="/assets/images/spark_conf-26116fec8d022b3e44c5dab02b093aa0.png"><br> |
| Figure 1 Management console configuration interface</p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="16-running-examples"></a>1.6 Running examples<a class="hash-link" href="#16-running-examples" title="Direct link to heading">#</a></h3><p>In the web browser, open the scriptis address, and the user can create a new sql, scala or pyspark script in the workspace on the left column. After writing the script code in the script editing area, click Run to submit your code to Linkis Execute in the background. After submission, the background will push the log, progress, status and other information to the user in real time through websocket. And after finishing, show the result to the user.</p><p><img alt="Spark running effect chart 1" src="/assets/images/spark_run1-70df1decddcfb87664f83455b1104d86.png"><br></p><center>Figure 2 Spark running effect Figure 1</center><p><img alt="Spark running effect chart 2" src="/assets/images/spark_run2-3f4ce5658f49cf53efecf02d866ddcea.png"><br> |
| Figure 3 Spark running effect Figure 2</p><p><img alt="Spark running effect chart 3" src="/assets/images/spark_run3-d7c957f26bc7605b191d51f56d725d6d.png"><br> |
| Figure 4 Spark running effect Figure 3</p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="2-how-the-spark-engine-is-implemented"></a>2 How the Spark engine is implemented<a class="hash-link" href="#2-how-the-spark-engine-is-implemented" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->The implementation of the Linkis-Python execution engine is based on <a href="/docs/0.11.0/development/new_engine_conn">How to implement a new engine</a> to implement the Entrance, EngineManager and Engine three The necessary interface of the module.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->In the EngineManager module, we chose to use the spark-submit command to start the java process, so Linkis took the rewrite of ProcessEngineBuilder's build method to configure the spark The startup parameters of is integrated with the spark-submit command to form a command to start the spark engine, and then the command is executed.</p><p>In the Engine module, Linkis uses the yarn-client mode by default to start spark sessions. Spark's Driver process will exist in the form of a Linkis engine and owned by the user who starts it.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->After obtaining the SparkSession instance, the SparkContext instance of the Driver process can be obtained through the API provided by Spark. Through the SparkContext instance, we can calculate and obtain the progress. You can cancel the user's task.</p><p>Spark execution engine now supports three types of spark jobs, sparksql, scala and pyspark. The code in the Engine module implements three SparkExecutors to execute separately, SQL is submitted using SparkSession, scala is submitted using Console, and pyspark is submitted using py4j.</p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="3-spark-version-adaptation"></a>3 Spark version adaptation<a class="hash-link" href="#3-spark-version-adaptation" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->The release version of Linkis0.5.0 and Linkis0.6.0 only supports spark2.1.0.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Starting from Linkis0.7.0, spark began to adapt to spark2.4+.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Of course, if the spark version used in your cluster is not compatible with our supported version, you may need to change the spark.version variable in the top-level pom.xml , And then recompile and package.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->If you encounter problems starting up and running, you can join a group to consult us.</p><p><img alt="WeChat group" src="/assets/images/group-8e93c2460f179fabea51336c34b1ddd2.png"><br></p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="4-future-goals"></a>4 Future goals<a class="hash-link" href="#4-future-goals" title="Direct link to heading">#</a></h2><p>-1. The deployment method is simpler, try to use the containerized method. |
| -2. Support the submission of spark jar package |
| -3. Better support for spark's yarn-cluster submission.</p></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/incubator-linkis-website/edit/dev/versioned_docs/version-0.11.0/engine_usage/spark.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_mS5F" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_mt2f"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/docs/0.11.0/engine_usage/python"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">« <!-- -->Python Engine</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/docs/0.11.0/api/login_api"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Login Api<!-- --> »</div></a></div></nav></div></div><div class="col col--3"><div class="tableOfContents_vrFS thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#1-use-of-spark-engine" class="table-of-contents__link">1 Use of Spark Engine</a><ul><li><a href="#11-environment-variable-configuration" class="table-of-contents__link">1.1 Environment variable configuration</a></li><li><a href="#12-start-dependent-services" class="table-of-contents__link">1.2 Start dependent services</a></li><li><a href="#13-custom-parameter-configuration" class="table-of-contents__link">1.3 Custom parameter configuration</a></li><li><a href="#14-front-end-deployment" class="table-of-contents__link">1.4 Front-end deployment</a></li><li><a href="#15-spark-startup-parameter-configuration" class="table-of-contents__link">1.5 spark startup parameter configuration</a></li><li><a href="#16-running-examples" class="table-of-contents__link">1.6 Running examples</a></li></ul></li><li><a href="#2-how-the-spark-engine-is-implemented" class="table-of-contents__link">2 How the Spark engine is implemented</a></li><li><a href="#3-spark-version-adaptation" class="table-of-contents__link">3 Spark version adaptation</a></li><li><a href="#4-future-goals" class="table-of-contents__link">4 Future goals</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">Linkis</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/latest/introduction">Document</a></li><li class="footer__item"><a class="footer__link-item" href="/faq/main">FAQ</a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/releases" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Releases<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items"><li class="footer__item"><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/issues" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Issue Tracker<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/pulls" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Pull Requests<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Subscribe Mailing List</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/community/how-to-subscribe">How to Subscribe</a></li><li class="footer__item"><a href="mailto:dev-subscribe@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Subscribe Mail<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://lists.apache.org/list.html?dev@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Mail Archive<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright"><div><img style="height:50px" alt="Apache Software Foundation" src="/img/incubator-logo.svg"><p style="color: #999999; padding: 0 20px 30px;font-weight:400;text-align:left">Apache Linkis is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p><p></p> |
| <p style="padding: 0 20px 30px;color: #999999;font-weight: 400;"> Copyright © 2022 The Apache Software Foundation. Licensed under the Apache License, Version 2.0. Apache Linkis, Apache Incubator, Apache, the Apache feather logo, the Apache Linkis logo and the Apache Incubator project logo are trademarks of The Apache Software Foundation.</p> |
| <div></div></div></div></div></div></footer></div> |
| <script src="/assets/js/runtime~main.a4c60f50.js"></script> |
| <script src="/assets/js/main.42471b13.js"></script> |
| </body> |
| </html> |
