| |
| |
| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <title>Livy - Getting Started</title> |
| |
| <meta name="author" content=""> |
| |
| <!-- Enable responsive viewport --> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <!-- Le HTML5 shim, for IE6-8 support of HTML elements --> |
| <!--[if lt IE 9]> |
| <script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script> |
| <![endif]--> |
| |
| <!-- Le styles --> |
| <link href="/assets/themes/apache/bootstrap/css/bootstrap.css" rel="stylesheet"> |
| <link href="/assets/themes/apache/css/style.css?body=1" rel="stylesheet" type="text/css"> |
| <link href="/assets/themes/apache/css/syntax.css" rel="stylesheet" type="text/css" media="screen" /> |
| <!-- Le fav and touch icons --> |
| <!-- Update these with your own images |
| <link rel="shortcut icon" href="images/favicon.ico"> |
| <link rel="apple-touch-icon" href="images/apple-touch-icon.png"> |
| <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png"> |
| <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png"> |
| --> |
| |
| </head> |
| |
| <body> |
| |
| <div class="navbar navbar-inverse navbar-fixed-top" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="navbar-brand" href="/"> |
| <img src="/assets/themes/apache/img/logo.png" width="50"> |
| Apache Livy |
| </a> |
| </div> |
| |
| <nav class="navbar-collapse collapse" role="navigation"> |
| <ul class="nav navbar-nav navbar-right"> |
| |
| <li id="get-started"> |
| |
| |
| <a href="/get-started" target="_self">Get Started</a> |
| |
| </li> |
| |
| <li id="documentation"> |
| |
| <a href="#" data-toggle="dropdown" class="dropdown-toggle">Documentation<b class="caret"></b></a> |
| <ul class="dropdown-menu dropdown-left"> |
| |
| |
| <li><a href="/docs/latest/rest-api.html" target="_self">REST API</a></li> |
| |
| |
| <li><a href="/docs/latest/programmatic-api.html" target="_self">Programmatic API</a></li> |
| |
| |
| <li><a href="/docs/latest/api/java/index.html" target="_self">JavaDocs</a></li> |
| |
| |
| <li><a href="/docs/latest/api/scala/index.html#org.apache.livy.scalaapi.package" target="_self">ScalaDocs</a></li> |
| |
| |
| <li><a href="/examples" target="_self">Examples</a></li> |
| |
| </ul> |
| |
| </li> |
| |
| <li id="community"> |
| |
| <a href="#" data-toggle="dropdown" class="dropdown-toggle">Community<b class="caret"></b></a> |
| <ul class="dropdown-menu dropdown-left"> |
| |
| |
| <li><a href="/community" target="_self">Get Involved</a></li> |
| |
| |
| <li><a href="/community-members" target="_self">Project Committers</a></li> |
| |
| |
| <li><a href="/third-party-projects" target="_self">Third-Party Projects</a></li> |
| |
| |
| <li><a href="https://issues.apache.org/jira/browse/LIVY" target="_blank">Issue Tracker</a></li> |
| |
| |
| <li><a href="https://github.com/apache/incubator-livy" target="_blank">Source Code</a></li> |
| |
| |
| <li><a href="https://github.com/apache/incubator-livy-website" target="_blank">Website Source Code</a></li> |
| |
| </ul> |
| |
| </li> |
| |
| <li id="apache"> |
| |
| <a href="#" data-toggle="dropdown" class="dropdown-toggle">Apache<b class="caret"></b></a> |
| <ul class="dropdown-menu dropdown-left"> |
| |
| |
| <li><a href="http://www.apache.org/foundation/how-it-works.html" target="_blank">Apache Software Foundation</a></li> |
| |
| |
| <li><a href="http://www.apache.org/licenses/" target="_blank">License</a></li> |
| |
| |
| <li><a href="http://www.apache.org/foundation/sponsorship" target="_blank">Sponsorship</a></li> |
| |
| |
| <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li> |
| |
| |
| <li><a href="http://www.apache.org/security/" target="_blank">Security</a></li> |
| |
| </ul> |
| |
| </li> |
| |
| </ul> |
| </nav><!--/.navbar-collapse --> |
| |
| </div> |
| </div> |
| |
| |
| |
| |
| <div class="container"> |
| |
| |
| <!--<div class="hero-unit Livy - Getting Started"> |
| <h1> <small>Getting Started</small></h1> |
| </div> |
| --> |
| |
| <div class="row"> |
| <div class="col-md-12"> |
| <!-- |
| |
| --> |
| |
| <h2 id="getting-started">Getting Started</h2> |
| |
| <h3 id="1-install-livy">1. Install Livy</h3> |
| <p>Download Livy packages from <a href="/download">here</a>.</p> |
| |
| <h3 id="2-run-livy">2. Run Livy</h3> |
| <p>To run the Livy server, you will also need an Apache Spark installation. You can get Spark releases at |
| <a href="https://spark.apache.org/downloads.html">https://spark.apache.org/downloads.html</a>. Livy requires at least Spark 1.6 and supports Scala 2.11 and 2.12 |
| builds of Spark. To run Livy with local sessions, first export these variables:</p> |
| |
| <p><code class="highlighter-rouge">export SPARK_HOME=/usr/lib/spark</code></p> |
| |
| <p><code class="highlighter-rouge">export HADOOP_CONF_DIR=/etc/hadoop/conf</code></p> |
| |
| <p>Then start the server with:</p> |
| |
| <p><code class="highlighter-rouge">./bin/livy-server start</code></p> |
| |
| <p>Livy uses the Spark configuration under <code class="highlighter-rouge">SPARK_HOME</code> by default. You can override the Spark configuration by setting the |
| <code class="highlighter-rouge">SPARK_CONF_DIR</code> environment variable before starting Livy.</p> |
| |
| <p>It is strongly recommended to configure Spark to submit applications in YARN cluster mode. That makes sure that user |
| sessions have their resources properly accounted for in the YARN cluster, and that the host running the Livy server |
| doesn’t become overloaded when multiple user sessions are running.</p> |
| |
| <h3 id="3-configure-livy">3. Configure Livy</h3> |
| <p>Livy uses a few configuration files under the configuration directory, which by default is the conf directory under the |
| Livy installation. An alternative configuration directory can be provided by setting the <code class="highlighter-rouge">LIVY_CONF_DIR</code> environment |
| variable when starting Livy.</p> |
| |
| <p>The configuration files used by Livy are:</p> |
| |
| <ul> |
| <li><strong>livy.conf:</strong> contains the server configuration. The Livy distribution ships with a default configuration file |
| template listing available configuration keys and their default values.</li> |
| <li><strong>spark-blacklist.conf:</strong> lists Spark configuration options that users are not allowed to override. These options will |
| be restricted to either their default values, or the values set in the Spark configuration used by Livy.</li> |
| <li><strong>log4j.properties:</strong> configuration for Livy logging. Defines log levels and where log messages will be written to. |
| The default configuration template will print log messages to stderr.</li> |
| </ul> |
| |
| <h3 id="4-start-using-livy">4. Start using Livy</h3> |
| |
| <p>Once the Livy server is running, you can connect to it on port 8998 (this can be changed with the |
| <code class="highlighter-rouge">livy.server.port</code> config option). Some examples to get started are provided <a href="/examples">here</a>, |
| or you can check out the API documentation:</p> |
| |
| <ul> |
| <li><a href="/docs/latest/rest-api.html">REST API</a></li> |
| <li><a href="/docs/latest/programmatic-api.html">Programmatic API</a></li> |
| </ul> |
| |
| </div> |
| </div> |
| |
| |
| <hr> |
| <footer> |
| <!-- <p>© 2024 </p>--> |
| <footer class="site-footer"> |
| <div class="wrapper"> |
| <div class="footer-col-wrapper"> |
| |
| Apache Livy is an effort undergoing <a href="https://incubator.apache.org/index.html">Incubation</a> |
| at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly |
| accepted projects until a further review indicates that the infrastructure, communications, and decision |
| making process have stabilized in a manner consistent with other successful ASF projects. While incubation |
| status is not necessarily a reflection of the completeness or stability of the code, it does indicate that |
| the project has yet to be fully endorsed by the ASF. |
| <hr> |
| |
| <div style="text-align:center;"> |
| |
| <div style="margin-top: 20px; margin-bottom: 20px;"> |
| <a href="http://incubator.apache.org"><img src="/assets/themes/apache/img/egg-logo.png" |
| alt="Apache Incubator" |
| height="30%" width="30%"/></a> |
| </div> |
| |
| <div> |
| Copyright © 2017 <a href="http://www.apache.org">The Apache Software Foundation</a>. |
| Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version |
| 2.0</a>. |
| <br> |
| |
| Apache, the Apache Feather logo, and the Apache Incubator project logo are trademarks of The Apache |
| Software Foundation. |
| |
| </div> |
| </div> |
| </div> |
| </div> |
| </footer> |
| |
| </footer> |
| </div> |
| |
| |
| <script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script> |
| |
| <script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script> |
| |
| |
| </body> |
| </html> |
| |
