| |
| |
| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <title>Livy Docs - Programmatic API</title> |
| |
| <meta name="author" content=""> |
| |
| <!-- Enable responsive viewport --> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <!-- Le HTML5 shim, for IE6-8 support of HTML elements --> |
| <!--[if lt IE 9]> |
| <script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script> |
| <![endif]--> |
| |
| <!-- Le styles --> |
| <link href="/assets/themes/apache/bootstrap/css/bootstrap.css" rel="stylesheet"> |
| <link href="/assets/themes/apache/css/style.css?body=1" rel="stylesheet" type="text/css"> |
| <link href="/assets/themes/apache/css/syntax.css" rel="stylesheet" type="text/css" media="screen" /> |
| <!-- Le fav and touch icons --> |
| <!-- Update these with your own images |
| <link rel="shortcut icon" href="images/favicon.ico"> |
| <link rel="apple-touch-icon" href="images/apple-touch-icon.png"> |
| <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png"> |
| <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png"> |
| --> |
| |
| </head> |
| |
| <body> |
| |
| <div class="navbar navbar-inverse navbar-fixed-top" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="navbar-brand" href="/"> |
| <img src="/assets/themes/apache/img/logo.png" width="50"> |
| Apache Livy |
| <span class="version">0.7.0-incubating</span> |
| </a> |
| </div> |
| |
| <nav class="navbar-collapse collapse" role="navigation"> |
| <ul class="nav navbar-nav navbar-right"> |
| |
| <li id="documentation"> |
| |
| <a href="#" data-toggle="dropdown" class="dropdown-toggle">Documentation<b class="caret"></b></a> |
| <ul class="dropdown-menu dropdown-left"> |
| |
| |
| <li><a href="rest-api.html" target="_self">REST API</a></li> |
| |
| |
| <li><a href="programmatic-api.html" target="_self">Programmatic API</a></li> |
| |
| |
| <li><a href="api/java/index.html" target="_self">Java API Docs</a></li> |
| |
| |
| <li><a href="api/scala/index.html#org.apache.livy.scalaapi.package" target="_self">Scala API Docs</a></li> |
| |
| </ul> |
| |
| </li> |
| |
| <li id="apache"> |
| |
| <a href="#" data-toggle="dropdown" class="dropdown-toggle">Apache<b class="caret"></b></a> |
| <ul class="dropdown-menu dropdown-left"> |
| |
| |
| <li><a href="http://www.apache.org/foundation/how-it-works.html" target="_blank">Apache Software Foundation</a></li> |
| |
| |
| <li><a href="http://www.apache.org/licenses/" target="_blank">Apache License</a></li> |
| |
| |
| <li><a href="http://www.apache.org/foundation/sponsorship" target="_blank">Sponsorship</a></li> |
| |
| |
| <li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li> |
| |
| </ul> |
| |
| </li> |
| |
| </ul> |
| </nav><!--/.navbar-collapse --> |
| |
| </div> |
| </div> |
| |
| |
| <div class="container"> |
| |
| |
| <!--<div class="hero-unit Livy Docs - Programmatic API"> |
| <h1> <small>Programmatic API</small></h1> |
| </div> |
| --> |
| |
| <div class="row"> |
| <div class="col-md-12"> |
| <!-- |
| |
| --> |
| |
| <h2 id="using-the-programmatic-api">Using the Programmatic API</h2> |
| |
| <p>Livy provides a programmatic Java/Scala and Python API that allows applications to run code inside |
| Spark without having to maintain a local Spark context. Here shows how to use the Java API.</p> |
| |
| <p>Add the Livy client dependency to your application’s POM:</p> |
| |
| <div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><dependency></span> |
| <span class="nt"><groupId></span>org.apache.livy<span class="nt"></groupId></span> |
| <span class="nt"><artifactId></span>livy-client-http<span class="nt"></artifactId></span> |
| <span class="nt"><version></span>0.7.0-incubating<span class="nt"></version></span> |
| <span class="nt"></dependency></span> |
| </code></pre> |
| </div> |
| |
| <p>To be able to compile code that uses Spark APIs, also add the correspondent Spark dependencies.</p> |
| |
| <p>To run Spark jobs within your applications, extend <code class="highlighter-rouge">org.apache.livy.Job</code> and implement |
| the functionality you need. Here’s an example job that calculates an approximate value for Pi:</p> |
| |
| <div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.util.*</span><span class="o">;</span> |
| |
| <span class="kn">import</span> <span class="nn">org.apache.spark.api.java.*</span><span class="o">;</span> |
| <span class="kn">import</span> <span class="nn">org.apache.spark.api.java.function.*</span><span class="o">;</span> |
| |
| <span class="kn">import</span> <span class="nn">org.apache.livy.*</span><span class="o">;</span> |
| |
| <span class="kd">public</span> <span class="kd">class</span> <span class="nc">PiJob</span> <span class="kd">implements</span> <span class="n">Job</span><span class="o"><</span><span class="n">Double</span><span class="o">>,</span> <span class="n">Function</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>,</span> |
| <span class="n">Function2</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="o">{</span> |
| |
| <span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">samples</span><span class="o">;</span> |
| |
| <span class="kd">public</span> <span class="nf">PiJob</span><span class="o">(</span><span class="kt">int</span> <span class="n">samples</span><span class="o">)</span> <span class="o">{</span> |
| <span class="k">this</span><span class="o">.</span><span class="na">samples</span> <span class="o">=</span> <span class="n">samples</span><span class="o">;</span> |
| <span class="o">}</span> |
| |
| <span class="nd">@Override</span> |
| <span class="kd">public</span> <span class="n">Double</span> <span class="nf">call</span><span class="o">(</span><span class="n">JobContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span> |
| <span class="n">List</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">sampleList</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Integer</span><span class="o">>();</span> |
| <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">samples</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span> |
| <span class="n">sampleList</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="o">);</span> |
| <span class="o">}</span> |
| |
| <span class="k">return</span> <span class="mf">4.0d</span> <span class="o">*</span> <span class="n">ctx</span><span class="o">.</span><span class="na">sc</span><span class="o">().</span><span class="na">parallelize</span><span class="o">(</span><span class="n">sampleList</span><span class="o">).</span><span class="na">map</span><span class="o">(</span><span class="k">this</span><span class="o">).</span><span class="na">reduce</span><span class="o">(</span><span class="k">this</span><span class="o">)</span> <span class="o">/</span> <span class="n">samples</span><span class="o">;</span> |
| <span class="o">}</span> |
| |
| <span class="nd">@Override</span> |
| <span class="kd">public</span> <span class="n">Integer</span> <span class="nf">call</span><span class="o">(</span><span class="n">Integer</span> <span class="n">v1</span><span class="o">)</span> <span class="o">{</span> |
| <span class="kt">double</span> <span class="n">x</span> <span class="o">=</span> <span class="n">Math</span><span class="o">.</span><span class="na">random</span><span class="o">();</span> |
| <span class="kt">double</span> <span class="n">y</span> <span class="o">=</span> <span class="n">Math</span><span class="o">.</span><span class="na">random</span><span class="o">();</span> |
| <span class="k">return</span> <span class="o">(</span><span class="n">x</span><span class="o">*</span><span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="o">*</span><span class="n">y</span> <span class="o"><</span> <span class="mi">1</span><span class="o">)</span> <span class="o">?</span> <span class="mi">1</span> <span class="o">:</span> <span class="mi">0</span><span class="o">;</span> |
| <span class="o">}</span> |
| |
| <span class="nd">@Override</span> |
| <span class="kd">public</span> <span class="n">Integer</span> <span class="nf">call</span><span class="o">(</span><span class="n">Integer</span> <span class="n">v1</span><span class="o">,</span> <span class="n">Integer</span> <span class="n">v2</span><span class="o">)</span> <span class="o">{</span> |
| <span class="k">return</span> <span class="n">v1</span> <span class="o">+</span> <span class="n">v2</span><span class="o">;</span> |
| <span class="o">}</span> |
| |
| <span class="o">}</span> |
| </code></pre> |
| </div> |
| |
| <p>To submit this code using Livy, create a LivyClient instance and upload your application code to |
| the Spark context. Here’s an example of code that submits the above job and prints the computed |
| value:</p> |
| |
| <div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">LivyClient</span> <span class="n">client</span> <span class="o">=</span> <span class="k">new</span> <span class="n">LivyClientBuilder</span><span class="o">()</span> |
| <span class="o">.</span><span class="na">setURI</span><span class="o">(</span><span class="k">new</span> <span class="n">URI</span><span class="o">(</span><span class="n">livyUrl</span><span class="o">))</span> |
| <span class="o">.</span><span class="na">build</span><span class="o">();</span> |
| |
| <span class="k">try</span> <span class="o">{</span> |
| <span class="n">System</span><span class="o">.</span><span class="na">err</span><span class="o">.</span><span class="na">printf</span><span class="o">(</span><span class="s">"Uploading %s to the Spark context...\n"</span><span class="o">,</span> <span class="n">piJar</span><span class="o">);</span> |
| <span class="n">client</span><span class="o">.</span><span class="na">uploadJar</span><span class="o">(</span><span class="k">new</span> <span class="n">File</span><span class="o">(</span><span class="n">piJar</span><span class="o">)).</span><span class="na">get</span><span class="o">();</span> |
| |
| <span class="n">System</span><span class="o">.</span><span class="na">err</span><span class="o">.</span><span class="na">printf</span><span class="o">(</span><span class="s">"Running PiJob with %d samples...\n"</span><span class="o">,</span> <span class="n">samples</span><span class="o">);</span> |
| <span class="kt">double</span> <span class="n">pi</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">submit</span><span class="o">(</span><span class="k">new</span> <span class="n">PiJob</span><span class="o">(</span><span class="n">samples</span><span class="o">)).</span><span class="na">get</span><span class="o">();</span> |
| |
| <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Pi is roughly: "</span> <span class="o">+</span> <span class="n">pi</span><span class="o">);</span> |
| <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span> |
| <span class="n">client</span><span class="o">.</span><span class="na">stop</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span> |
| <span class="o">}</span> |
| </code></pre> |
| </div> |
| |
| <p>To learn about all the functionality available to applications, read the javadoc documentation for |
| the classes under the <code class="highlighter-rouge">api</code> module.</p> |
| |
| </div> |
| </div> |
| |
| |
| <hr> |
| <footer> |
| <!-- <p>© 2020 </p>--> |
| <footer class="site-footer"> |
| <div class="wrapper"> |
| <div class="footer-col-wrapper"> |
| |
| Apache Livy is an effort undergoing <a href="https://incubator.apache.org/index.html">Incubation</a> |
| at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly |
| accepted projects until a further review indicates that the infrastructure, communications, and decision |
| making process have stabilized in a manner consistent with other successful ASF projects. While incubation |
| status is not necessarily a reflection of the completeness or stability of the code, it does indicate that |
| the project has yet to be fully endorsed by the ASF. |
| <hr> |
| |
| <div style="text-align:center;"> |
| |
| <div style="margin-top: 20px; margin-bottom: 20px;"> |
| <a href="http://incubator.apache.org"><img src="/assets/themes/apache/img/egg-logo.png" |
| alt="Apache Incubator" |
| height="30%" width="30%"/></a> |
| </div> |
| |
| <div> |
| Copyright © 2017 <a href="http://www.apache.org">The Apache Software Foundation</a>. |
| Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version |
| 2.0</a>. |
| <br> |
| |
| Apache, the Apache Feather logo, and the Apache Incubator project logo are trademarks of The Apache |
| Software Foundation. |
| |
| </div> |
| </div> |
| </div> |
| </div> |
| </footer> |
| |
| </footer> |
| </div> |
| |
| <script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script> |
| |
| <script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script> |
| |
| |
| </body> |
| </html> |
| |
