Google Git
Sign in
apache / incubator-livy-website / 2c1f9b93b37c98f9257c04b432a97dbf82e05998 / . / site / docs / 0.4.0-incubating / rest-api.html
blob: 4ccc0c158661dfdc7240b71c3774ca5be955e96d [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Livy Docs - REST API</title>
<meta name="author" content="">
<!-- Enable responsive viewport -->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- Le HTML5 shim, for IE6-8 support of HTML elements -->
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
<!-- Le styles -->
<link href="/assets/themes/apache/bootstrap/css/bootstrap.css" rel="stylesheet">
<link href="/assets/themes/apache/css/style.css?body=1" rel="stylesheet" type="text/css">
<link href="/assets/themes/apache/css/syntax.css" rel="stylesheet" type="text/css" media="screen" />
<!-- Le fav and touch icons -->
<!-- Update these with your own images
<link rel="shortcut icon" href="images/favicon.ico">
<link rel="apple-touch-icon" href="images/apple-touch-icon.png">
<link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png">
<link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png">
-->
</head>
<body>
<div class="navbar navbar-inverse navbar-fixed-top" role="navigation">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="/">
<img src="/assets/themes/apache/img/logo.png" width="50">
Apache Livy
<span class="version">0.4.0-incubating</span>
</a>
</div>
<nav class="navbar-collapse collapse" role="navigation">
<ul class="nav navbar-nav navbar-right">
<li id="documentation">
<a href="#" data-toggle="dropdown" class="dropdown-toggle">Documentation<b class="caret"></b></a>
<ul class="dropdown-menu dropdown-left">
<li><a href="rest-api.html" target="_self">REST API</a></li>
<li><a href="programmatic-api.html" target="_self">Programmatic API</a></li>
</ul>
</li>
<li id="apache">
<a href="#" data-toggle="dropdown" class="dropdown-toggle">Apache<b class="caret"></b></a>
<ul class="dropdown-menu dropdown-left">
<li><a href="http://www.apache.org/foundation/how-it-works.html" target="_blank">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/licenses/" target="_blank">Apache License</a></li>
<li><a href="http://www.apache.org/foundation/sponsorship" target="_blank">Sponsorship</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
</ul>
</li>
</ul>
</nav><!--/.navbar-collapse -->
</div>
</div>
<div class="container">
<!--<div class="hero-unit Livy Docs - REST API">
<h1> <small>REST API</small></h1>
</div>
-->
<div class="row">
<div class="col-md-12">
<!--
-->
<h2 id="rest-api">REST API</h2>
<h3 id="get-sessions">GET /sessions</h3>
<p>Returns all the active interactive sessions.</p>
<h4 id="request-parameters">Request Parameters</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>from</td>
<td>The start index to fetch sessions</td>
<td>int</td>
</tr>
<tr>
<td>size</td>
<td>Number of sessions to fetch</td>
<td>int</td>
</tr>
</table>
<h4 id="response-body">Response Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>from</td>
<td>The start index to fetch sessions</td>
<td>int</td>
</tr>
<tr>
<td>size</td>
<td>Number of sessions to fetch</td>
<td>int</td>
</tr>
<tr>
<td>sessions</td>
<td><a href="#session">Session</a> list</td>
<td>list</td>
</tr>
</table>
<h3 id="post-sessions">POST /sessions</h3>
<p>Creates a new interactive Scala, Python, or R shell in the cluster.</p>
<h4 id="request-body">Request Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>kind</td>
<td>The session kind (required)</td>
<td><a href="#session-kind">session kind</a></td>
</tr>
<tr>
<td>proxyUser</td>
<td>User to impersonate when starting the session</td>
<td>string</td>
</tr>
<tr>
<td>jars</td>
<td>jars to be used in this session</td>
<td>List of string</td>
</tr>
<tr>
<td>pyFiles</td>
<td>Python files to be used in this session</td>
<td>List of string</td>
</tr>
<tr>
<td>files</td>
<td>files to be used in this session</td>
<td>List of string</td>
</tr>
<tr>
<td>driverMemory</td>
<td>Amount of memory to use for the driver process</td>
<td>string</td>
</tr>
<tr>
<td>driverCores</td>
<td>Number of cores to use for the driver process</td>
<td>int</td>
</tr>
<tr>
<td>executorMemory</td>
<td>Amount of memory to use per executor process</td>
<td>string</td>
</tr>
<tr>
<td>executorCores</td>
<td>Number of cores to use for each executor</td>
<td>int</td>
</tr>
<tr>
<td>numExecutors</td>
<td>Number of executors to launch for this session</td>
<td>int</td>
</tr>
<tr>
<td>archives</td>
<td>Archives to be used in this session</td>
<td>List of string</td>
</tr>
<tr>
<td>queue</td>
<td>The name of the YARN queue to which submitted</td>
<td>string</td>
</tr>
<tr>
<td>name</td>
<td>The name of this session</td>
<td>string</td>
</tr>
<tr>
<td>conf</td>
<td>Spark configuration properties</td>
<td>Map of key=val</td>
</tr>
<tr>
<td>heartbeatTimeoutInSecond</td>
<td>Timeout in second to which session be orphaned</td>
<td>int</td>
</tr>
</table>
<h4 id="response-body-1">Response Body</h4>
<p>The created <a href="#session">Session</a>.</p>
<h3 id="get-sessionssessionid">GET /sessions/{sessionId}</h3>
<p>Returns the session information.</p>
<h4 id="response-body-2">Response Body</h4>
<p>The <a href="#session">Session</a>.</p>
<h3 id="get-sessionssessionidstate">GET /sessions/{sessionId}/state</h3>
<p>Returns the state of session</p>
<h4 id="response">Response</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>id</td>
<td>Session id</td>
<td>int</td>
</tr>
<tr>
<td>state</td>
<td>The current state of session</td>
<td>string</td>
</tr>
</table>
<h3 id="delete-sessionssessionid">DELETE /sessions/{sessionId}</h3>
<p>Kills the <a href="#session">Session</a> job.</p>
<h3 id="get-sessionssessionidlog">GET /sessions/{sessionId}/log</h3>
<p>Gets the log lines from this session.</p>
<h4 id="request-parameters-1">Request Parameters</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>from</td>
<td>Offset</td>
<td>int</td>
</tr>
<tr>
<td>size</td>
<td>Max number of log lines to return</td>
<td>int</td>
</tr>
</table>
<h4 id="response-body-3">Response Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>id</td>
<td>The session id</td>
<td>int</td>
</tr>
<tr>
<td>from</td>
<td>Offset from start of log</td>
<td>int</td>
</tr>
<tr>
<td>size</td>
<td>Max number of log lines</td>
<td>int</td>
</tr>
<tr>
<td>log</td>
<td>The log lines</td>
<td>list of strings</td>
</tr>
</table>
<h3 id="get-sessionssessionidstatements">GET /sessions/{sessionId}/statements</h3>
<p>Returns all the statements in a session.</p>
<h4 id="response-body-4">Response Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>statements</td>
<td><a href="#statement">statement</a> list</td>
<td>list</td>
</tr>
</table>
<h3 id="post-sessionssessionidstatements">POST /sessions/{sessionId}/statements</h3>
<p>Runs a statement in a session.</p>
<h4 id="request-body-1">Request Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>code</td>
<td>The code to execute</td>
<td>string</td>
</tr>
</table>
<h4 id="response-body-5">Response Body</h4>
<p>The <a href="#statement">statement</a> object.</p>
<h3 id="get-sessionssessionidstatementsstatementid">GET /sessions/{sessionId}/statements/{statementId}</h3>
<p>Returns a specified statement in a session.</p>
<h4 id="response-body-6">Response Body</h4>
<p>The <a href="#statement">statement</a> object.</p>
<h3 id="post-sessionssessionidstatementsstatementidcancel">POST /sessions/{sessionId}/statements/{statementId}/cancel</h3>
<p>Cancel the specified statement in this session.</p>
<h4 id="response-body-7">Response Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>msg</td>
<td>is always "cancelled"</td>
<td>string</td>
</tr>
</table>
<h3 id="get-batches">GET /batches</h3>
<p>Returns all the active batch sessions.</p>
<h4 id="request-parameters-2">Request Parameters</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>from</td>
<td>The start index to fetch sessions</td>
<td>int</td>
</tr>
<tr>
<td>size</td>
<td>Number of sessions to fetch</td>
<td>int</td>
</tr>
</table>
<h4 id="response-body-8">Response Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>from</td>
<td>The start index of fetched sessions</td>
<td>int</td>
</tr>
<tr>
<td>total</td>
<td>Number of sessions fetched</td>
<td>int</td>
</tr>
<tr>
<td>sessions</td>
<td><a href="#batch">Batch</a> list</td>
<td>list</td>
</tr>
</table>
<h3 id="post-batches">POST /batches</h3>
<h4 id="request-body-2">Request Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>file</td>
<td>File containing the application to execute</td>
<td>path (required)</td>
</tr>
<tr>
<td>proxyUser</td>
<td>User to impersonate when running the job</td>
<td>string</td>
</tr>
<tr>
<td>className</td>
<td>Application Java/Spark main class</td>
<td>string</td>
</tr>
<tr>
<td>args</td>
<td>Command line arguments for the application</td>
<td>list of strings</td>
</tr>
<tr>
<td>jars</td>
<td>jars to be used in this session</td>
<td>list of strings</td>
</tr>
<tr>
<td>pyFiles</td>
<td>Python files to be used in this session</td>
<td>list of strings</td>
</tr>
<tr>
<td>files</td>
<td>files to be used in this session</td>
<td>list of strings</td>
</tr>
<tr>
<td>driverMemory</td>
<td>Amount of memory to use for the driver process</td>
<td>string</td>
</tr>
<tr>
<td>driverCores</td>
<td>Number of cores to use for the driver process</td>
<td>int</td>
</tr>
<tr>
<td>executorMemory</td>
<td>Amount of memory to use per executor process</td>
<td>string</td>
</tr>
<tr>
<td>executorCores</td>
<td>Number of cores to use for each executor</td>
<td>int</td>
</tr>
<tr>
<td>numExecutors</td>
<td>Number of executors to launch for this session</td>
<td>int</td>
</tr>
<tr>
<td>archives</td>
<td>Archives to be used in this session</td>
<td>List of string</td>
</tr>
<tr>
<td>queue</td>
<td>The name of the YARN queue to which submitted</td>
<td>string</td>
</tr>
<tr>
<td>name</td>
<td>The name of this session</td>
<td>string</td>
</tr>
<tr>
<td>conf</td>
<td>Spark configuration properties</td>
<td>Map of key=val</td>
</tr>
</table>
<h4 id="response-body-9">Response Body</h4>
<p>The created <a href="#batch">Batch</a> object.</p>
<h3 id="get-batchesbatchid">GET /batches/{batchId}</h3>
<p>Returns the batch session information.</p>
<h4 id="response-body-10">Response Body</h4>
<p>The <a href="#batch">Batch</a>.</p>
<h3 id="get-batchesbatchidstate">GET /batches/{batchId}/state</h3>
<p>Returns the state of batch session</p>
<h4 id="response-1">Response</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>id</td>
<td>Batch session id</td>
<td>int</td>
</tr>
<tr>
<td>state</td>
<td>The current state of batch session</td>
<td>string</td>
</tr>
</table>
<h3 id="delete-batchesbatchid">DELETE /batches/{batchId}</h3>
<p>Kills the <a href="#batch">Batch</a> job.</p>
<h3 id="get-batchesbatchidlog">GET /batches/{batchId}/log</h3>
<p>Gets the log lines from this batch.</p>
<h4 id="request-parameters-3">Request Parameters</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>from</td>
<td>Offset</td>
<td>int</td>
</tr>
<tr>
<td>size</td>
<td>Max number of log lines to return</td>
<td>int</td>
</tr>
</table>
<h4 id="response-body-11">Response Body</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>id</td>
<td>The batch id</td>
<td>int</td>
</tr>
<tr>
<td>from</td>
<td>Offset from start of log</td>
<td>int</td>
</tr>
<tr>
<td>size</td>
<td>Number of log lines</td>
<td>int</td>
</tr>
<tr>
<td>log</td>
<td>The log lines</td>
<td>list of strings</td>
</tr>
</table>
<h2 id="rest-objects">REST Objects</h2>
<h3 id="session">Session</h3>
<p>A session represents an interactive shell.</p>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>id</td>
<td>The session id</td>
<td>int</td>
</tr>
<tr>
<td>appId</td>
<td>The application id of this session</td>
<td>string</td>
</tr>
<tr>
<td>owner</td>
<td>Remote user who submitted this session</td>
<td>string</td>
</tr>
<tr>
<td>proxyUser</td>
<td>User to impersonate when running</td>
<td>string</td>
</tr>
<tr>
<td>kind</td>
<td>Session kind (spark, pyspark, or sparkr)</td>
<td><a href="#session-kind">session kind</a></td>
</tr>
<tr>
<td>log</td>
<td>The log lines</td>
<td>list of strings</td>
</tr>
<tr>
<td>state</td>
<td>The session state</td>
<td>string</td>
</tr>
<tr>
<td>appInfo</td>
<td>The detailed application info</td>
<td>Map of key=val</td>
</tr>
</table>
<h4 id="session-state">Session State</h4>
<table class="table">
<tr><th>Value</th><th>Description</th></tr>
<tr>
<td>not_started</td>
<td>Session has not been started</td>
</tr>
<tr>
<td>starting</td>
<td>Session is starting</td>
</tr>
<tr>
<td>idle</td>
<td>Session is waiting for input</td>
</tr>
<tr>
<td>busy</td>
<td>Session is executing a statement</td>
</tr>
<tr>
<td>shutting_down</td>
<td>Session is shutting down</td>
</tr>
<tr>
<td>error</td>
<td>Session errored out</td>
</tr>
<tr>
<td>dead</td>
<td>Session has exited</td>
</tr>
<tr>
<td>success</td>
<td>Session is successfully stopped</td>
</tr>
</table>
<h4 id="session-kind">Session Kind</h4>
<table class="table">
<tr><th>Value</th><th>Description</th></tr>
<tr>
<td>spark</td>
<td>Interactive Scala Spark session</td>
</tr>
<tr>
<td><a href="#pyspark">pyspark</a></td>
<td>Interactive Python 2 Spark session</td>
</tr>
<tr>
<td><a href="#pyspark3">pyspark3</a></td>
<td>Interactive Python 3 Spark session</td>
</tr>
<tr>
<td>sparkr</td>
<td>Interactive R Spark session</td>
</tr>
</table>
<h4 id="pyspark">pyspark</h4>
<p>To change the Python executable the session uses, Livy reads the path from environment variable
<code class="highlighter-rouge">PYSPARK_PYTHON</code> (Same as pyspark).</p>
<p>Like pyspark, if Livy is running in <code class="highlighter-rouge">local</code> mode, just set the environment variable.
If the session is running in <code class="highlighter-rouge">yarn-cluster</code> mode, please set
<code class="highlighter-rouge">spark.yarn.appMasterEnv.PYSPARK_PYTHON</code> in SparkConf so the environment variable is passed to
the driver.</p>
<h4 id="pyspark3">pyspark3</h4>
<p>To change the Python executable the session uses, Livy reads the path from environment variable
<code class="highlighter-rouge">PYSPARK3_PYTHON</code>.</p>
<p>Like pyspark, if Livy is running in <code class="highlighter-rouge">local</code> mode, just set the environment variable.
If the session is running in <code class="highlighter-rouge">yarn-cluster</code> mode, please set
<code class="highlighter-rouge">spark.yarn.appMasterEnv.PYSPARK3_PYTHON</code> in SparkConf so the environment variable is passed to
the driver.</p>
<h3 id="statement">Statement</h3>
<p>A statement represents the result of an execution statement.</p>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>id</td>
<td>The statement id</td>
<td>integer</td>
</tr>
<tr>
<td>code</td>
<td>The execution code</td>
<td>string</td>
</tr>
<tr>
<td>state</td>
<td>The execution state</td>
<td>statement state</td>
</tr>
<tr>
<td>output</td>
<td>The execution output</td>
<td>statement output</td>
</tr>
</table>
<h4 id="statement-state">Statement State</h4>
<table class="table">
<tr><th>Value</th><th>Description</th></tr>
<tr>
<td>waiting</td>
<td>Statement is enqueued but execution hasn't started</td>
</tr>
<tr>
<td>running</td>
<td>Statement is currently running</td>
</tr>
<tr>
<td>available</td>
<td>Statement has a response ready</td>
</tr>
<tr>
<td>error</td>
<td>Statement failed</td>
</tr>
<tr>
<td>cancelling</td>
<td>Statement is being cancelling</td>
</tr>
<tr>
<td>cancelled</td>
<td>Statement is cancelled</td>
</tr>
</table>
<h4 id="statement-output">Statement Output</h4>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>status</td>
<td>Execution status</td>
<td>string</td>
</tr>
<tr>
<td>execution_count</td>
<td>A monotonically increasing number</td>
<td>integer</td>
</tr>
<tr>
<td>data</td>
<td>Statement output</td>
<td>An object mapping a mime type to the result. If the mime type is
``application/json``, the value is a JSON value.</td>
</tr>
</table>
<h3 id="batch">Batch</h3>
<table class="table">
<tr><th>Name</th><th>Description</th><th>Type</th></tr>
<tr>
<td>id</td>
<td>The session id</td>
<td>int</td>
</tr>
<tr>
<td>appId</td>
<td>The application id of this session</td>
<td>string</td>
</tr>
<tr>
<td>appInfo</td>
<td>The detailed application info</td>
<td>Map of key=val</td>
</tr>
<tr>
<td>log</td>
<td>The log lines</td>
<td>list of strings</td>
</tr>
<tr>
<td>state</td>
<td>The batch state</td>
<td>string</td>
</tr>
</table>
</div>
</div>
<hr>
<footer>
<!-- <p>&copy; 2017 </p>-->
<footer class="site-footer">
<div class="wrapper">
<div class="footer-col-wrapper">
Apache Livy is an effort undergoing <a href="https://incubator.apache.org/index.html">Incubation</a>
at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly
accepted projects until a further review indicates that the infrastructure, communications, and decision
making process have stabilized in a manner consistent with other successful ASF projects. While incubation
status is not necessarily a reflection of the completeness or stability of the code, it does indicate that
the project has yet to be fully endorsed by the ASF.
<hr>
<div style="text-align:center;">
<div style="margin-top: 20px; margin-bottom: 20px;">
<a href="http://incubator.apache.org"><img src="/assets/themes/apache/img/egg-logo.png"
alt="Apache Incubator"
height="30%" width="30%"/></a>
</div>
<div>
Copyright &copy; 2017 <a href="http://www.apache.org">The Apache Software Foundation</a>.
Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version
2.0</a>.
<br>
Apache, the Apache Feather logo, and the Apache Incubator project logo are trademarks of The Apache
Software Foundation.
</div>
</div>
</div>
</div>
</footer>
</footer>
</div>
<script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script>
<script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script>
</body>
</html>