{% include JB/setup %}
Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.
Additional requirements for the Livy interpreter are:
We added some common configurations for spark, and you can set any configuration you want. You can find all Spark configurations in here. And instead of starting property with spark.
it should be replaced with livy.spark.
. Example: spark.driver.memory
to livy.spark.driver.memory
We remove livy.spark.master in zeppelin-0.7. Because we sugguest user to use livy 0.3 in zeppelin-0.7. And livy 0.3 don't allow to specify livy.spark.master, it enfornce yarn-cluster mode.
You can load dynamic library to livy interpreter by set livy.spark.jars.packages
property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.
Example
Basically, you can use
spark
%livy.spark sc.version
pyspark
%livy.pyspark print "1"
sparkR
%livy.sparkr hello <- function( name ) { sprintf( "Hello, %s", name ); } hello("livy")
When Zeppelin server is running with authentication enabled, then this interpreter utilizes Livy’s user impersonation feature i.e. sends extra parameter for creating and running a session (“proxyUser”: “${loggedInUser}”). This is particularly useful when multi users are sharing a Notebook server.
You can leverage Zeppelin Dynamic Form. Form templates is only avalible for livy sql interpreter.
%livy.sql select * from products where ${product_id=1}
And creating dynamic formst programmatically is not feasible in livy interpreter, because ZeppelinContext is not available in livy interpreter.
Starting from livy 0.5 which is supported by Zeppelin 0.8.0, SparkContext is shared between scala, python, r and sql. That means you can query the table via %livy.sql
when this table is registered in %livy.spark
, %livy.pyspark
, $livy.sparkr
.
Livy debugging: If you see any of these in error console
Connect to livyhost:8998 [livyhost/127.0.0.1, livyhost/0:0:0:0:0:0:0:1] failed: Connection refused
Looks like the livy server is not up yet or the config is wrong
Exception: Session not found, Livy server would have restarted, or lost session.
The session would have timed out, you may need to restart the interpreter.
Blacklisted configuration values in session config: spark.master
Edit conf/spark-blacklist.conf
file in livy server and comment out #spark.master
line.
If you choose to work on livy in apps/spark/java
directory in https://github.com/cloudera/hue, copy spark-user-configurable-options.template
to spark-user-configurable-options.conf
file in livy server and comment out #spark.master
.