commit | 5cd1727089eff26f553f990ab9dfe49d2be14fe2 | [log] [tgz] |
---|---|---|
author | Ryan Blue <blue@apache.org> | Sun Feb 05 18:18:36 2017 -0800 |
committer | Ryan Blue <blue@apache.org> | Thu Jun 15 12:16:12 2017 -0700 |
tree | 3de7c9d6248deef8debc5c13b98b910d05e534d6 | |
parent | 5cfbc83322c290e84f0cc0848c4eba41508ac54d [diff] |
TOREE-380: Allow interpreters to produce output by MIME type. This updates the interpreter API to return a Map of MIME type to content instead of a String that is rendered as text. This allows interpreters to show HTML as cell output. The output of magic functions is now a similar MIME type to content structure. This is cleaner and no longer requires hacky post-processing before relaying a cell's output. Scala now uses Jupyter's jvm-repr API for JVM objects. This allows libraries to register functions that will produce display representations of results by MIME type, that is hopefully compatible with any JVM kernel. This includes the following squashed commits: 5434046 TOREE-380: Allow interpreters to produce output by MIME type. a78e9cb TOREE-380: Add support for Jupyter's jvm-repr API. fc98890 TOREE-380: Fix tests. 9c9c9dc TOREE-380: Add JVMReprSpec. b4c1510 TOREE-380: Fix Python integration tests. 7a50f20 TOREE-380: Fix AddJar integration test. 1eaf4a0 TOREE-380: Add copyright header to new files. e7d4004 TOREE-380: Add comment to ScalaDisplayers.ensureLoaded.
The main goal of the Toree is to provide the foundation for interactive applications to connect to and use Apache Spark.
Toree provides an interface that allows clients to interact with a Spark Cluster. Clients can send libraries and snippets of code that are interpreted and ran against a preconfigured Spark context. These snippets can do a variety of things:
The main supported language is Scala
, but it is also capable of processing both Python
and R
. It implements the latest Jupyter message protocol (5.0), so it can easily plug into the latest releases of Jupyter/IPython (3.2.x+ and 4.x+) for quick, interactive data exploration.
A version of Toree is deployed as part of the Try Jupyter! site. Select Scala 2.10.4 (Spark 1.4.1)
under the New
dropdown. Note that this version only supports Scala
.
This project uses make
as the entry point for build, test, and packaging. It supports 2 modes, local and vagrant. The default is local and all command (i.e. sbt) will be ran locally on your machine. This means that you need to install sbt
, jupyter/ipython
, and other development requirements locally on your machine. The 2nd mode uses Vagrant to simplify the development experience. In vagrant mode, all commands are sent to the vagrant box that has all necessary dependencies pre-installed. To run in vagrant mode, run export USE_VAGRANT=true
.
To build and interact with Toree using Jupyter, run
make dev
This will start a Jupyter notebook server. Depending on your mode, it will be accessible at http://localhost:8888
or http://192.168.44.44:8888
. From here you can create notebooks that use Toree configured for Spark local mode.
Tests can be run by doing make test
.
NOTE: Do not use
sbt
directly.
To build and package up Toree, run
make release
This results in 2 packages.
./dist/toree-<VERSION>-binary-release.tar.gz
is a simple package that contains JAR and executable./dist/toree-<VERSION>.tar.gz
is a pip
installable package that adds Toree as a Jupyter kernel.NOTE: make release
uses docker
. Please refer to docker
installation instructions for your system. USE_VAGRANT
is not supported by this make
target.
To play with the example notebooks, run
make jupyter
A notebook server will be launched in a Docker
container with Toree and some other dependencies installed. Refer to your Docker
setup for the ip address. The notebook will be at http://<ip>:8888/
.
Dev snapshots of Toree are located at https://dist.apache.org/repos/dist/dev/incubator/toree. To install using one of those packages, you can use the following:
pip install <PIP_RELEASE_URL> jupyter toree install
where PIP_RELEASE_URL
is one of the pip
packages. For example:
pip install https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshots/dev1/toree-pip/toree-0.2.0.dev1.tar.gz jupyter toree install
Refer to and open issue here
You can reach us through gitter or our mailing list
We are working on publishing binary releases of Toree soon. As part of our move into Apache Incubator, Toree will start a new version sequence starting at 0.1
.
Our goal is to keep master
up to date with the latest version of Spark. When new versions of Spark require specific code changes to Toree, we will branch out older Spark version support.
As it stands, we maintain several branches for legacy versions of Spark. The table below shows what is available now.
Branch | Apache Spark Version |
---|---|
master | 2.0 |
0.1.x | 1.6+ |
Please note that for the most part, new features will mainly be added to the master
branch.
We are working on porting our documentation into Apache. For the time being, you can refer to this Wiki and our Getting Started guide. You may also visit our website.