commit | ecfa5dd2abe8f992e54b12f4c5e411855b72e898 | [log] [tgz] |
---|---|---|
author | Chip Senkbeil <chip.senkbeil@gmail.com> | Wed Nov 25 11:26:18 2015 -0600 |
committer | Chip Senkbeil <chip.senkbeil@gmail.com> | Wed Nov 25 11:26:18 2015 -0600 |
tree | 7cf9bb2ec54d41bb32d76ddf86da8ec9f36a95d1 | |
parent | 6f46c20fbfd4a7eb7e9fe4dd559e6d404bdca4e4 [diff] | |
parent | bc834e873d9668d8060b43bd871bd861368e3a54 [diff] |
Merge pull request #196 from ibm-et/SPARK_HOME_REQUIRED Require SPARK-HOME to run and do not pack spark dependencies
The main goal of the Spark Kernel is to provide the foundation for interactive applications to connect to and use Apache Spark.
The Spark Kernel provides an interface that allows clients to interact with a Spark Cluster. Clients can send libraries and snippets of code that are interpreted and ran against a preconfigured Spark context. These snippets can do a variety of things:
The kernel's main supported language is Scala
, but it is also capable of processing both Python
and R
. It implements the latest Jupyter message protocol (5.0), so it can easily plug into the 3.x branch of Jupyter/IPython for quick, interactive data exploration.
A version of the Spark Kernel is deployed as part of the Try Jupyter! site. Select Scala 2.10.4 (Spark 1.4.1)
under the New
dropdown. Note that this version only supports Scala
.
Vagrant is used to simplify the development experience. It is the only requirement to be able to build, package and test the Spark Kernel on your development machine.
To build and interact with the Spark Kernel using Jupyter, run
make dev
This will start a Jupyter notebook server accessible at http://192.168.44.44:8888
. From here you can create notebooks that use the Spark Kernel configured for local mode.
Tests can be run by doing make test
.
To build and package up the Spark Kernel, run
make dist
The resulting package of the kernel will be located at ./dist/spark-kernel-<VERSION>.tar.gz
. The uncompressed package is what is used is ran by Jupyter when doing make dev
.
Our goal is to keep master
up to date with the latest version of Spark. When new versions of Spark require code changes, we create a separate branch. The table below shows what is available now.
Branch | Spark Kernel Version | Apache Spark Version |
---|---|---|
master | 0.1.5 | 1.5.1 |
branch-0.1.4 | 0.1.4 | 1.4.1 |
branch-0.1.3 | 0.1.3 | 1.3.1 |
Please note that for the most part, new features to Spark Kernel will only be added to the master
branch.
There is more detailed information available in our Wiki and our Getting Started guide.