Google Git
Sign in
apache / incubator-livy / d39ab356b4e36d8780bd44b9d5020bfeec90cb2f
commitd39ab356b4e36d8780bd44b9d5020bfeec90cb2f[log] [tgz]
authorMarcelo Vanzin <vanzin@cloudera.com>Thu Sep 20 11:49:22 2018 +0800
committerjerryshao <sai.sai.shao@gmail.com>Thu Sep 20 11:49:22 2018 +0800
tree601a41ac21b34cd44bc2957ba76450c8af557ae7
parentf99cc32df3e0f8cf5f2a3925df166d6b531f2b50 [diff]
[LIVY-510] Remove support for JDK7.

With this change Livy will only build and run with JDK8. While this
doesn't necessarily preclude using Java 7 for running Spark itself,
that is not encouraged due to possible issues with serialization.

There are a few small changes elsewhere that I made so tests would pass,
but aren't related to the java 8 change:

- a small tweak to the fallback code that initializes the python
  interpreter, which was failing with a weird reflection error.

- expose the user's home directory in the MiniYarnCluster, to allow
  the tests to see pip packages installed in the user's home.

- tweak the python API tests so that the pytest cache does not pollute
  the source directory.

- changed Travis to use Oracle's JDK, since the Open JDK path was
  mixing Java 8 and 9 and causing errors.

- updated jacoco to latest stable.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #111 from vanzin/LIVY-510.
8 files changed
tree: 601a41ac21b34cd44bc2957ba76450c8af557ae7
  1. .github/
  2. api/
  3. assembly/
  4. bin/
  5. client-common/
  6. client-http/
  7. conf/
  8. core/
  9. coverage/
  10. dev/
  11. docs/
  12. examples/
  13. integration-test/
  14. python-api/
  15. repl/
  16. rsc/
  17. scala/
  18. scala-api/
  19. server/
  20. test-lib/
  21. thriftserver/
  22. .gitignore
  23. .rat-excludes
  24. .travis.yml
  25. checkstyle-suppressions.xml
  26. checkstyle.xml
  27. DISCLAIMER
  28. LICENSE
  29. NOTICE
  30. pom.xml
  31. README.md
  32. scalastyle.xml
README.md

Apache Livy

Build Status

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN.

  • Interactive Scala, Python and R shells
  • Batch submissions in Scala, Java, Python
  • Multiple users can share the same server (impersonation support)
  • Can be used for submitting jobs from anywhere with REST
  • Does not require any code change to your programs

Pull requests are welcomed! But before you begin, please check out the Contributing section on the Community page of our website.

Online Documentation

Guides and documentation on getting started using Livy, example code snippets, and Livy API documentation can be found at livy.incubator.apache.org.

Before Building Livy

To build Livy, you will need:

Debian/Ubuntu:

  • mvn (from maven package or maven3 tarball)
  • openjdk-8-jdk (or Oracle JDK 8)
  • Python 2.6+
  • R 3.x

Redhat/CentOS:

  • mvn (from maven package or maven3 tarball)
  • java-1.8.0-openjdk (or Oracle JDK 8)
  • Python 2.6+
  • R 3.x

MacOS:

  • Xcode command line tools
  • Oracle's JDK 1.8
  • Maven (Homebrew)
  • Python 2.6+
  • R 3.x

Required python packages for building Livy:

  • cloudpickle
  • requests
  • requests-kerberos
  • flake8
  • flaky
  • pytest

To run Livy, you will also need a Spark installation. You can get Spark releases at https://spark.apache.org/downloads.html.

Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark, Livy will automatically pick repl dependencies through detecting the Scala version of Spark.

Livy also supports Spark 2.0+ for both interactive and batch submission, you could seamlessly switch to different versions of Spark through SPARK_HOME configuration, without needing to rebuild Livy.

Building Livy

Livy is built using Apache Maven. To check out and build Livy, run:

git clone https://github.com/apache/incubator-livy.git
cd livy
mvn package

By default Livy is built against Apache Spark 1.6.2, but the version of Spark used when running Livy does not need to match the version used to build Livy. Livy internally uses reflection to mitigate the gaps between different Spark versions, also Livy package itself does not contain a Spark distribution, so it will work with any supported version of Spark (Spark 1.6+) without needing to rebuild against specific version of Spark.