| commit | 7d8fa69f1804025d95971bcfb389951075dc6b98 | [log] [tgz] |
|---|---|---|
| author | Marco Gaido <mgaido@hortonworks.com> | Mon Sep 10 11:42:10 2018 +0800 |
| committer | jerryshao <sai.sai.shao@gmail.com> | Mon Sep 10 11:42:10 2018 +0800 |
| tree | f85f23f7ba03c3c31a77ee86c351b23dbb91503b | |
| parent | 8027ca708fdc3df9a5b08d2d33d0436018154bcc [diff] |
[LIVY-491][LIVY-492][LIVY-493] Add Thriftserver module and implementation ## What changes were proposed in this pull request? The PR contains an implementation of a JDBC API for Livy server based on the Hive Thriftserver. The implementation is based on the version 3.0 of Hive Thriftserver. This initial PR contains the thriftserver module added to Livy and its implementation. It doesn't contain any binding for starting it, this will be added later. Some Hive classes have been ported here because they needed come modifications in order to works properly in Livy. Long term solution is to get of them all by re-implementing the needed parts in the Livy thriftserver itself, without relying on Hive code (other than the PRC interface) anymore. Those classes/changes can be summarized in three categories: 1. Changes to make the Hive classes easy to extend: for instance, some visibility modifiers were changes (moving from `private` or `package private` to `protected` or `public`); 2. Changes in order to reduce the dependencies on Hive modules/classes: for instance all the classes in the `operation` package were modified in order to get rid of the usage of `HiveSession` and the `HiveServer2` was changed in order not to use `CLIService`. 3. The UGI management which is currently performed in Livy is definitely very different from the Hive one, this required changes to the `HiveAuthFactory` in order not to interfere with the existing codebase. ## How was this patch tested? added integration tests as UTs Author: Marco Gaido <mgaido@hortonworks.com> Closes #104 from mgaido91/LIVY-491.
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN.
Pull requests are welcomed! But before you begin, please check out the Contributing section on the Community page of our website.
Guides and documentation on getting started using Livy, example code snippets, and Livy API documentation can be found at livy.incubator.apache.org.
To build Livy, you will need:
Debian/Ubuntu:
maven package or maven3 tarball)Redhat/CentOS:
maven package or maven3 tarball)MacOS:
Required python packages for building Livy:
To run Livy, you will also need a Spark installation. You can get Spark releases at https://spark.apache.org/downloads.html.
Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark, Livy will automatically pick repl dependencies through detecting the Scala version of Spark.
Livy also supports Spark 2.0+ for both interactive and batch submission, you could seamlessly switch to different versions of Spark through SPARK_HOME configuration, without needing to rebuild Livy.
Livy is built using Apache Maven. To check out and build Livy, run:
git clone https://github.com/apache/incubator-livy.git cd livy mvn package
By default Livy is built against Apache Spark 1.6.2, but the version of Spark used when running Livy does not need to match the version used to build Livy. Livy internally uses reflection to mitigate the gaps between different Spark versions, also Livy package itself does not contain a Spark distribution, so it will work with any supported version of Spark (Spark 1.6+) without needing to rebuild against specific version of Spark.