| commit | 2ea7f17e22d25d0be718e71de03dd014272853fb | [log] [tgz] |
|---|---|---|
| author | fwang12 <fwang12@ebay.com> | Thu Apr 27 10:25:01 2023 +0800 |
| committer | fwang12 <fwang12@ebay.com> | Thu Apr 27 10:25:10 2023 +0800 |
| tree | 9ae21fa49793d10d8cdecddaba1e3b94ec14f736 | |
| parent | 3f0153a80948a62a215aa4285e33eff3a48cc014 [diff] |
[KYUUBI #4767] Correct the submit time for BatchJobSubmission and check applicationInfo if submitted application ### _Why are the changes needed?_ - if the kyuubi instance is unreachable, we should not transfer the batch metadata create time as batch submit time - we should always wait the kyuubi instance recovery - here using a fake submit time to prevent that the batch be marked as terminated if application state is NOT_FOUND - Inside the BatchJobSubmission, using the first get application info time as batch submit time. In this pr, I also record whether the batch operation submit batch. If it did submit batch application and the app is not started, we need to mark the batch state as ERROR. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4767 from turboFei/submit_time. Closes #4767 9d4df0f91 [fwang12] save 3e56a39cb [fwang12] runtime exception -> kyuubi exception 5cac15ec5 [fwang12] nit 92d5000be [fwang12] nit 3678f8f2c [fwang12] wait the app to monitor d51fb2636 [fwang12] save 708ad20ce [fwang12] refactor 98d49c64e [fwang12] wait 1adbefd59 [fwang12] revert f3e4f2a11 [fwang12] wait app id ready before monitoring a3bfe6f56 [fwang12] check app started 7530b5118 [fwang12] check submit app and final state a41e81d0e [fwang12] refactor e4217da03 [fwang12] _app start time 3d1e8f022 [fwang12] fake submit timeout 06c8f0a22 [fwang12] correct submit time Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> (cherry picked from commit 94c72734ca2d64328f99715ed648305d73a0718b) Signed-off-by: fwang12 <fwang12@ebay.com>
Apache Kyuubi™ is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
Kyuubi provides a pure SQL gateway through Thrift JDBC/ODBC interface for end-users to manipulate large-scale data with pre-programmed and extensible Spark SQL engines. This “out-of-the-box” model minimizes the barriers and costs for end-users to use Spark at the client side. At the server-side, Kyuubi server and engines' multi-tenant architecture provides the administrators a way to achieve computing resource isolation, data security, high availability, high client concurrency, etc.
Kyuubi's goal is to make it easy and efficient for anyone to use Spark(maybe other engines soon) and facilitate users to handle big data like ordinary data. Here, anyone means that users do not need to have a Spark technical background but a human language, SQL only. Sometimes, SQL skills are unnecessary when integrating Kyuubi with Apache Superset, which supports rich visualizations and dashboards.
In typical big data production environments with Kyuubi, there should be system administrators and end-users.
Additionally, the Kyuubi community will continuously optimize the whole system with various features, such as History-Based Optimizer, Auto-tuning, Materialized View, SQL Dialects, Functions, e.t.c.
In typical big data production environments, especially secured ones, all bundled services manage access control lists to restricting access to authorized users. For example, Hadoop YARN divides compute resources into queues. With Queue ACLs, it can identify and control which users/groups can take actions on particular queues. Similarly, HDFS ACLs control access of HDFS files by providing a way to set different permissions for specific users/groups.
Apache Spark is a unified analytics engine for large-scale data processing. It provides a Distributed SQL Engine, a.k.a, the Spark Thrift Server(STS), designed to be seamlessly compatible with HiveServer2 and get even better performance.
HiveServer2 can identify and authenticate a caller, and then if the caller also has permissions for the YARN queue and HDFS files, it succeeds. Otherwise, it fails. However, on the one hand, STS is a single Spark application. The user and queue to which STS belongs are uniquely determined at startup. Consequently, STS cannot leverage cluster managers such as YARN and Kubernetes for resource isolation and sharing or control the access for callers by the single user inside the whole system. On the other hand, the Thrift Server is coupled in the Spark driver's JVM process. This coupled architecture puts a high risk on server stability and makes it unable to handle high client concurrency or apply high availability such as load balancing as it is stateful.
Kyuubi extends the use of STS in a multi-tenant model based on a unified interface and relies on the concept of multi-tenancy to interact with cluster managers to finally gain the ability of resources sharing/isolation and data security. The loosely coupled architecture of the Kyuubi server and engine dramatically improves the client concurrency and service stability of the service itself.
The vision of Kyuubi is to unify the portal and become an easy-to-use data lake management platform. Different kinds of workloads, such as ETL processing and BI analytics, can be supported by one platform, using one copy of data, with one SQL interface.
Kyuubi can deploy its engines on different kinds of Cluster Managers, such as, Hadoop YARN, Kubernetes, etc.
The figure below shows our vision for the Kyuubi Ecosystem. Some of them have been realized, some in development, and others would not be possible without your help.
Since Kyuubi 1.3.0-incubating, the Kyuubi online documentation is hosted by https://kyuubi.apache.org/. You can find the latest Kyuubi documentation on this web page. For 1.2 and earlier versions, please check the Readthedocs directly.
Ready? Getting Started with Kyuubi.
The project took its name from a character of a popular Japanese manga - Naruto. The character is named Kyuubi Kitsune/Kurama, which is a nine-tailed fox in mythology. Kyuubi spread the power and spirit of fire, which is used here to represent the powerful Apache Spark. Its nine tails stand for end-to-end multi-tenancy support of this project.
This project is licensed under the Apache 2.0 License. See the LICENSE file for details.