| commit | e119e4578a96b1c04286ea4f79015dbea5107360 | [log] [tgz] |
|---|---|---|
| author | Wilfred Spiegelenburg <wilfreds@apache.org> | Mon Aug 10 19:08:32 2020 +1000 |
| committer | Wilfred Spiegelenburg <wilfreds@apache.org> | Mon Aug 10 19:08:32 2020 +1000 |
| tree | 8217d6a356eddc86ecf4952e27ebc50790eb56f8 | |
| parent | f674ea91ec17e633ec4c3732c5db090d12835c43 [diff] |
[YUNIKORN-358] tag and go.mod update for release
Apache YuniKorn (Incubating) is a light-weight, universal resource scheduler for container orchestrator systems. It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.
YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn's architecture design also allows adding different shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN, or any other systems.
Following chart illustrates the high-level architecture of YuniKorn.
YuniKorn consists of the following components spread over multiple code repositories, most of the following projects are written in GoLang.
Scheduler core: the brain of the scheduler, which makes placement decisions (Allocate container X on node Y) according to pre configured policies. See more in current repo yunikorn-core. Scheduler core is implemented in a way to be agnostic to scheduler implementation.
Scheduler interface: the common scheduler interface used by shims and the core scheduler. Contains the API layer (with GRPC/programming language bindings) which is agnostic to container orchestrator systems like YARN/K8s. See more in yunikorn-scheduler-interface.
Resource Manager shims: allow container orchestrator systems talks to yunikorn-core through scheduler-interface. Which can be configured on existing clusters without code change.
Currently, yunikorn-k8shim is available for Kubernetes integration. Supporting other Resource Manager is our next priority.
Scheduler User Interface: the YuniKorn web interface for app/queue management. See more in yunikorn-web.
The k8shim provides the integration for K8s scheduling and supported features include:
kubectl describe pod.We love high-performance software, and we made tremendous efforts to make it to support large scale cluster and high-churning tasks. Here's Performance Test Result
Yunikorn can be deployed as a K8s custom scheduler, and take over all POD scheduling. An enhancement is open to improve coexistence behaviour of the YuniKorn scheduler with other Kubernetes schedulers, like the default scheduler, in the cluster: Co-existing with other K8s schedulers.
| K8s Version | Support? |
|---|---|
| 1.12.x (or earlier) | X |
| 1.13.x | √ |
| 1.14.x | √ |
| 1.15.x | √ |
| 1.16.x | To be verified |
| 1.17.x | To be verified |
YuniKorn has builtin web UIs for queue hierarchies and apps. See below:
Want to learn more about future of YuniKorn? You can find more information about what are already supported and future plans in the Road Map.
The simplest way to run YuniKorn is to build a docker image and then deployed to Kubernetes with a yaml file, running as a customized scheduler. Then you can run workloads with this scheduler. See more instructions from the User Guide.
Apache YuniKorn (Incubating) community includes engineers from Alibaba, Apple, Cloudera, Linkedin, Microsoft, Nvidia, Tencent, Uber, etc. (sorted by alphabet). Want to join the community? We welcome any form of contributions, code, documentation or suggestions!
To get involved, please read following resources.
Demo videos
Communication channels
Mailing lists are:
We use Slack as our collaboration system, you can join us by accessing this link. Currently, we have following channels in the workspace: #yunikorn-dev and #yunikorn-user.
Blog posts
Apache based blogs:
3rd party blog posts: