Apache YuniKorn (Incubating) is a light-weight, universal resource scheduler for container orchestrator systems. It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.
YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn's architecture design also allows adding different shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN, or any other systems.
Following chart illustrates the high-level architecture of YuniKorn.
YuniKorn consists of the following components spread over multiple code repositories, most of the following projects are written in GoLang.
Scheduler core: the brain of the scheduler, which makes placement decisions (Allocate container X on node Y) according to pre configured policies. See more in current repo yunikorn-core. Scheduler core is implemented in a way to be agnostic to scheduler implementation.
Scheduler interface: the common scheduler interface used by shims and the core scheduler. Contains the API layer (with GRPC/programming language bindings) which is agnostic to container orchestrator systems like YARN/K8s. See more in yunikorn-scheduler-interface.
Resource Manager shims: allow container orchestrator systems talks to yunikorn-core through scheduler-interface. Which can be configured on existing clusters without code change.
Currently, yunikorn-k8shim is available for Kubernetes integration. Supporting other Resource Manager is our next priority.
Scheduler User Interface: the YuniKorn web interface for app/queue management. See more in yunikorn-web.
k8shim provides the integration for K8s scheduling and supported features include:
kubectl describe pod.
We love high-performance software, and we made tremendous efforts to make it to support large scale cluster and high-churning tasks. Here's Performance Test Result
Yunikorn can be deployed as a K8s custom scheduler, and take over all POD scheduling. An enhancement is open to improve coexistence behaviour of the YuniKorn scheduler with other Kubernetes schedulers, like the default scheduler, in the cluster: Co-existing with other K8s schedulers.
|1.12.x (or earlier)||X|
|1.16.x||To be verified|
|1.17.x||To be verified|
YuniKorn has builtin web UIs for queue hierarchies and apps. See below: