Apache YuniKorn Core

Clone this repo:
  1. 357d71f [YUNIKORN-331] Reduce the verbosity of the logs when predicate fails (#191) by Adam Antal · 6 days ago master
  2. d2a76c0 [YUNIKORN-302] use same reservation delay in all tests (#187) by Wilfred Spiegelenburg · 2 weeks ago
  3. f3e1c76 [YUNIKORN-287] reserved ask release double release (#185) by Wilfred Spiegelenburg · 3 weeks ago
  4. 699243c [YUNIKORN-278] Support upscaling for unlimited queues (#184) by Weiwei Yang · 3 weeks ago
  5. 06d2bd9 Removed VolumesPlugin related code (#182) by manirajv06 · 4 weeks ago

Apache YuniKorn (Incubating) - A Universal Scheduler

Build Status codecov Go Report Card License Repo Size


Apache YuniKorn (Incubating) is a light-weight, universal resource scheduler for container orchestrator systems. It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.

YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn's architecture design also allows adding different shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN, or any other systems.

Architecture

Following chart illustrates the high-level architecture of YuniKorn.

Architecture

YuniKorn consists of the following components spread over multiple code repositories, most of the following projects are written in GoLang.

  • Scheduler core: the brain of the scheduler, which makes placement decisions (Allocate container X on node Y) according to pre configured policies. See more in current repo yunikorn-core. Scheduler core is implemented in a way to be agnostic to scheduler implementation.

  • Scheduler interface: the common scheduler interface used by shims and the core scheduler. Contains the API layer (with GRPC/programming language bindings) which is agnostic to container orchestrator systems like YARN/K8s. See more in yunikorn-scheduler-interface.

  • Resource Manager shims: allow container orchestrator systems talks to yunikorn-core through scheduler-interface. Which can be configured on existing clusters without code change.

    Currently, yunikorn-k8shim is available for Kubernetes integration. Supporting other Resource Manager is our next priority.

  • Scheduler User Interface: the YuniKorn web interface for app/queue management. See more in yunikorn-web.

Feature highlights

Main scheduling features:

  • Features to support both batch jobs and long-running/stateful services
  • Hierarchy queues with min/max resource quotas.
  • Resource fairness between queues, users and apps.
  • Cross-queue preemption based on fairness.
  • Customized resource types (like GPU) scheduling support.
  • Rich placement constraints support.
  • Automatically map incoming container requests to queues by policies.
  • Node partition: partition cluster to sub-clusters with dedicated quota/ACL management.

Integration with K8s:

The k8shim provides the integration for K8s scheduling and supported features include:

  • Predicates: All kinds of predicates such as node-selector, pod affinity/anti-affinity, taint/tolerant, etc.
  • Persistent volumes: We have verified hostpath, EBS, NFS, etc.
  • K8s namespace awareness: YuniKorn support hierarchical of queues, does it mean you need to give up K8s namespace? Answer is no, with simple config, YuniKorn is able to support automatically map K8s namespaces to YuniKorn queues. All K8s-namespace-related ResourceQuota, permissions will be still valid.
  • Metrics: Prometheus, Grafana integration.
  • Cluster AutoScaler: YuniKorn can nicely work with Cluster AutoScaler (https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) to drive cluster scales up and down.
  • K8s Events: YuniKorn also integrated with K8s events, so lots of information can be retrieved by using kubectl describe pod.

Performance testing

We love high-performance software, and we made tremendous efforts to make it to support large scale cluster and high-churning tasks. Here's Performance Test Result

Deployment model

Yunikorn can be deployed as a K8s custom scheduler, and take over all POD scheduling. An enhancement is open to improve coexistence behaviour of the YuniKorn scheduler with other Kubernetes schedulers, like the default scheduler, in the cluster: Co-existing with other K8s schedulers.

Verified K8s versions

K8s VersionSupport?
1.12.x (or earlier)X
1.13.x
1.14.x
1.15.x
1.16.xTo be verified
1.17.xTo be verified

Web UI

YuniKorn has builtin web UIs for queue hierarchies and apps. See below:

Queues-UI