Apache incubator-yunikorn-core - incubating

Clone this repo:
  1. 4e9e0b7 resize to 200 by Weiwei Yang · 10 days ago master
  2. 5c946f4 use blue logo by Weiwei Yang · 10 days ago
  3. a37f804 add correct logo file by Weiwei Yang · 10 days ago
  4. 4de4740 remove wrong file by Weiwei Yang · 10 days ago
  5. 9f7ac6a fix dead link by Weiwei Yang · 10 days ago

YuniKorn - A Universal Scheduler

Green Build


YuniKorn is a light-weighted, universal resource scheduler for container orchestrator systems. It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform scheduling experience for mixed workloads consists of stateless batch workloads and stateful services. Support for but not limited to, YARN and Kubernetes.

Architecture

Following chart illustrates the high-level architecture of YuniKorn.

Architecture

YuniKorn consists of the following components spread over multiple code repositories.

  • Scheduler core: the brain of the scheduler, which makes placement decisions (Allocate container X on node Y) according to pre configured policies. See more in current repo yunikorn-core.
  • Scheduler interface: the common scheduler interface used by shims and the core scheduler. Contains the API layer (with GRPC/programming language bindings) which is agnostic to container orchestrator systems like YARN/K8s. See more in yunikorn-scheduler-interface.
  • Resource Manager shims: allow container orchestrator systems talks to yunikorn-core through scheduler-interface. Which can be configured on existing clusters without code change. Currently, yunikorn-k8shim is available for Kubernetes integration.
  • Scheduler User Interface: the YuniKorn web interface for app/queue management. See more in yunikorn-web.

Key features

Here are some key features of YuniKorn.

  • Features to support both batch jobs and long-running/stateful services
  • Hierarchy queues with min/max resource quotas.
  • Resource fairness between queues, users and apps.
  • Cross-queue preemption based on fairness.
  • Customized resource types (like GPU) scheduling support.
  • Rich placement constraints support.
  • Automatically map incoming container requests to queues by policies.
  • Node partition: partition cluster to sub-clusters with dedicated quota/ACL management.

You can find more information about what are already supported and future plans in the Road Map.

How to use

The simplest way to run YuniKorn is to build a docker image and then deployed to Kubernetes with a yaml file, running as a customized scheduler. Then you can run workloads with this scheduler. See more instructions from the User Guide.

How can I get involved?

We welcome any form of contributions, code, documentation or even discussions. To get involved, please read following resources.

Other Resources

Demo videos

Communication channels

Blog posts