| commit | 8c6f4c533a5d18f10baea495b6b668f8a7040e4f | [log] [tgz] |
|---|---|---|
| author | Peter Bacsko <pbacsko@cloudera.com> | Thu Jan 25 16:59:04 2024 +1100 |
| committer | Wilfred Spiegelenburg <wilfreds@apache.org> | Thu Jan 25 16:59:04 2024 +1100 |
| tree | 4f21b9e3e02b505f14f4c16789c43c0ff68bb143 | |
| parent | 681a4cbf4b9929e579674e44e6309acf0e728e29 [diff] |
[YUNIKORN-2212] Outstanding request collection optimisation (#745) Auto scaling n K8s relies on the pod to be marked unschedulable. That is done via a regular callback from the core to the shim. The logic for building the pod list in this callback is simple. This could cause over scaling nodes or excessive K8s API server calls. Improvements added: * Don't collect requests that have not been scheduled yet. * If already triggered scale up do not sent again * Check for gang placeholders for any real ask * Don't scale if pod has required node set (daemon set pod) Track the state of scheduling and auto scale triggering. Closes: #745 Signed-off-by: Wilfred Spiegelenburg <wilfreds@apache.org>
Apache YuniKorn is a light-weight, universal resource scheduler for container orchestrator systems. It is created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.
YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn's architecture design also allows adding different shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN, or any other systems.
See how to get started with running YuniKorn on Kubernetes, please read the documentation on yunikorn.apache.org.
Want to know more about the value of the YuniKorn project, and what YuniKorn can do? Here are some session recordings and demos.
Please read get involved document if you want to discuss issues, contribute your ideas, explore use cases, or participate the development.
If you want to contribute code to this repo, please read the developer doc. All the design docs are available here.
Apache YuniKorn project has the following git repositories:
The yunikorn-core is the brain of the scheduler, which makes placement decisions (allocate container X on node Y) according to the builtin rich scheduling policies. Scheduler core implementation is agnostic to the underneath resource manager system.
