At the end of 2020, Apache Uniffle (incubating)'s first line of code was written inside Tencent. It was open-sourced in November 2021, and then donated to the Apache Foundation in the mid-2022. Since its donation, it has attracted many developers from various companies. This article makes a brief summary of Apache Uniffle (incubating) in 2022.
Apache Uniffle (Incubating) was accepted as an Apache incubator project on June 6, 2022. As of the end of December 2022 (current writing time: 2022.12.26), a total of 157 new issues have been created (76 of which have been closed or resolved), and new 272 PRs (of which 264 were merged or closed).
Apache Uniffle (incubating) has released two versions in 2022: 0.6.0 and 0.6.1. Among them:
Version 0.6.0 is the first version released after Uniffle entered the Apache incubator. It features:
Shuffle Server
Version 0.6.1 is an important bug fix version after version 0.6.0. It mainly fixes the following problems:
Apache Uniffle (incubating) has added 22 contributors since entering the incubator, with a total of 33 contributors. Contributors come from Tencent, iQiyi, Ebay, Didi, SF Express, Baidu, ByteDance, JD.com, Bilibili, Databricks And other Internet companies. Among the 22 new contributors, Apache Uniffle (incubating) PMC voted to add 2 new committers according to their contribution. The new recruited committers are expected to make continuous contributions to Apache Uniffle (incubating) and we are planing to added more committers in 2023.
According to the online/offline communication with contributors and users, Apache Uniffle (incubating) is currently in production and used in companies such as Tencent, iQiyi, Didi, SF Express, VIPShop, Bilibili, etc. The amount of shuffle data processed by Uniffle of many companies exceeds PBs per day, and tens of thousands of apps are running. In addition to solving the stability and scalability problems of native Spark shuffle, it also improves the overall resource utilization.
In 2023, Apache Uniffle (incubating) will continue to provide an efficient and pervasive Shuffle Service. Currently, the following work is on the planning list: