apache /
kudu /
10efaf2c77dfe5e4474505e0267c583c011703be [master] KUDU-3390 support auto rebalance tablet leaders across TServers
The number of leader replicas per tablet server can become imbalanced
over time, which lead to load skew on some nodes. This patch adds
auto leader rebalance task to avoid leader replicas skew.
Two reasons of load skew:
- The main reason. Scan Requests has two modes: LeaderOnly(default) and CLOSEST_REPLICA.
For more accurate results, users will choose the LeaderOnly(default) mode.
Mostly, the scan load is positive correlation with leader numbers.
- The other reason. Write requests, leaders receive write requests and followers receive
UpdateConsensus request(raft called AppendEntries), the flow of processing is a little
different: the queue, pipeline, batch size, that may cause imbalanced load. Leader
rebalance will make leader and followers balanced and make service more stable.
An experiment present leader rebalance's effect (more details at KUDU-3390):
|leader ratio| scan cost | cpu usage | io
before|40: 0: 0 | 811.586 s | 47%, 18%, 19% | 102MB/s ioutil:55%, 8KB/s 2%, 64KB/s 3%
after |13: 14: 13 | 611.012 s | 39%, 45%, 35% | 53MB/s ioutil:31%, 80MB/s 18%, 45MB/s 24%
Change-Id: Ibfb60d8759a93b6a19238637c27df4f6b1cac918
Reviewed-on: http://gerrit.cloudera.org:8080/18454
Reviewed-by: Yuqi Du <shenxingwuying@gmail.com>
Reviewed-by: Yingchun Lai <acelyc1112009@gmail.com>
Tested-by: Kudu Jenkins
8 files changed
