commit a1aba60a6701692f42ed650cae88827a26e93183
author Jiajun Wang <1803880+jiajunwang@users.noreply.github.com> Wed May 06 15:48:15 2020 -0700
committer GitHub <noreply@github.com> Wed May 06 15:48:15 2020 -0700
tree ee85f4ba7476bcc7e7e6fcb518bbdd12a0f1783f
parent 6c9fdb0aa73bf7c50e131c99d5ff3704ca20f887

Adjust the auto rebalancer state assignment logic to reduce top state transition. (#986)

The old state assignment logic assign the states to selected nodes according to the priority of the current replica state that is on the instance. Moreover, the sorting algorithm is designed to prioritize both current topstate and current secondary states equally. The result is that we will have premature mastership handoff to a current seconardy state host before the real desired master host is ready.
For example,
1. The current states are: [N1:M, N2:S, N3,S]
2. The desired states are: [N4:M, N2:S, N1:S]
3. Due to the sorting logic based on current states, we will have a transient preference list ordered like: [N2, N1, N4]. In which case, the controller will assign master to N2 before N4 has a slave state replica.
4. When N4 finishes the Offline to Slave transition, the same sorting logic will sort the preference list to be: [N4, N2, N1]. Then we have another mastership handoff.
To be clear, we don't want step 3. But only the state transition in step 4.

In this PR, we refactor the sorting logic so that it will only move the master whenever the candidate has a "ready" state replica, in which case, only one mastership handoff happens.