tree 631c06d2154e074fd69c8381fac5948448d67dfb
parent 38832f17633b186952a677d2653504a7333ef31b
author Andrew Wong <awong@cloudera.com> 1588392554 -0700
committer Andrew Wong <awong@cloudera.com> 1588826791 +0000

KUDU-3113: fix auto-rebalancer move execution

When executing moves, the auto-rebalancer would try to resolve the
leader's address by passing its UUID instead of its host. This fixes it
to use an appropriate host.

This includes some light cleanup, and updates auto_rebalancer-test to
verify the moves lead to the copying of bytes on tablet servers.

The following flakiness is also addressed:
- NoRebalancingIfReplicasRecovering would sometimes schedule some moves
  before shutting down the tablet server, and we'd time out waiting to
  iterate without scheduling moves. I adjusted the ordering of the
  shutdown so the rebalancer doesn't get a chance to schedule moves.
- Rarely, TestHandlingFailedTservers would see a different error than
  expected when checking for failed sent RPCs. I updated the test to
  expect a couple of messages.

I looped auto_rebalancer-test in DEBUG mode and it pased 1000/1000
times, compared to failing 4/10 times with the change to actually
execute moves.

I also validated this on a real cluster:
- First, I enabled auto-rebalancing on the master.
- I put a tablet server into maintenance mode.
- I then moved all replicas off the tablet server using the rebalancer
  tool's --move_replicas_from_ignored_tservers option.
- I verified that even with the significant skew, since one of the
  tablet servers was in maintenance mode (i.e. unavailable for
  placement), the master didn't automatically move any replicas.
- Once I took the tablet server out of maintenance mode, moves were
  scheduled to repopulate it.
- Steady state was reached with a cluster skew of 1.

Change-Id: If658997dc9bcb709c27d981db56cf2db13ba235f
Reviewed-on: http://gerrit.cloudera.org:8080/15850
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>