commit 28f30f0eb70421feee8d4e7b775483d47e9ce888
author Bankim Bhavsar <bankim@cloudera.com> Fri Feb 19 10:40:41 2021 -0800
committer Bankim Bhavsar <bankim@cloudera.com> Mon Feb 22 22:03:19 2021 +0000
tree 8a543959f245261842bd161b1a6ef3e1680c9f4d
parent 78bd6c04ef37ca8906379a587140bc646433e28c

[test] Fix flakiness with detecting a master in FAILED_UNRECOVERABLE state

Observed one test failure on the flakiness dashboard in the
dynamic_multi_master test where system catalog WAL is expected to be
GC'ed and hence new master can't be caught up from WAL which should
result in the new master going to FAILED_UNRECOVERABLE state.
However the new master gets caught up from WAL and is in HEALTHY
state.

Expected: consensus::HealthReportPB::FAILED_UNRECOVERABLE
  Which is: 2
To be equal to: peer.health_report().overall_health()
  Which is: 1

This is because the check for system catalog WAL was made against
master index 0 which may not be the leader master and that's what
happened in case of the test failure.

The fix makes the GC count check against all masters as master
leadership could change.

This change also includes another place where master index 0 was used
and it'd be good to explicitly use the leader index to future-proof
it in case order of validation is changed.

Change-Id: Id6017f1601eaed22be28c8a5babd6e35e93b1d2e
Reviewed-on: http://gerrit.cloudera.org:8080/17089
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>