[catalog_manager] Tighten leader UUID fallback

It is safe to assume that if the term is the same in the current cstate
as in the previous cstate then even if the leader is not set, it will
be the same. But it is possible that cmeta file is deleted then
recreated with "local_replica cmeta unsafe_recreate" command. In this
case the leader_uuid is empty in the new cmeta file. This means that the
peer doesn't consider itself a leader, so no health report is generated
in tablet report and it has no leader_uuid set either. When a master
receives tablet report like this and there isn't a new term, then the
catalog master will treat this peer as a leader, but it will fail on a
check because the leader has to be in healthy status. This happened in
ToolTest::TestRecreateCMeta. As a reproduction step, the same test now
runs with a single TServer configuration, too. In this configuration the
error is reproducible 100% of the times, since the term is not increased
and the leader's cmeta file is changed.

The solution is that catalog manager only assumes the previous leader
for the peer if the previous leader is not the peer itself. This gives
time for the peers to form a consensus about the leader.

Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Reviewed-on: http://gerrit.cloudera.org:8080/21004
Tested-by: Alexey Serbin <alexey@apache.org>
Reviewed-by: Alexey Serbin <alexey@apache.org>
2 files changed