[==========] Running 10 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 10 tests from RaftConsensusElectionITest
[ RUN      ] RaftConsensusElectionITest.RunLeaderElection
I0426 18:02:11.515056 20091 test_util.cc:200] Using random seed: 801827415
I0426 18:02:11.664279 20091 ts_itest-base.cc:113] Starting cluster with:
I0426 18:02:11.664485 20091 ts_itest-base.cc:114] --------------
I0426 18:02:11.664600 20091 ts_itest-base.cc:115] 3 tablet servers
I0426 18:02:11.664726 20091 ts_itest-base.cc:116] 3 replicas per TS
I0426 18:02:11.664837 20091 ts_itest-base.cc:117] --------------
I0426 18:02:11.669464 20091 external_mini_cluster.cc:819] Running /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-master
/tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-master
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.158.254:0
--webserver_interface=127.19.158.254
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0426 18:02:11.789261 20103 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0426 18:02:11.789655 20103 flags.cc:412] Enabled unsafe flag: --never_fsync=true
W0426 18:02:11.799146 20103 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024
W0426 18:02:11.799295 20103 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024
W0426 18:02:11.799396 20103 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512
I0426 18:02:11.814257 20103 master_main.cc:66] Master server non-default flags:
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=1024
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.158.254:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.158.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision c9307f8eb89fd99c1492b5e53812f2cdc1adffb4
build type FASTDEBUG
built by None at 26 Apr 2018 18:00:48 UTC on jenkins-slave-vn0g
build id 13197
TSAN enabled
I0426 18:02:11.822619 20103 master_main.cc:73] Initializing master server...
I0426 18:02:11.824295 20103 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 13547us
I0426 18:02:11.825196 20103 fs_manager.cc:260] Metadata directory not provided
I0426 18:02:11.825330 20103 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0426 18:02:11.825552 20103 server_base.cc:433] Could not load existing FS layout: Not found: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2)
I0426 18:02:11.825667 20103 server_base.cc:434] Attempting to create new FS layout instead
I0426 18:02:11.833616 20103 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "08f726b6ca8e483486158eacd80e933a"
format_stamp: "Formatted at 2018-04-26 18:02:11 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:11.834422 20103 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "08f726b6ca8e483486158eacd80e933a"
format_stamp: "Formatted at 2018-04-26 18:02:11 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:11.844794 20103 fs_manager.cc:495] Time spent creating directory manager: real 0.010s	user 0.008s	sys 0.000s
I0426 18:02:11.846194 20103 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go
I0426 18:02:11.846695 20103 file_cache.cc:470] Constructed file cache lbm with capacity 26214
I0426 18:02:11.853688 20103 fs_manager.cc:417] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.000s
I0426 18:02:11.853826 20103 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/wal
uuid: "08f726b6ca8e483486158eacd80e933a"
format_stamp: "Formatted at 2018-04-26 18:02:11 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:11.854120 20103 fs_report.cc:347] Block manager report
--------------------
1 data directories: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0426 18:02:12.025734 20103 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go
I0426 18:02:12.027698 20103 master_main.cc:76] Starting Master server...
I0426 18:02:12.029160 20120 process_memory.cc:182] Process hard memory limit is 11.777631 GB
I0426 18:02:12.029335 20120 process_memory.cc:184] Process soft memory limit is 9.422105 GB
I0426 18:02:12.029428 20120 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB
I0426 18:02:12.070179 20103 rpc_server.cc:200] RPC server started. Bound to: 127.19.158.254:32815
I0426 18:02:12.083832 20103 webserver.cc:173] Starting webserver on 127.19.158.254:0
I0426 18:02:12.083942 20103 webserver.cc:184] Document root disabled
I0426 18:02:12.085273 20103 webserver.cc:311] Webserver started. Bound to: http://127.19.158.254:35161/
I0426 18:02:12.086097 20103 server_base.cc:606] Dumped server information to /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/master-0/data/info.pb
I0426 18:02:12.089023 20164 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0426 18:02:12.093897 20091 external_mini_cluster.cc:881] Started /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-master as pid 20103
I0426 18:02:12.095029 20091 external_mini_cluster.cc:819] Running /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver
/tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.19.158.193:0
--local_ip_for_outbound_sockets=127.19.158.193
--webserver_interface=127.19.158.193
--webserver_port=0
--tserver_master_addrs=127.19.158.254:32815
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I0426 18:02:12.104182 20164 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a: Bootstrap starting.
I0426 18:02:12.106734 20164 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a: No blocks or log segments found. Creating new log.
I0426 18:02:12.108031 20164 log.cc:520] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a: Log is configured to *not* fsync() on all Append() calls
I0426 18:02:12.110610 20164 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a: No bootstrap required, opened a new log
I0426 18:02:12.123486 20164 raft_consensus.cc:303] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f726b6ca8e483486158eacd80e933a" member_type: VOTER }
I0426 18:02:12.123858 20164 raft_consensus.cc:329] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0426 18:02:12.123996 20164 raft_consensus.cc:604] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f726b6ca8e483486158eacd80e933a, State: Initialized, Role: FOLLOWER
I0426 18:02:12.124481 20164 consensus_queue.cc:227] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f726b6ca8e483486158eacd80e933a" member_type: VOTER }
I0426 18:02:12.124832 20164 raft_consensus.cc:343] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I0426 18:02:12.125000 20164 raft_consensus.cc:435] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I0426 18:02:12.125151 20164 raft_consensus.cc:2699] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a: Snoozing failure detection for 2.180s (starting election)
I0426 18:02:12.125298 20164 raft_consensus.cc:2751] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 0 FOLLOWER]: Advancing to term 1
I0426 18:02:12.128233 20164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f726b6ca8e483486158eacd80e933a" member_type: VOTER }
I0426 18:02:12.128779 20164 leader_election.cc:257] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0426 18:02:12.130381 20167 raft_consensus.cc:2699] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a: Snoozing failure detection for 2.030s (election complete)
I0426 18:02:12.130643 20167 raft_consensus.cc:2506] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 1 FOLLOWER]: Leader election won for term 1
I0426 18:02:12.138551 20167 raft_consensus.cc:571] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [term 1 LEADER]: Becoming Leader. State: Replica: 08f726b6ca8e483486158eacd80e933a, State: Running, Role: LEADER
I0426 18:02:12.138965 20167 consensus_queue.cc:204] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f726b6ca8e483486158eacd80e933a" member_type: VOTER }
I0426 18:02:12.142307 20164 sys_catalog.cc:335] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "08f726b6ca8e483486158eacd80e933a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f726b6ca8e483486158eacd80e933a" member_type: VOTER } }
I0426 18:02:12.142688 20164 sys_catalog.cc:338] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [sys.catalog]: This master's current role is: LEADER
I0426 18:02:12.144579 20169 sys_catalog.cc:335] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f726b6ca8e483486158eacd80e933a. Latest consensus state: current_term: 1 leader_uuid: "08f726b6ca8e483486158eacd80e933a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f726b6ca8e483486158eacd80e933a" member_type: VOTER } }
I0426 18:02:12.144220 20168 sys_catalog.cc:335] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08f726b6ca8e483486158eacd80e933a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f726b6ca8e483486158eacd80e933a" member_type: VOTER } }
I0426 18:02:12.145058 20164 sys_catalog.cc:422] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [sys.catalog]: configured and running, proceeding with master startup.
I0426 18:02:12.145534 20168 sys_catalog.cc:338] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [sys.catalog]: This master's current role is: LEADER
I0426 18:02:12.145277 20171 catalog_manager.cc:990] Loading table and tablet metadata into memory...
I0426 18:02:12.145545 20169 sys_catalog.cc:338] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a [sys.catalog]: This master's current role is: LEADER
I0426 18:02:12.147105 20103 master_main.cc:79] Master server successfully started.
I0426 18:02:12.150617 20171 catalog_manager.cc:1001] Initializing Kudu internal certificate authority...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0426 18:02:12.207855 20165 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0426 18:02:12.208323 20165 flags.cc:412] Enabled unsafe flag: --never_fsync=true
W0426 18:02:12.217430 20165 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024
W0426 18:02:12.217840 20165 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.158.193
I0426 18:02:12.220098 20171 catalog_manager.cc:888] Generated new certificate authority record
I0426 18:02:12.221678 20171 catalog_manager.cc:1010] Loading token signing keys...
I0426 18:02:12.235308 20171 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 08f726b6ca8e483486158eacd80e933a: Generated new TSK 0
I0426 18:02:12.250535 20165 tablet_server_main.cc:72] Tablet server non-default flags:
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.19.158.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.158.193
--webserver_port=0
--tserver_master_addrs=127.19.158.254:32815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.158.193
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision c9307f8eb89fd99c1492b5e53812f2cdc1adffb4
build type FASTDEBUG
built by None at 26 Apr 2018 18:00:48 UTC on jenkins-slave-vn0g
build id 13197
TSAN enabled
I0426 18:02:12.259595 20165 tablet_server_main.cc:79] Initializing tablet server...
I0426 18:02:12.261610 20165 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 14047us
I0426 18:02:12.262135 20165 fs_manager.cc:260] Metadata directory not provided
I0426 18:02:12.262276 20165 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0426 18:02:12.262487 20165 server_base.cc:433] Could not load existing FS layout: Not found: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2)
I0426 18:02:12.262609 20165 server_base.cc:434] Attempting to create new FS layout instead
I0426 18:02:12.270684 20165 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "849e2ce84972434a997a3dfa1f77ae22"
format_stamp: "Formatted at 2018-04-26 18:02:12 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:12.271466 20165 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "849e2ce84972434a997a3dfa1f77ae22"
format_stamp: "Formatted at 2018-04-26 18:02:12 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:12.303690 20165 fs_manager.cc:495] Time spent creating directory manager: real 0.032s	user 0.008s	sys 0.024s
I0426 18:02:12.305124 20165 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go
I0426 18:02:12.305610 20165 file_cache.cc:470] Constructed file cache lbm with capacity 26214
I0426 18:02:12.321331 20165 fs_manager.cc:417] Time spent opening block manager: real 0.011s	user 0.008s	sys 0.004s
I0426 18:02:12.321472 20165 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/wal
uuid: "849e2ce84972434a997a3dfa1f77ae22"
format_stamp: "Formatted at 2018-04-26 18:02:12 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:12.321710 20165 fs_report.cc:347] Block manager report
--------------------
1 data directories: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0426 18:02:12.527070 20165 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go
I0426 18:02:12.530267 20165 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0426 18:02:12.542541 20165 tablet_server_main.cc:84] Starting tablet server...
I0426 18:02:13.084795 20165 rpc_server.cc:200] RPC server started. Bound to: 127.19.158.193:39806
I0426 18:02:13.085728 20165 webserver.cc:173] Starting webserver on 127.19.158.193:0
I0426 18:02:13.085840 20165 webserver.cc:184] Document root disabled
I0426 18:02:13.087486 20165 webserver.cc:311] Webserver started. Bound to: http://127.19.158.193:37023/
I0426 18:02:13.088268 20165 server_base.cc:606] Dumped server information to /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-0/data/info.pb
I0426 18:02:13.093343 20297 process_memory.cc:182] Process hard memory limit is 11.777631 GB
I0426 18:02:13.094067 20297 process_memory.cc:184] Process soft memory limit is 9.422105 GB
I0426 18:02:13.094192 20297 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB
I0426 18:02:13.095690 20091 external_mini_cluster.cc:881] Started /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver as pid 20165
I0426 18:02:13.096597 20091 external_mini_cluster.cc:819] Running /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver
/tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--rpc_bind_addresses=127.19.158.194:0
--local_ip_for_outbound_sockets=127.19.158.194
--webserver_interface=127.19.158.194
--webserver_port=0
--tserver_master_addrs=127.19.158.254:32815
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I0426 18:02:13.093793 20165 tablet_server_main.cc:87] Tablet server successfully started.
I0426 18:02:13.113649 20296 heartbeater.cc:346] Connected to a master server at 127.19.158.254:32815
I0426 18:02:13.113927 20296 heartbeater.cc:426] Registering TS with master...
I0426 18:02:13.114444 20296 heartbeater.cc:475] Master 127.19.158.254:32815 requested a full tablet report, sending...
I0426 18:02:13.116511 20131 ts_manager.cc:81] Registered new tserver with Master: 849e2ce84972434a997a3dfa1f77ae22 (127.19.158.193:39806)
I0426 18:02:13.118386 20131 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.19.158.193:48198
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0426 18:02:13.214220 20300 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0426 18:02:13.214615 20300 flags.cc:412] Enabled unsafe flag: --never_fsync=true
W0426 18:02:13.223816 20300 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024
W0426 18:02:13.224174 20300 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.158.194
I0426 18:02:13.238559 20300 tablet_server_main.cc:72] Tablet server non-default flags:
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.19.158.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.158.194
--webserver_port=0
--tserver_master_addrs=127.19.158.254:32815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.158.194
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision c9307f8eb89fd99c1492b5e53812f2cdc1adffb4
build type FASTDEBUG
built by None at 26 Apr 2018 18:00:48 UTC on jenkins-slave-vn0g
build id 13197
TSAN enabled
I0426 18:02:13.247191 20300 tablet_server_main.cc:79] Initializing tablet server...
I0426 18:02:13.249528 20300 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 14547us
I0426 18:02:13.250048 20300 fs_manager.cc:260] Metadata directory not provided
I0426 18:02:13.250216 20300 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0426 18:02:13.250466 20300 server_base.cc:433] Could not load existing FS layout: Not found: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2)
I0426 18:02:13.250630 20300 server_base.cc:434] Attempting to create new FS layout instead
I0426 18:02:13.259886 20300 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e8cb9d247c364de2956352d4368be4f3"
format_stamp: "Formatted at 2018-04-26 18:02:13 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:13.260687 20300 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e8cb9d247c364de2956352d4368be4f3"
format_stamp: "Formatted at 2018-04-26 18:02:13 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:13.270881 20300 fs_manager.cc:495] Time spent creating directory manager: real 0.010s	user 0.008s	sys 0.004s
I0426 18:02:13.272297 20300 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go
I0426 18:02:13.272773 20300 file_cache.cc:470] Constructed file cache lbm with capacity 26214
I0426 18:02:13.279654 20300 fs_manager.cc:417] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.004s
I0426 18:02:13.279789 20300 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/wal
uuid: "e8cb9d247c364de2956352d4368be4f3"
format_stamp: "Formatted at 2018-04-26 18:02:13 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:13.280829 20300 fs_report.cc:347] Block manager report
--------------------
1 data directories: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0426 18:02:13.499577 20300 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go
I0426 18:02:13.502252 20300 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0426 18:02:13.503315 20300 tablet_server_main.cc:84] Starting tablet server...
I0426 18:02:13.608496 20300 rpc_server.cc:200] RPC server started. Bound to: 127.19.158.194:34029
I0426 18:02:13.609426 20300 webserver.cc:173] Starting webserver on 127.19.158.194:0
I0426 18:02:13.609536 20300 webserver.cc:184] Document root disabled
I0426 18:02:13.610909 20300 webserver.cc:311] Webserver started. Bound to: http://127.19.158.194:38362/
I0426 18:02:13.611703 20300 server_base.cc:606] Dumped server information to /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-1/data/info.pb
I0426 18:02:13.614548 20091 external_mini_cluster.cc:881] Started /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver as pid 20300
I0426 18:02:13.615696 20091 external_mini_cluster.cc:819] Running /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver
/tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--rpc_bind_addresses=127.19.158.195:0
--local_ip_for_outbound_sockets=127.19.158.195
--webserver_interface=127.19.158.195
--webserver_port=0
--tserver_master_addrs=127.19.158.254:32815
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I0426 18:02:13.614572 20422 process_memory.cc:182] Process hard memory limit is 11.777631 GB
I0426 18:02:13.617153 20422 process_memory.cc:184] Process soft memory limit is 9.422105 GB
I0426 18:02:13.617282 20422 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB
I0426 18:02:13.617504 20300 tablet_server_main.cc:87] Tablet server successfully started.
I0426 18:02:13.635696 20421 heartbeater.cc:346] Connected to a master server at 127.19.158.254:32815
I0426 18:02:13.636018 20421 heartbeater.cc:426] Registering TS with master...
I0426 18:02:13.636514 20421 heartbeater.cc:475] Master 127.19.158.254:32815 requested a full tablet report, sending...
I0426 18:02:13.638370 20131 ts_manager.cc:81] Registered new tserver with Master: e8cb9d247c364de2956352d4368be4f3 (127.19.158.194:34029)
I0426 18:02:13.639716 20131 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.19.158.194:55036
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0426 18:02:13.733669 20425 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0426 18:02:13.734097 20425 flags.cc:412] Enabled unsafe flag: --never_fsync=true
W0426 18:02:13.743257 20425 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024
W0426 18:02:13.743616 20425 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.158.195
I0426 18:02:13.759395 20425 tablet_server_main.cc:72] Tablet server non-default flags:
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.19.158.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.158.195
--webserver_port=0
--tserver_master_addrs=127.19.158.254:32815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.158.195
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision c9307f8eb89fd99c1492b5e53812f2cdc1adffb4
build type FASTDEBUG
built by None at 26 Apr 2018 18:00:48 UTC on jenkins-slave-vn0g
build id 13197
TSAN enabled
I0426 18:02:13.768307 20425 tablet_server_main.cc:79] Initializing tablet server...
I0426 18:02:13.770530 20425 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 14547us
I0426 18:02:13.771080 20425 fs_manager.cc:260] Metadata directory not provided
I0426 18:02:13.771211 20425 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0426 18:02:13.771442 20425 server_base.cc:433] Could not load existing FS layout: Not found: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2)
I0426 18:02:13.771569 20425 server_base.cc:434] Attempting to create new FS layout instead
I0426 18:02:13.779753 20425 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "667824c990864570a89a4c4f2c4e5ab5"
format_stamp: "Formatted at 2018-04-26 18:02:13 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:13.780661 20425 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "667824c990864570a89a4c4f2c4e5ab5"
format_stamp: "Formatted at 2018-04-26 18:02:13 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:13.791282 20425 fs_manager.cc:495] Time spent creating directory manager: real 0.010s	user 0.004s	sys 0.008s
I0426 18:02:13.792649 20425 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go
I0426 18:02:13.793131 20425 file_cache.cc:470] Constructed file cache lbm with capacity 26214
I0426 18:02:13.800127 20425 fs_manager.cc:417] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.000s
I0426 18:02:13.800271 20425 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/wal
uuid: "667824c990864570a89a4c4f2c4e5ab5"
format_stamp: "Formatted at 2018-04-26 18:02:13 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:13.800509 20425 fs_report.cc:347] Block manager report
--------------------
1 data directories: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0426 18:02:13.992527 20425 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go
I0426 18:02:13.995317 20425 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0426 18:02:13.996407 20425 tablet_server_main.cc:84] Starting tablet server...
I0426 18:02:14.104907 20425 rpc_server.cc:200] RPC server started. Bound to: 127.19.158.195:44330
I0426 18:02:14.105895 20425 webserver.cc:173] Starting webserver on 127.19.158.195:0
I0426 18:02:14.106009 20425 webserver.cc:184] Document root disabled
I0426 18:02:14.107342 20425 webserver.cc:311] Webserver started. Bound to: http://127.19.158.195:42457/
I0426 18:02:14.108196 20425 server_base.cc:606] Dumped server information to /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/info.pb
I0426 18:02:14.114238 20547 process_memory.cc:182] Process hard memory limit is 11.777631 GB
I0426 18:02:14.114547 20547 process_memory.cc:184] Process soft memory limit is 9.422105 GB
I0426 18:02:14.114681 20547 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB
I0426 18:02:14.115056 20091 external_mini_cluster.cc:881] Started /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver as pid 20425
I0426 18:02:14.121412 20296 heartbeater.cc:467] Master 127.19.158.254:32815 was elected leader, sending a full tablet report...
I0426 18:02:14.114364 20425 tablet_server_main.cc:87] Tablet server successfully started.
I0426 18:02:14.143072 20546 heartbeater.cc:346] Connected to a master server at 127.19.158.254:32815
I0426 18:02:14.143391 20546 heartbeater.cc:426] Registering TS with master...
I0426 18:02:14.143918 20546 heartbeater.cc:475] Master 127.19.158.254:32815 requested a full tablet report, sending...
I0426 18:02:14.145822 20131 ts_manager.cc:81] Registered new tserver with Master: 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330)
I0426 18:02:14.147114 20131 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.19.158.195:59229
I0426 18:02:14.155324 20091 external_mini_cluster.cc:509] 3 TS(s) registered with all masters
I0426 18:02:14.187079 20131 catalog_manager.cc:1316] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35703:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0426 18:02:14.196913 20131 catalog_manager.cc:1472] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0426 18:02:14.234140 20483 tablet_service.cc:756] Processing CreateTablet for tablet cdcca06f278544a7b2cc376ef47fde17 (table=TestTable [id=20d4403e67364c6aa058efb2c94fcb71]), partition=RANGE (key) PARTITION UNBOUNDED
I0426 18:02:14.235191 20483 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cdcca06f278544a7b2cc376ef47fde17. 1 dirs total, 0 dirs full, 0 dirs failed
I0426 18:02:14.238070 20233 tablet_service.cc:756] Processing CreateTablet for tablet cdcca06f278544a7b2cc376ef47fde17 (table=TestTable [id=20d4403e67364c6aa058efb2c94fcb71]), partition=RANGE (key) PARTITION UNBOUNDED
I0426 18:02:14.238284 20358 tablet_service.cc:756] Processing CreateTablet for tablet cdcca06f278544a7b2cc376ef47fde17 (table=TestTable [id=20d4403e67364c6aa058efb2c94fcb71]), partition=RANGE (key) PARTITION UNBOUNDED
I0426 18:02:14.239253 20233 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cdcca06f278544a7b2cc376ef47fde17. 1 dirs total, 0 dirs full, 0 dirs failed
I0426 18:02:14.239549 20358 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cdcca06f278544a7b2cc376ef47fde17. 1 dirs total, 0 dirs full, 0 dirs failed
I0426 18:02:14.251497 20483 ts_tablet_manager.cc:1173] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Registered tablet (data state: TABLET_DATA_READY)
I0426 18:02:14.253331 20566 ts_tablet_manager.cc:1028] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Bootstrapping tablet
I0426 18:02:14.254082 20566 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Bootstrap starting.
I0426 18:02:14.257663 20566 tablet_bootstrap.cc:587] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: No blocks or log segments found. Creating new log.
I0426 18:02:14.259575 20566 log.cc:520] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Log is configured to *not* fsync() on all Append() calls
I0426 18:02:14.262907 20566 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: No bootstrap required, opened a new log
I0426 18:02:14.263509 20566 ts_tablet_manager.cc:1045] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I0426 18:02:14.264040 20358 ts_tablet_manager.cc:1173] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Registered tablet (data state: TABLET_DATA_READY)
I0426 18:02:14.274152 20233 ts_tablet_manager.cc:1173] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Registered tablet (data state: TABLET_DATA_READY)
I0426 18:02:14.276777 20568 ts_tablet_manager.cc:1028] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Bootstrapping tablet
I0426 18:02:14.277443 20568 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Bootstrap starting.
I0426 18:02:14.280378 20568 tablet_bootstrap.cc:587] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: No blocks or log segments found. Creating new log.
I0426 18:02:14.280479 20566 raft_consensus.cc:303] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.281390 20566 raft_consensus.cc:329] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0426 18:02:14.281605 20566 raft_consensus.cc:604] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667824c990864570a89a4c4f2c4e5ab5, State: Initialized, Role: FOLLOWER
I0426 18:02:14.281891 20568 log.cc:520] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Log is configured to *not* fsync() on all Append() calls
I0426 18:02:14.282387 20566 consensus_queue.cc:227] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.284567 20569 ts_tablet_manager.cc:1028] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Bootstrapping tablet
I0426 18:02:14.285219 20569 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Bootstrap starting.
I0426 18:02:14.288468 20569 tablet_bootstrap.cc:587] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: No blocks or log segments found. Creating new log.
I0426 18:02:14.289912 20569 log.cc:520] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Log is configured to *not* fsync() on all Append() calls
I0426 18:02:14.291940 20546 heartbeater.cc:467] Master 127.19.158.254:32815 was elected leader, sending a full tablet report...
I0426 18:02:14.292465 20566 ts_tablet_manager.cc:1073] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Time spent starting tablet: real 0.029s	user 0.024s	sys 0.004s
I0426 18:02:14.292706 20568 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: No bootstrap required, opened a new log
I0426 18:02:14.293184 20568 ts_tablet_manager.cc:1045] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.000s
I0426 18:02:14.295348 20569 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: No bootstrap required, opened a new log
I0426 18:02:14.295907 20569 ts_tablet_manager.cc:1045] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.004s
I0426 18:02:14.311352 20568 raft_consensus.cc:303] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.312263 20568 raft_consensus.cc:329] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0426 18:02:14.312481 20568 raft_consensus.cc:604] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8cb9d247c364de2956352d4368be4f3, State: Initialized, Role: FOLLOWER
I0426 18:02:14.313243 20569 raft_consensus.cc:303] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.313241 20568 consensus_queue.cc:227] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.314180 20569 raft_consensus.cc:329] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0426 18:02:14.314438 20569 raft_consensus.cc:604] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849e2ce84972434a997a3dfa1f77ae22, State: Initialized, Role: FOLLOWER
I0426 18:02:14.315445 20569 consensus_queue.cc:227] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.316835 20421 heartbeater.cc:467] Master 127.19.158.254:32815 was elected leader, sending a full tablet report...
I0426 18:02:14.318500 20568 ts_tablet_manager.cc:1073] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Time spent starting tablet: real 0.025s	user 0.020s	sys 0.004s
I0426 18:02:14.319715 20569 ts_tablet_manager.cc:1073] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Time spent starting tablet: real 0.024s	user 0.016s	sys 0.008s
W0426 18:02:14.345257 20297 tablet.cc:1678] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0426 18:02:14.365144 20547 tablet.cc:1678] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0426 18:02:14.367998 20422 tablet.cc:1678] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0426 18:02:14.368721 20571 raft_consensus.cc:435] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0426 18:02:14.369001 20571 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Snoozing failure detection for 1.847s (starting election)
I0426 18:02:14.369236 20571 raft_consensus.cc:457] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.370364 20571 leader_election.cc:230] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e8cb9d247c364de2956352d4368be4f3
I0426 18:02:14.370854 20571 leader_election.cc:230] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 849e2ce84972434a997a3dfa1f77ae22
I0426 18:02:14.381652 20378 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "cdcca06f278544a7b2cc376ef47fde17" candidate_uuid: "667824c990864570a89a4c4f2c4e5ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8cb9d247c364de2956352d4368be4f3" is_pre_election: true
I0426 18:02:14.382345 20378 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Snoozing failure detection for 1.990s (vote granted)
I0426 18:02:14.382639 20378 raft_consensus.cc:2214] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667824c990864570a89a4c4f2c4e5ab5 in term 0.
I0426 18:02:14.382711 20253 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "cdcca06f278544a7b2cc376ef47fde17" candidate_uuid: "667824c990864570a89a4c4f2c4e5ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849e2ce84972434a997a3dfa1f77ae22" is_pre_election: true
I0426 18:02:14.383342 20253 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Snoozing failure detection for 1.783s (vote granted)
I0426 18:02:14.383558 20253 raft_consensus.cc:2214] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667824c990864570a89a4c4f2c4e5ab5 in term 0.
I0426 18:02:14.383882 20433 leader_election.cc:362] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 pre-election: Vote granted by peer e8cb9d247c364de2956352d4368be4f3
I0426 18:02:14.384081 20433 leader_election.cc:257] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0426 18:02:14.384575 20435 leader_election.cc:362] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 849e2ce84972434a997a3dfa1f77ae22
I0426 18:02:14.384584 20571 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Snoozing failure detection for 1.929s (election complete)
I0426 18:02:14.384896 20571 raft_consensus.cc:2506] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0426 18:02:14.385104 20571 raft_consensus.cc:435] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0426 18:02:14.385280 20571 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Snoozing failure detection for 1.558s (starting election)
I0426 18:02:14.385442 20571 raft_consensus.cc:2751] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 0 FOLLOWER]: Advancing to term 1
I0426 18:02:14.388785 20571 raft_consensus.cc:457] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.389542 20571 leader_election.cc:230] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 election: Requesting vote from peer e8cb9d247c364de2956352d4368be4f3
I0426 18:02:14.389896 20571 leader_election.cc:230] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 election: Requesting vote from peer 849e2ce84972434a997a3dfa1f77ae22
I0426 18:02:14.391108 20253 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "cdcca06f278544a7b2cc376ef47fde17" candidate_uuid: "667824c990864570a89a4c4f2c4e5ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849e2ce84972434a997a3dfa1f77ae22"
I0426 18:02:14.391088 20378 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "cdcca06f278544a7b2cc376ef47fde17" candidate_uuid: "667824c990864570a89a4c4f2c4e5ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8cb9d247c364de2956352d4368be4f3"
I0426 18:02:14.391474 20253 raft_consensus.cc:2751] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 0 FOLLOWER]: Advancing to term 1
I0426 18:02:14.391496 20378 raft_consensus.cc:2751] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 0 FOLLOWER]: Advancing to term 1
I0426 18:02:14.391685 20253 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Snoozing failure detection for 2.201s (vote granted)
I0426 18:02:14.391723 20378 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Snoozing failure detection for 1.518s (vote granted)
I0426 18:02:14.395119 20253 raft_consensus.cc:2214] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667824c990864570a89a4c4f2c4e5ab5 in term 1.
I0426 18:02:14.395923 20378 raft_consensus.cc:2214] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667824c990864570a89a4c4f2c4e5ab5 in term 1.
I0426 18:02:14.396093 20435 leader_election.cc:362] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 election: Vote granted by peer 849e2ce84972434a997a3dfa1f77ae22
I0426 18:02:14.396289 20435 leader_election.cc:257] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0426 18:02:14.396751 20571 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Snoozing failure detection for 1.836s (election complete)
I0426 18:02:14.396967 20571 raft_consensus.cc:2506] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 1 FOLLOWER]: Leader election won for term 1
I0426 18:02:14.396976 20433 leader_election.cc:362] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [CANDIDATE]: Term 1 election: Vote granted by peer e8cb9d247c364de2956352d4368be4f3
I0426 18:02:14.398319 20571 raft_consensus.cc:571] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 667824c990864570a89a4c4f2c4e5ab5, State: Running, Role: LEADER
I0426 18:02:14.398808 20571 consensus_queue.cc:204] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:14.405359 20131 catalog_manager.cc:3716] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195). New cstate: current_term: 1 leader_uuid: "667824c990864570a89a4c4f2c4e5ab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } health_report { overall_health: UNKNOWN } } }
I0426 18:02:14.440295 20091 external_mini_cluster.cc:509] 3 TS(s) registered with all masters
I0426 18:02:14.443260 20091 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 849e2ce84972434a997a3dfa1f77ae22 to finish bootstrapping
I0426 18:02:14.455464 20091 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e8cb9d247c364de2956352d4368be4f3 to finish bootstrapping
I0426 18:02:14.466791 20091 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 667824c990864570a89a4c4f2c4e5ab5 to finish bootstrapping
I0426 18:02:14.513281 20378 raft_consensus.cc:1057] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 1 FOLLOWER]: Refusing update from remote peer 667824c990864570a89a4c4f2c4e5ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0426 18:02:14.515671 20576 consensus_queue.cc:963] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0426 18:02:14.516553 20253 raft_consensus.cc:1057] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 1 FOLLOWER]: Refusing update from remote peer 667824c990864570a89a4c4f2c4e5ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0426 18:02:14.518234 20576 consensus_queue.cc:963] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 08f726b6ca8e483486158eacd80e933a | 127.19.158.254:32815 | HEALTHY
Fetched info from all 1 masters
Connected to the leader master
Tablet Server Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 667824c990864570a89a4c4f2c4e5ab5 | 127.19.158.195:44330 | HEALTHY
 849e2ce84972434a997a3dfa1f77ae22 | 127.19.158.193:39806 | HEALTHY
 e8cb9d247c364de2956352d4368be4f3 | 127.19.158.194:34029 | HEALTHY
Fetched info from all 3 tablet servers
Table TestTable is HEALTHY (1 tablet(s) checked)

Summary by table
   Name    | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+---------+---------------+---------+------------+------------------+-------------
 TestTable | HEALTHY | 1             | 1       | 0          | 0                | 0
The metadata for 1 table(s) is HEALTHY
Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 150 rows summed)
-----------------------
TestTable
-----------------------
T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 (127.19.158.194:34029): Checksum: 108787617684
T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 (127.19.158.193:39806): Checksum: 108787617684
T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330): Checksum: 108787617684

I0426 18:02:14.745784 20091 log_verifier.cc:121] Checking tablet cdcca06f278544a7b2cc376ef47fde17
I0426 18:02:14.866740 20091 log_verifier.cc:172] Verified matching terms for 6 ops in tablet cdcca06f278544a7b2cc376ef47fde17
I0426 18:02:15.092761 20091 external_mini_cluster.cc:1011] Killing /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver with pid 20425
W0426 18:02:15.156270 20111 connection.cc:512] client connection to 127.19.158.195:44330 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.195:44330): Connection reset by peer (error 104)
W0426 18:02:15.158399 20310 connection.cc:512] server connection from 127.19.158.195:42376 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.195:42376): Connection reset by peer (error 104)
W0426 18:02:15.162487 20112 connection.cc:512] server connection from 127.19.158.195:59229 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.195:59229): Connection reset by peer (error 104)
W0426 18:02:15.165395 20184 connection.cc:512] server connection from 127.19.158.195:43998 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.195:43998): Connection reset by peer (error 104)
I0426 18:02:15.166775 20253 tablet_service.cc:1106] Received Run Leader Election RPC: tablet_id: "cdcca06f278544a7b2cc376ef47fde17"
dest_uuid: "849e2ce84972434a997a3dfa1f77ae22"
 from {username='slave'} at 127.0.0.1:58542
I0426 18:02:15.167244 20253 raft_consensus.cc:435] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I0426 18:02:15.167485 20253 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Snoozing failure detection for 1.936s (starting election)
I0426 18:02:15.167692 20253 raft_consensus.cc:2751] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 1 FOLLOWER]: Advancing to term 2
I0426 18:02:15.173108 20253 raft_consensus.cc:457] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:15.182974 20253 leader_election.cc:230] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [CANDIDATE]: Term 2 election: Requesting vote from peer e8cb9d247c364de2956352d4368be4f3
I0426 18:02:15.189709 20253 leader_election.cc:230] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [CANDIDATE]: Term 2 election: Requesting vote from peer 667824c990864570a89a4c4f2c4e5ab5
W0426 18:02:15.157677 20551 connection.cc:512] client connection to 127.19.158.195:44330 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.195:44330): Connection reset by peer (error 104)
W0426 18:02:15.207605 20183 leader_election.cc:286] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 667824c990864570a89a4c4f2c4e5ab5: Network error: Client connection negotiation failed: client connection to 127.19.158.195:44330: connect: Connection refused (error 111)
I0426 18:02:15.209192 20378 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "cdcca06f278544a7b2cc376ef47fde17" candidate_uuid: "849e2ce84972434a997a3dfa1f77ae22" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e8cb9d247c364de2956352d4368be4f3"
I0426 18:02:15.209715 20378 raft_consensus.cc:2751] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 1 FOLLOWER]: Advancing to term 2
I0426 18:02:15.209977 20378 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3: Snoozing failure detection for 1.726s (vote granted)
W0426 18:02:15.163483 20102 connection.cc:512] client connection to 127.19.158.195:44330 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.195:44330): Connection reset by peer (error 104)
I0426 18:02:15.215559 20378 raft_consensus.cc:2214] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849e2ce84972434a997a3dfa1f77ae22 in term 2.
W0426 18:02:15.164324 20097 connection.cc:512] client connection to 127.19.158.195:44330 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.195:44330): Connection reset by peer (error 104)
I0426 18:02:15.216960 20183 leader_election.cc:362] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [CANDIDATE]: Term 2 election: Vote granted by peer e8cb9d247c364de2956352d4368be4f3
I0426 18:02:15.217273 20183 leader_election.cc:257] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won.
W0426 18:02:15.218904 20551 meta_cache.cc:216] Tablet cdcca06f278544a7b2cc376ef47fde17: Replica 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330) has failed: Network error: Client connection negotiation failed: client connection to 127.19.158.195:44330: connect: Connection refused (error 111)
I0426 18:02:15.230994 20622 raft_consensus.cc:2699] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22: Snoozing failure detection for 2.212s (election complete)
I0426 18:02:15.231458 20622 raft_consensus.cc:2506] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 2 FOLLOWER]: Leader election won for term 2
I0426 18:02:15.246575 20622 raft_consensus.cc:571] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [term 2 LEADER]: Becoming Leader. State: Replica: 849e2ce84972434a997a3dfa1f77ae22, State: Running, Role: LEADER
I0426 18:02:15.247300 20622 consensus_queue.cc:204] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:15.252295 20131 catalog_manager.cc:3716] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 reported cstate change: term changed from 1 to 2, leader changed from 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195) to 849e2ce84972434a997a3dfa1f77ae22 (127.19.158.193). New cstate: current_term: 2 leader_uuid: "849e2ce84972434a997a3dfa1f77ae22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } health_report { overall_health: HEALTHY } } }
W0426 18:02:15.297312 20183 consensus_peers.cc:434] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 -> Peer 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330): Couldn't send request to peer 667824c990864570a89a4c4f2c4e5ab5 for tablet cdcca06f278544a7b2cc376ef47fde17. Status: Network error: Client connection negotiation failed: client connection to 127.19.158.195:44330: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times.
I0426 18:02:15.298542 20378 raft_consensus.cc:1057] T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 [term 2 FOLLOWER]: Refusing update from remote peer 849e2ce84972434a997a3dfa1f77ae22: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I0426 18:02:15.300113 20624 consensus_queue.cc:963] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I0426 18:02:15.418908 20091 external_mini_cluster.cc:819] Running /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver
/tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--rpc_bind_addresses=127.19.158.195:44330
--local_ip_for_outbound_sockets=127.19.158.195
--webserver_port=42457
--webserver_interface=127.19.158.195
--tserver_master_addrs=127.19.158.254:32815
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0426 18:02:15.600291 20634 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0426 18:02:15.600889 20634 flags.cc:412] Enabled unsafe flag: --never_fsync=true
W0426 18:02:15.616897 20634 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024
W0426 18:02:15.617478 20634 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.158.195
I0426 18:02:15.641549 20634 tablet_server_main.cc:72] Tablet server non-default flags:
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.19.158.195:44330
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.158.195
--webserver_port=42457
--tserver_master_addrs=127.19.158.254:32815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.158.195
--log_dir=/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision c9307f8eb89fd99c1492b5e53812f2cdc1adffb4
build type FASTDEBUG
built by None at 26 Apr 2018 18:00:48 UTC on jenkins-slave-vn0g
build id 13197
TSAN enabled
I0426 18:02:15.655854 20634 tablet_server_main.cc:79] Initializing tablet server...
I0426 18:02:15.658774 20634 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 15547us
I0426 18:02:15.659525 20634 fs_manager.cc:260] Metadata directory not provided
I0426 18:02:15.659799 20634 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0426 18:02:15.668844 20634 fs_manager.cc:397] Time spent opening directory manager: real 0.006s	user 0.004s	sys 0.000s
I0426 18:02:15.669895 20634 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go
I0426 18:02:15.670699 20634 file_cache.cc:470] Constructed file cache lbm with capacity 26214
I0426 18:02:15.682560 20634 fs_manager.cc:417] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
I0426 18:02:15.682811 20634 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/wal
uuid: "667824c990864570a89a4c4f2c4e5ab5"
format_stamp: "Formatted at 2018-04-26 18:02:13 on dist-test-slave-dist-test-slave-qvfl"
I0426 18:02:15.685031 20634 fs_report.cc:347] Block manager report
--------------------
1 data directories: /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
W0426 18:02:15.743782 20183 consensus_peers.cc:434] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 -> Peer 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330): Couldn't send request to peer 667824c990864570a89a4c4f2c4e5ab5 for tablet cdcca06f278544a7b2cc376ef47fde17. Status: Network error: Client connection negotiation failed: client connection to 127.19.158.195:44330: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times.
I0426 18:02:15.980309 20634 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go
I0426 18:02:15.984465 20634 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete)
I0426 18:02:15.984674 20634 ts_tablet_manager.cc:1005] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Loading tablet metadata
I0426 18:02:15.992148 20634 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets)
I0426 18:02:15.997766 20634 ts_tablet_manager.cc:1173] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Registered tablet (data state: TABLET_DATA_READY)
I0426 18:02:16.002583 20651 ts_tablet_manager.cc:1028] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Bootstrapping tablet
I0426 18:02:16.003321 20651 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Bootstrap starting.
I0426 18:02:16.042874 20651 log.cc:520] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Log is configured to *not* fsync() on all Append() calls
I0426 18:02:16.015276 20634 tablet_server_main.cc:84] Starting tablet server...
I0426 18:02:16.113045 20651 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I0426 18:02:16.116505 20651 tablet_bootstrap.cc:436] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Bootstrap complete.
I0426 18:02:16.118551 20651 ts_tablet_manager.cc:1045] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Time spent bootstrapping tablet: real 0.116s	user 0.064s	sys 0.012s
I0426 18:02:16.148969 20651 raft_consensus.cc:303] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:16.158123 20651 raft_consensus.cc:604] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667824c990864570a89a4c4f2c4e5ab5, State: Initialized, Role: FOLLOWER
I0426 18:02:16.159394 20651 consensus_queue.cc:227] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8cb9d247c364de2956352d4368be4f3" member_type: VOTER last_known_addr { host: "127.19.158.194" port: 34029 } } peers { permanent_uuid: "667824c990864570a89a4c4f2c4e5ab5" member_type: VOTER last_known_addr { host: "127.19.158.195" port: 44330 } } peers { permanent_uuid: "849e2ce84972434a997a3dfa1f77ae22" member_type: VOTER last_known_addr { host: "127.19.158.193" port: 39806 } }
I0426 18:02:16.165856 20651 ts_tablet_manager.cc:1073] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5: Time spent starting tablet: real 0.047s	user 0.028s	sys 0.012s
W0426 18:02:16.272392 20183 consensus_peers.cc:434] T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 -> Peer 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330): Couldn't send request to peer 667824c990864570a89a4c4f2c4e5ab5 for tablet cdcca06f278544a7b2cc376ef47fde17. Status: Network error: Client connection negotiation failed: client connection to 127.19.158.195:44330: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times.
I0426 18:02:16.385583 20634 rpc_server.cc:200] RPC server started. Bound to: 127.19.158.195:44330
I0426 18:02:16.387074 20634 webserver.cc:173] Starting webserver on 127.19.158.195:42457
I0426 18:02:16.387246 20634 webserver.cc:184] Document root disabled
I0426 18:02:16.389330 20634 webserver.cc:311] Webserver started. Bound to: http://127.19.158.195:42457/
I0426 18:02:16.390461 20634 server_base.cc:606] Dumped server information to /tmp/dist-test-taskEpRJ5y/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1524765731449908-20091/raft_consensus-itest-cluster/ts-2/data/info.pb
I0426 18:02:16.399819 20091 external_mini_cluster.cc:881] Started /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver as pid 20634
I0426 18:02:16.403182 20634 tablet_server_main.cc:87] Tablet server successfully started.
I0426 18:02:16.404460 20760 process_memory.cc:182] Process hard memory limit is 11.777631 GB
I0426 18:02:16.405865 20760 process_memory.cc:184] Process soft memory limit is 9.422105 GB
I0426 18:02:16.406112 20760 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB
I0426 18:02:16.420740 20759 heartbeater.cc:346] Connected to a master server at 127.19.158.254:32815
I0426 18:02:16.421133 20759 heartbeater.cc:426] Registering TS with master...
I0426 18:02:16.421840 20759 heartbeater.cc:475] Master 127.19.158.254:32815 requested a full tablet report, sending...
I0426 18:02:16.424929 20131 ts_manager.cc:87] Re-registered known tserver with Master: 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330)
I0426 18:02:16.427646 20131 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.19.158.195:44078
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 08f726b6ca8e483486158eacd80e933a | 127.19.158.254:32815 | HEALTHY
Fetched info from all 1 masters
Connected to the leader master
Tablet Server Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 667824c990864570a89a4c4f2c4e5ab5 | 127.19.158.195:44330 | HEALTHY
 849e2ce84972434a997a3dfa1f77ae22 | 127.19.158.193:39806 | HEALTHY
 e8cb9d247c364de2956352d4368be4f3 | 127.19.158.194:34029 | HEALTHY
Fetched info from all 3 tablet servers
Tablet cdcca06f278544a7b2cc376ef47fde17 of table 'TestTable' is conflicted: Tablet cdcca06f278544a7b2cc376ef47fde17 of table 'TestTable' replicas' active configs disagree with the master's
  667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330): RUNNING
  849e2ce84972434a997a3dfa1f77ae22 (127.19.158.193:39806): RUNNING [LEADER]
  e8cb9d247c364de2956352d4368be4f3 (127.19.158.194:34029): RUNNING
  All the peers reported by the master and tablet servers are:
  A = 667824c990864570a89a4c4f2c4e5ab5
  B = 849e2ce84972434a997a3dfa1f77ae22
  C = e8cb9d247c364de2956352d4368be4f3
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A   B*  C    |              |              | Yes
 A             | A   B   C    | 1            | -1           | Yes
 B             | A   B*  C    | 2            | -1           | Yes
 C             | A   B*  C    | 2            | -1           | Yes

Table TestTable has 1 tablet(s) with mismatched consensus

Summary by table
   Name    |   Status    | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+-------------+---------------+---------+------------+------------------+-------------
 TestTable | UNAVAILABLE | 1             | 0       | 0          | 0                | 1
I0426 18:02:16.616920 20091 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Corruption: 1 out of 1 table(s) are not healthy
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 08f726b6ca8e483486158eacd80e933a | 127.19.158.254:32815 | HEALTHY
Fetched info from all 1 masters
Connected to the leader master
I0426 18:02:16.866853 20716 raft_consensus.cc:2751] T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 [term 1 FOLLOWER]: Advancing to term 2
Tablet Server Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 667824c990864570a89a4c4f2c4e5ab5 | 127.19.158.195:44330 | HEALTHY
 849e2ce84972434a997a3dfa1f77ae22 | 127.19.158.193:39806 | HEALTHY
 e8cb9d247c364de2956352d4368be4f3 | 127.19.158.194:34029 | HEALTHY
Fetched info from all 3 tablet servers
Tablet cdcca06f278544a7b2cc376ef47fde17 of table 'TestTable' is conflicted: Tablet cdcca06f278544a7b2cc376ef47fde17 of table 'TestTable' replicas' active configs disagree with the master's
  667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330): RUNNING
  849e2ce84972434a997a3dfa1f77ae22 (127.19.158.193:39806): RUNNING [LEADER]
  e8cb9d247c364de2956352d4368be4f3 (127.19.158.194:34029): RUNNING
  All the peers reported by the master and tablet servers are:
  A = 667824c990864570a89a4c4f2c4e5ab5
  B = 849e2ce84972434a997a3dfa1f77ae22
  C = e8cb9d247c364de2956352d4368be4f3
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A   B*  C    |              |              | Yes
 A             | A   B   C    | 1            | -1           | Yes
 B             | A   B*  C    | 2            | -1           | Yes
 C             | A   B*  C    | 2            | -1           | Yes

Table TestTable has 1 tablet(s) with mismatched consensus

Summary by table
I0426 18:02:16.872921 20759 heartbeater.cc:467] Master 127.19.158.254:32815 was elected leader, sending a full tablet report...
   Name    |   Status    | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+-------------+---------------+---------+------------+------------------+-------------
 TestTable | UNAVAILABLE | 1             | 0       | 0          | 0                | 1
I0426 18:02:16.883785 20091 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Corruption: 1 out of 1 table(s) are not healthy
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 08f726b6ca8e483486158eacd80e933a | 127.19.158.254:32815 | HEALTHY
Fetched info from all 1 masters
Connected to the leader master
Tablet Server Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 667824c990864570a89a4c4f2c4e5ab5 | 127.19.158.195:44330 | HEALTHY
 849e2ce84972434a997a3dfa1f77ae22 | 127.19.158.193:39806 | HEALTHY
 e8cb9d247c364de2956352d4368be4f3 | 127.19.158.194:34029 | HEALTHY
Fetched info from all 3 tablet servers
Table TestTable is HEALTHY (1 tablet(s) checked)

Summary by table
   Name    | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+---------+---------------+---------+------------+------------------+-------------
 TestTable | HEALTHY | 1             | 1       | 0          | 0                | 0
The metadata for 1 table(s) is HEALTHY
Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 300 rows summed)
-----------------------
TestTable
-----------------------
T cdcca06f278544a7b2cc376ef47fde17 P 849e2ce84972434a997a3dfa1f77ae22 (127.19.158.193:39806): Checksum: 214634478652
T cdcca06f278544a7b2cc376ef47fde17 P e8cb9d247c364de2956352d4368be4f3 (127.19.158.194:34029): Checksum: 214634478652
T cdcca06f278544a7b2cc376ef47fde17 P 667824c990864570a89a4c4f2c4e5ab5 (127.19.158.195:44330): Checksum: 214634478652

I0426 18:02:17.218503 20091 log_verifier.cc:121] Checking tablet cdcca06f278544a7b2cc376ef47fde17
I0426 18:02:17.296936 20091 log_verifier.cc:172] Verified matching terms for 12 ops in tablet cdcca06f278544a7b2cc376ef47fde17
==================
WARNING: ThreadSanitizer: data race (pid=20091)
  Write of size 8 at 0x7b2c00001558 by main thread:
    #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (raft_consensus_election-itest+0x4cc9d1)
    #1 std::__1::default_delete<kudu::ThreadPoolToken>::operator()(kudu::ThreadPoolToken*) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libkudu_util.so+0x1c42f3)
    #2 std::__1::unique_ptr<kudu::ThreadPoolToken, std::__1::default_delete<kudu::ThreadPoolToken> >::reset(kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:2598 (libkudu_util.so+0x1c42f3)
    #3 std::__1::unique_ptr<kudu::ThreadPoolToken, std::__1::default_delete<kudu::ThreadPoolToken> >::~unique_ptr() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:2552 (libkudu_util.so+0x1c42f3)
    #4 kudu::ThreadPool::~ThreadPool() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:342 (libkudu_util.so+0x1c42f3)
    #5 kudu::DefaultDeleter<kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:145:5 (libksck.so+0x4e4ae)
    #6 kudu::internal::gscoped_ptr_impl<kudu::ThreadPool, kudu::DefaultDeleter<kudu::ThreadPool> >::~gscoped_ptr_impl() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:228:7 (libksck.so+0x4e479)
    #7 gscoped_ptr<kudu::ThreadPool, kudu::DefaultDeleter<kudu::ThreadPool> >::~gscoped_ptr() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:318:7 (libksck.so+0x48269)
    #8 kudu::DnsResolver::~DnsResolver() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/net/dns_resolver.cc:47:1 (libkudu_util.so+0x18a9fa)
    #9 kudu::DefaultDeleter<kudu::DnsResolver>::operator()(kudu::DnsResolver*) const /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:145:5 (libkudu_client.so+0xd568e)
    #10 kudu::internal::gscoped_ptr_impl<kudu::DnsResolver, kudu::DefaultDeleter<kudu::DnsResolver> >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:254:7 (libkudu_client.so+0xd5644)
    #11 gscoped_ptr<kudu::DnsResolver, kudu::DefaultDeleter<kudu::DnsResolver> >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:375:46 (libkudu_client.so+0xca410)
    #12 kudu::client::KuduClient::Data::~Data() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/client-internal.cc:370:17 (libkudu_client.so+0xdaffd)
    #13 kudu::client::KuduClient::~KuduClient() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/client.cc:378:3 (libkudu_client.so+0xbff7c)
    #14 std::__1::default_delete<kudu::client::KuduClient>::operator()(kudu::client::KuduClient*) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libkudu_client.so+0xd504b)
    #15 std::__1::__shared_ptr_pointer<kudu::client::KuduClient*, std::__1::default_delete<kudu::client::KuduClient>, std::__1::allocator<kudu::client::KuduClient> >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:3586 (libkudu_client.so+0xd504b)
    #16 std::__1::__shared_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:3490:9 (raft_consensus_election-itest+0x4daeae)
    #17 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:3532 (raft_consensus_election-itest+0x4daeae)
    #18 std::__1::shared_ptr<kudu::client::KuduClient>::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:4468 (raft_consensus_election-itest+0x4daeae)
    #19 kudu::tserver::TabletServerIntegrationTestBase::~TabletServerIntegrationTestBase() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/ts_itest-base.h:46:7 (raft_consensus_election-itest+0x4dad9f)
    #20 kudu::tserver::RaftConsensusITestBase::~RaftConsensusITestBase() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/raft_consensus-itest-base.h:43:7 (raft_consensus_election-itest+0x4da1af)
    #21 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::~RaftConsensusElectionITest_RunLeaderElection_Test() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/raft_consensus_election-itest.cc:143:1 (raft_consensus_election-itest+0x4da1d9)
    #22 testing::Test::DeleteSelf_() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/include/gtest/gtest.h:453:24 (libgmock.so+0x55a07)
    #23 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef)
    #24 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef)
    #25 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2661:3 (libgmock.so+0x357b8)
    #26 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226)
    #27 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa)
    #28 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f)
    #29 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f)
    #30 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2)
    #31 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x340b)
    #32 main /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2bd6)

  Previous read of size 8 at 0x7b2c00001558 by thread T9:
    #0 scoped_refptr<kudu::Histogram>::operator kudu::Histogram* scoped_refptr<kudu::Histogram>::*() const /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/ref_counted.h:280:38 (libkrpc.so+0xc2e39)
    #1 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr<kudu::Runnable>, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:551:7 (libkudu_util.so+0x1c3180)
    #2 kudu::ThreadPool::Submit(std::__1::shared_ptr<kudu::Runnable>) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c4c5f)
    #3 kudu::ThreadPool::SubmitFunc(boost::function<void ()>) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:454:10 (libkudu_util.so+0x1c4cf1)
    #4 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector<kudu::Sockaddr, std::__1::allocator<kudu::Sockaddr> >*, kudu::Callback<void (kudu::Status const&)> const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/net/dns_resolver.cc:59:21 (libkudu_util.so+0x18aaab)
    #5 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback<void (kudu::Status const&)> const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/meta_cache.cc:144:33 (libkudu_client.so+0xff663)
    #6 kudu::client::internal::MetaCacheServerPicker::PickLeader(kudu::Callback<void (kudu::Status const&, kudu::client::internal::RemoteTabletServer*)> const&, kudu::MonoTime const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/meta_cache.cc:450:11 (libkudu_client.so+0x10176b)
    #7 kudu::rpc::RetriableRpc<kudu::client::internal::RemoteTabletServer, kudu::tserver::WriteRequestPB, kudu::tserver::WriteResponsePB>::SendRpc() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/retriable_rpc.h:144:19 (libkudu_client.so+0xb6988)
    #8 kudu::rpc::RpcRetrier::DelayedRetryCb(kudu::rpc::Rpc*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/rpc.cc:94:10 (libkrpc.so+0xd5438)
    #9 boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>::operator()(kudu::rpc::RpcRetrier*, kudu::rpc::Rpc*, kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:280:29 (libkrpc.so+0xd5e1a)
    #10 void boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>, boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1> >::operator()<boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>, boost::_bi::rrlist1<kudu::Status const&> >(boost::_bi::type<void>, boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>&, boost::_bi::rrlist1<kudu::Status const&>&, int) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/bind.hpp:398:9 (libkrpc.so+0xd5d52)
    #11 void boost::_bi::bind_t<void, boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>, boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>, boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1> > >::operator()<kudu::Status const&>(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1234:16 (libkrpc.so+0xd5c78)
    #12 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>, boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>, boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1> > >, void, kudu::Status const&>::invoke(boost::detail::function::function_buffer&, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xd5a13)
    #13 boost::function1<void, kudu::Status const&>::operator()(kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xc4014)
    #14 kudu::rpc::DelayedTask::TimerHandler(ev::timer&, int) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:705:5 (libkrpc.so+0xc18a6)
    #15 void ev::base<ev_timer, ev::timer>::method_thunk<kudu::rpc::DelayedTask, &kudu::rpc::DelayedTask::TimerHandler>(ev_loop*, ev_timer*, int) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc970a)
    #16 ev_invoke_pending /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9800)
    #17 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbdaa5)
    #18 ev_run /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa90d)
    #19 ev::loop_ref::run(int) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc3708)
    #20 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:471:9 (libkrpc.so+0xbdbea)
    #21 boost::_mfi::mf0<void, kudu::rpc::ReactorThread>::operator()(kudu::rpc::ReactorThread*) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc79a9)
    #22 void boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc78fa)
    #23 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc7883)
    #24 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc7679)
    #25 boost::function0<void>::operator()() const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb7781)
    #26 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bcff4)

  As if synchronized via sleep:
    #0 nanosleep /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:355 (raft_consensus_election-itest+0x442f6a)
    #1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/sysinfo.cc:89:10 (libgutil.so+0x61c92)
    #2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/monotime.cc:267:3 (libkudu_util.so+0x1893b6)
    #3 kudu::ClusterVerifier::CheckCluster() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/cluster_verifier.cc:85:5 (libitest_util.so+0x5725d)
    #4 kudu::tserver::TabletServerIntegrationTestBase::AssertAllReplicasAgree(int) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/ts_itest-base.cc:554:3 (libitest_util.so+0x73d9a)
    #5 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/raft_consensus_election-itest.cc:182:3 (raft_consensus_election-itest+0x4cffa3)
    #6 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef)
    #7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef)
    #8 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8)
    #9 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c)
    #10 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226)
    #11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa)
    #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f)
    #13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f)
    #14 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2)
    #15 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x340b)
    #16 main /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2bd6)

  Thread T9 'rpc reactor-205' (tid=20551, running) created by main thread at:
    #0 pthread_create /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (raft_consensus_election-itest+0x43ced6)
    #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1bca1f)
    #2 kudu::Status kudu::Thread::Create<void (kudu::rpc::ReactorThread::*)(), kudu::rpc::ReactorThread*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.h:164:12 (libkrpc.so+0xc2dd5)
    #3 kudu::rpc::ReactorThread::Init() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:168:10 (libkrpc.so+0xbd80e)
    #4 kudu::rpc::Reactor::Init() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:722:18 (libkrpc.so+0xc1cd1)
    #5 kudu::rpc::Messenger::Init() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/messenger.cc:436:5 (libkrpc.so+0xa8402)
    #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr<kudu::rpc::Messenger>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/messenger.cc:199:3 (libkrpc.so+0xa7e5d)
    #7 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr<kudu::client::KuduClient>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/client.cc:332:3 (libkudu_client.so+0xbf3e1)
    #8 kudu::tserver::TabletServerIntegrationTestBase::CreateClient(std::__1::shared_ptr<kudu::client::KuduClient>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/ts_itest-base.cc:520:3 (libitest_util.so+0x73302)
    #9 kudu::tserver::TabletServerIntegrationTestBase::BuildAndStart(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/ts_itest-base.cc:545:3 (libitest_util.so+0x73a1f)
    #10 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/raft_consensus_election-itest.cc:147:3 (raft_consensus_election-itest+0x4cfb63)
    #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef)
    #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8)
    #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c)
    #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226)
    #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa)
    #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f)
    #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f)
    #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2)
    #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x340b)
    #21 main /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2bd6)

SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*)
==================
==================
WARNING: ThreadSanitizer: data race (pid=20091)
  Write of size 8 at 0x7b5000001ba0 by main thread:
    #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (raft_consensus_election-itest+0x4cc9d1)
    #1 kudu::DefaultDeleter<kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:145:5 (libksck.so+0x4e4b6)
    #2 kudu::internal::gscoped_ptr_impl<kudu::ThreadPool, kudu::DefaultDeleter<kudu::ThreadPool> >::~gscoped_ptr_impl() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:228:7 (libksck.so+0x4e479)
    #3 gscoped_ptr<kudu::ThreadPool, kudu::DefaultDeleter<kudu::ThreadPool> >::~gscoped_ptr() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:318:7 (libksck.so+0x48269)
    #4 kudu::DnsResolver::~DnsResolver() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/net/dns_resolver.cc:47:1 (libkudu_util.so+0x18a9fa)
    #5 kudu::DefaultDeleter<kudu::DnsResolver>::operator()(kudu::DnsResolver*) const /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:145:5 (libkudu_client.so+0xd568e)
    #6 kudu::internal::gscoped_ptr_impl<kudu::DnsResolver, kudu::DefaultDeleter<kudu::DnsResolver> >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:254:7 (libkudu_client.so+0xd5644)
    #7 gscoped_ptr<kudu::DnsResolver, kudu::DefaultDeleter<kudu::DnsResolver> >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/gscoped_ptr.h:375:46 (libkudu_client.so+0xca410)
    #8 kudu::client::KuduClient::Data::~Data() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/client-internal.cc:370:17 (libkudu_client.so+0xdaffd)
    #9 kudu::client::KuduClient::~KuduClient() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/client.cc:378:3 (libkudu_client.so+0xbff7c)
    #10 std::__1::default_delete<kudu::client::KuduClient>::operator()(kudu::client::KuduClient*) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libkudu_client.so+0xd504b)
    #11 std::__1::__shared_ptr_pointer<kudu::client::KuduClient*, std::__1::default_delete<kudu::client::KuduClient>, std::__1::allocator<kudu::client::KuduClient> >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:3586 (libkudu_client.so+0xd504b)
    #12 std::__1::__shared_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:3490:9 (raft_consensus_election-itest+0x4daeae)
    #13 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:3532 (raft_consensus_election-itest+0x4daeae)
    #14 std::__1::shared_ptr<kudu::client::KuduClient>::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/memory:4468 (raft_consensus_election-itest+0x4daeae)
    #15 kudu::tserver::TabletServerIntegrationTestBase::~TabletServerIntegrationTestBase() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/ts_itest-base.h:46:7 (raft_consensus_election-itest+0x4dad9f)
    #16 kudu::tserver::RaftConsensusITestBase::~RaftConsensusITestBase() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/raft_consensus-itest-base.h:43:7 (raft_consensus_election-itest+0x4da1af)
    #17 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::~RaftConsensusElectionITest_RunLeaderElection_Test() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/raft_consensus_election-itest.cc:143:1 (raft_consensus_election-itest+0x4da1d9)
    #18 testing::Test::DeleteSelf_() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/include/gtest/gtest.h:453:24 (libgmock.so+0x55a07)
    #19 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef)
    #20 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef)
    #21 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2661:3 (libgmock.so+0x357b8)
    #22 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226)
    #23 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa)
    #24 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f)
    #25 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f)
    #26 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2)
    #27 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x340b)
    #28 main /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2bd6)

  Previous read of size 8 at 0x7b5000001ba0 by thread T9:
    #0 scoped_refptr<kudu::Histogram>::operator kudu::Histogram* scoped_refptr<kudu::Histogram>::*() const /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/ref_counted.h:280:38 (libkrpc.so+0xc2e39)
    #1 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr<kudu::Runnable>, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:548:7 (libkudu_util.so+0x1c3151)
    #2 kudu::ThreadPool::Submit(std::__1::shared_ptr<kudu::Runnable>) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c4c5f)
    #3 kudu::ThreadPool::SubmitFunc(boost::function<void ()>) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:454:10 (libkudu_util.so+0x1c4cf1)
    #4 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector<kudu::Sockaddr, std::__1::allocator<kudu::Sockaddr> >*, kudu::Callback<void (kudu::Status const&)> const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/net/dns_resolver.cc:59:21 (libkudu_util.so+0x18aaab)
    #5 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback<void (kudu::Status const&)> const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/meta_cache.cc:144:33 (libkudu_client.so+0xff663)
    #6 kudu::client::internal::MetaCacheServerPicker::PickLeader(kudu::Callback<void (kudu::Status const&, kudu::client::internal::RemoteTabletServer*)> const&, kudu::MonoTime const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/meta_cache.cc:450:11 (libkudu_client.so+0x10176b)
    #7 kudu::rpc::RetriableRpc<kudu::client::internal::RemoteTabletServer, kudu::tserver::WriteRequestPB, kudu::tserver::WriteResponsePB>::SendRpc() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/retriable_rpc.h:144:19 (libkudu_client.so+0xb6988)
    #8 kudu::rpc::RpcRetrier::DelayedRetryCb(kudu::rpc::Rpc*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/rpc.cc:94:10 (libkrpc.so+0xd5438)
    #9 boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>::operator()(kudu::rpc::RpcRetrier*, kudu::rpc::Rpc*, kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:280:29 (libkrpc.so+0xd5e1a)
    #10 void boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>, boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1> >::operator()<boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>, boost::_bi::rrlist1<kudu::Status const&> >(boost::_bi::type<void>, boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>&, boost::_bi::rrlist1<kudu::Status const&>&, int) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/bind.hpp:398:9 (libkrpc.so+0xd5d52)
    #11 void boost::_bi::bind_t<void, boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>, boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>, boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1> > >::operator()<kudu::Status const&>(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1234:16 (libkrpc.so+0xd5c78)
    #12 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>, boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>, boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1> > >, void, kudu::Status const&>::invoke(boost::detail::function::function_buffer&, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xd5a13)
    #13 boost::function1<void, kudu::Status const&>::operator()(kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xc4014)
    #14 kudu::rpc::DelayedTask::TimerHandler(ev::timer&, int) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:705:5 (libkrpc.so+0xc18a6)
    #15 void ev::base<ev_timer, ev::timer>::method_thunk<kudu::rpc::DelayedTask, &kudu::rpc::DelayedTask::TimerHandler>(ev_loop*, ev_timer*, int) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc970a)
    #16 ev_invoke_pending /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9800)
    #17 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbdaa5)
    #18 ev_run /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa90d)
    #19 ev::loop_ref::run(int) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc3708)
    #20 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:471:9 (libkrpc.so+0xbdbea)
    #21 boost::_mfi::mf0<void, kudu::rpc::ReactorThread>::operator()(kudu::rpc::ReactorThread*) const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc79a9)
    #22 void boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc78fa)
    #23 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc7883)
    #24 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc7679)
    #25 boost::function0<void>::operator()() const /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb7781)
    #26 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bcff4)

  As if synchronized via sleep:
    #0 nanosleep /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:355 (raft_consensus_election-itest+0x442f6a)
    #1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/gutil/sysinfo.cc:89:10 (libgutil.so+0x61c92)
    #2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/monotime.cc:267:3 (libkudu_util.so+0x1893b6)
    #3 kudu::ClusterVerifier::CheckCluster() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/cluster_verifier.cc:85:5 (libitest_util.so+0x5725d)
    #4 kudu::tserver::TabletServerIntegrationTestBase::AssertAllReplicasAgree(int) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/ts_itest-base.cc:554:3 (libitest_util.so+0x73d9a)
    #5 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/raft_consensus_election-itest.cc:182:3 (raft_consensus_election-itest+0x4cffa3)
    #6 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef)
    #7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef)
    #8 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8)
    #9 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c)
    #10 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226)
    #11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa)
    #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f)
    #13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f)
    #14 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2)
    #15 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x340b)
    #16 main /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2bd6)

  Thread T9 'rpc reactor-205' (tid=20551, running) created by main thread at:
    #0 pthread_create /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (raft_consensus_election-itest+0x43ced6)
    #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1bca1f)
    #2 kudu::Status kudu::Thread::Create<void (kudu::rpc::ReactorThread::*)(), kudu::rpc::ReactorThread*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.h:164:12 (libkrpc.so+0xc2dd5)
    #3 kudu::rpc::ReactorThread::Init() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:168:10 (libkrpc.so+0xbd80e)
    #4 kudu::rpc::Reactor::Init() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:722:18 (libkrpc.so+0xc1cd1)
    #5 kudu::rpc::Messenger::Init() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/messenger.cc:436:5 (libkrpc.so+0xa8402)
    #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr<kudu::rpc::Messenger>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/messenger.cc:199:3 (libkrpc.so+0xa7e5d)
    #7 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr<kudu::client::KuduClient>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/client/client.cc:332:3 (libkudu_client.so+0xbf3e1)
    #8 kudu::tserver::TabletServerIntegrationTestBase::CreateClient(std::__1::shared_ptr<kudu::client::KuduClient>*) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/ts_itest-base.cc:520:3 (libitest_util.so+0x73302)
    #9 kudu::tserver::TabletServerIntegrationTestBase::BuildAndStart(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/ts_itest-base.cc:545:3 (libitest_util.so+0x73a1f)
    #10 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/0/src/kudu/integration-tests/raft_consensus_election-itest.cc:147:3 (raft_consensus_election-itest+0x4cfb63)
    #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef)
    #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8)
    #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c)
    #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226)
    #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa)
    #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f)
    #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f)
    #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2)
    #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x340b)
    #21 main /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2bd6)

SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*)
==================
I0426 18:02:17.518096 20091 external_mini_cluster.cc:1011] Killing /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver with pid 20165
W0426 18:02:17.570519 20113 connection.cc:512] client connection to 127.19.158.193:39806 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.193:39806): Connection reset by peer (error 104)
W0426 18:02:17.570564 20309 connection.cc:512] server connection from 127.19.158.193:42428 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.193:42428): Connection reset by peer (error 104)
W0426 18:02:17.570650 20102 connection.cc:512] client connection to 127.19.158.193:39806 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.193:39806): Connection reset by peer (error 104)
W0426 18:02:17.571547 20113 connection.cc:512] server connection from 127.19.158.193:48198 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.193:48198): Connection reset by peer (error 104)
W0426 18:02:17.571811 20099 connection.cc:512] client connection to 127.19.158.193:39806 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.193:39806): Connection reset by peer (error 104)
W0426 18:02:17.572953 20642 connection.cc:512] server connection from 127.19.158.193:59360 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.193:59360): Connection reset by peer (error 104)
I0426 18:02:17.573035 20091 external_mini_cluster.cc:1011] Killing /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver with pid 20300
W0426 18:02:17.603667 20111 connection.cc:512] client connection to 127.19.158.194:34029 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.194:34029): Connection reset by peer (error 104)
W0426 18:02:17.604725 20115 connection.cc:512] server connection from 127.19.158.194:55036 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.194:55036): Connection reset by peer (error 104)
W0426 18:02:17.605597 20102 connection.cc:512] client connection to 127.19.158.194:34029 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.194:34029): Connection reset by peer (error 104)
W0426 18:02:17.606722 20097 connection.cc:512] client connection to 127.19.158.194:34029 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.194:34029): Connection reset by peer (error 104)
I0426 18:02:17.607700 20091 external_mini_cluster.cc:1011] Killing /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-tserver with pid 20634
I0426 18:02:17.633913 20091 external_mini_cluster.cc:1011] Killing /tmp/dist-test-taskEpRJ5y/build/tsan/bin/kudu-master with pid 20103
W0426 18:02:17.633883 20111 connection.cc:512] server connection from 127.19.158.195:44078 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.195:44078): Connection reset by peer (error 104)
W0426 18:02:17.661109 20102 connection.cc:512] client connection to 127.19.158.254:32815 recv error: Network error: failed to read from TLS socket (remote: 127.19.158.254:32815): Connection reset by peer (error 104)
[       OK ] RaftConsensusElectionITest.RunLeaderElection (6165 ms)
[----------] 10 tests from RaftConsensusElectionITest (127508 ms total)

[----------] Global test environment tear-down
[==========] 10 tests from 1 test case ran. (127508 ms total)
[  PASSED  ] 10 tests.
ThreadSanitizer: reported 2 warnings