KUDU-1186. Work around NTP not synchronized initialization issue
The Master initialization code makes a "first run" decision at the very
beginning of startup based on whether it sees tablet data on disk and
then assumes that if the tablet data is there then the rest of the
metadata is there. One possible cause for a partially created master
tablet is a crash on startup due to an unsynchronized clock. Since this
is likely to be a common issue people run into when trying to run Kudu
for the first time, this works around the issue by doing that check
early.
However, we should still rework that startup logic to be more robust in
a separate patch.
Tested this manually and it seems to do the trick.
Change-Id: I29525101297db3b3bfbfd540ed5a3ed824ff62b0
Reviewed-on: http://gerrit.cloudera.org:8080/1023
Reviewed-by: Todd Lipcon <todd@cloudera.com>
Tested-by: Todd Lipcon <todd@cloudera.com>
(cherry picked from commit 7ce36ed83908a6c3e05e683332f1c3fb205cfac3)
Reviewed-on: http://gerrit.cloudera.org:8080/1034
Tested-by: Internal Jenkins
diff --git a/src/kudu/server/hybrid_clock.cc b/src/kudu/server/hybrid_clock.cc
index ef7b649..b79855e 100644
--- a/src/kudu/server/hybrid_clock.cc
+++ b/src/kudu/server/hybrid_clock.cc
@@ -104,6 +104,11 @@
} // anonymous namespace
+Status CheckClockSynchronized() {
+ ntptimeval junk;
+ return GetClockTime(&junk);
+}
+
// Left shifting 12 bits gives us 12 bits for the logical value
// and should still keep accurate microseconds time until 2100+
const int HybridClock::kBitsToShift = 12;
diff --git a/src/kudu/server/hybrid_clock.h b/src/kudu/server/hybrid_clock.h
index c3bd2d0..3b7c782 100644
--- a/src/kudu/server/hybrid_clock.h
+++ b/src/kudu/server/hybrid_clock.h
@@ -28,6 +28,9 @@
namespace kudu {
namespace server {
+// Returns OK if the clock is considered synchronized.
+Status CheckClockSynchronized();
+
// The HybridTime clock.
class HybridClock : public Clock {
public:
diff --git a/src/kudu/server/server_base.cc b/src/kudu/server/server_base.cc
index 37b6b18..56d4fd1 100644
--- a/src/kudu/server/server_base.cc
+++ b/src/kudu/server/server_base.cc
@@ -151,6 +151,13 @@
InitSpinLockContentionProfiling();
+ // Check for NTP errors so it's less likely to get into a partially
+ // initialized state on disk during startup.
+ // TODO: Remove this check after fully fixing KUDU-1186.
+ if (FLAGS_use_hybrid_clock) {
+ RETURN_NOT_OK_PREPEND(CheckClockSynchronized(), "Clock is not synchronized");
+ }
+
Status s = fs_manager_->Open();
if (s.IsNotFound()) {
LOG(INFO) << "Could not load existing FS layout: " << s.ToString();
