rpc: add a flag for keepalive time, bump default to 65s
We're seeing KUDU-279 fairly often in stress tests on a cluster.
Until that is fixed, it makes sense to bump the RPC keepalive time
significantly. Bumping to 65 seconds should be enough that any
"once a minute" type heartbeats would keep the connection alive.
In particular, this will be long enough that Impala's scanner keepalive
heartbeats will keep RPC connections to the tablet servers alive,
hopefully preventing the errors we're seeing in some large Impala
queries.
Change-Id: I3059ca764447b8073b69972728ad1ebd41b3e8c4
Reviewed-on: http://gerrit.cloudera.org:8080/1497
Reviewed-by: Jean-Daniel Cryans
Tested-by: Internal Jenkins
(cherry picked from commit ff502b6f22ef9eadf3674a08fa4078d802923a0e)
Reviewed-on: http://gerrit.cloudera.org:8080/1499
Tested-by: Jean-Daniel Cryans
diff --git a/src/kudu/rpc/messenger.cc b/src/kudu/rpc/messenger.cc
index 91b9c19..ccac5d8 100644
--- a/src/kudu/rpc/messenger.cc
+++ b/src/kudu/rpc/messenger.cc
@@ -20,6 +20,7 @@
#include <unistd.h>
#include <boost/foreach.hpp>
+#include <gflags/gflags.h>
#include <glog/logging.h>
#include <list>
#include <set>
@@ -38,6 +39,7 @@
#include "kudu/rpc/sasl_common.h"
#include "kudu/rpc/transfer.h"
#include "kudu/util/errno.h"
+#include "kudu/util/flag_tags.h"
#include "kudu/util/metrics.h"
#include "kudu/util/monotime.h"
#include "kudu/util/net/socket.h"
@@ -49,6 +51,11 @@
using std::tr1::shared_ptr;
using strings::Substitute;
+DEFINE_int32(rpc_default_keepalive_time_ms, 65000,
+ "If an RPC connection from a client is idle for this amount of time, the server "
+ "will disconnect the client.");
+TAG_FLAG(rpc_default_keepalive_time_ms, advanced);
+
namespace kudu {
namespace rpc {
@@ -57,7 +64,7 @@
MessengerBuilder::MessengerBuilder(const std::string &name)
: name_(name),
- connection_keepalive_time_(MonoDelta::FromSeconds(10)),
+ connection_keepalive_time_(MonoDelta::FromMilliseconds(FLAGS_rpc_default_keepalive_time_ms)),
num_reactors_(4),
num_negotiation_threads_(4),
coarse_timer_granularity_(MonoDelta::FromMilliseconds(100)) {
