webserver: enable HTTP keep-alive

This is a port of IMPALA-8869. The motivation is, to quote from IMPALA-8869:
"...we mishandle HTTP keep-alive semantics when returning a 401 because we
close the connection but don't return a 'Connection: close' header, even
though we're using HTTP/1.1 where keep-alive is assumed, which can cause
clients to incorrectly believe that the connection has remained open."

More tangibly, this leads to issues when proxying via Apache Knox.
Specifically, every other request fails because the browser expected its
connection to remain open, but Kudu closed it.

The main challenge is that now we need to buffer up all parts of the
response to avoid triggering a combination of Nagle's algorithm and TCP
delayed acks, which add ~40ms to the RTT. To be safe, I refactored all
response handling to go through a single function. Since I was knee deep in
refactoring, I took the opportunity to fix the "SPNEGO header not returned
with 200 response" issue.

Change-Id: Ic08ef5a268fdf6dea6a8c428b4ab8dac27418dd6
Reviewed-on: http://gerrit.cloudera.org:8080/14440
Tested-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Thomas Tauber-Marshall <tmarshall@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
13 files changed