tree 47255eaa8c1d3b8dcd65dd5be41357b6cab703f4
parent 6f9a842a5113d76778b40241136dc28e4f27a6c1
author Tim Armstrong <tarmstrong@cloudera.com> 1573772963 -0800
committer Impala Public Jenkins <impala-public-jenkins@cloudera.com> 1574206898 +0000

IMPALA-9134: pipeline data stream sender flush

This changes the logic so that the last TransmitData
and EndDataStream RPCs can be sent to receivers in
parallel, instead of them being serialized in FlushAndSendEos().

This was a problem because each call to FlushAndSendEos() would
wait for one or two network round-trips, so previously the flush
could easily take sum(2 * RTT). After this change the sending
and waiting for RPCs is pipeline so the time taken should be
closer to 2 * RTT.

Testing:
This code is exercised by existing tests. Ran exhaustive tests.

Perf:
No measurable change on local TPC-H (where network latency isn't a
significant factor).

+----------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format           | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) |
+----------+-----------------------+---------+------------+------------+----------------+
| TPCH(30) | parquet / none / none | 7.64    | -1.37%     | 5.00       | +0.29%         |
+----------+-----------------------+---------+------------+------------+----------------+

+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
| Workload | Query    | File Format           | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%)  | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval  |
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
| TPCH(30) | TPCH-Q14 | parquet / none / none | 3.75   | 3.39        |   +10.48%  | * 19.43% * |   8.88%        | 7     |   +7.26%       | 0.48    | 1.19  |
| TPCH(30) | TPCH-Q13 | parquet / none / none | 13.55  | 12.78       |   +6.05%   |   5.43%    |   4.75%        | 7     |   +5.09%       | 1.76    | 2.14  |
| TPCH(30) | TPCH-Q2  | parquet / none / none | 1.30   | 1.26        |   +3.63%   |   5.53%    |   3.27%        | 7     |   +3.93%       | 1.28    | 1.46  |
| TPCH(30) | TPCH-Q12 | parquet / none / none | 2.65   | 2.56        |   +3.44%   |   4.23%    |   2.92%        | 7     |   +3.68%       | 1.28    | 1.73  |
| TPCH(30) | TPCH-Q15 | parquet / none / none | 3.80   | 3.76        |   +0.91%   |   0.81%    |   1.80%        | 7     |   +1.42%       | 1.12    | 1.22  |
| TPCH(30) | TPCH-Q8  | parquet / none / none | 5.23   | 5.13        |   +2.11%   |   5.41%    |   0.48%        | 7     |   +0.04%       | 0.16    | 1.01  |
| TPCH(30) | TPCH-Q10 | parquet / none / none | 5.77   | 5.62        |   +2.70%   |   9.84%    |   2.79%        | 7     |   -0.80%       | -0.96   | 0.68  |
| TPCH(30) | TPCH-Q21 | parquet / none / none | 28.82  | 28.62       |   +0.70%   |   0.59%    |   0.37%        | 7     |   +0.68%       | 1.92    | 2.67  |
| TPCH(30) | TPCH-Q6  | parquet / none / none | 1.32   | 1.31        |   +0.82%   |   0.36%    |   1.53%        | 7     |   +0.23%       | 1.28    | 1.38  |
| TPCH(30) | TPCH-Q11 | parquet / none / none | 1.30   | 1.29        |   +0.60%   |   5.11%    |   2.94%        | 7     |   +0.14%       | 0.16    | 0.27  |
| TPCH(30) | TPCH-Q19 | parquet / none / none | 4.52   | 4.49        |   +0.57%   |   2.98%    |   1.58%        | 7     |   +0.12%       | 0.32    | 0.44  |
| TPCH(30) | TPCH-Q16 | parquet / none / none | 2.78   | 2.77        |   +0.35%   |   3.21%    |   2.02%        | 7     |   +0.17%       | 0.64    | 0.25  |
| TPCH(30) | TPCH-Q3  | parquet / none / none | 5.65   | 5.65        |   -0.04%   |   1.62%    |   1.91%        | 7     |   -0.06%       | -0.32   | -0.04 |
| TPCH(30) | TPCH-Q1  | parquet / none / none | 10.88  | 10.89       |   -0.11%   |   0.76%    |   0.90%        | 7     |   +0.01%       | 0.00    | -0.26 |
| TPCH(30) | TPCH-Q4  | parquet / none / none | 2.18   | 2.18        |   -0.38%   |   1.33%    |   2.48%        | 7     |   -0.04%       | -0.16   | -0.36 |
| TPCH(30) | TPCH-Q5  | parquet / none / none | 3.94   | 3.95        |   -0.27%   |   2.37%    |   1.33%        | 7     |   -0.26%       | -1.12   | -0.26 |
| TPCH(30) | TPCH-Q7  | parquet / none / none | 22.37  | 22.41       |   -0.19%   |   2.94%    |   1.92%        | 7     |   -0.66%       | -0.80   | -0.15 |
| TPCH(30) | TPCH-Q22 | parquet / none / none | 3.68   | 3.72        |   -1.14%   |   4.95%    |   3.99%        | 7     |   +0.13%       | 0.00    | -0.48 |
| TPCH(30) | TPCH-Q20 | parquet / none / none | 2.83   | 2.86        |   -1.21%   |   2.69%    |   2.20%        | 7     |   -1.60%       | -0.64   | -0.93 |
| TPCH(30) | TPCH-Q17 | parquet / none / none | 4.92   | 5.04        |   -2.46%   |   3.11%    |   2.63%        | 7     |   -2.00%       | -1.44   | -1.62 |
| TPCH(30) | TPCH-Q9  | parquet / none / none | 16.32  | 17.19       |   -5.03%   |   2.76%    |   4.56%        | 7     |   -4.00%       | -2.24   | -2.53 |
| TPCH(30) | TPCH-Q18 | parquet / none / none | 20.45  | 23.45       |   -12.82%  |   5.49%    | * 23.11% *     | 7     |   -4.89%       | -0.96   | -1.44 |
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+

Change-Id: If66add99e4370f8faf22761b336205cc9f9f1867
Reviewed-on: http://gerrit.cloudera.org:8080/14717
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>