tree e19b6b8c2ea80fb1689ad2d5bdab52ae6aab01df
parent ceca6abc506fc4c1012ced6e8a6564897b3af4ad
author Alexey Serbin <alexey@apache.org> 1614411789 -0800
committer Alexey Serbin <aserbin@cloudera.com> 1614732880 +0000

KUDU-2612 add perf scenario to TxnWriteOpsITest

This patch adds WriteOpPerf scenario to TxnWriteOpsITest.  The new
scenario is to evaluate --tablet_max_pending_txn_write_ops setting for
tablet servers: it runs for a short time to count number of completed
Write RPCs in context of a transactional session.  The scenario focuses
on single-row write operations to pinpoint the latency of processing
txn write operations when performing registration of transaction
participants.  Probably, the current locking approach for the
TxnOpDispatcher's queue while submitting the accumulated operations
isn't optimal.  Apparently, the exponential back-off timing built into
the client's RPC retry logic is also an important factor: I saw
significant deviation in the number of completed RPCs from run to run.

Below are results averaged for 100 runs of the benchmark scenario with
varying --max_pending_txn_write_ops accordingly and the following
settings fixed:
  --prepare_connections_to_tservers=true
  --clients=8
  --sessions_per_client=1
  --benchmark_run_time_ms=50

I used the following script to get the accumulated results for
writes in a transactional context:
  for i in {0..99}; do
    ./bin/txn_write_ops-itest --gtest_filter='*WriteOpPerf' \
        --max_pending_txn_write_ops=<X> 2>&1 | grep 'write RPCs' | \
        awk '{print $9}'; done | \
        awk 'BEGIN {sum=0} {sum += $0} END {print sum}'

RELEASE build:
  --max_pending_txn_write_ops=0  : 442.13 RPCs
  --max_pending_txn_write_ops=2  : 494.33 RPCs
  --max_pending_txn_write_ops=5  : 471.90 RPCs
  --max_pending_txn_write_ops=10 : 490.22 RPCs
  --max_pending_txn_write_ops=20 : 469.21 RPCs

DEBUG   build:
  --max_pending_txn_write_ops=0  : 83.74 RPCs
  --max_pending_txn_write_ops=2  : 98.18 RPCs
  --max_pending_txn_write_ops=5  : 95.23 RPCs
  --max_pending_txn_write_ops=10 : 98.12 RPCs
  --max_pending_txn_write_ops=20 : 94.40 RPCs

I also measured the performance of transactional vs non-transactional
for various time intervals in RELEASE build, where for transactional
writes I used --max_pending_txn_write_ops=2 setting:

50ms:
  non-transactional:    588.33 RPCs (11767 req/sec)
      transactional:    487.82 RPCs ( 9756 req/sec)

3000ms:
  non-transactional:  40041.63 RPCs (13347 req/sec)
      transactional:  39759.07 RPCs (13253 req/sec)

8000ms:
  non-transactional: 106832.37 RPCs (13354 req/sec)
      transactional: 105922.65 RPCs (13240 req/sec)

Change-Id: I0370dbb289a4e1cfc154205ae92e13da510682b4
Reviewed-on: http://gerrit.cloudera.org:8080/17105
Tested-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>