tree 839d62623f46a82b3fc00b0877f897bee7159d9e
parent 7c4edbc9e79f9cee327cf1dc1b86a0e5c6eda855
author Thomas Tauber-Marshall <tmarshall@cloudera.com> 1571333856 -0700
committer Impala Public Jenkins <impala-public-jenkins@cloudera.com> 1574893678 +0000

IMPALA-8138: Reintroduce rpc debugging options

In the past, Impala had a very simple 'fault injection' framework for
simulating failed rpcs between impalads. With the move to KRPC, that
framework was not carried over, and we lost the ability to test
certain failure scenarios.

This patch reintroduces this functionality. It removes the prior fault
injection framework in favor of the existing debug action framework,
which is more flexible.

To facilitate this, a few modifications are made to the debug action
framework:
- In addition to matching on a label, debug actions may now match on
  optional arguments. In this patch, the debug action
  IMPALA_SERVICE_POOL takes the arguments 'host', 'port', and
  'rpc name' to allow simulating the failure of specific rpcs to
  specific impalads.
- The FAIL action now takes an optional 'error message' parameter. In
  this patch, the debug action IMPALA_SERVICE_POOL uses this to
  simulate different types of rpc errors, eg. 'service too busy'.
- The FAIL action increments a metric, 'impala.debug_action.fail', so
  that tests can check that it has actually been hit. Prior to this
  patch the tests in test_rpc_exception.py where all passing
  spuriously as the faults they were supposed to be testing were no
  longer being injected.

This patch uses these new mechanisms to introduce tests that simulate
failures in DataStreamService rpcs. Follow up patches will add test
cases for ControlService rpcs.

Change-Id: I9c047ebce6d32c5ae461f70279391fa2df4c2029
Reviewed-on: http://gerrit.cloudera.org:8080/14641
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>