commit 2ef6184ee1010d29fdaa5cd4ba5f0c95ef9abc0d
author Qifan Chen <qchen@cloudera.com> Mon Jul 20 13:20:11 2020 -0700
committer Impala Public Jenkins <impala-public-jenkins@cloudera.com> Fri Aug 28 20:22:57 2020 +0000
tree c4fa721fce50f9e535177b60d43d73f584d8399d
parent 3733c4cc2cfb78d7f13463fb1ee9e1c4560d4a3d

IMPALA-9989 Improve admission control pool stats logging

This work addresses the current limitation in admission controller by
appending the last known memory consumption statistics about the set of
queries running or waiting on a host or in a pool to the existing memory
exhaustion message. The statistics is logged in impalad.INFO when a
query is queued or queued and then timed out due to memory pressure in
the pool or on the host. The statistics can also be part of the query
profile.

The new memory consumption statistics can be either stats on host or
aggregated pool stats. The stats on host describes memory consumption
for every pool on a host. The aggregated pool stats describes the
aggregated memory consumption on all hosts for a pool. For each stats
type, information such as query Ids and memory consumption of up to top
5 queries is provided, in addition to the min, the max, the average and
the total memory consumption for the query set.

When a query request is queued due to memory exhaustion, the above
new consumption statistics is logged when the BE logging level is set
at 2.

When a query request is timed out due to memory exhaustion, the above
new consumption statistics is logged when the BE logging level is set
at 1.

Testing:
1. Added a new test TopNQueryCheck in admission-controller-test.cc to
   verify that the topN query memory consumption details are reported
   correctly.
2. Add two new tests in test_admission_controller.py to simulate
   queries being queued and then timed out due to pool or host memory
   pressure.
3. Added a new test TopN in mem-tracker-test.cc to
   verify that the topN query memory consumption details are computed
   correctly from a mem tracker hierarchy.
4. Ran Core tests successfully.

Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781
Reviewed-on: http://gerrit.cloudera.org:8080/16220
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>