commit 79b17db83a2f97a409e468c5dd5cfaeb9454f5dd
author Tim Armstrong <tarmstrong@cloudera.com> Sat Feb 22 16:48:14 2020 -0800
committer Impala Public Jenkins <impala-public-jenkins@cloudera.com> Mon Sep 21 22:02:24 2020 +0000
tree 80193707f6486e4d5fd5deb8577e797406129454
parent 201eea6d779b73017935e2539dce791b003e26ba

IMPALA-9382: part 1: transposed profile prototype

This adds an experimental profile representation
that is denser than the traditional representation.
Counters, info strings and other information for all
instances of a fragment are merged into a single
tree. Descriptive stats (min, max, mean) are shown for
each counter, along with the values for each instance. It
can be enabled by setting --gen_experimental_profile=true.
The default behaviour is unchanged, aside from including
a few extra counters in existing profiles.

An example of the pretty-printed profile is attached
to the JIRA.

The thrift representation of the profile is extended
so that all instances of a fragment can be merged
together into a single "aggregated" fragment, with
vectors of counters.

The in-memory representation is transformed in
a similar way. The RuntimeProfile class is
restructured so that there is a common RuntimeProfileBase
class, with RuntimeProfile and AggregatedRuntimeProfile
subclasses. Execution fills in counters in RuntimeProfile
for each instances, then these are aggregated together into
an AggregatedRuntimeProfile on the coordinator. This replaces
the "averaged" profile concept with an abstraction that
more clearly distinguishes what operations apply to aggregated
and unaggregated profiles.

In a future change, we could use AggregatedRuntimeProfile
for status reports so that less data needs to be sent to
the coordinator, and the coordinator needs to do less
processing.

The new profile removes the bad practice of including aggregated
stats as strings from the new profile. These stats can now be
automatically as aggregations of counters. The legacy uses of
InfoString are preserved so as to not lose information but
can be removed when we switch to the transposed profile.

Also make TotalTime and InactiveTime behave like other counters -
they are pretty-printed the same as other counters. Inactive time
is also now subtracted from local time in the averaged profile,
which fixes IMPALA-2794.

TODO in later patches for IMPALA-9382:
These will need to be fixed before this can be considered production
ready.
* The JSON profile generation is not fully implemented for aggregated
  profiles.
* Not all counter times are included in aggregated profile, e.g. time
  series counters.
* The pretty-printing of the various profile counters will need to be
  improved to be more readable, e.g. grouping by host, improving
  formatting.
* The aggregated profile is only updated at the end of the query.
  We need to support live updating.
* Consider how to show local time per instance - make it a first-class
  counter in the profile?

Possible extensions:
* We could better highlight outliers when pretty-printing the profile.

Testing:
* I diffed the text profile of TPC-DS Q1 to make sure there were no
  unexpected changes.
* Added unit test for stats computation in AveragedCounter.
* Passed core tests.
* exhaustive tests
* ASAN tests
* Ran some tests locally with TSAN

Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a
Reviewed-on: http://gerrit.cloudera.org:8080/15798
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>