commit | 79b17db83a2f97a409e468c5dd5cfaeb9454f5dd | [log] [tgz] |
---|---|---|
author | Tim Armstrong <tarmstrong@cloudera.com> | Sat Feb 22 16:48:14 2020 -0800 |
committer | Impala Public Jenkins <impala-public-jenkins@cloudera.com> | Mon Sep 21 22:02:24 2020 +0000 |
tree | 80193707f6486e4d5fd5deb8577e797406129454 | |
parent | 201eea6d779b73017935e2539dce791b003e26ba [diff] |
IMPALA-9382: part 1: transposed profile prototype This adds an experimental profile representation that is denser than the traditional representation. Counters, info strings and other information for all instances of a fragment are merged into a single tree. Descriptive stats (min, max, mean) are shown for each counter, along with the values for each instance. It can be enabled by setting --gen_experimental_profile=true. The default behaviour is unchanged, aside from including a few extra counters in existing profiles. An example of the pretty-printed profile is attached to the JIRA. The thrift representation of the profile is extended so that all instances of a fragment can be merged together into a single "aggregated" fragment, with vectors of counters. The in-memory representation is transformed in a similar way. The RuntimeProfile class is restructured so that there is a common RuntimeProfileBase class, with RuntimeProfile and AggregatedRuntimeProfile subclasses. Execution fills in counters in RuntimeProfile for each instances, then these are aggregated together into an AggregatedRuntimeProfile on the coordinator. This replaces the "averaged" profile concept with an abstraction that more clearly distinguishes what operations apply to aggregated and unaggregated profiles. In a future change, we could use AggregatedRuntimeProfile for status reports so that less data needs to be sent to the coordinator, and the coordinator needs to do less processing. The new profile removes the bad practice of including aggregated stats as strings from the new profile. These stats can now be automatically as aggregations of counters. The legacy uses of InfoString are preserved so as to not lose information but can be removed when we switch to the transposed profile. Also make TotalTime and InactiveTime behave like other counters - they are pretty-printed the same as other counters. Inactive time is also now subtracted from local time in the averaged profile, which fixes IMPALA-2794. TODO in later patches for IMPALA-9382: These will need to be fixed before this can be considered production ready. * The JSON profile generation is not fully implemented for aggregated profiles. * Not all counter times are included in aggregated profile, e.g. time series counters. * The pretty-printing of the various profile counters will need to be improved to be more readable, e.g. grouping by host, improving formatting. * The aggregated profile is only updated at the end of the query. We need to support live updating. * Consider how to show local time per instance - make it a first-class counter in the profile? Possible extensions: * We could better highlight outliers when pretty-printing the profile. Testing: * I diffed the text profile of TPC-DS Q1 to make sure there were no unexpected changes. * Added unit test for stats computation in AveragedCounter. * Passed core tests. * exhaustive tests * ASAN tests * Ran some tests locally with TSAN Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Reviewed-on: http://gerrit.cloudera.org:8080/15798 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Lightning-fast, distributed SQL queries for petabytes of data stored in Apache Hadoop clusters.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
To learn more about Impala as a business user, or to try Impala live or in a VM, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Impala only supports Linux at the moment. Impala supports x86_64 and has experimental support for arm64 (as of Impala 4.0). Impala Requirements contains more detailed information on the minimum CPU requirements.
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.