commit | 8e745a778078b8b0f07573d6c2f53d80bc079a12 | [log] [tgz] |
---|---|---|
author | stiga-huang <huangquanlong@gmail.com> | Wed Aug 26 14:28:37 2020 +0800 |
committer | Impala Public Jenkins <impala-public-jenkins@cloudera.com> | Sat Sep 19 06:03:58 2020 +0000 |
tree | 2bdfaf57560864a8e2532b3e0cc7243090982058 | |
parent | 5dccf8024b17148e3736473d03426588f434af48 [diff] |
IMPALA-10076: Reduce partition level update logs Catalogd and the legacy mode coordinator log catalog topic items one by a line. A table can have more than 100K partitions. It's too verbose to log all partition level updates. This patch aggregates the partition level updates of each table and only logs the aggregated info in a line. For instance, here is an aggregated log: Collected 24 partition update(s): 1:HDFS_PARTITION:test.tbl:(p=1,p=2,...,p=24), version=1451, original size=(avg=646, min=646, max=648, sum=15516), compressed size=(avg=478, min=475, max=485, sum=11487) If there are more than 3 partitions, we only show the lexicographically smallest, the second smallest and the largest partition names. If there are only one partition update, the log format is the same as before: Collected 1 partition update(s): 1:HDFS_PARTITION:test.tbl:p=0, version=1451, original size=648, compressed size=475 To support these, this patch implements a helper class, PartitionMetaSummary, to aggregate these information and get a summary for logging. The compressed size is calculated by BE. So the jni method for NativeAddPendingTopicItem is extended to return the actual compressed size. Tests: - Added unit tests for PartitionMetaSummary. - Start the cluster in mixed catalog mode and manually verified the logs. Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Reviewed-on: http://gerrit.cloudera.org:8080/16375 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Lightning-fast, distributed SQL queries for petabytes of data stored in Apache Hadoop clusters.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
To learn more about Impala as a business user, or to try Impala live or in a VM, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Impala only supports Linux at the moment.
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.