commit | 21c50f8dbb7ef75dbb56bee07590a710993f4d64 | [log] [tgz] |
---|---|---|
author | Vihang Karajgaonkar <vihangk1@apache.org> | Tue Aug 25 12:22:26 2020 -0700 |
committer | Vihang Karajgaonkar <vihang@cloudera.com> | Wed Aug 26 17:08:43 2020 +0000 |
tree | ed61b9d7e7cbab769b714b5f23d0f638e4eba4a0 | |
parent | 28d94851b1418ddd57e4c046646f7a2df0e702d7 [diff] |
IMPALA-4364: [Addendum] Compare specific fields in StorageDescriptor The query option REFRESH_UPDATED_HMS_PARTITIONS was introduced earlier in IMPALA-4364 to detect changes in the partition objects in HMS when a refresh table command is issued. Originally, it relied on using the StorageDescriptor#equals() method to determine if the Partition in catalogd is same as partition in HMS with while executing the refresh statement. However, using StorageDescriptor#equals() is dependent on HMS version and may introduce inconsistent behaviors after upgrades. For example, when we backported the original patch to older distribution which uses Hive-2, the SkewedInfo field of StorageDescriptor is not null. This field causes the comparison logic to fail, since catalogd doesn't store the SkewedInfo field in the cached StorageDescriptor to optimize memory usage. This patch modifies the comparison logic to use explicit implementation in HdfsPartition class which compares only some fields which are cached in the HdfsPartition object. Testing: 1. Added a new test for the comparison method. 2. Modified existing test for the query option. Change-Id: I90c797060265f8f508d0b150e15da3d0f9961b9b Reviewed-on: http://gerrit.cloudera.org:8080/16363 Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Reviewed-by: Vihang Karajgaonkar <vihang@cloudera.com>
Lightning-fast, distributed SQL queries for petabytes of data stored in Apache Hadoop clusters.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
To learn more about Impala as a business user, or to try Impala live or in a VM, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Impala only supports Linux at the moment.
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.