commit | cd52932125e5636ff154c3cdb6a740877b255998 | [log] [tgz] |
---|---|---|
author | Vihang Karajgaonkar <vihangk1@apache.org> | Tue Aug 04 12:27:24 2020 -0700 |
committer | Quanlong Huang <huangquanlong@gmail.com> | Sat Aug 15 02:01:05 2020 +0000 |
tree | e3cc523ca2b87c71060703177bcfc292e0895201 | |
parent | e0681615c2f8f38763437dc9440e8eda42334f5c [diff] |
IMPALA-4364: Query option to refresh updated HMS partitions This patch introduces a new boolean query option REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However for certain use-cases currently there is no way to detect changed partitions using refresh table command. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. Ran exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Reviewed-on: http://gerrit.cloudera.org:8080/16308 Reviewed-by: Quanlong Huang <huangquanlong@gmail.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Lightning-fast, distributed SQL queries for petabytes of data stored in Apache Hadoop clusters.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
To learn more about Impala as a business user, or to try Impala live or in a VM, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Impala only supports Linux at the moment.
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.