IMPALA-14739: Harden CatalogdMetaProvider.Weigher for edge cases Handle two cases safer: 1. > 2GB cache entries These were truncated before the patch to 2GB to fit to int32, and this underreporting meant that the cache could grow beyond its supposed limit. This is improved by using byte_size / 16 as weight, allowing sizes up to 32GB, which seems unrealistically large to me. 2. eviction of currently loaded entries The "piggy-backing" mechanism uses CompletableFuture as values while loading objects. Evicting this before loading finishes leads to not writing back the loaded object to cache as it is assumed that it was invalidated. The patch protects against weight based eviction by weighing these entries as 0, which leads the weight based eviction to ignore them. This is the recommended way to "lock" entries in weight bounded guava / caffeine caches. Time based evection can still remove entries while loading (1 hour by default). Both time and weight based eviction should be rare - one needs >1hour loading time, the other needs many new entries added while loading to push out the entry from LRU cache. Change-Id: Id525b9b0578fb7f9cb3e0f8f4fa32f6fdae313b9 Reviewed-on: http://gerrit.cloudera.org:8080/24037 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Lightning-fast, distributed SQL queries for petabytes of data stored in open data and table formats.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
The fastest way to try out Impala is a quickstart Docker container. You can try out running queries and processing data sets in Impala on a single machine without installing dependencies. It can automatically load test data sets into Apache Kudu and Apache Parquet formats and you can start playing around with Apache Impala SQL within minutes.
To learn more about Impala as a user or administrator, or to try Impala, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Impala only supports Linux at the moment. Impala supports x86_64 and has experimental support for arm64 (as of Impala 4.0). Impala Requirements contains more detailed information on the minimum CPU requirements.
Impala runs on Linux systems only. The supported distros are
Other systems, e.g. SLES12, may also be supported but are not tested by the community.
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.