commit | 5508be2b6273edd8074f466a7a64e70ecf304ddf | [log] [tgz] |
---|---|---|
author | Riza Suminto <riza.suminto@cloudera.com> | Tue Sep 22 14:24:44 2020 -0700 |
committer | Impala Public Jenkins <impala-public-jenkins@cloudera.com> | Sat Sep 26 11:11:47 2020 +0000 |
tree | 0ce602151b8ecce6d64fb2aa6d37b87f9b59bc84 | |
parent | 5d9daf40511e8b43d9bfd186b19efc06588cd1ce [diff] |
IMPALA-10112: Remove FpRateTooHigh() check for bloom filter FpRateTooHigh() check for bloom filter that can be disabled if the observed false-positive probability (FPP) rate is higher than FLAGS_max_filter_error_rate. This patch remove FpRateTooHigh() check for several reasons: 1. Partition filters are probably still worth evaluating even if there are false positives because it is cheap and beneficial to eliminate a partition. 2. Runtime filters are dynamically disabled on the scan side if they are ineffective. Even an ALWAYS_TRUE_FILTER that substitutes a disabled filter is also still being evaluated and not entirely free. 3. The disabling is less likely to kick in for partitioned joins because it only applied to a small subset of the filters before the Or() operation. 4. FpRateTooHigh() use num_build_rows to approximate the FPP rate of the resulting filter. This can be inaccurate because it does not take account of duplicate values of the filter key on the build side. This patch also removes some tests in test_runtime_filters.py that check cancellation of filters having a high FPP rate. Testing: - Run and pass core tests. - Manually test and verify in a real large cluster (TPC-DS 30TB scale) that there is only a little (< 2.1%) to no performance regression incurred from the removal of high FPP check. The test used TPC-DS queries Q14a, Q50, Q64, Q71, Q84, Q93, and a modified Q93 with the left outer join replaced by an inner join. The query time comparison between having an FPP check versus without having an FPP check are the following: | Query | With check (ms) | Without check (ms) | |-------+-----------------+--------------------| | Q14a | 129801 | 125992 | | Q50 | 72519 | 72652 | | Q64 | 150684 | 145241 | | Q71 | 21009 | 20358 | | Q84 | 29334 | 29944 | | Q93 | 91782 | 91923 | | Q93m | 92897 | 92135 | Change-Id: Id9f8f40764b4f6664cc81b0da428afea8e3588d4 Reviewed-on: http://gerrit.cloudera.org:8080/16499 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Lightning-fast, distributed SQL queries for petabytes of data stored in Apache Hadoop clusters.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
To learn more about Impala as a business user, or to try Impala live or in a VM, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Impala only supports Linux at the moment. Impala supports x86_64 and has experimental support for arm64 (as of Impala 4.0). Impala Requirements contains more detailed information on the minimum CPU requirements.
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.