tree 0ce602151b8ecce6d64fb2aa6d37b87f9b59bc84
parent 5d9daf40511e8b43d9bfd186b19efc06588cd1ce
author Riza Suminto <riza.suminto@cloudera.com> 1600809884 -0700
committer Impala Public Jenkins <impala-public-jenkins@cloudera.com> 1601118707 +0000

IMPALA-10112: Remove FpRateTooHigh() check for bloom filter

FpRateTooHigh() check for bloom filter that can be disabled if the
observed false-positive probability (FPP) rate is higher than
FLAGS_max_filter_error_rate. This patch remove FpRateTooHigh() check for
several reasons:

1. Partition filters are probably still worth evaluating even if there
   are false positives because it is cheap and beneficial to eliminate a
   partition.

2. Runtime filters are dynamically disabled on the scan side if they are
   ineffective. Even an ALWAYS_TRUE_FILTER that substitutes a disabled
   filter is also still being evaluated and not entirely free.

3. The disabling is less likely to kick in for partitioned joins because
   it only applied to a small subset of the filters before the Or()
   operation.

4. FpRateTooHigh() use num_build_rows to approximate the FPP rate of the
   resulting filter. This can be inaccurate because it does not take
   account of duplicate values of the filter key on the build side.

This patch also removes some tests in test_runtime_filters.py that check
cancellation of filters having a high FPP rate.

Testing:
- Run and pass core tests.
- Manually test and verify in a real large cluster (TPC-DS 30TB scale)
  that there is only a little (< 2.1%) to no performance regression
  incurred from the removal of high FPP check. The test used TPC-DS
  queries Q14a, Q50, Q64, Q71, Q84, Q93, and a modified Q93 with the
  left outer join replaced by an inner join.
  The query time comparison between having an FPP check versus without
  having an FPP check are the following:

  | Query | With check (ms) | Without check (ms) |
  |-------+-----------------+--------------------|
  | Q14a  |          129801 |             125992 |
  | Q50   |           72519 |              72652 |
  | Q64   |          150684 |             145241 |
  | Q71   |           21009 |              20358 |
  | Q84   |           29334 |              29944 |
  | Q93   |           91782 |              91923 |
  | Q93m  |           92897 |              92135 |

Change-Id: Id9f8f40764b4f6664cc81b0da428afea8e3588d4
Reviewed-on: http://gerrit.cloudera.org:8080/16499
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>