docs/topics/impala_runtime_bloom_filter_size.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="runtime_bloom_filter_size" rev="2.5.0">

  <title>RUNTIME_BLOOM_FILTER_SIZE Query Option (<keyword keyref="impala25"/> or higher only)</title>
  <titlealts audience="PDF"><navtitle>RUNTIME_BLOOM_FILTER_SIZE</navtitle></titlealts>
  <prolog>
    <metadata>
      <data name="Category" value="Impala"/>
      <data name="Category" value="Impala Query Options"/>
      <data name="Category" value="Performance"/>
      <data name="Category" value="Developers"/>
      <data name="Category" value="Data Analysts"/>
    </metadata>
  </prolog>

  <conbody>

    <p rev="2.5.0">
      <indexterm audience="hidden">RUNTIME_BLOOM_FILTER_SIZE query option</indexterm>
      Size (in bytes) of Bloom filter data structure used by the runtime filtering
      feature.
    </p>

    <note type="important">
      <p rev="2.6.0 IMPALA-3007">
        In <keyword keyref="impala26_full"/> and higher, this query option only applies as a fallback, when statistics
        are not available. By default, Impala estimates the optimal size of the Bloom filter structure
        regardless of the setting for this option. (This is a change from the original behavior in
        <keyword keyref="impala25_full"/>.)
      </p>
      <p rev="2.6.0 IMPALA-3007">
        In <keyword keyref="impala26_full"/> and higher, when the value of this query option is used for query planning,
        it is constrained by the minimum and maximum sizes specified by the
        <codeph>RUNTIME_FILTER_MIN_SIZE</codeph> and <codeph>RUNTIME_FILTER_MAX_SIZE</codeph> query options.
        The filter size is adjusted upward or downward if necessary to fit within the minimum/maximum range.
      </p>
    </note>

    <p conref="../shared/impala_common.xml#common/type_integer"/>

    <p>
      <b>Default:</b> 1048576 (1 MB)
    </p>

    <p>
      <b>Maximum:</b> 16 MB
    </p>

    <p conref="../shared/impala_common.xml#common/added_in_250"/>

    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>

    <p>
      This setting affects optimizations for large and complex queries, such
      as dynamic partition pruning for partitioned tables, and join optimization
      for queries that join large tables.
      Larger filters are more effective at handling
      higher cardinality input sets, but consume more memory per filter.
      <!--
      See <xref href="impala_runtime_filtering.xml#runtime_filtering"/> for details about when to change
      the setting for this query option.
      -->
    </p>

    <p>
      If your query filters on high-cardinality columns (for example, millions of different values)
      and you do not get the expected speedup from the runtime filtering mechanism, consider
      doing some benchmarks with a higher value for <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph>.
      The extra memory devoted to the Bloom filter data structures can help make the filtering
      more accurate.
    </p>

    <p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>

    <p>
      Because the effectiveness of this setting depends so much on query characteristics and data distribution,
      you typically only use it for specific queries that need some extra tuning, and the ideal value depends
      on the query. Consider setting this query option immediately before the expensive query and
      unsetting it immediately afterward.
    </p>

    <p conref="../shared/impala_common.xml#common/kudu_blurb"/>

    <p rev="2.11.0 IMPALA-4252" conref="../shared/impala_common.xml#common/filter_option_bloom_only"/>

    <p conref="../shared/impala_common.xml#common/related_info"/>
    <p>
      <xref href="impala_runtime_filtering.xml"/>,
      <!-- , <xref href="impala_partitioning.xml#dynamic_partition_pruning"/> -->
      <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/>,
      <xref href="impala_runtime_filter_min_size.xml#runtime_filter_min_size"/>,
      <xref href="impala_runtime_filter_max_size.xml#runtime_filter_max_size"/>
    </p>

  </conbody>
</concept>