Google Git
Sign in
apache / impala / 8a582472527eb7300cf5e855d401a90e662de556 / . / docs / build / asf-site-html / topics / impala_mem_limit.html
blob: 220577f49780aa316319c4557ff088e47e656b79 [file] [log] [blame]
<!DOCTYPE html
SYSTEM "about:legacy-compat">
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2023"><meta name="DC.rights.owner" content="(C) Copyright 2023"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.4.x"><meta name="version" content="Impala 3.4.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="mem_limit"><link rel="stylesheet" type="text/css" href="../css/commonltr.css"><link rel="stylesheet" type="text/css" href="../css/dita-ot-doc.css"><title>MEM_LIMIT Query Option</title></head><body id="mem_limit"><header role="banner"><!--
The DITA Open Toolkit is licensed for use under the the Apache
Software Foundation License v2.0.
A copy of the Apache Software Foundation License 2.0 is
available at http://opensource.org/licenses/apache2.0.php
This statement must be included in any copies of DITA Open
Toolkit code.
--><div class="header">
<p>Apache Impala</p>
<hr>
</div></header><nav role="toc"><ul><li><a href="../topics/impala_intro.html">Introducing Apache Impala</a></li><li><a href="../topics/impala_concepts.html">Concepts and Architecture</a></li><li><a href="../topics/impala_planning.html">Deployment Planning</a></li><li><a href="../topics/impala_install.html">Installing Impala</a></li><li><a href="../topics/impala_config.html">Managing Impala</a></li><li><a href="../topics/impala_upgrading.html">Upgrading Impala</a></li><li><a href="../topics/impala_processes.html">Starting Impala</a></li><li><a href="../topics/impala_tutorial.html">Tutorials</a></li><li><a href="../topics/impala_admin.html">Administration</a></li><li><a href="../topics/impala_security.html">Impala Security</a></li><li><a href="../topics/impala_langref.html">SQL Reference</a><ul><li><a href="../topics/impala_comments.html">Comments</a></li><li><a href="../topics/impala_datatypes.html">Data Types</a></li><li><a href="../topics/impala_literals.html">Literals</a></li><li><a href="../topics/impala_operators.html">SQL Operators</a></li><li><a href="../topics/impala_schema_objects.html">Schema Objects and Object Names</a></li><li><a href="../topics/impala_transactions.html">Transactions</a></li><li><a href="../topics/impala_langref_sql.html">SQL Statements</a><ul><li><a href="../topics/impala_ddl.html">DDL Statements</a></li><li><a href="../topics/impala_dml.html">DML Statements</a></li><li><a href="../topics/impala_alter_database.html">ALTER DATABASE</a></li><li><a href="../topics/impala_alter_table.html">ALTER TABLE</a></li><li><a href="../topics/impala_alter_view.html">ALTER VIEW</a></li><li><a href="../topics/impala_comment.html">COMMENT</a></li><li><a href="../topics/impala_compute_stats.html">COMPUTE STATS</a></li><li><a href="../topics/impala_create_database.html">CREATE DATABASE</a></li><li><a href="../topics/impala_create_function.html">CREATE FUNCTION</a></li><li><a href="../topics/impala_create_role.html">CREATE ROLE</a></li><li><a href="../topics/impala_create_table.html">CREATE TABLE</a></li><li><a href="../topics/impala_create_view.html">CREATE VIEW</a></li><li><a href="../topics/impala_delete.html">DELETE</a></li><li><a href="../topics/impala_describe.html">DESCRIBE</a></li><li><a href="../topics/impala_drop_database.html">DROP DATABASE</a></li><li><a href="../topics/impala_drop_function.html">DROP FUNCTION</a></li><li><a href="../topics/impala_drop_role.html">DROP ROLE</a></li><li><a href="../topics/impala_drop_stats.html">DROP STATS</a></li><li><a href="../topics/impala_drop_table.html">DROP TABLE</a></li><li><a href="../topics/impala_drop_view.html">DROP VIEW</a></li><li><a href="../topics/impala_explain.html">EXPLAIN</a></li><li><a href="../topics/impala_grant.html">GRANT</a></li><li><a href="../topics/impala_insert.html">INSERT</a></li><li><a href="../topics/impala_invalidate_metadata.html">INVALIDATE METADATA</a></li><li><a href="../topics/impala_load_data.html">LOAD DATA</a></li><li><a href="../topics/impala_refresh.html">REFRESH</a></li><li><a href="../topics/impala_refresh_authorization.html">REFRESH AUTHORIZATION</a></li><li><a href="../topics/impala_refresh_functions.html">REFRESH FUNCTIONS</a></li><li><a href="../topics/impala_revoke.html">REVOKE</a></li><li><a href="../topics/impala_select.html">SELECT</a></li><li><a href="../topics/impala_set.html">SET</a><ul><li><a href="../topics/impala_query_options.html">Query Options for the SET Statement</a><ul><li><a href="../topics/impala_abort_on_error.html">ABORT_ON_ERROR</a></li><li><a href="../topics/impala_allow_erasure_coded_files.html">ALLOW_ERASURE_CODED_FILES</a></li><li><a href="../topics/impala_allow_unsupported_formats.html">ALLOW_UNSUPPORTED_FORMATS</a></li><li><a href="../topics/impala_appx_count_distinct.html">APPX_COUNT_DISTINCT</a></li><li><a href="../topics/impala_batch_size.html">BATCH_SIZE</a></li><li><a href="../topics/impala_broadcast_bytes_limit.html">BROADCAST_BYTES_LIMIT</a></li><li><a href="../topics/impala_buffer_pool_limit.html">BUFFER_POOL_LIMIT</a></li><li><a href="../topics/impala_compression_codec.html">COMPRESSION_CODEC</a></li><li><a href="../topics/impala_compute_stats_min_sample_size.html">COMPUTE_STATS_MIN_SAMPLE_SIZE</a></li><li><a href="../topics/impala_debug_action.html">DEBUG_ACTION</a></li><li><a href="../topics/impala_decimal_v2.html">DECIMAL_V2</a></li><li><a href="../topics/impala_default_file_format.html">DEFAULT_FILE_FORMAT</a></li><li><a href="../topics/impala_default_hints_insert_statement.html">DEFAULT_HINTS_INSERT_STATEMENT</a></li><li><a href="../topics/impala_default_join_distribution_mode.html">DEFAULT_JOIN_DISTRIBUTION_MODE</a></li><li><a href="../topics/impala_default_spillable_buffer_size.html">DEFAULT_SPILLABLE_BUFFER_SIZE</a></li><li><a href="../topics/impala_default_transactional_type.html">DEFAULT_TRANSACTIONAL_TYPE</a></li><li><a href="../topics/impala_delete_stats_in_truncate.html">DELETE_STATS_IN_TRUNCATE</a></li><li><a href="../topics/impala_disable_codegen.html">DISABLE_CODEGEN</a></li><li><a href="../topics/impala_disable_codegen_rows_threshold.html">DISABLE_CODEGEN_ROWS_THRESHOLD</a></li><li><a href="../topics/impala_disable_hbase_num_rows_estimate.html">DISABLE_HBASE_NUM_ROWS_ESTIMATE</a></li><li><a href="../topics/impala_disable_row_runtime_filtering.html">DISABLE_ROW_RUNTIME_FILTERING</a></li><li><a href="../topics/impala_disable_streaming_preaggregations.html">DISABLE_STREAMING_PREAGGREGATIONS</a></li><li><a href="../topics/impala_disable_unsafe_spills.html">DISABLE_UNSAFE_SPILLS</a></li><li><a href="../topics/impala_enable_expr_rewrites.html">ENABLE_EXPR_REWRITES</a></li><li><a href="../topics/impala_exec_single_node_rows_threshold.html">EXEC_SINGLE_NODE_ROWS_THRESHOLD</a></li><li><a href="../topics/impala_exec_time_limit_s.html">EXEC_TIME_LIMIT_S</a></li><li><a href="../topics/impala_explain_level.html">EXPLAIN_LEVEL</a></li><li><a href="../topics/impala_fetch_rows_timeout_ms.html">FETCH_ROWS_TIMEOUT_MS</a></li><li><a href="../topics/impala_join_rows_produced_limit.html">JOIN_ROWS_PRODUCED_LIMIT</a></li><li><a href="../topics/impala_hbase_cache_blocks.html">HBASE_CACHE_BLOCKS</a></li><li><a href="../topics/impala_hbase_caching.html">HBASE_CACHING</a></li><li><a href="../topics/impala_idle_session_timeout.html">IDLE_SESSION_TIMEOUT</a></li><li><a href="../topics/impala_kudu_read_mode.html">KUDU_READ_MODE</a></li><li><a href="../topics/impala_live_progress.html">LIVE_PROGRESS</a></li><li><a href="../topics/impala_live_summary.html">LIVE_SUMMARY</a></li><li><a href="../topics/impala_max_errors.html">MAX_ERRORS</a></li><li><a href="../topics/impala_max_mem_estimate_for_admission.html">MAX_MEM_ESTIMATE_FOR_ADMISSION</a></li><li><a href="../topics/impala_max_result_spooling_mem.html">MAX_RESULT_SPOOLING_MEM</a></li><li><a href="../topics/impala_max_row_size.html">MAX_ROW_SIZE</a></li><li><a href="../topics/impala_max_scan_range_length.html">MAX_SCAN_RANGE_LENGTH</a></li><li><a href="../topics/impala_max_spilled_result_spooling_mem.html">MAX_SPILLED_RESULT_SPOOLING_MEM</a></li><li class="active"><a href="../topics/impala_mem_limit.html">MEM_LIMIT</a></li><li><a href="../topics/impala_min_spillable_buffer_size.html">MIN_SPILLABLE_BUFFER_SIZE</a></li><li><a href="../topics/impala_mt_dop.html">MT_DOP</a></li><li><a href="../topics/impala_num_nodes.html">NUM_NODES</a></li><li><a href="../topics/impala_num_rows_produced_limit.html">NUM_ROWS_PRODUCED_LIMIT</a></li><li><a href="../topics/impala_num_scanner_threads.html">NUM_SCANNER_THREADS</a></li><li><a href="../topics/impala_optimize_partition_key_scans.html">OPTIMIZE_PARTITION_KEY_SCANS</a></li><li><a href="../topics/impala_parquet_compression_codec.html">PARQUET_COMPRESSION_CODEC</a></li><li><a href="../topics/impala_parquet_annotate_strings_utf8.html">PARQUET_ANNOTATE_STRINGS_UTF8</a></li><li><a href="../topics/impala_parquet_array_resolution.html">PARQUET_ARRAY_RESOLUTION</a></li><li><a href="../topics/impala_parquet_dictionary_filtering.html">PARQUET_DICTIONARY_FILTERING</a></li><li><a href="../topics/impala_parquet_fallback_schema_resolution.html">PARQUET_FALLBACK_SCHEMA_RESOLUTION</a></li><li><a href="../topics/impala_parquet_file_size.html">PARQUET_FILE_SIZE</a></li><li><a href="../topics/impala_parquet_object_store_split_size.html">PARQUET_OBJECT_STORE_SPLIT_SIZE</a></li><li><a href="../topics/impala_parquet_page_row_count_limit.html">PARQUET_PAGE_ROW_COUNT_LIMIT</a></li><li><a href="../topics/impala_parquet_read_statistics.html">PARQUET_READ_STATISTICS</a></li><li><a href="../topics/impala_parquet_read_page_index.html">PARQUET_READ_PAGE_INDEX</a></li><li><a href="../topics/impala_parquet_write_page_index.html">PARQUET_WRITE_PAGE_INDEX</a></li><li><a href="../topics/impala_prefetch_mode.html">PREFETCH_MODE</a></li><li><a href="../topics/impala_query_timeout_s.html">QUERY_TIMEOUT_S</a></li><li><a href="../topics/impala_refresh_updated_hms.html">REFRESH_UPDATED_HMS_PARTITIONS</a></li><li><a href="../topics/impala_replica_preference.html">REPLICA_PREFERENCE</a></li><li><a href="../topics/impala_request_pool.html">REQUEST_POOL</a></li><li><a href="../topics/impala_resource_trace_ratio.html">RESOURCE_TRACE_RATIO</a></li><li><a href="../topics/impala_retry_failed_queries.html">RETRY_FAILED_QUERIES</a></li><li><a href="../topics/impala_runtime_bloom_filter_size.html">RUNTIME_BLOOM_FILTER_SIZE</a></li><li><a href="../topics/impala_runtime_filter_max_size.html">RUNTIME_FILTER_MAX_SIZE</a></li><li><a href="../topics/impala_runtime_filter_min_size.html">RUNTIME_FILTER_MIN_SIZE</a></li><li><a href="../topics/impala_runtime_filter_mode.html">RUNTIME_FILTER_MODE</a></li><li><a href="../topics/impala_runtime_filter_wait_time_ms.html">RUNTIME_FILTER_WAIT_TIME_MS</a></li><li><a href="../topics/impala_s3_skip_insert_staging.html">S3_SKIP_INSERT_STAGING</a></li><li><a href="../topics/impala_scan_bytes_limit.html">SCAN_BYTES_LIMIT</a></li><li><a href="../topics/impala_schedule_random_replica.html">SCHEDULE_RANDOM_REPLICA</a></li><li><a href="../topics/impala_scratch_limit.html">SCRATCH_LIMIT</a></li><li><a href="../topics/impala_shuffle_distinct_exprs.html">SHUFFLE_DISTINCT_EXPRS</a></li><li><a href="../topics/impala_spool_query_results.html">SPOOL_QUERY_RESULTS</a></li><li><a href="../topics/impala_support_start_over.html">SUPPORT_START_OVER</a></li><li><a href="../topics/impala_sync_ddl.html">SYNC_DDL</a></li><li><a href="../topics/impala_thread_reservation_aggregate_limit.html">THREAD_RESERVATION_AGGREGATE_LIMIT</a></li><li><a href="../topics/impala_thread_reservation_limit.html">THREAD_RESERVATION_LIMIT</a></li><li><a href="../topics/impala_timezone.html">TIMEZONE</a></li><li><a href="../topics/impala_topn_bytes_limit.html">TOPN_BYTES_LIMIT</a></li><li><a href="../topics/impala_utf8_mode.html">UTF8_MODE</a></li><li><a href="../topics/impala_expand_complex_types.html">EXPAND_COMPLEX_TYPES</a></li></ul></li></ul></li><li><a href="../topics/impala_show.html">SHOW</a></li><li><a href="../topics/impala_shutdown.html">SHUTDOWN</a></li><li><a href="../topics/impala_truncate_table.html">TRUNCATE TABLE</a></li><li><a href="../topics/impala_update.html">UPDATE</a></li><li><a href="../topics/impala_upsert.html">UPSERT</a></li><li><a href="../topics/impala_use.html">USE</a></li><li><a href="../topics/impala_values.html">VALUES</a></li><li><a href="../topics/impala_hints.html">Optimizer Hints</a></li></ul></li><li><a href="../topics/impala_functions.html">Built-In Functions</a></li><li><a href="../topics/impala_udf.html">User-Defined Functions (UDFs)</a></li><li><a href="../topics/impala_langref_unsupported.html">SQL Differences Between Impala and Hive</a></li><li><a href="../topics/impala_porting.html">Porting SQL</a></li><li><a href="../topics/impala_utf_8.html">UTF-8 Support</a></li></ul></li><li><a href="../topics/impala_performance.html">Performance Tuning</a></li><li><a href="../topics/impala_scalability.html">Scalability Considerations</a></li><li><a href="../topics/impala_resource_management.html">Resource Management</a></li><li><a href="../topics/impala_partitioning.html">Partitioning</a></li><li><a href="../topics/impala_file_formats.html">File Formats</a></li><li><a href="../topics/impala_kudu.html">Using Impala to Query Kudu Tables</a></li><li><a href="../topics/impala_hbase.html">HBase Tables</a></li><li><a href="../topics/impala_iceberg.html">Iceberg Tables</a></li><li><a href="../topics/impala_s3.html">S3 Tables</a></li><li><a href="../topics/impala_adls.html">ADLS Tables</a></li><li><a href="../topics/impala_isilon.html">Isilon Storage</a></li><li><a href="../topics/impala_ozone.html">Ozone Storage</a></li><li><a href="../topics/impala_logging.html">Logging</a></li><li><a href="../topics/impala_client.html">Client Access</a></li><li><a href="../topics/impala_fault_tolerance.html">Fault Tolerance</a></li><li><a href="../topics/impala_troubleshooting.html">Troubleshooting Impala</a></li><li><a href="../topics/impala_ports.html">Ports Used by Impala</a></li><li><a href="../topics/impala_reserved_words.html">Impala Reserved Words</a></li><li><a href="../topics/impala_faq.html">Impala Frequently Asked Questions</a></li><li><a href="../topics/impala_release_notes.html">Impala Release Notes</a></li></ul></nav><main role="main"><article role="article" aria-labelledby="ariaid-title1">
<h1 class="title topictitle1" id="ariaid-title1">MEM_LIMIT Query Option</h1>
<div class="body conbody">
<p class="p">
The MEM_LIMIT query option defines the maximum amount of memory a query can allocate on
each node. The total memory that can be used by a query is the <code class="ph codeph">MEM_LIMIT</code>
times the number of nodes.
</p>
<p class="p">
There are two levels of memory limit for Impala. The
<code class="ph codeph">‑‑mem_limit</code> startup option sets an overall limit for the
<span class="keyword cmdname">impalad</span> process (which handles multiple queries concurrently). That
process memory limit can be expressed either as a percentage of RAM available to the
process such as <code class="ph codeph">‑‑mem_limit=70%</code> or as a fixed amount of
memory, such as <code class="ph codeph">100gb</code>. The memory available to the process is based on
the host's physical memory and, since Impala 3.2, memory limits from Linux Control Groups.
E.g. if an <span class="keyword cmdname">impalad</span> process is running in a Docker container on a host
with 100GB of memory, the memory available is 100GB or the Docker container's memory
limit, whichever is less.
</p>
<p class="p">
The <code class="ph codeph">MEM_LIMIT</code> query option, which you set through
<span class="keyword cmdname">impala-shell</span> or the <code class="ph codeph">SET</code> statement in a JDBC or ODBC
application, applies to each individual query. The <code class="ph codeph">MEM_LIMIT</code> query option
is usually expressed as a fixed size such as <code class="ph codeph">10gb</code>, and must always be
less than the <span class="keyword cmdname">impalad</span> memory limit.
</p>
<p class="p">
If query processing approaches the specified memory limit on any node, either the
per-query limit or the impalad limit, then the SQL operations will start to reduce
their memory consumption, for example by writing the temporary data to disk (known as spilling to disk).
The result is a query that completes successfully, rather than failing with an out-of-memory error.
The tradeoff is decreased performance due to the extra disk I/O to write the temporary data and
read it back in. The slowdown could potentially be significant. Thus, while this feature improves
reliability, you should optimize your queries, system parameters, and hardware configuration to
make this spilling a rare occurrence.
</p>
<p class="p">
<strong class="ph b">Type:</strong> numeric
</p>
<p class="p">
<strong class="ph b">Units:</strong> A numeric argument represents memory size in bytes; you can also use a
suffix of <code class="ph codeph">m</code> or <code class="ph codeph">mb</code> for megabytes, or more commonly
<code class="ph codeph">g</code> or <code class="ph codeph">gb</code> for gigabytes. If you specify a value with
unrecognized formats, subsequent queries fail with an error.
</p>
<p class="p">
<strong class="ph b">Default:</strong> 0 (unlimited)
</p>
<p class="p">
<strong class="ph b">Usage notes:</strong>
</p>
<p class="p">
The <code class="ph codeph">MEM_LIMIT</code> setting is primarily useful for production workloads.
Impala's Admission Controller can be configured to automatically assign memory limits to
queries and limit memory consumption of resource pools. See <a class="xref" href="impala_admission.html#admission_concurrency">Concurrent Queries and Admission Control</a>
and <a class="xref" href="impala_admission.html#admission_memory">Memory Limits and Admission Control</a> for more information on configuring
the resource usage through admission control.
</p>
<p class="p">
Use the output of the <code class="ph codeph">SUMMARY</code> command in <span class="keyword cmdname">impala-shell</span>
to get a report of memory used for each phase of your most heavyweight queries on each
node, and then set a <code class="ph codeph">MEM_LIMIT</code> somewhat higher than that. See
<a class="xref" href="impala_explain_plan.html#perf_summary">Using the SUMMARY Report for Performance Tuning</a> for usage information about the
<code class="ph codeph">SUMMARY</code> command.
</p>
<p class="p">
<strong class="ph b">Examples:</strong>
</p>
<p class="p">
The following examples show how to set the <code class="ph codeph">MEM_LIMIT</code> query option using a
fixed number of bytes, or suffixes representing gigabytes or megabytes.
</p>
<pre class="pre codeblock"><code>
[localhost:21000] &gt; set mem_limit=3000000000;
MEM_LIMIT set to 3000000000
[localhost:21000] &gt; select 5;
Query: select 5
+---+
| 5 |
+---+
| 5 |
+---+
[localhost:21000] &gt; set mem_limit=3g;
MEM_LIMIT set to 3g
[localhost:21000] &gt; select 5;
Query: select 5
+---+
| 5 |
+---+
| 5 |
+---+
[localhost:21000] &gt; set mem_limit=3gb;
MEM_LIMIT set to 3gb
[localhost:21000] &gt; select 5;
+---+
| 5 |
+---+
| 5 |
+---+
[localhost:21000] &gt; set mem_limit=3m;
MEM_LIMIT set to 3m
[localhost:21000] &gt; select 5;
+---+
| 5 |
+---+
| 5 |
+---+
[localhost:21000] &gt; set mem_limit=3mb;
MEM_LIMIT set to 3mb
[localhost:21000] &gt; select 5;
+---+
| 5 |
+---+
</code></pre>
<p class="p">
The following examples show how unrecognized <code class="ph codeph">MEM_LIMIT</code> values lead to
errors for subsequent queries.
</p>
<pre class="pre codeblock"><code>
[localhost:21000] &gt; set mem_limit=3pb;
MEM_LIMIT set to 3pb
[localhost:21000] &gt; select 5;
ERROR: Failed to parse query memory limit from '3pb'.
[localhost:21000] &gt; set mem_limit=xyz;
MEM_LIMIT set to xyz
[localhost:21000] &gt; select 5;
Query: select 5
ERROR: Failed to parse query memory limit from 'xyz'.
</code></pre>
<p class="p">
The following examples shows the automatic query cancellation when the
<code class="ph codeph">MEM_LIMIT</code> value is exceeded on any host involved in the Impala query.
First it runs a successful query and checks the largest amount of memory used on any node
for any stage of the query. Then it sets an artificially low <code class="ph codeph">MEM_LIMIT</code>
setting so that the same query cannot run.
</p>
<pre class="pre codeblock"><code>
[localhost:21000] &gt; select count(*) from customer;
Query: select count(*) from customer
+----------+
| count(*) |
+----------+
| 150000 |
+----------+
[localhost:21000] &gt; select count(distinct c_name) from customer;
Query: select count(distinct c_name) from customer
+------------------------+
| count(distinct c_name) |
+------------------------+
| 150000 |
+------------------------+
[localhost:21000] &gt; summary;
+--------------+--------+--------+----------+----------+---------+------------+----------+---------------+---------------+
| Operator | #Hosts | #Inst | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail |
+--------------+--------+--------+----------+----------+---------+------------+----------+---------------+---------------+
| 06:AGGREGATE | 1 | 1 | 230.00ms | 230.00ms | 1 | 1 | 16.00 KB | -1 B | FINALIZE |
| 05:EXCHANGE | 1 | 1 | 43.44us | 43.44us | 1 | 1 | 0 B | -1 B | UNPARTITIONED |
| 02:AGGREGATE | 1 | 1 | 227.14ms | 227.14ms | 1 | 1 | 12.00 KB | 10.00 MB | |
| 04:AGGREGATE | 1 | 1 | 126.27ms | 126.27ms | 150.00K | 150.00K | 15.17 MB | 10.00 MB | |
| 03:EXCHANGE | 1 | 1 | 44.07ms | 44.07ms | 150.00K | 150.00K | 0 B | 0 B | HASH(c_name) |
<strong class="ph b">| 01:AGGREGATE | 1 | 1 | 361.94ms | 361.94ms | 150.00K | 150.00K | 23.04 MB | 10.00 MB | |</strong>
| 00:SCAN HDFS | 1 | 1 | 43.64ms | 43.64ms | 150.00K | 150.00K | 24.19 MB | 64.00 MB | tpch.customer |
+--------------+--------+--------+----------+----------+---------+------------+----------+---------------+---------------+
[localhost:21000] &gt; set mem_limit=15mb;
MEM_LIMIT set to 15mb
[localhost:21000] &gt; select count(distinct c_name) from customer;
Query: select count(distinct c_name) from customer
ERROR:
Rejected query from pool default-pool: minimum memory reservation is greater than memory available to the query
for buffer reservations. Memory reservation needed given the current plan: 38.00 MB. Adjust either the mem_limit
or the pool config (max-query-mem-limit, min-query-mem-limit) for the query to allow the query memory limit to be
at least 70.00 MB. Note that changing the mem_limit may also change the plan. See the query profile for more
information about the per-node memory requirements.</code></pre>
</div>
<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_query_options.html">Query Options for the SET Statement</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="mem_limit__mem_limit_executors">
<h2 class="title topictitle2" id="ariaid-title2">MEM_LIMIT_EXECUTORS Query Option</h2>
<div class="body conbody">
<div class="note note note_note"><span class="note__title notetitle">Note:</span> This is an advanced query option. Setting this query option is not recommended
unless specifically advised.</div>
<p class="p">The existing <code class="ph codeph">MEM_LIMIT</code> query option applies to all impala coordinators and
executors. This means that the same amount of memory gets reserved but coordinators
typically just do the job of coordinating the query and thus do not necessarily need all the
estimated memory. Blocking the estimated memory on coordinators blocks the memory to be used
for other queries.</p>
<p class="p">The new <code class="ph codeph">MEM_LIMIT_EXECUTORS</code> query option functions similarly to the
<code class="ph codeph">MEM_LIMIT</code> option but sets the query memory limit only on executors. This
new option addresses the issue related to <code class="ph codeph">MEM_LIMIT</code> and is recommended in
scenarios where the query needs much higher memory on executors compared with
coordinators.</p>
<p class="p">Note that the <code class="ph codeph">MEM_LIMIT_EXECUTORS</code> option does not work with
<code class="ph codeph">MEM_LIMIT</code>. If you set both, only <code class="ph codeph">MEM_LIMIT</code> applies.</p>
</div>
</article>
<article class="topic concept nested1" aria-labelledby="ariaid-title3" id="mem_limit__mem_limit_coordinators">
<h2 class="title topictitle2" id="ariaid-title3">MEM_LIMIT_COORDINATORS Query Option</h2>
<div class="body conbody">
<div class="note note note_note"><span class="note__title notetitle">Note:</span> This is an advanced query option. Setting this query option is not recommended
unless specifically advised.</div>
<p class="p">The existing <code class="ph codeph">MEM_LIMIT</code> query option applies to all impala coordinators and
executors. This means that the same amount of memory gets reserved but coordinators
typically just do the job of coordinating the query and thus do not necessarily need all the
estimated memory. Blocking the estimated memory on coordinators blocks the memory to be used
for other queries.</p>
<p class="p">The new <code class="ph codeph">MEM_LIMIT_COORDINATORS</code> query option functions similarly to the
<code class="ph codeph">MEM_LIMIT</code> option but sets the query memory limit only on coordinators. This
new option addresses the issue related to <code class="ph codeph">MEM_LIMIT</code> and is recommended in
scenarios where the query needs higher or lower memory on coordinators compared to the planner
estimates.</p>
<p class="p">Note that the <code class="ph codeph">MEM_LIMIT_COORDINATORS</code> option does not work with
<code class="ph codeph">MEM_LIMIT</code>. If you set both, only <code class="ph codeph">MEM_LIMIT</code> applies.</p>
</div>
</article>
</article></main></body></html>