| <!DOCTYPE html |
| SYSTEM "about:legacy-compat"> |
| <html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2023"><meta name="DC.rights.owner" content="(C) Copyright 2023"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_set.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_max_num_runtime_filters.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.4.x"><meta name="version" content="Impala 3.4.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="explain_level"><link rel="stylesheet" type="text/css" href="../css/commonltr.css"><link rel="stylesheet" type="text/css" href="../css/dita-ot-doc.css"><title>EXPLAIN_LEVEL Query Option</title></head><body id="explain_level"><header role="banner"><!-- |
| The DITA Open Toolkit is licensed for use under the the Apache |
| Software Foundation License v2.0. |
| |
| A copy of the Apache Software Foundation License 2.0 is |
| available at http://opensource.org/licenses/apache2.0.php |
| |
| This statement must be included in any copies of DITA Open |
| Toolkit code. |
| --><div class="header"> |
| <p>Apache Impala</p> |
| <hr> |
| </div></header><nav role="toc"><ul><li><a href="../topics/impala_intro.html">Introducing Apache Impala</a></li><li><a href="../topics/impala_concepts.html">Concepts and Architecture</a></li><li><a href="../topics/impala_planning.html">Deployment Planning</a></li><li><a href="../topics/impala_install.html">Installing Impala</a></li><li><a href="../topics/impala_config.html">Managing Impala</a></li><li><a href="../topics/impala_upgrading.html">Upgrading Impala</a></li><li><a href="../topics/impala_processes.html">Starting Impala</a></li><li><a href="../topics/impala_tutorial.html">Tutorials</a></li><li><a href="../topics/impala_admin.html">Administration</a></li><li><a href="../topics/impala_security.html">Impala Security</a></li><li><a href="../topics/impala_langref.html">SQL Reference</a><ul><li><a href="../topics/impala_comments.html">Comments</a></li><li><a href="../topics/impala_datatypes.html">Data Types</a></li><li><a href="../topics/impala_literals.html">Literals</a></li><li><a href="../topics/impala_operators.html">SQL Operators</a></li><li><a href="../topics/impala_schema_objects.html">Schema Objects and Object Names</a></li><li><a href="../topics/impala_transactions.html">Transactions</a></li><li><a href="../topics/impala_langref_sql.html">SQL Statements</a><ul><li><a href="../topics/impala_ddl.html">DDL Statements</a></li><li><a href="../topics/impala_dml.html">DML Statements</a></li><li><a href="../topics/impala_alter_database.html">ALTER DATABASE</a></li><li><a href="../topics/impala_alter_table.html">ALTER TABLE</a></li><li><a href="../topics/impala_alter_view.html">ALTER VIEW</a></li><li><a href="../topics/impala_comment.html">COMMENT</a></li><li><a href="../topics/impala_compute_stats.html">COMPUTE STATS</a></li><li><a href="../topics/impala_create_database.html">CREATE DATABASE</a></li><li><a href="../topics/impala_create_function.html">CREATE FUNCTION</a></li><li><a href="../topics/impala_create_role.html">CREATE ROLE</a></li><li><a href="../topics/impala_create_table.html">CREATE TABLE</a></li><li><a href="../topics/impala_create_view.html">CREATE VIEW</a></li><li><a href="../topics/impala_delete.html">DELETE</a></li><li><a href="../topics/impala_describe.html">DESCRIBE</a></li><li><a href="../topics/impala_drop_database.html">DROP DATABASE</a></li><li><a href="../topics/impala_drop_function.html">DROP FUNCTION</a></li><li><a href="../topics/impala_drop_role.html">DROP ROLE</a></li><li><a href="../topics/impala_drop_stats.html">DROP STATS</a></li><li><a href="../topics/impala_drop_table.html">DROP TABLE</a></li><li><a href="../topics/impala_drop_view.html">DROP VIEW</a></li><li><a href="../topics/impala_explain.html">EXPLAIN</a></li><li><a href="../topics/impala_grant.html">GRANT</a></li><li><a href="../topics/impala_insert.html">INSERT</a></li><li><a href="../topics/impala_invalidate_metadata.html">INVALIDATE METADATA</a></li><li><a href="../topics/impala_load_data.html">LOAD DATA</a></li><li><a href="../topics/impala_refresh.html">REFRESH</a></li><li><a href="../topics/impala_refresh_authorization.html">REFRESH AUTHORIZATION</a></li><li><a href="../topics/impala_refresh_functions.html">REFRESH FUNCTIONS</a></li><li><a href="../topics/impala_revoke.html">REVOKE</a></li><li><a href="../topics/impala_select.html">SELECT</a></li><li><a href="../topics/impala_set.html">SET</a><ul><li><a href="../topics/impala_abort_on_error.html">ABORT ON ERROR</a></li><li><a href="../topics/impala_allow_erasure_coded_files.html">ALLOW ERASURE CODED FILES</a></li><li><a href="../topics/impala_allow_unsupported_formats.html">ALLOW UNSUPPORTED FORMATS</a></li><li><a href="../topics/impala_appx_count_distinct.html">APPX COUNT DISTINCT</a></li><li><a href="../topics/impala_batch_size.html">BATCH SIZE</a></li><li><a href="../topics/impala_broadcast_bytes_limit.html">BROADCAST BYTES LIMIT</a></li><li><a href="../topics/impala_buffer_pool_limit.html">BUFFER POOL LIMIT</a></li><li><a href="../topics/impala_compression_codec.html">COMPRESSION CODEC</a></li><li><a href="../topics/impala_compute_stats_min_sample_size.html">COMPUTE STATS MIN SAMPLE SIZE</a></li><li><a href="../topics/impala_debug_action.html">DEBUG ACTION</a></li><li><a href="../topics/impala_decimal_v2.html">DECIMAL V2</a></li><li><a href="../topics/impala_default_file_format.html">DEFAULT FILE FORMAT</a></li><li><a href="../topics/impala_default_hints_insert_statement.html">DEFAULT HINTS INSERT STATEMENT</a></li><li><a href="../topics/impala_default_join_distribution_mode.html">DEFAULT JOIN DISTRIBUTION MODE</a></li><li><a href="../topics/impala_default_spillable_buffer_size.html">DEFAULT SPILLABLE BUFFER SIZE</a></li><li><a href="../topics/impala_default_transactional_type.html">DEFAULT TRANSACTIONAL TYPE</a></li><li><a href="../topics/impala_delete_stats_in_truncate.html">DELETE STATS IN TRUNCATE</a></li><li><a href="../topics/impala_disable_codegen.html">DISABLE CODEGEN</a></li><li><a href="../topics/impala_disable_codegen_rows_threshold.html">DISABLE CODEGEN ROWS THRESHOLD</a></li><li><a href="../topics/impala_disable_hbase_num_rows_estimate.html">DISABLE HBASE NUM ROWS ESTIMATE</a></li><li><a href="../topics/impala_disable_row_runtime_filtering.html">DISABLE ROW RUNTIME FILTERING</a></li><li><a href="../topics/impala_disable_streaming_preaggregations.html">DISABLE STREAMING PREAGGREGATIONS</a></li><li><a href="../topics/impala_disable_unsafe_spills.html">DISABLE UNSAFE SPILLS</a></li><li><a href="../topics/impala_enable_expr_rewrites.html">ENABLE EXPR REWRITES</a></li><li><a href="../topics/impala_exec_single_node_rows_threshold.html">EXEC SINGLE NODE ROWS THRESHOLD</a></li><li><a href="../topics/impala_exec_time_limit_s.html">EXEC TIME LIMIT S</a></li><li class="active"><a href="../topics/impala_explain_level.html">EXPLAIN LEVEL</a><ul><li><a href="../topics/impala_max_num_runtime_filters.html">MAX NUM RUNTIME FILTERS</a></li></ul></li><li><a href="../topics/impala_fetch_rows_timeout_ms.html">FETCH ROWS TIMEOUT MS</a></li><li><a href="../topics/impala_join_rows_produced_limit.html">JOIN ROWS PRODUCED LIMIT</a></li><li><a href="../topics/impala_hbase_cache_blocks.html">HBASE CACHE BLOCKS</a></li><li><a href="../topics/impala_hbase_caching.html">HBASE CACHING</a></li><li><a href="../topics/impala_idle_session_timeout.html">IDLE SESSION TIMEOUT</a></li><li><a href="../topics/impala_kudu_read_mode.html">KUDU READ MODE</a></li><li><a href="../topics/impala_live_progress.html">LIVE PROGRESS</a></li><li><a href="../topics/impala_live_summary.html">LIVE SUMMARY</a></li><li><a href="../topics/impala_max_errors.html">MAX ERRORS</a></li><li><a href="../topics/impala_max_mem_estimate_for_admission.html">MAX MEM ESTIMATE FOR ADMISSION</a></li><li><a href="../topics/impala_max_result_spooling_mem.html">MAX RESULT SPOOLING MEM</a></li><li><a href="../topics/impala_max_row_size.html">MAX ROW SIZE</a></li><li><a href="../topics/impala_max_scan_range_length.html">MAX SCAN RANGE LENGTH</a></li><li><a href="../topics/impala_max_spilled_result_spooling_mem.html">MAX SPILLED RESULT SPOOLING MEM</a></li><li><a href="../topics/impala_mem_limit.html">MEM LIMIT</a></li><li><a href="../topics/impala_min_spillable_buffer_size.html">MIN SPILLABLE BUFFER SIZE</a></li><li><a href="../topics/impala_mt_dop.html">MT DOP</a></li><li><a href="../topics/impala_num_nodes.html">NUM NODES</a></li><li><a href="../topics/impala_num_rows_produced_limit.html">NUM ROWS PRODUCED LIMIT</a></li><li><a href="../topics/impala_num_scanner_threads.html">NUM SCANNER THREADS</a></li><li><a href="../topics/impala_optimize_partition_key_scans.html">OPTIMIZE PARTITION KEY SCANS</a></li><li><a href="../topics/impala_parquet_compression_codec.html">PARQUET COMPRESSION CODEC</a></li><li><a href="../topics/impala_parquet_annotate_strings_utf8.html">PARQUET ANNOTATE STRINGS UTF8</a></li><li><a href="../topics/impala_parquet_array_resolution.html">PARQUET ARRAY RESOLUTION</a></li><li><a href="../topics/impala_parquet_dictionary_filtering.html">PARQUET DICTIONARY FILTERING</a></li><li><a href="../topics/impala_parquet_fallback_schema_resolution.html">PARQUET FALLBACK SCHEMA RESOLUTION</a></li><li><a href="../topics/impala_parquet_file_size.html">PARQUET FILE SIZE</a></li><li><a href="../topics/impala_parquet_object_store_split_size.html">PARQUET OBJECT STORE SPLIT SIZE</a></li><li><a href="../topics/impala_parquet_page_row_count_limit.html">PARQUET PAGE ROW COUNT LIMIT</a></li><li><a href="../topics/impala_parquet_read_statistics.html">PARQUET READ STATISTICS</a></li><li><a href="../topics/impala_parquet_read_page_index.html">PARQUET READ PAGE INDEX</a></li><li><a href="../topics/impala_parquet_write_page_index.html">PARQUET WRITE PAGE INDEX</a></li><li><a href="../topics/impala_prefetch_mode.html">PREFETCH MODE</a></li><li><a href="../topics/impala_query_timeout_s.html">QUERY TIMEOUT S</a></li><li><a href="../topics/impala_refresh_updated_hms.html">REFRESH UPDATED HMS PARTITIONS</a></li><li><a href="../topics/impala_replica_preference.html">REPLICA PREFERENCE</a></li><li><a href="../topics/impala_request_pool.html">REQUEST POOL</a></li><li><a href="../topics/impala_resource_trace_ratio.html">RESOURCE TRACE RATIO</a></li><li><a href="../topics/impala_retry_failed_queries.html">RETRY FAILED QUERIES</a></li><li><a href="../topics/impala_runtime_bloom_filter_size.html">RUNTIME BLOOM FILTER SIZE</a></li><li><a href="../topics/impala_runtime_filter_max_size.html">RUNTIME FILTER MAX SIZE</a></li><li><a href="../topics/impala_runtime_filter_min_size.html">RUNTIME FILTER MIN SIZE</a></li><li><a href="../topics/impala_runtime_filter_mode.html">RUNTIME FILTER MODE</a></li><li><a href="../topics/impala_runtime_filter_wait_time_ms.html">RUNTIME FILTER WAIT TIME MS</a></li><li><a href="../topics/impala_s3_skip_insert_staging.html">S3 SKIP INSERT STAGING</a></li><li><a href="../topics/impala_scan_bytes_limit.html">SCAN BYTES LIMIT</a></li><li><a href="../topics/impala_schedule_random_replica.html">SCHEDULE RANDOM REPLICA</a></li><li><a href="../topics/impala_scratch_limit.html">SCRATCH LIMIT</a></li><li><a href="../topics/impala_shuffle_distinct_exprs.html">SHUFFLE DISTINCT EXPRS</a></li><li><a href="../topics/impala_spool_query_results.html">SPOOL QUERY RESULTS</a></li><li><a href="../topics/impala_support_start_over.html">SUPPORT START OVER</a></li><li><a href="../topics/impala_sync_ddl.html">SYNC DDL</a></li><li><a href="../topics/impala_thread_reservation_aggregate_limit.html">THREAD RESERVATION AGGREGATE LIMIT</a></li><li><a href="../topics/impala_thread_reservation_limit.html">THREAD RESERVATION LIMIT</a></li><li><a href="../topics/impala_timezone.html">TIMEZONE</a></li><li><a href="../topics/impala_topn_bytes_limit.html">TOPN BYTES LIMIT</a></li><li><a href="../topics/impala_utf8_mode.html">UTF8 MODE</a></li><li><a href="../topics/impala_expand_complex_types.html">EXPAND COMPLEX TYPES</a></li></ul></li><li><a href="../topics/impala_show.html">SHOW</a></li><li><a href="../topics/impala_shutdown.html">SHUTDOWN</a></li><li><a href="../topics/impala_truncate_table.html">TRUNCATE TABLE</a></li><li><a href="../topics/impala_update.html">UPDATE</a></li><li><a href="../topics/impala_upsert.html">UPSERT</a></li><li><a href="../topics/impala_use.html">USE</a></li><li><a href="../topics/impala_values.html">VALUES</a></li><li><a href="../topics/impala_hints.html">Optimizer Hints</a></li></ul></li><li><a href="../topics/impala_functions.html">Built-In Functions</a></li><li><a href="../topics/impala_udf.html">User-Defined Functions (UDFs)</a></li><li><a href="../topics/impala_langref_unsupported.html">SQL Differences Between Impala and Hive</a></li><li><a href="../topics/impala_porting.html">Porting SQL</a></li><li><a href="../topics/impala_utf_8.html">UTF-8 Support</a></li></ul></li><li><a href="../topics/impala_performance.html">Performance Tuning</a></li><li><a href="../topics/impala_scalability.html">Scalability Considerations</a></li><li><a href="../topics/impala_resource_management.html">Resource Management</a></li><li><a href="../topics/impala_partitioning.html">Partitioning</a></li><li><a href="../topics/impala_file_formats.html">File Formats</a></li><li><a href="../topics/impala_kudu.html">Using Impala to Query Kudu Tables</a></li><li><a href="../topics/impala_hbase.html">HBase Tables</a></li><li><a href="../topics/impala_iceberg.html">Iceberg Tables</a></li><li><a href="../topics/impala_s3.html">S3 Tables</a></li><li><a href="../topics/impala_adls.html">ADLS Tables</a></li><li><a href="../topics/impala_isilon.html">Isilon Storage</a></li><li><a href="../topics/impala_ozone.html">Ozone Storage</a></li><li><a href="../topics/impala_logging.html">Logging</a></li><li><a href="../topics/impala_client.html">Client Access</a></li><li><a href="../topics/impala_fault_tolerance.html">Fault Tolerance</a></li><li><a href="../topics/impala_troubleshooting.html">Troubleshooting Impala</a></li><li><a href="../topics/impala_ports.html">Ports Used by Impala</a></li><li><a href="../topics/impala_reserved_words.html">Impala Reserved Words</a></li><li><a href="../topics/impala_faq.html">Impala Frequently Asked Questions</a></li><li><a href="../topics/impala_release_notes.html">Impala Release Notes</a></li></ul></nav><main role="main"><article role="article" aria-labelledby="ariaid-title1"> |
| |
| <h1 class="title topictitle1" id="ariaid-title1">EXPLAIN_LEVEL Query Option</h1> |
| |
| |
| |
| <div class="body conbody"> |
| |
| <p class="p"> Controls the amount of detail provided in the output of the |
| <code class="ph codeph">EXPLAIN</code> statement. The basic output can help you |
| identify high-level performance issues such as scanning a higher volume of |
| data or more partitions than you expect. The higher levels of detail show |
| how intermediate results flow between nodes and how different SQL |
| operations such as <code class="ph codeph">ORDER BY</code>, <code class="ph codeph">GROUP BY</code>, |
| joins, and <code class="ph codeph">WHERE</code> clauses are implemented within a |
| distributed query. </p> |
| |
| <p class="p"> |
| <strong class="ph b">Type:</strong> <code class="ph codeph">STRING</code> or <code class="ph codeph">INT</code> |
| </p> |
| |
| <p class="p"> |
| <strong class="ph b">Default:</strong> <code class="ph codeph">1</code> |
| </p> |
| |
| <p class="p"> |
| <strong class="ph b">Arguments:</strong> |
| </p> |
| |
| <p class="p"> |
| The allowed range of numeric values for this option is 0 to 3: |
| </p> |
| |
| <ul class="ul"> |
| <li class="li"> |
| <code class="ph codeph">0</code> or <code class="ph codeph">MINIMAL</code>: A barebones list, one line per operation. Primarily useful |
| for checking the join order in very long queries where the regular <code class="ph codeph">EXPLAIN</code> output is too |
| long to read easily. |
| </li> |
| |
| <li class="li"> |
| <code class="ph codeph">1</code> or <code class="ph codeph">STANDARD</code>: The default level of detail, showing the logical way that |
| work is split up for the distributed query. |
| </li> |
| |
| <li class="li"> |
| <code class="ph codeph">2</code> or <code class="ph codeph">EXTENDED</code>: Includes additional |
| detail about how the query planner uses statistics in its |
| decision-making process, to understand how a query could be tuned by |
| gathering statistics, using query hints, adding or removing predicates, |
| and so on. In <span class="keyword">Impala 3.2</span> and higher, the output |
| also includes the analyzed query with the cast information in the output |
| header, and the implicit cast info in the Predicate section.</li> |
| |
| <li class="li"> |
| <code class="ph codeph">3</code> or <code class="ph codeph">VERBOSE</code>: The maximum level of detail, showing how work is split up |
| within each node into <span class="q">"query fragments"</span> that are connected in a pipeline. This extra detail is |
| primarily useful for low-level performance testing and tuning within Impala itself, rather than for |
| rewriting the SQL code at the user level. |
| </li> |
| </ul> |
| |
| <div class="note note note_note"><span class="note__title notetitle">Note:</span> |
| Prior to Impala 1.3, the allowed argument range for <code class="ph codeph">EXPLAIN_LEVEL</code> was 0 to 1: level 0 had |
| the mnemonic <code class="ph codeph">NORMAL</code>, and level 1 was <code class="ph codeph">VERBOSE</code>. In Impala 1.3 and higher, |
| <code class="ph codeph">NORMAL</code> is not a valid mnemonic value, and <code class="ph codeph">VERBOSE</code> still applies to the |
| highest level of detail but now corresponds to level 3. You might need to adjust the values if you have any |
| older <code class="ph codeph">impala-shell</code> script files that set the <code class="ph codeph">EXPLAIN_LEVEL</code> query option. |
| </div> |
| |
| <p class="p"> |
| Changing the value of this option controls the amount of detail in the output of the <code class="ph codeph">EXPLAIN</code> |
| statement. The extended information from level 2 or 3 is especially useful during performance tuning, when |
| you need to confirm whether the work for the query is distributed the way you expect, particularly for the |
| most resource-intensive operations such as join queries against large tables, queries against tables with |
| large numbers of partitions, and insert operations for Parquet tables. The extended information also helps to |
| check estimated resource usage when you use the admission control or resource management features explained |
| in <a class="xref" href="impala_resource_management.html#resource_management">Resource Management</a>. See |
| <a class="xref" href="impala_explain.html#explain">EXPLAIN Statement</a> for the syntax of the <code class="ph codeph">EXPLAIN</code> statement, and |
| <a class="xref" href="impala_explain_plan.html#perf_explain">Using the EXPLAIN Plan for Performance Tuning</a> for details about how to use the extended information. |
| </p> |
| |
| <p class="p"> |
| <strong class="ph b">Usage notes:</strong> |
| </p> |
| |
| <p class="p"> |
| As always, read the <code class="ph codeph">EXPLAIN</code> output from bottom to top. The lowest lines represent the |
| initial work of the query (scanning data files), the lines in the middle represent calculations done on each |
| node and how intermediate results are transmitted from one node to another, and the topmost lines represent |
| the final results being sent back to the coordinator node. |
| </p> |
| |
| <p class="p"> |
| The numbers in the left column are generated internally during the initial planning phase and do not |
| represent the actual order of operations, so it is not significant if they appear out of order in the |
| <code class="ph codeph">EXPLAIN</code> output. |
| </p> |
| |
| <p class="p"> |
| At all <code class="ph codeph">EXPLAIN</code> levels, the plan contains a warning if any tables in the query are missing |
| statistics. Use the <code class="ph codeph">COMPUTE STATS</code> statement to gather statistics for each table and suppress |
| this warning. See <a class="xref" href="impala_perf_stats.html#perf_stats">Table and Column Statistics</a> for details about how the statistics help |
| query performance. |
| </p> |
| |
| <p class="p"> |
| The <code class="ph codeph">PROFILE</code> command in <span class="keyword cmdname">impala-shell</span> always starts with an explain plan |
| showing full detail, the same as with <code class="ph codeph">EXPLAIN_LEVEL=3</code>. <span class="ph">After the explain |
| plan comes the executive summary, the same output as produced by the <code class="ph codeph">SUMMARY</code> command in |
| <span class="keyword cmdname">impala-shell</span>.</span> |
| </p> |
| |
| <p class="p"> |
| <strong class="ph b">Examples:</strong> |
| </p> |
| |
| <p class="p"> |
| These examples use a trivial, empty table to illustrate how the essential aspects of query planning are shown |
| in <code class="ph codeph">EXPLAIN</code> output: |
| </p> |
| |
| <pre class="pre codeblock"><code>[localhost:21000] > create table t1 (x int, s string); |
| [localhost:21000] > set explain_level=1; |
| [localhost:21000] > explain select count(*) from t1; |
| +------------------------------------------------------------------------+ |
| | Explain String | |
| +------------------------------------------------------------------------+ |
| | Estimated Per-Host Requirements: Memory=10.00MB VCores=1 | |
| | WARNING: The following tables are missing relevant table and/or column | |
| | statistics. | |
| | explain_plan.t1 | |
| | | |
| | 03:AGGREGATE [MERGE FINALIZE] | |
| | | output: sum(count(*)) | |
| | | | |
| | 02:EXCHANGE [PARTITION=UNPARTITIONED] | |
| | | | |
| | 01:AGGREGATE | |
| | | output: count(*) | |
| | | | |
| | 00:SCAN HDFS [explain_plan.t1] | |
| | partitions=1/1 size=0B | |
| +------------------------------------------------------------------------+ |
| [localhost:21000] > explain select * from t1; |
| +------------------------------------------------------------------------+ |
| | Explain String | |
| +------------------------------------------------------------------------+ |
| | Estimated Per-Host Requirements: Memory=-9223372036854775808B VCores=0 | |
| | WARNING: The following tables are missing relevant table and/or column | |
| | statistics. | |
| | explain_plan.t1 | |
| | | |
| | 01:EXCHANGE [PARTITION=UNPARTITIONED] | |
| | | | |
| | 00:SCAN HDFS [explain_plan.t1] | |
| | partitions=1/1 size=0B | |
| +------------------------------------------------------------------------+ |
| [localhost:21000] > set explain_level=2; |
| [localhost:21000] > explain select * from t1; |
| +------------------------------------------------------------------------+ |
| | Explain String | |
| +------------------------------------------------------------------------+ |
| | Estimated Per-Host Requirements: Memory=-9223372036854775808B VCores=0 | |
| | WARNING: The following tables are missing relevant table and/or column | |
| | statistics. | |
| | explain_plan.t1 | |
| | | |
| | 01:EXCHANGE [PARTITION=UNPARTITIONED] | |
| | | hosts=0 per-host-mem=unavailable | |
| | | tuple-ids=0 row-size=19B cardinality=unavailable | |
| | | | |
| | 00:SCAN HDFS [explain_plan.t1, PARTITION=RANDOM] | |
| | partitions=1/1 size=0B | |
| | table stats: unavailable | |
| | column stats: unavailable | |
| | hosts=0 per-host-mem=0B | |
| | tuple-ids=0 row-size=19B cardinality=unavailable | |
| +------------------------------------------------------------------------+ |
| [localhost:21000] > set explain_level=3; |
| [localhost:21000] > explain select * from t1; |
| +------------------------------------------------------------------------+ |
| | Explain String | |
| +------------------------------------------------------------------------+ |
| | Estimated Per-Host Requirements: Memory=-9223372036854775808B VCores=0 | |
| <strong class="ph b">| WARNING: The following tables are missing relevant table and/or column |</strong> |
| <strong class="ph b">| statistics. |</strong> |
| <strong class="ph b">| explain_plan.t1 |</strong> |
| | | |
| | F01:PLAN FRAGMENT [PARTITION=UNPARTITIONED] | |
| | 01:EXCHANGE [PARTITION=UNPARTITIONED] | |
| | hosts=0 per-host-mem=unavailable | |
| | tuple-ids=0 row-size=19B cardinality=unavailable | |
| | | |
| | F00:PLAN FRAGMENT [PARTITION=RANDOM] | |
| | DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=01, PARTITION=UNPARTITIONED] | |
| | 00:SCAN HDFS [explain_plan.t1, PARTITION=RANDOM] | |
| | partitions=1/1 size=0B | |
| <strong class="ph b">| table stats: unavailable |</strong> |
| <strong class="ph b">| column stats: unavailable |</strong> |
| | hosts=0 per-host-mem=0B | |
| | tuple-ids=0 row-size=19B cardinality=unavailable | |
| +------------------------------------------------------------------------+ |
| </code></pre> |
| |
| <p class="p"> |
| As the warning message demonstrates, most of the information needed for Impala to do efficient query |
| planning, and for you to understand the performance characteristics of the query, requires running the |
| <code class="ph codeph">COMPUTE STATS</code> statement for the table: |
| </p> |
| |
| <pre class="pre codeblock"><code>[localhost:21000] > compute stats t1; |
| +-----------------------------------------+ |
| | summary | |
| +-----------------------------------------+ |
| | Updated 1 partition(s) and 2 column(s). | |
| +-----------------------------------------+ |
| [localhost:21000] > explain select * from t1; |
| +------------------------------------------------------------------------+ |
| | Explain String | |
| +------------------------------------------------------------------------+ |
| | Estimated Per-Host Requirements: Memory=-9223372036854775808B VCores=0 | |
| | | |
| | F01:PLAN FRAGMENT [PARTITION=UNPARTITIONED] | |
| | 01:EXCHANGE [PARTITION=UNPARTITIONED] | |
| | hosts=0 per-host-mem=unavailable | |
| | tuple-ids=0 row-size=20B cardinality=0 | |
| | | |
| | F00:PLAN FRAGMENT [PARTITION=RANDOM] | |
| | DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=01, PARTITION=UNPARTITIONED] | |
| | 00:SCAN HDFS [explain_plan.t1, PARTITION=RANDOM] | |
| | partitions=1/1 size=0B | |
| <strong class="ph b">| table stats: 0 rows total |</strong> |
| <strong class="ph b">| column stats: all |</strong> |
| | hosts=0 per-host-mem=0B | |
| | tuple-ids=0 row-size=20B cardinality=0 | |
| +------------------------------------------------------------------------+ |
| </code></pre> |
| |
| <p class="p"> |
| Joins and other complicated, multi-part queries are the ones where you most commonly need to examine the |
| <code class="ph codeph">EXPLAIN</code> output and customize the amount of detail in the output. This example shows the |
| default <code class="ph codeph">EXPLAIN</code> output for a three-way join query, then the equivalent output with a |
| <code class="ph codeph">[SHUFFLE]</code> hint to change the join mechanism between the first two tables from a broadcast |
| join to a shuffle join. |
| </p> |
| |
| <pre class="pre codeblock"><code>[localhost:21000] > set explain_level=1; |
| [localhost:21000] > explain select one.*, two.*, three.* from t1 one, t1 two, t1 three where one.x = two.x and two.x = three.x; |
| +---------------------------------------------------------+ |
| | Explain String | |
| +---------------------------------------------------------+ |
| | Estimated Per-Host Requirements: Memory=4.00GB VCores=3 | |
| | | |
| | 07:EXCHANGE [PARTITION=UNPARTITIONED] | |
| | | | |
| <strong class="ph b">| 04:HASH JOIN [INNER JOIN, BROADCAST] |</strong> |
| | | hash predicates: two.x = three.x | |
| | | | |
| <strong class="ph b">| |--06:EXCHANGE [BROADCAST] |</strong> |
| | | | | |
| | | 02:SCAN HDFS [explain_plan.t1 three] | |
| | | partitions=1/1 size=0B | |
| | | | |
| <strong class="ph b">| 03:HASH JOIN [INNER JOIN, BROADCAST] |</strong> |
| | | hash predicates: one.x = two.x | |
| | | | |
| <strong class="ph b">| |--05:EXCHANGE [BROADCAST] |</strong> |
| | | | | |
| | | 01:SCAN HDFS [explain_plan.t1 two] | |
| | | partitions=1/1 size=0B | |
| | | | |
| | 00:SCAN HDFS [explain_plan.t1 one] | |
| | partitions=1/1 size=0B | |
| +---------------------------------------------------------+ |
| [localhost:21000] > explain select one.*, two.*, three.* |
| > from t1 one join [shuffle] t1 two join t1 three |
| > where one.x = two.x and two.x = three.x; |
| +---------------------------------------------------------+ |
| | Explain String | |
| +---------------------------------------------------------+ |
| | Estimated Per-Host Requirements: Memory=4.00GB VCores=3 | |
| | | |
| | 08:EXCHANGE [PARTITION=UNPARTITIONED] | |
| | | | |
| <strong class="ph b">| 04:HASH JOIN [INNER JOIN, BROADCAST] |</strong> |
| | | hash predicates: two.x = three.x | |
| | | | |
| <strong class="ph b">| |--07:EXCHANGE [BROADCAST] |</strong> |
| | | | | |
| | | 02:SCAN HDFS [explain_plan.t1 three] | |
| | | partitions=1/1 size=0B | |
| | | | |
| <strong class="ph b">| 03:HASH JOIN [INNER JOIN, PARTITIONED] |</strong> |
| | | hash predicates: one.x = two.x | |
| | | | |
| <strong class="ph b">| |--06:EXCHANGE [PARTITION=HASH(two.x)] |</strong> |
| | | | | |
| | | 01:SCAN HDFS [explain_plan.t1 two] | |
| | | partitions=1/1 size=0B | |
| | | | |
| <strong class="ph b">| 05:EXCHANGE [PARTITION=HASH(one.x)] |</strong> |
| | | | |
| | 00:SCAN HDFS [explain_plan.t1 one] | |
| | partitions=1/1 size=0B | |
| +---------------------------------------------------------+ |
| </code></pre> |
| |
| <p class="p"> |
| For a join involving many different tables, the default <code class="ph codeph">EXPLAIN</code> output might stretch over |
| several pages, and the only details you care about might be the join order and the mechanism (broadcast or |
| shuffle) for joining each pair of tables. In that case, you might set <code class="ph codeph">EXPLAIN_LEVEL</code> to its |
| lowest value of 0, to focus on just the join order and join mechanism for each stage. The following example |
| shows how the rows from the first and second joined tables are hashed and divided among the nodes of the |
| cluster for further filtering; then the entire contents of the third table are broadcast to all nodes for the |
| final stage of join processing. |
| </p> |
| |
| <pre class="pre codeblock"><code>[localhost:21000] > set explain_level=0; |
| [localhost:21000] > explain select one.*, two.*, three.* |
| > from t1 one join [shuffle] t1 two join t1 three |
| > where one.x = two.x and two.x = three.x; |
| +---------------------------------------------------------+ |
| | Explain String | |
| +---------------------------------------------------------+ |
| | Estimated Per-Host Requirements: Memory=4.00GB VCores=3 | |
| | | |
| | 08:EXCHANGE [PARTITION=UNPARTITIONED] | |
| <strong class="ph b">| 04:HASH JOIN [INNER JOIN, BROADCAST] |</strong> |
| <strong class="ph b">| |--07:EXCHANGE [BROADCAST] |</strong> |
| | | 02:SCAN HDFS [explain_plan.t1 three] | |
| <strong class="ph b">| 03:HASH JOIN [INNER JOIN, PARTITIONED] |</strong> |
| <strong class="ph b">| |--06:EXCHANGE [PARTITION=HASH(two.x)] |</strong> |
| | | 01:SCAN HDFS [explain_plan.t1 two] | |
| <strong class="ph b">| 05:EXCHANGE [PARTITION=HASH(one.x)] |</strong> |
| | 00:SCAN HDFS [explain_plan.t1 one] | |
| +---------------------------------------------------------+ |
| </code></pre> |
| |
| |
| |
| </div> |
| <nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_max_num_runtime_filters.html">MAX_NUM_RUNTIME_FILTERS Query Option (Impala 2.5 or higher only)</a></strong><br></li></ul><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_set.html">SET Statement</a></div></div></nav></article></main></body></html> |
