| <?xml version="1.0" encoding="UTF-8"?> |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| <!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> |
| <concept rev="2.3.0" id="live_summary"> |
| |
| <title>LIVE_SUMMARY Query Option (<keyword keyref="impala23"/> or higher only)</title> |
| <titlealts audience="PDF"><navtitle>LIVE_SUMMARY</navtitle></titlealts> |
| <prolog> |
| <metadata> |
| <data name="Category" value="Impala"/> |
| <data name="Category" value="Impala Query Options"/> |
| <data name="Category" value="Querying"/> |
| <data name="Category" value="Performance"/> |
| <data name="Category" value="Reports"/> |
| <data name="Category" value="impala-shell"/> |
| <data name="Category" value="Developers"/> |
| <data name="Category" value="Data Analysts"/> |
| </metadata> |
| </prolog> |
| |
| <conbody> |
| <p rev="2.3.0"> When the <codeph>LIVE_SUMMARY</codeph> query option is set |
| to <codeph>TRUE</codeph>, Impala displays the same output as the |
| <codeph>SUMMARY</codeph> command for queries submitted through the |
| <cmdname>impala-shell</cmdname> command, with the measurements updated |
| in real time as the query progresses. When the query finishes, the final |
| <codeph>SUMMARY</codeph> output remains visible in the |
| <cmdname>impala-shell</cmdname> console output. </p> |
| <p>Starting in <keyword keyref="impala31"/>, the summary output also |
| includes the queuing status consisting of whether the query was queued and |
| what was the latest queuing reason.</p> |
| <p><b>Type:</b> |
| <codeph>Boolean</codeph></p> |
| <p><b>Default:</b> |
| <codeph>FALSE (0)</codeph></p> |
| <p conref="../shared/impala_common.xml#common/command_line_blurb"/> |
| <p> You can enable this query option within <cmdname>impala-shell</cmdname> |
| by starting the shell with the <codeph>--live_summary</codeph> |
| command-line option. You can still turn this setting off and on again |
| within the shell through the <codeph>SET</codeph> command. </p> |
| <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/> |
| <p> The live summary output can be useful for evaluating long-running |
| queries, to evaluate which phase of execution takes up the most time, or |
| if some hosts take much longer than others for certain operations, |
| dragging overall performance down. By making the information available in |
| real time, this feature lets you decide what action to take even before |
| you cancel a query that is taking much longer than normal. </p> |
| <p> For example, you might see the HDFS scan phase taking a long time, and |
| therefore revisit performance-related aspects of your schema design such |
| as constructing a partitioned table, switching to the Parquet file format, |
| running the <codeph>COMPUTE STATS</codeph> statement for the table, and so |
| on. Or you might see a wide variation between the average and maximum |
| times for all hosts to perform some phase of the query, and therefore |
| investigate if one particular host needed more memory or was experiencing |
| a network problem. </p> |
| <p conref="../shared/impala_common.xml#common/live_reporting_details"/> |
| <p> For a simple and concise way of tracking the progress of an interactive |
| query, see <xref href="impala_live_progress.xml#live_progress"/>. </p> |
| <p conref="../shared/impala_common.xml#common/restrictions_blurb"/> |
| <p |
| conref="../shared/impala_common.xml#common/impala_shell_progress_reports_compute_stats_caveat"/> |
| <p |
| conref="../shared/impala_common.xml#common/impala_shell_progress_reports_shell_only_caveat"/> |
| <p conref="../shared/impala_common.xml#common/added_in_230"/> |
| <p conref="../shared/impala_common.xml#common/example_blurb"/> |
| <p> The following example shows a series of <codeph>LIVE_SUMMARY</codeph> |
| reports that are displayed during the course of a query, showing how the |
| numbers increase to show the progress of different phases of the |
| distributed query. When you do the same in |
| <cmdname>impala-shell</cmdname>, only a single report is displayed at any |
| one time, with each update overwriting the previous numbers. </p> |
| <codeblock><![CDATA[[localhost:21000] > set live_summary=true; |
| LIVE_SUMMARY set to true |
| [localhost:21000] > select count(*) from customer t1 cross join customer t2; |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 0 | 0ns | 0ns | 0 | 22.50B | 0 B | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 0 | 0ns | 0ns | 0 | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 0 | 0ns | 0ns | 0 | 150.00K | 0 B | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 1 | 0ns | 0ns | 0 | 1 | 20.00 KB | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 1 | 17.62s | 17.62s | 81.14M | 22.50B | 3.23 MB | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 1 | 26.29ms | 26.29ms | 150.00K | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 1 | 247.53ms | 247.53ms | 1.02K | 150.00K | 24.39 MB | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 1 | 0ns | 0ns | 0 | 1 | 20.00 KB | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 1 | 61.85s | 61.85s | 283.43M | 22.50B | 3.23 MB | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 1 | 26.29ms | 26.29ms | 150.00K | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 1 | 247.59ms | 247.59ms | 2.05K | 150.00K | 24.39 MB | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| ]]> |
| </codeblock> |
| <!-- Keeping this sample output that illustrates a couple of glitches in the LIVE_SUMMARY display, hidden, to help filing JIRAs. --> |
| <codeblock audience="hidden"><![CDATA[[ |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 1 | 0ns | 0ns | 0 | 1 | 20.00 KB | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 1 | 91.34s | 91.34s | 419.48M | 22.50B | 3.23 MB | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 1 | 26.29ms | 26.29ms | 150.00K | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 1 | 247.63ms | 247.63ms | 3.07K | 150.00K | 24.39 MB | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 1 | 0ns | 0ns | 0 | 1 | 20.00 KB | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 1 | 140.49s | 140.49s | 646.82M | 22.50B | 3.23 MB | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 1 | 26.29ms | 26.29ms | 150.00K | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 1 | 247.73ms | 247.73ms | 5.12K | 150.00K | 24.39 MB | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 1 | 0ns | 0ns | 0 | 1 | 20.00 KB | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 1 | 228.96s | 228.96s | 1.06B | 22.50B | 3.23 MB | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 1 | 26.29ms | 26.29ms | 150.00K | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 1 | 247.83ms | 247.83ms | 7.17K | 150.00K | 24.39 MB | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 1 | 0ns | 0ns | 0 | 1 | 20.00 KB | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 1 | 563.11s | 563.11s | 2.59B | 22.50B | 3.23 MB | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 1 | 26.29ms | 26.29ms | 150.00K | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 1 | 248.11ms | 248.11ms | 17.41K | 150.00K | 24.39 MB | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 1 | 0ns | 0ns | 0 | 1 | 20.00 KB | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 1 | 985.71s | 985.71s | 4.54B | 22.50B | 3.23 MB | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 1 | 26.29ms | 26.29ms | 150.00K | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 1 | 248.49ms | 248.49ms | 30.72K | 150.00K | 24.39 MB | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | 06:AGGREGATE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | FINALIZE | |
| | 05:EXCHANGE | 0 | 0ns | 0ns | 0 | 1 | 0 B | -1 B | UNPARTITIONED | |
| | 03:AGGREGATE | 1 | 0ns | 0ns | 0 | 1 | 20.00 KB | 10.00 MB | | |
| | 02:NESTED LOOP JOIN | 1 | None | None | 5.42B | 22.50B | 3.23 MB | 0 B | CROSS JOIN, BROADCAST | |
| | |--04:EXCHANGE | 1 | 26.29ms | 26.29ms | 150.00K | 150.00K | 0 B | 0 B | BROADCAST | |
| | | 01:SCAN HDFS | 1 | 503.57ms | 503.57ms | 150.00K | 150.00K | 24.09 MB | 64.00 MB | tpch.customer t2 | |
| | 00:SCAN HDFS | 1 | 248.66ms | 248.66ms | 36.86K | 150.00K | 24.39 MB | 64.00 MB | tpch.customer t1 | |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| |
| [localhost:21000] > select count(*) from customer t1 cross join customer t2; |
| Query: select count(*) from customer t1 cross join customer t2 |
| [####################################################################################################] 100% |
| +---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+ |
| | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | |
| ]]> |
| </codeblock> |
| <p |
| conref="../shared/impala_common.xml#common/live_progress_live_summary_asciinema" |
| /> |
| </conbody> |
| </concept> |