blob: 80b7363d3ce4c64373a0697b3fc2a950c871f750 [file] [log] [blame]
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="troubleshooting">
<title>Troubleshooting Impala</title>
<prolog>
<metadata>
<data name="Category" value="Impala"/>
<data name="Category" value="Troubleshooting"/>
<data name="Category" value="Administrators"/>
<data name="Category" value="Developers"/>
<data name="Category" value="Data Analysts"/>
</metadata>
</prolog>
<conbody>
<p>
<indexterm audience="hidden">troubleshooting</indexterm>
Troubleshooting for Impala requires being able to diagnose and debug problems
with performance, network connectivity, out-of-memory conditions, disk space usage,
and crash or hang conditions in any of the Impala-related daemons.
</p>
<p outputclass="toc inpage" audience="PDF">
The following sections describe the general troubleshooting procedures to diagnose
different kinds of problems:
</p>
</conbody>
<concept id="trouble_sql">
<title>Troubleshooting Impala SQL Syntax Issues</title>
<conbody>
<p>
In general, if queries issued against Impala fail, you can try running these same queries against Hive.
</p>
<ul>
<li>
If a query fails against both Impala and Hive, it is likely that there is a problem with your query or
other elements of your <keyword keyref="distro"/> environment:
<ul>
<li>
Review the <xref href="impala_langref.xml#langref">Language Reference</xref> to ensure your query is
valid.
</li>
<li>
Check <xref href="impala_reserved_words.xml#reserved_words"/> to see if any database, table,
column, or other object names in your query conflict with Impala reserved words.
Quote those names with backticks (<codeph>``</codeph>) if so.
</li>
<li>
Check <xref href="impala_functions.xml#builtins"/> to confirm whether Impala supports all the
built-in functions being used by your query, and whether argument and return types are the
same as you expect.
</li>
<li>
Review the <xref href="impala_logging.xml#logs_debug">contents of the Impala logs</xref> for any information that may be useful in identifying the
source of the problem.
</li>
</ul>
</li>
<li>
If a query fails against Impala but not Hive, it is likely that there is a problem with your Impala
installation.
</li>
</ul>
</conbody>
</concept>
<concept id="IMPALA-5605">
<title>Troubleshooting Crashes Caused by Memory Resource Limit</title>
<conbody>
<p>Under very high concurrency, Impala could encounter a serious error due
to usage of various operating system resources. Errors similar to the
following may be caused by operating system resource exhaustion:</p>
<codeblock>F0629 08:20:02.956413 29088 llvm-codegen.cc:111] LLVM hit fatal error: Unable to allocate section memory!
terminate called after throwing an instance of 'boost::exception_detail::clone_impl&lt;boost::exception_detail::error_info_injector&lt;boost::thread_resource_error> >'</codeblock>
<p>The KRPC implementation in Impala 2.12 / 3.0 greatly reduces thread
counts and the chances of hitting a resource limit.</p>
<p>If you still get an error similar to the above in Impala 3.0 and
higher, try increasing the <codeph>max_map_count</codeph> OS virtual
memory parameter. <codeph>max_map_count</codeph> defines the maximum
number of memory map areas that a process can use. Configure each host
running an <codeph>impalad</codeph> daemon with the command to increase
<codeph>max_map_count</codeph> to 8 GB.</p>
<codeblock outputclass="cdoc-input">echo 8000000 > /proc/sys/vm/max_map_count</codeblock>
<p>To make the above settings durable, refer to your OS documentation. For
example, on RHEL 6.x:<ol>
<li>Add the following line to
<codeph>/etc/sysctl.conf</codeph>:<codeblock>vm.max_map_count=8000000</codeblock></li>
<li>Run the following
command:<codeblock outputclass="cdoc-input">sysctl -p</codeblock></li>
</ol></p>
</conbody>
</concept>
<concept id="trouble_io" rev="">
<title>Troubleshooting I/O Capacity Problems</title>
<conbody>
<p> Impala queries are typically I/O-intensive. If there is an I/O problem
with storage devices, or with HDFS itself, Impala queries could show
slow response times with no obvious cause on the Impala side. Slow I/O
on even a single Impala daemon could result in an overall slowdown,
because queries involving clauses such as <codeph>ORDER BY</codeph>,
<codeph>GROUP BY</codeph>, or <codeph>JOIN</codeph> do not start
returning results until all executor Impala daemons have finished their
work. </p>
<p> To test whether the Linux I/O system itself is performing as expected,
run Linux commands like the following on each host Impala daemon is
running: </p>
<codeblock>
$ sudo sysctl -w vm.drop_caches=3 vm.drop_caches=0
vm.drop_caches = 3
vm.drop_caches = 0
$ sudo dd if=/dev/sda bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.60373 s, 192 MB/s
$ sudo dd if=/dev/sdb bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.51145 s, 195 MB/s
$ sudo dd if=/dev/sdc bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.58096 s, 192 MB/s
$ sudo dd if=/dev/sdd bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.43924 s, 197 MB/s
</codeblock>
<p>
On modern hardware, a throughput rate of less than 100 MB/s typically indicates
a performance issue with the storage device. Correct the hardware problem before
continuing with Impala tuning or benchmarking.
</p>
</conbody>
</concept>
<concept id="trouble_cookbook">
<title>Impala Troubleshooting Quick Reference</title>
<conbody>
<p>
The following table lists common problems and potential solutions.
</p>
<table>
<tgroup cols="3">
<colspec colname="1" colwidth="10*"/>
<colspec colname="2" colwidth="30*"/>
<colspec colname="3" colwidth="30*"/>
<thead>
<row>
<entry>
Symptom
</entry>
<entry>
Explanation
</entry>
<entry>
Recommendation
</entry>
</row>
</thead>
<tbody>
<row>
<entry>
Impala takes a long time to start.
</entry>
<entry>
Impala instances with large numbers of tables, partitions, or data files take longer to start
because the metadata for these objects is broadcast to all <cmdname>impalad</cmdname> nodes and
cached.
</entry>
<entry>
Adjust timeout and synchronicity settings.
</entry>
</row>
<row>
<entry>
<p>
Joins fail to complete.
</p>
</entry>
<entry>
<p>
There may be insufficient memory. During a join, data from the second, third, and so on sets to
be joined is loaded into memory. If Impala chooses an inefficient join order or join mechanism,
the query could exceed the total memory available.
</p>
</entry>
<entry>
<p>
Start by gathering statistics with the <codeph>COMPUTE STATS</codeph> statement for each table
involved in the join. Consider specifying the <codeph>[SHUFFLE]</codeph> hint so that data from
the joined tables is split up between nodes rather than broadcast to each node. If tuning at the
SQL level is not sufficient, add more memory to your system or join smaller data sets.
</p>
</entry>
</row>
<row>
<entry>
<p>
Queries return incorrect results.
</p>
</entry>
<entry>
<p>
Impala metadata may be outdated after changes are performed in Hive.
</p>
</entry>
<entry>
<p>
Where possible, use the appropriate Impala statement (<codeph>INSERT</codeph>, <codeph>LOAD
DATA</codeph>, <codeph>CREATE TABLE</codeph>, <codeph>ALTER TABLE</codeph>, <codeph>COMPUTE
STATS</codeph>, and so on) rather than switching back and forth between Impala and Hive. Impala
automatically broadcasts the results of DDL and DML operations to all Impala nodes in the
cluster, but does not automatically recognize when such changes are made through Hive. After
inserting data, adding a partition, or other operation in Hive, refresh the metadata for the
table as described in <xref href="impala_refresh.xml#refresh"/>.
</p>
</entry>
</row>
<row>
<entry>
<p>
Queries are slow to return results.
</p>
</entry>
<entry>
<p>
Some <codeph>impalad</codeph> instances may not have started. Using a browser, connect to the
host running the Impala state store. Connect using an address of the form
<codeph>http://<varname>hostname</varname>:<varname>port</varname>/metrics</codeph>.
</p>
<p>
<note> Replace <varname>hostname</varname> and
<varname>port</varname> with the hostname and port of your
Impala state store host machine and web server port. The
default port is 25010. </note> The number of
<codeph>impalad</codeph> instances listed should match the
expected number of <codeph>impalad</codeph> instances
installed in the cluster. There should also be one
<codeph>impalad</codeph> instance installed on each
DataNode.</p>
</entry>
<entry>
<p>
Ensure Impala is installed on all DataNodes. Start any <codeph>impalad</codeph> instances that
are not running.
</p>
</entry>
</row>
<row>
<entry>
<p>
Queries are slow to return results.
</p>
</entry>
<entry>
<p>
Impala may not be configured to use native checksumming. Native checksumming uses
machine-specific instructions to compute checksums over HDFS data very quickly. Review Impala
logs. If you find instances of "<codeph>INFO util.NativeCodeLoader: Loaded the
native-hadoop</codeph>" messages, native checksumming is not enabled.
</p>
</entry>
<entry>
<p>
Ensure Impala is configured to use native checksumming as described in
<xref href="impala_config_performance.xml#config_performance"/>.
</p>
</entry>
</row>
<row>
<entry>
<p>
Queries are slow to return results.
</p>
</entry>
<entry>
<p>
Impala may not be configured to use data locality tracking.
</p>
</entry>
<entry>
<p>
Test Impala for data locality tracking and make configuration changes as necessary. Information
on this process can be found in <xref href="impala_config_performance.xml#config_performance"/>.
</p>
</entry>
</row>
<row>
<entry>
<p>
Attempts to complete Impala tasks such as executing INSERT-SELECT actions fail. The Impala logs
include notes that files could not be opened due to permission denied.
</p>
</entry>
<entry>
<p>
This can be the result of permissions issues. For example, you could use the Hive shell as the
hive user to create a table. After creating this table, you could attempt to complete some
action, such as an INSERT-SELECT on the table. Because the table was created using one user and
the INSERT-SELECT is attempted by another, this action may fail due to permissions issues.
</p>
</entry>
<entry>
<p>
In general, ensure the Impala user has sufficient permissions. In the preceding example, ensure
the Impala user has sufficient permissions to the table that the Hive user created.
</p>
</entry>
</row>
<row rev="IMP-1210">
<entry>
<p>
Impala fails to start up, with the <cmdname>impalad</cmdname> logs referring to errors connecting
to the statestore service and attempts to re-register.
</p>
</entry>
<entry>
<p>
A large number of databases, tables, partitions, and so on can require metadata synchronization,
particularly on startup, that takes longer than the default timeout for the statestore service.
</p>
</entry>
<entry>
<p>
Configure the statestore timeout value and possibly other settings related to the frequency of
statestore updates and metadata loading. See
<xref href="impala_timeouts.xml#statestore_timeout"/> and
<xref href="impala_scalability.xml#statestore_scalability"/>.
</p>
</entry>
</row>
</tbody>
</tgroup>
</table>
<p audience="hidden">
Some or all of these settings might also be useful.
<codeblock>NUM_SCANNER_THREADS: 0
ABORT_ON_DEFAULT_LIMIT_EXCEEDED: 0
MAX_IO_BUFFERS: 0
DEFAULT_ORDER_BY_LIMIT: -1
BATCH_SIZE: 0
NUM_NODES: 0
DISABLE_CODEGEN: 0
MAX_ERRORS: 0
ABORT_ON_ERROR: 0
MAX_SCAN_RANGE_LENGTH: 0
ALLOW_UNSUPPORTED_FORMATS: 0
SUPPORT_START_OVER: false
DEBUG_ACTION:
MEM_LIMIT: 0
</codeblock>
</p>
</conbody>
</concept>
<concept audience="hidden" id="core_dumps">
<title>Enabling Core Dumps for Impala</title>
<conbody>
<p>
Fill in details, then unhide.
</p>
<p>
From <xref href="impala_config_options.xml#config_options"/>:
</p>
<codeblock>export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-false}</codeblock>
<note conref="../shared/impala_common.xml#common/core_dump_considerations"/>
<p></p>
</conbody>
</concept>
<concept audience="hidden" id="io_throughput">
<title>Verifying I/O Throughput</title>
<conbody>
<p>
Optimal Impala query performance depends on being able to perform I/O across multiple storage devices
in parallel, with the data transferred at or close to the maximum throughput for each device.
If a hardware or configuration issue causes a reduction in I/O throughput, even if the problem only
affects a subset of storage devices, you might experience
slow query performance that cannot be improved by using regular SQL tuning techniques.
</p>
<p>
As a general guideline, expect each commodity storage device (for example, a standard rotational
hard drive) to be able to transfer approximately 100 MB per second. If you see persistent slow query
perormance, examine the Impala logs to check
</p>
<codeblock>
<![CDATA[
Useful test for I/O throughput of hardware.
Symptoms:
* Queries running slow
* Scan rate of IO in Impala logs show noticeably less than expected IO rate for each disk (typical commodity disk should provide ~100 MB/s
Actions:
* Validate disk read from OS to confirm no issue at hardware or OS level
* Validate disk read at HDFS to see if issue at HDFS config
Specifics:
Testing Linux and hardware IO:
# First running:
sudo sysctl -w vm.drop_caches=3 vm.drop_caches=0
# Then Running:
sudo dd if=/dev/sda bs=1M of=/dev/null count=1k
& sudo dd if=/dev/sdb bs=1M of=/dev/null count=1k
& sudo dd if=/dev/sdc bs=1M of=/dev/null count=1k
& sudo dd if=/dev/sdd bs=1M of=/dev/null count=1k & wait
Testing HDFS IO:
# You can use TestDFSIO. Its documented here ; http://answers.oreilly.com/topic/460-how-to-benchmark-a-hadoop-cluster/
# You can also use sar, dd and iostat for monitoring the disk.
# writes 10 files each of 1000 MB
hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -write -nrFiles 10 -fileSize 1000
# run the read benchmark
hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -read -nrFiles 10 -fileSize 1000
# clean up the data
hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -clean
]]>
</codeblock>
</conbody>
</concept>
</concept>