| <?xml version="1.0" encoding="UTF-8"?> |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| <!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> |
| <concept id="breakpad" rev="2.6.0 IMPALA-2686"> |
| |
| <title>Breakpad Minidumps for Impala (<keyword keyref="impala26"/> or higher only)</title> |
| <titlealts audience="PDF"><navtitle>Breakpad Minidumps</navtitle></titlealts> |
| <prolog> |
| <metadata> |
| <data name="Category" value="Impala"/> |
| <data name="Category" value="Troubleshooting"/> |
| <data name="Category" value="Support"/> |
| <data name="Category" value="Administrators"/> |
| </metadata> |
| </prolog> |
| |
| <conbody> |
| |
| <p rev="2.6.0 IMPALA-2686"> |
| The <xref href="https://chromium.googlesource.com/breakpad/breakpad/" scope="external" format="html">breakpad</xref> |
| project is an open-source framework for crash reporting. |
| In <keyword keyref="impala26_full"/> and higher, Impala can use <codeph>breakpad</codeph> to record stack information and |
| register values when any of the Impala-related daemons crash due to an error such as <codeph>SIGSEGV</codeph> |
| or unhandled exceptions. |
| The dump files are much smaller than traditional core dump files. The dump mechanism itself uses very little |
| memory, which improves reliability if the crash occurs while the system is low on memory. |
| </p> |
| |
| <note type="important"> |
| Because of the internal mechanisms involving Impala memory allocation and Linux |
| signalling for out-of-memory (OOM) errors, if an Impala-related daemon experiences a |
| crash due to an OOM condition, it does <i>not</i> generate a minidump for that error. |
| <p> |
| |
| </p> |
| </note> |
| |
| |
| <p outputclass="toc inpage" audience="PDF"/> |
| |
| </conbody> |
| |
| <concept id="breakpad_minidump_enable"> |
| <title>Enabling or Disabling Minidump Generation</title> |
| <conbody> |
| <p> |
| By default, a minidump file is generated when an Impala-related daemon |
| crashes. |
| </p> |
| |
| <p> |
| To turn off generation of the minidump files, use one of the following |
| options: |
| |
| <ul> |
| <li> |
| Set the <codeph>--enable_minidumps</codeph> configuration setting |
| to <codeph>false</codeph>. Restart the corresponding services or |
| daemons. |
| </li> |
| |
| <li> |
| Set the <codeph>--minidump_path</codeph> configuration setting to |
| an empty string. Restart the corresponding services or daemons. |
| </li> |
| </ul> |
| </p> |
| |
| <p rev="IMPALA-3677"> |
| In <keyword keyref="impala27_full"/> and higher, |
| you can send a <codeph>SIGUSR1</codeph> signal to any Impala-related daemon to write a |
| Breakpad minidump. For advanced troubleshooting, you can now produce a minidump |
| without triggering a crash. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="breakpad_minidump_location" rev="IMPALA-3581"> |
| <title>Specifying the Location for Minidump Files</title> |
| <conbody> |
| <p> |
| By default, all minidump files are written to the following location |
| on the host where a crash occurs: |
| <!-- Location stated in IMPALA-3581; overridden by different location from IMPALA-2686? |
| <filepath><varname>log_directory</varname>/minidumps/<varname>daemon_name</varname></filepath> --> |
| <ul> |
| <li> |
| <p> |
| Clusters not managed by cluster management software: |
| <filepath><varname>impala_log_dir</varname>/<varname>daemon_name</varname>/minidumps/<varname>daemon_name</varname></filepath> |
| </p> |
| </li> |
| </ul> |
| The minidump files for <cmdname>impalad</cmdname>, <cmdname>catalogd</cmdname>, |
| and <cmdname>statestored</cmdname> are each written to a separate directory. |
| </p> |
| <p> |
| To specify a different location, set the |
| <!-- Again, IMPALA-3581 says one thing and IMPALA-2686 says another. |
| log_dir vs. minidump_path --> |
| <uicontrol>minidump_path</uicontrol> |
| configuration setting of one or more Impala-related daemons, and restart the corresponding services or daemons. |
| </p> |
| <p> |
| If you specify a relative path for this setting, the value is interpreted relative to |
| the default <uicontrol>minidump_path</uicontrol> directory. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="breakpad_minidump_number"> |
| <title>Controlling the Number of Minidump Files</title> |
| <conbody> |
| <p> |
| Like any files used for logging or troubleshooting, consider limiting the number of |
| minidump files, or removing unneeded ones, depending on the amount of free storage |
| space on the hosts in the cluster. |
| </p> |
| <p> |
| Because the minidump files are only used for problem resolution, you can remove any such files that |
| are not needed to debug current issues. |
| </p> |
| <p> |
| To control how many minidump files Impala keeps around at any one time, |
| set the <uicontrol>max_minidumps</uicontrol> configuration setting for |
| of one or more Impala-related daemon, and restart the corresponding services or daemons. |
| The default for this setting is 9. A zero or negative value is interpreted as |
| <q>unlimited</q>. |
| </p> |
| </conbody> |
| </concept> |
| |
| <concept id="breakpad_minidump_logging"> |
| <title>Detecting Crash Events</title> |
| <conbody> |
| |
| <p> |
| You can see in the Impala log files when crash events occur that generate |
| minidump files. Because each restart begins a new log file, the <q>crashed</q> message |
| is always at or near the bottom of the log file. There might be another later message |
| if core dumps are also enabled. |
| </p> |
| |
| </conbody> |
| </concept> |
| |
| <concept id="breakpad_demo"> |
| <title>Demonstration of Breakpad Feature</title> |
| <conbody> |
| <p> |
| The following example uses the command <cmdname>kill -11</cmdname> to |
| simulate a <codeph>SIGSEGV</codeph> crash for an <cmdname>impalad</cmdname> |
| process on a single DataNode, then examines the relevant log files and minidump file. |
| </p> |
| |
| <p> |
| First, as root on a worker node, kill the <cmdname>impalad</cmdname> process with a |
| <codeph>SIGSEGV</codeph> error. The original process ID was 23114. |
| </p> |
| |
| <codeblock><![CDATA[ |
| # ps ax | grep impalad |
| 23114 ? Sl 0:18 /opt/local/parcels/<parcel_version>/lib/impala/sbin/impalad --flagfile=/var/run/impala/process/114-impala-IMPALAD/impala-conf/impalad_flags |
| 31259 pts/0 S+ 0:00 grep impalad |
| # |
| # kill -11 23114 |
| # |
| # ps ax | grep impalad |
| 31374 ? Rl 0:04 /opt/local/parcels/<parcel_version>/lib/impala/sbin/impalad --flagfile=/var/run/impala/process/114-impala-IMPALAD/impala-conf/impalad_flags |
| 31475 pts/0 S+ 0:00 grep impalad |
| ]]> |
| </codeblock> |
| |
| <p> |
| We locate the log directory underneath <filepath>/var/log</filepath>. |
| There is a <codeph>.INFO</codeph>, <codeph>.WARNING</codeph>, and <codeph>.ERROR</codeph> |
| log file for the 23114 process ID. The minidump message is written to the |
| <codeph>.INFO</codeph> file and the <codeph>.ERROR</codeph> file, but not the |
| <codeph>.WARNING</codeph> file. In this case, a large core file was also produced. |
| </p> |
| <codeblock><![CDATA[ |
| # cd /var/log/impalad |
| # ls -la | grep 23114 |
| -rw------- 1 impala impala 3539079168 Jun 23 15:20 core.23114 |
| -rw-r--r-- 1 impala impala 99057 Jun 23 15:20 hs_err_pid23114.log |
| -rw-r--r-- 1 impala impala 351 Jun 23 15:20 impalad.worker_node_123.impala.log.ERROR.20160623-140343.23114 |
| -rw-r--r-- 1 impala impala 29101 Jun 23 15:20 impalad.worker_node_123.impala.log.INFO.20160623-140343.23114 |
| -rw-r--r-- 1 impala impala 228 Jun 23 14:03 impalad.worker_node_123.impala.log.WARNING.20160623-140343.23114 |
| ]]> |
| </codeblock> |
| <p> |
| The <codeph>.INFO</codeph> log includes the location of the minidump file, followed by |
| a report of a core dump. With the breakpad minidump feature enabled, now we might |
| disable core dumps or keep fewer of them around. |
| </p> |
| <codeblock><![CDATA[ |
| # cat impalad.worker_node_123.impala.log.INFO.20160623-140343.23114 |
| ... |
| Wrote minidump to /var/log/impala-minidumps/impalad/0980da2d-a905-01e1-25ff883a-04ee027a.dmp |
| # |
| # A fatal error has been detected by the Java Runtime Environment: |
| # |
| # SIGSEGV (0xb) at pc=0x00000030c0e0b68a, pid=23114, tid=139869541455968 |
| # |
| # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01) |
| # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops) |
| # Problematic frame: |
| # C [libpthread.so.0+0xb68a] pthread_cond_wait+0xca |
| # |
| # Core dump written. Default location: /var/log/impalad/core or core.23114 |
| # |
| # An error report file with more information is saved as: |
| # /var/log/impalad/hs_err_pid23114.log |
| # |
| # If you would like to submit a bug report, please visit: |
| # http://bugreport.sun.com/bugreport/crash.jsp |
| # The crash happened outside the Java Virtual Machine in native code. |
| # See problematic frame for where to report the bug. |
| ... |
| |
| # cat impalad.worker_node_123.impala.log.ERROR.20160623-140343.23114 |
| |
| Log file created at: 2016/06/23 14:03:43 |
| Running on machine:.worker_node_123 |
| Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg |
| E0623 14:03:43.911002 23114 logging.cc:118] stderr will be logged to this file. |
| Wrote minidump to /var/log/impala-minidumps/impalad/0980da2d-a905-01e1-25ff883a-04ee027a.dmp |
| ]]> |
| </codeblock> |
| |
| <p> |
| The resulting minidump file is much smaller than the corresponding core file, |
| making it much easier to supply diagnostic information to <keyword keyref="support_org"/>. |
| </p> |
| |
| <codeblock><![CDATA[ |
| # pwd |
| /var/log/impalad |
| # cd ../impala-minidumps/impalad |
| # ls |
| 0980da2d-a905-01e1-25ff883a-04ee027a.dmp |
| # du -kh * |
| 2.4M 0980da2d-a905-01e1-25ff883a-04ee027a.dmp |
| ]]> |
| </codeblock> |
| </conbody> |
| </concept> |
| |
| </concept> |