<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="breakpad" rev="2.6.0 IMPALA-2686">

  <title>Breakpad Minidumps for Impala (<keyword keyref="impala26"/> or higher only)</title>
  <titlealts audience="PDF"><navtitle>Breakpad Minidumps</navtitle></titlealts>
  <prolog>
    <metadata>
      <data name="Category" value="Impala"/>
      <data name="Category" value="Troubleshooting"/>
      <data name="Category" value="Support"/>
      <data name="Category" value="Administrators"/>
    </metadata>
  </prolog>

  <conbody>

    <p rev="2.6.0 IMPALA-2686">
      The <xref href="https://chromium.googlesource.com/breakpad/breakpad/" scope="external" format="html">breakpad</xref>
      project is an open-source framework for crash reporting.
      In <keyword keyref="impala26_full"/> and higher, Impala can use <codeph>breakpad</codeph> to record stack information and
      register values when any of the Impala-related daemons crash due to an error such as <codeph>SIGSEGV</codeph>
      or unhandled exceptions.
      The dump files are much smaller than traditional core dump files. The dump mechanism itself uses very little
      memory, which improves reliability if the crash occurs while the system is low on memory.
    </p>

    <note type="important">
      Because of the internal mechanisms involving Impala memory allocation and Linux
      signalling for out-of-memory (OOM) errors, if an Impala-related daemon experiences a
      crash due to an OOM condition, it does <i>not</i> generate a minidump for that error.
    <p>

    </p>
    </note>


    <p outputclass="toc inpage" audience="PDF"/>

  </conbody>

  <concept id="breakpad_minidump_enable">
    <title>Enabling or Disabling Minidump Generation</title>
    <conbody>
      <p>
        By default, a minidump file is generated when an Impala-related daemon
        crashes.
      </p>

      <p>
        To turn off generation of the minidump files, use one of the following
        options:

        <ul>
          <li>
            Set the <codeph>--enable_minidumps</codeph> configuration setting
            to <codeph>false</codeph>. Restart the corresponding services or
            daemons.
          </li>

          <li>
            Set the <codeph>--minidump_path</codeph> configuration setting to
            an empty string. Restart the corresponding services or daemons.
          </li>
        </ul>
      </p>

      <p rev="IMPALA-3677">
        In <keyword keyref="impala27_full"/> and higher,
        you can send a <codeph>SIGUSR1</codeph> signal to any Impala-related daemon to write a
        Breakpad minidump. For advanced troubleshooting, you can now produce a minidump
        without triggering a crash.
      </p>
    </conbody>
  </concept>

  <concept id="breakpad_minidump_location" rev="IMPALA-3581">
    <title>Specifying the Location for Minidump Files</title>
    <conbody>
      <p>
        By default, all minidump files are written to the following location
        on the host where a crash occurs:
        <!-- Location stated in IMPALA-3581; overridden by different location from IMPALA-2686?
             <filepath><varname>log_directory</varname>/minidumps/<varname>daemon_name</varname></filepath> -->
         <ul>
          <li>
            <p>
              Clusters not managed by cluster management software:
              <filepath><varname>impala_log_dir</varname>/<varname>daemon_name</varname>/minidumps/<varname>daemon_name</varname></filepath>
            </p>
          </li>
        </ul>
        The minidump files for <cmdname>impalad</cmdname>, <cmdname>catalogd</cmdname>,
        and <cmdname>statestored</cmdname> are each written to a separate directory.
      </p>
      <p>
        To specify a different location, set the
        <!-- Again, IMPALA-3581 says one thing and IMPALA-2686 says another.
             log_dir vs. minidump_path -->
        <uicontrol>minidump_path</uicontrol>
        configuration setting of one or more Impala-related daemons, and restart the corresponding services or daemons.
      </p>
      <p>
        If you specify a relative path for this setting, the value is interpreted relative to
        the default <uicontrol>minidump_path</uicontrol> directory.
      </p>
    </conbody>
  </concept>

  <concept id="breakpad_minidump_number">
    <title>Controlling the Number of Minidump Files</title>
    <conbody>
      <p>
        Like any files used for logging or troubleshooting, consider limiting the number of
        minidump files, or removing unneeded ones, depending on the amount of free storage
        space on the hosts in the cluster.
      </p>
      <p>
        Because the minidump files are only used for problem resolution, you can remove any such files that
        are not needed to debug current issues.
      </p>
      <p>
        To control how many minidump files Impala keeps around at any one time,
        set the <uicontrol>max_minidumps</uicontrol> configuration setting for
        of one or more Impala-related daemon, and restart the corresponding services or daemons.
        The default for this setting is 9. A zero or negative value is interpreted as
        <q>unlimited</q>.
      </p>
    </conbody>
  </concept>

  <concept id="breakpad_minidump_logging">
    <title>Detecting Crash Events</title>
    <conbody>

      <p>
        You can see in the Impala log files when crash events occur that generate
        minidump files. Because each restart begins a new log file, the <q>crashed</q> message
        is always at or near the bottom of the log file. There might be another later message
        if core dumps are also enabled.
      </p>

    </conbody>
  </concept>

  <concept id="breakpad_demo">
    <title>Demonstration of Breakpad Feature</title>
    <conbody>
      <p>
        The following example uses the command <cmdname>kill -11</cmdname> to
        simulate a <codeph>SIGSEGV</codeph> crash for an <cmdname>impalad</cmdname>
        process on a single DataNode, then examines the relevant log files and minidump file.
      </p>

      <p>
        First, as root on a worker node, kill the <cmdname>impalad</cmdname> process with a
        <codeph>SIGSEGV</codeph> error. The original process ID was 23114.
      </p>

<codeblock><![CDATA[
# ps ax | grep impalad
23114 ?        Sl     0:18 /opt/local/parcels/<parcel_version>/lib/impala/sbin/impalad --flagfile=/var/run/impala/process/114-impala-IMPALAD/impala-conf/impalad_flags
31259 pts/0    S+     0:00 grep impalad
#
# kill -11 23114
#
# ps ax | grep impalad
31374 ?        Rl     0:04 /opt/local/parcels/<parcel_version>/lib/impala/sbin/impalad --flagfile=/var/run/impala/process/114-impala-IMPALAD/impala-conf/impalad_flags
31475 pts/0    S+     0:00 grep impalad
]]>
</codeblock>

      <p>
        We locate the log directory underneath <filepath>/var/log</filepath>.
        There is a <codeph>.INFO</codeph>, <codeph>.WARNING</codeph>, and <codeph>.ERROR</codeph>
        log file for the 23114 process ID. The minidump message is written to the
        <codeph>.INFO</codeph> file and the <codeph>.ERROR</codeph> file, but not the
        <codeph>.WARNING</codeph> file. In this case, a large core file was also produced.
      </p>
<codeblock><![CDATA[
# cd /var/log/impalad
# ls -la | grep 23114
-rw-------   1 impala impala 3539079168 Jun 23 15:20 core.23114
-rw-r--r--   1 impala impala      99057 Jun 23 15:20 hs_err_pid23114.log
-rw-r--r--   1 impala impala        351 Jun 23 15:20 impalad.worker_node_123.impala.log.ERROR.20160623-140343.23114
-rw-r--r--   1 impala impala      29101 Jun 23 15:20 impalad.worker_node_123.impala.log.INFO.20160623-140343.23114
-rw-r--r--   1 impala impala        228 Jun 23 14:03 impalad.worker_node_123.impala.log.WARNING.20160623-140343.23114
]]>
</codeblock>
      <p>
        The <codeph>.INFO</codeph> log includes the location of the minidump file, followed by
        a report of a core dump. With the breakpad minidump feature enabled, now we might
        disable core dumps or keep fewer of them around.
      </p>
<codeblock><![CDATA[
# cat impalad.worker_node_123.impala.log.INFO.20160623-140343.23114
...
Wrote minidump to /var/log/impala-minidumps/impalad/0980da2d-a905-01e1-25ff883a-04ee027a.dmp
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000030c0e0b68a, pid=23114, tid=139869541455968
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libpthread.so.0+0xb68a]  pthread_cond_wait+0xca
#
# Core dump written. Default location: /var/log/impalad/core or core.23114
#
# An error report file with more information is saved as:
# /var/log/impalad/hs_err_pid23114.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
...

# cat impalad.worker_node_123.impala.log.ERROR.20160623-140343.23114

Log file created at: 2016/06/23 14:03:43
Running on machine:.worker_node_123
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0623 14:03:43.911002 23114 logging.cc:118] stderr will be logged to this file.
Wrote minidump to /var/log/impala-minidumps/impalad/0980da2d-a905-01e1-25ff883a-04ee027a.dmp
]]>
</codeblock>

      <p>
        The resulting minidump file is much smaller than the corresponding core file,
        making it much easier to supply diagnostic information to <keyword keyref="support_org"/>.
      </p>

<codeblock><![CDATA[
# pwd
/var/log/impalad
# cd ../impala-minidumps/impalad
# ls
0980da2d-a905-01e1-25ff883a-04ee027a.dmp
# du -kh *
2.4M  0980da2d-a905-01e1-25ff883a-04ee027a.dmp
]]>
</codeblock>
    </conbody>
  </concept>

</concept>