docs/build/plain-html/topics/impala_logging.html - impala - Git at Google

 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html
   PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

 <meta name="copyright" content="(C) Copyright 2025" />
 <meta name="DC.rights.owner" content="(C) Copyright 2025" />
 <meta name="DC.Type" content="concept" />
 <meta name="DC.Title" content="Using Impala Logging" />
 <meta name="prodname" content="Impala" />
 <meta name="prodname" content="Impala" />
 <meta name="prodname" content="Impala" />
 <meta name="prodname" content="Impala" />
 <meta name="prodname" content="Impala" />
 <meta name="prodname" content="Impala" />
 <meta name="prodname" content="Impala" />
 <meta name="prodname" content="Impala" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="DC.Format" content="XHTML" />
 <meta name="DC.Identifier" content="logging" />
 <link rel="stylesheet" type="text/css" href="../commonltr.css" />
 <title>Using Impala Logging</title>
 </head>
 <body id="logging">


   <h1 class="title topictitle1" id="ariaid-title1">Using Impala Logging</h1>


   <div class="body conbody">

     <p class="p"> The Impala logs record information about: </p>


     <ul class="ul">
       <li class="li"> Any errors Impala encountered. If Impala experienced a serious error during startup, you
         must diagnose and troubleshoot that problem before you can do anything further with Impala. </li>


       <li class="li"> How Impala is configured. </li>


       <li class="li"> Jobs Impala has completed. </li>

     </ul>


     <div class="note note"><span class="notetitle">Note:</span>
       <p class="p"> Formerly, the logs contained the query profile for each query, showing low-level details
         of how the work is distributed among nodes and how intermediate and final results are
         transmitted across the network. To save space, those query profiles are now stored in
         zlib-compressed files in <span class="ph filepath">/var/log/impala/profiles</span>. You can access them
         through the Impala web user interface. For example, at
             <code class="ph codeph">http://<var class="keyword varname">impalad-node-hostname</var>:25000/queries</code>, each
         query is followed by a <code class="ph codeph">Profile</code> link leading to a page showing extensive
         analytical data for the query execution. </p>

       <p class="p"> The auditing feature introduced in Impala 1.1.1 produces a separate set of
         audit log files when enabled. See <a class="xref" href="impala_auditing.html#auditing">Auditing Impala Operations</a> for details. </p>

       <p class="p"> In <span class="keyword">Impala 2.9</span> and higher, you can control
         how many audit event log files are kept on each host through the
           <code class="ph codeph">‑‑max_audit_event_log_files</code> startup option for the
           <span class="keyword cmdname">impalad</span> daemon, similar to the
           <code class="ph codeph">‑‑max_log_files</code> option for regular log files. </p>

       <p class="p"> The lineage feature introduced in Impala 2.2.0 produces a separate lineage log
         file when enabled. See <a class="xref" href="impala_lineage.html#lineage">Viewing Lineage Information for Impala Data</a> for details. </p>

     </div>


     <p class="p toc inpage"></p>


   </div>


   <div class="topic concept nested1" aria-labelledby="ariaid-title2" id="logs_details">

     <h2 class="title topictitle2" id="ariaid-title2">Locations and Names of Impala Log Files</h2>


     <div class="body conbody">

       <ul class="ul">
         <li class="li"> By default, the log files are under the directory <span class="ph filepath">/var/log/impala</span>.
           To change log file locations, modify the defaults file described in <a class="xref" href="impala_processes.html#processes">Starting Impala</a>. </li>


         <li class="li"> The significant files for the <code class="ph codeph">impalad</code> process are
             <span class="ph filepath">impalad.INFO</span>, <span class="ph filepath">impalad.WARNING</span>, and
             <span class="ph filepath">impalad.ERROR</span>. You might also see a file
             <span class="ph filepath">impalad.FATAL</span>, although this is only present in rare conditions. </li>


         <li class="li"> The significant files for the <code class="ph codeph">statestored</code> process are
             <span class="ph filepath">statestored.INFO</span>, <span class="ph filepath">statestored.WARNING</span>, and
             <span class="ph filepath">statestored.ERROR</span>. You might also see a file
             <span class="ph filepath">statestored.FATAL</span>, although this is only present in rare
           conditions. </li>


         <li class="li"> The significant files for the <code class="ph codeph">catalogd</code> process are
             <span class="ph filepath">catalogd.INFO</span>, <span class="ph filepath">catalogd.WARNING</span>, and
             <span class="ph filepath">catalogd.ERROR</span>. You might also see a file
             <span class="ph filepath">catalogd.FATAL</span>, although this is only present in rare conditions. </li>


         <li class="li"> Examine the <code class="ph codeph">.INFO</code> files to see configuration settings for the
           processes. </li>


         <li class="li"> Examine the <code class="ph codeph">.WARNING</code> files to see all kinds of problem information,
           including such things as suboptimal settings and also serious runtime errors. </li>


         <li class="li"> Examine the <code class="ph codeph">.ERROR</code> and/or <code class="ph codeph">.FATAL</code> files to see only
           the most serious errors, if the processes crash, or queries fail to complete. These
           messages are also in the <code class="ph codeph">.WARNING</code> file. </li>


         <li class="li"> A new set of log files is produced each time the associated daemon is restarted. These
           log files have long names including a timestamp. The <code class="ph codeph">.INFO</code>,
             <code class="ph codeph">.WARNING</code>, and <code class="ph codeph">.ERROR</code> files are physically represented
           as symbolic links to the latest applicable log files. </li>

       </ul>


       <p class="p"> Impala stores information using the <code class="ph codeph">glog_v</code> logging system. You will see
         some messages referring to C++ file names. Logging is affected by: </p>


       <ul class="ul">
         <li class="li"> The <code class="ph codeph">GLOG_v</code> environment variable specifies which types of messages are
           logged. See <a class="xref" href="#log_levels">Setting Logging Levels</a> for details. </li>


         <li class="li"> The <code class="ph codeph">‑‑logbuflevel</code> startup flag for the
             <span class="keyword cmdname">impalad</span> daemon specifies how often the log information is written to
           disk. The default is 0, meaning that the log is immediately flushed to disk when Impala
           outputs an important messages such as a warning or an error, but less important messages
           such as informational ones are buffered in memory rather than being flushed to disk
           immediately. </li>

       </ul>


     </div>


   </div>


   <div class="topic concept nested1" aria-labelledby="ariaid-title3" id="logs_rotate">

     <h2 class="title topictitle2" id="ariaid-title3">Rotating Impala Logs</h2>


     <div class="body conbody">

       <p class="p"> Impala periodically switches the physical files representing the current log files, after
         which it is safe to remove the old files if they are no longer needed. </p>


       <p class="p"> Impala can automatically remove older unneeded log files, a feature known as <dfn class="term">log
           rotation</dfn>. </p>


       <div class="p"> In Impala 2.2 and higher, the <code class="ph codeph">‑‑max_log_files</code> configuration
         option specifies how many log files to keep at each severity level (<code class="ph codeph">INFO</code>,
           <code class="ph codeph">WARNING</code>, <code class="ph codeph">ERROR</code>, and <code class="ph codeph">FATAL</code>). You can
         specify an appropriate setting for each Impala-related daemon (<span class="keyword cmdname">impalad</span>,
           <span class="keyword cmdname">statestored</span>, and <span class="keyword cmdname">catalogd</span>). <ul class="ul">
           <li class="li"> A value of 0 preserves all log files, in which case you would set up set up manual
             log rotation using your Linux tool or technique of choice. </li>


           <li class="li"> A value of 1 preserves only the very latest log file. </li>


           <li class="li"> The default value is 10. </li>

         </ul>

       </div>


       <p class="p"> Impala checks to see if any old logs need to be removed based on the interval specified in
         the <code class="ph codeph">‑‑logbufsecs</code> setting, every 5 seconds by default. </p>


       <p class="p"> For some log levels, Impala logs are first temporarily buffered in memory and only written
         to disk periodically. The <code class="ph codeph">‑‑logbufsecs</code> setting controls the
         maximum time that log messages are buffered for. For example, with the default value of 5
         seconds, there may be up to a 5 second delay before a logged message shows up in the log
         file. </p>


       <p class="p"> It is not recommended that you set <code class="ph codeph">‑‑logbufsecs</code> to 0 as the
         setting makes the Impala daemon to spin in the thread that tries to delete old log files. </p>


     </div>


   </div>


   <div class="topic concept nested1" aria-labelledby="ariaid-title4" id="dynamic_log_levels">
     <h2 class="title topictitle2" id="ariaid-title4">Changing Log Levels Dynamically</h2>

     <div class="body conbody">
       <p class="p">For debugging purposes you may be adjusting the logging configuration for Catalog and
         impalad servers. This required restarting the services. Impala supports adjusting the log
         levels dynamically without the need to restart the server. There is a
           <code class="ph codeph">/log_level</code> tab in the debug page of all Impala servers. You can query the
           <code class="ph codeph">log4j</code> log level of <code class="ph codeph">root</code> or
           <code class="ph codeph">org.apache.impala</code> by using the <code class="ph codeph">Get Java Log Level</code>
         button. Also you can change the <code class="ph codeph">vlog/log4j</code> levels to any supported levels
         of logging. You can select the log level using the <code class="ph codeph">LOG LEVEL</code> drop down box.
         You also have an option to restore the log levels to their original configuration by using
         the <code class="ph codeph">RESET</code> button.</p>

       <p class="p">Here is the format of a Glog:</p>

       <pre class="pre codeblock"><code>${level}${month}${day} HH:MM:SS.${us} ${thread-id} ${source-file}:${line}] ${query-id}] ${message}</code></pre>
       <p class="p">where</p>

       <ul class="ul" id="dynamic_log_levels__ul_yqb_ynv_plb">
         <li class="li">${level} — Log Levels; displays the levels as <code class="ph codeph">I</code> for
             <code class="ph codeph">INFO</code>, <code class="ph codeph">W</code> for <code class="ph codeph">WARNING</code>,
             <code class="ph codeph">E</code> for <code class="ph codeph">ERROR</code>, <code class="ph codeph">F</code> for
             <code class="ph codeph">FATAL.</code>
         </li>

         <li class="li">${month}${day} — Month and Date. </li>

         <li class="li">HH:MM:SS — Hours, Minutes, Seconds. </li>

         <li class="li">${us} — Microseconds. </li>

         <li class="li">${thread-id} — TID of the thread. </li>

         <li class="li">${source-file}:${line}] — File name and line number. </li>

         <li class="li">${query-id}] — An unique id for each and every query that is run in Impala. </li>

         <li class="li">${message} — Actual log message.</li>

       </ul>

     </div>

   </div>


   <div class="topic concept nested1" aria-labelledby="ariaid-title5" id="logs_debug">

     <h2 class="title topictitle2" id="ariaid-title5">Reviewing Impala Logs</h2>


     <div class="body conbody">

       <p class="p"> By default, the Impala log is stored at <code class="ph codeph">/var/log/impalad/</code>. The most
         comprehensive log, showing informational, warning, and error messages, is in the file name
           <span class="ph filepath">impalad.INFO</span>. View log file contents by using the web interface or by
         examining the contents of the log file. (When you examine the logs through the file system,
         you can troubleshoot problems by reading the <span class="ph filepath">impalad.WARNING</span> and/or
           <span class="ph filepath">impalad.ERROR</span> files, which contain the subsets of messages indicating
         potential problems.) </p>


       <div class="note note"><span class="notetitle">Note:</span>
         <p class="p"> The web interface limits the amount of logging information displayed. To view every log
           entry, access the log files directly through the file system. </p>

       </div>


       <p class="p"> You can view the contents of the <code class="ph codeph">impalad.INFO</code> log file in the file
         system. With the default configuration settings, the start of the log file appears as
         follows: </p>


       <pre class="pre codeblock"><code>[user@example impalad]$ pwd
 /var/log/impalad
 [user@example impalad]$ more impalad.INFO
 Log file created at: 2013/01/07 08:42:12
 Running on machine: impala.example.com
 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
 I0107 08:42:12.292155 14876 daemon.cc:34] impalad version 0.4 RELEASE (build 9d7fadca0461ab40b9e9df8cdb47107ec6b27cff)
 Built on Fri, 21 Dec 2012 12:55:19 PST
 I0107 08:42:12.292484 14876 daemon.cc:35] Using hostname: impala.example.com
 I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug webserver):
 --dump_ir=false
 --module_output=
 --be_port=22000
 --classpath=
 --hostname=impala.example.com</code></pre>

       <div class="note note"><span class="notetitle">Note:</span>  The preceding example shows only a small part of the log file. Impala log files are
         often several megabytes in size. </div>


     </div>


   </div>


   <div class="topic concept nested1" aria-labelledby="ariaid-title6" id="log_format">

     <h2 class="title topictitle2" id="ariaid-title6">Understanding Impala Log Contents</h2>


     <div class="body conbody">

       <p class="p"> The logs store information about Impala startup options. This information appears once for
         each time Impala is started and may include: </p>


       <ul class="ul">
         <li class="li"> Machine name. </li>


         <li class="li"> Impala version number. </li>


         <li class="li"> Flags used to start Impala. </li>


         <li class="li"> CPU information. </li>


         <li class="li"> The number of available disks. </li>

       </ul>


     </div>


   </div>


   <div class="topic concept nested1" aria-labelledby="ariaid-title7" id="log_levels">

     <h2 class="title topictitle2" id="ariaid-title7">Setting Logging Levels</h2>


     <div class="body conbody">

       <p class="p"> Impala uses the GLOG system, which supports three logging levels. You can adjust logging
         levels by exporting variable settings. To change logging settings manually, use a command
         similar to the following on each node before starting <code class="ph codeph">impalad</code>: </p>


       <pre class="pre codeblock"><code>export GLOG_v=1</code></pre>

       <div class="note note"><span class="notetitle">Note:</span>  For performance reasons, do not enable the most verbose logging level of 3 unless there
         is no other alternative for troubleshooting. </div>


       <p class="p"> For more information on how to configure GLOG, including how to set variable logging
         levels for different system components, see <a class="xref" href="https://github.com/google/glog" target="_blank">documentation for the
           glog project on github</a>. </p>


       <div class="section" id="log_levels__loglevels_details"><h3 class="title sectiontitle">Understanding What is Logged at Different Logging Levels</h3>

         <p class="p"> As logging levels increase, the categories of information logged are cumulative. For
           example, GLOG_v=2 records everything GLOG_v=1 records, as well as additional information. </p>

         <p class="p"> Increasing logging levels imposes performance overhead and increases log size. Where
           practical, use GLOG_v=1 for most cases: this level has minimal performance impact but
           still captures useful troubleshooting information. </p>

         <p class="p"> Additional information logged at each level is as follows: </p>

         <ul class="ul">
           <li class="li"> GLOG_v=1 - The default level. Logs information about each connection and query that
             is initiated to an <code class="ph codeph">impalad</code> instance, including runtime profiles. </li>


           <li class="li"> GLOG_v=2 - Everything from the previous level plus information for each RPC
             initiated. This level also records query execution progress information, including
             details on each file that is read. </li>


           <li class="li"> GLOG_v=3 - Everything from the previous level plus logging of every row that is read.
             This level is only applicable for the most serious troubleshooting and tuning scenarios,
             because it can produce exceptionally large and detailed log files, potentially leading
             to its own set of performance and capacity problems. </li>

         </ul>

       </div>


     </div>


   </div>


   <div class="topic concept nested1" aria-labelledby="ariaid-title8" id="redaction">

     <h2 class="title topictitle2" id="ariaid-title8">Redacting Sensitive Information from Impala Log Files</h2>


     <div class="body conbody">

       <p class="p">
         <dfn class="term">Log redaction</dfn> is a security feature that prevents sensitive information from
         being displayed in locations used by administrators for monitoring and troubleshooting, such
         as log files and the Impala debug web user interface. You configure regular expressions that
         match sensitive types of information processed by your system, such as credit card numbers
         or tax IDs, and literals matching these patterns are obfuscated wherever they would normally
         be recorded in log files or displayed in administration or debugging user interfaces. </p>


       <p class="p"> In a security context, the log redaction feature is complementary to the Ranger
         authorization framework. Ranger prevents unauthorized users from being able to directly
         access table data. Redaction prevents administrators or support personnel from seeing the
         smaller amounts of sensitive or personally identifying information (PII) that might appear
         in queries issued by those authorized users. </p>


       <p class="p"> See <span class="xref">the documentation for your Apache Hadoop distribution</span> for details about how to enable this feature and set up
         the regular expressions to detect and redact sensitive information within SQL statement
         text. </p>


     </div>


   </div>


 </body>
 </html>
	<?xml version="1.0" encoding="UTF-8"?>
	<!DOCTYPE html
	PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

	<meta name="copyright" content="(C) Copyright 2025" />
	<meta name="DC.rights.owner" content="(C) Copyright 2025" />
	<meta name="DC.Type" content="concept" />
	<meta name="DC.Title" content="Using Impala Logging" />
	<meta name="prodname" content="Impala" />
	<meta name="prodname" content="Impala" />
	<meta name="prodname" content="Impala" />
	<meta name="prodname" content="Impala" />
	<meta name="prodname" content="Impala" />
	<meta name="prodname" content="Impala" />
	<meta name="prodname" content="Impala" />
	<meta name="prodname" content="Impala" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="DC.Format" content="XHTML" />
	<meta name="DC.Identifier" content="logging" />
	<link rel="stylesheet" type="text/css" href="../commonltr.css" />
	<title>Using Impala Logging</title>
	</head>
	<body id="logging">


	<h1 class="title topictitle1" id="ariaid-title1">Using Impala Logging</h1>






	<div class="body conbody">

	<p class="p"> The Impala logs record information about: </p>


	<ul class="ul">
	<li class="li"> Any errors Impala encountered. If Impala experienced a serious error during startup, you
	must diagnose and troubleshoot that problem before you can do anything further with Impala. </li>


	<li class="li"> How Impala is configured. </li>


	<li class="li"> Jobs Impala has completed. </li>

	</ul>


	<div class="note note"><span class="notetitle">Note:</span>
	<p class="p"> Formerly, the logs contained the query profile for each query, showing low-level details
	of how the work is distributed among nodes and how intermediate and final results are
	transmitted across the network. To save space, those query profiles are now stored in
	zlib-compressed files in <span class="ph filepath">/var/log/impala/profiles</span>. You can access them
	through the Impala web user interface. For example, at
	<code class="ph codeph">http://<var class="keyword varname">impalad-node-hostname</var>:25000/queries</code>, each
	query is followed by a <code class="ph codeph">Profile</code> link leading to a page showing extensive
	analytical data for the query execution. </p>

	<p class="p"> The auditing feature introduced in Impala 1.1.1 produces a separate set of
	audit log files when enabled. See <a class="xref" href="impala_auditing.html#auditing">Auditing Impala Operations</a> for details. </p>

	<p class="p"> In <span class="keyword">Impala 2.9</span> and higher, you can control
	how many audit event log files are kept on each host through the
	<code class="ph codeph">‑‑max_audit_event_log_files</code> startup option for the
	<span class="keyword cmdname">impalad</span> daemon, similar to the
	<code class="ph codeph">‑‑max_log_files</code> option for regular log files. </p>

	<p class="p"> The lineage feature introduced in Impala 2.2.0 produces a separate lineage log
	file when enabled. See <a class="xref" href="impala_lineage.html#lineage">Viewing Lineage Information for Impala Data</a> for details. </p>

	</div>


	<p class="p toc inpage"></p>


	</div>


	<div class="topic concept nested1" aria-labelledby="ariaid-title2" id="logs_details">

	<h2 class="title topictitle2" id="ariaid-title2">Locations and Names of Impala Log Files</h2>


	<div class="body conbody">

	<ul class="ul">
	<li class="li"> By default, the log files are under the directory <span class="ph filepath">/var/log/impala</span>.
	To change log file locations, modify the defaults file described in <a class="xref" href="impala_processes.html#processes">Starting Impala</a>. </li>


	<li class="li"> The significant files for the <code class="ph codeph">impalad</code> process are
	<span class="ph filepath">impalad.INFO</span>, <span class="ph filepath">impalad.WARNING</span>, and
	<span class="ph filepath">impalad.ERROR</span>. You might also see a file
	<span class="ph filepath">impalad.FATAL</span>, although this is only present in rare conditions. </li>


	<li class="li"> The significant files for the <code class="ph codeph">statestored</code> process are
	<span class="ph filepath">statestored.INFO</span>, <span class="ph filepath">statestored.WARNING</span>, and
	<span class="ph filepath">statestored.ERROR</span>. You might also see a file
	<span class="ph filepath">statestored.FATAL</span>, although this is only present in rare
	conditions. </li>


	<li class="li"> The significant files for the <code class="ph codeph">catalogd</code> process are
	<span class="ph filepath">catalogd.INFO</span>, <span class="ph filepath">catalogd.WARNING</span>, and
	<span class="ph filepath">catalogd.ERROR</span>. You might also see a file
	<span class="ph filepath">catalogd.FATAL</span>, although this is only present in rare conditions. </li>


	<li class="li"> Examine the <code class="ph codeph">.INFO</code> files to see configuration settings for the
	processes. </li>


	<li class="li"> Examine the <code class="ph codeph">.WARNING</code> files to see all kinds of problem information,
	including such things as suboptimal settings and also serious runtime errors. </li>


	<li class="li"> Examine the <code class="ph codeph">.ERROR</code> and/or <code class="ph codeph">.FATAL</code> files to see only
	the most serious errors, if the processes crash, or queries fail to complete. These
	messages are also in the <code class="ph codeph">.WARNING</code> file. </li>


	<li class="li"> A new set of log files is produced each time the associated daemon is restarted. These
	log files have long names including a timestamp. The <code class="ph codeph">.INFO</code>,
	<code class="ph codeph">.WARNING</code>, and <code class="ph codeph">.ERROR</code> files are physically represented
	as symbolic links to the latest applicable log files. </li>

	</ul>


	<p class="p"> Impala stores information using the <code class="ph codeph">glog_v</code> logging system. You will see
	some messages referring to C++ file names. Logging is affected by: </p>


	<ul class="ul">
	<li class="li"> The <code class="ph codeph">GLOG_v</code> environment variable specifies which types of messages are
	logged. See <a class="xref" href="#log_levels">Setting Logging Levels</a> for details. </li>


	<li class="li"> The <code class="ph codeph">‑‑logbuflevel</code> startup flag for the
	<span class="keyword cmdname">impalad</span> daemon specifies how often the log information is written to
	disk. The default is 0, meaning that the log is immediately flushed to disk when Impala
	outputs an important messages such as a warning or an error, but less important messages
	such as informational ones are buffered in memory rather than being flushed to disk
	immediately. </li>

	</ul>


	</div>


	</div>


	<div class="topic concept nested1" aria-labelledby="ariaid-title3" id="logs_rotate">

	<h2 class="title topictitle2" id="ariaid-title3">Rotating Impala Logs</h2>




	<div class="body conbody">

	<p class="p"> Impala periodically switches the physical files representing the current log files, after
	which it is safe to remove the old files if they are no longer needed. </p>


	<p class="p"> Impala can automatically remove older unneeded log files, a feature known as <dfn class="term">log
	rotation</dfn>. </p>


	<div class="p"> In Impala 2.2 and higher, the <code class="ph codeph">‑‑max_log_files</code> configuration
	option specifies how many log files to keep at each severity level (<code class="ph codeph">INFO</code>,
	<code class="ph codeph">WARNING</code>, <code class="ph codeph">ERROR</code>, and <code class="ph codeph">FATAL</code>). You can
	specify an appropriate setting for each Impala-related daemon (<span class="keyword cmdname">impalad</span>,
	<span class="keyword cmdname">statestored</span>, and <span class="keyword cmdname">catalogd</span>). <ul class="ul">
	<li class="li"> A value of 0 preserves all log files, in which case you would set up set up manual
	log rotation using your Linux tool or technique of choice. </li>


	<li class="li"> A value of 1 preserves only the very latest log file. </li>


	<li class="li"> The default value is 10. </li>

	</ul>

	</div>


	<p class="p"> Impala checks to see if any old logs need to be removed based on the interval specified in
	the <code class="ph codeph">‑‑logbufsecs</code> setting, every 5 seconds by default. </p>


	<p class="p"> For some log levels, Impala logs are first temporarily buffered in memory and only written
	to disk periodically. The <code class="ph codeph">‑‑logbufsecs</code> setting controls the
	maximum time that log messages are buffered for. For example, with the default value of 5
	seconds, there may be up to a 5 second delay before a logged message shows up in the log
	file. </p>


	<p class="p"> It is not recommended that you set <code class="ph codeph">‑‑logbufsecs</code> to 0 as the
	setting makes the Impala daemon to spin in the thread that tries to delete old log files. </p>


	</div>


	</div>


	<div class="topic concept nested1" aria-labelledby="ariaid-title4" id="dynamic_log_levels">
	<h2 class="title topictitle2" id="ariaid-title4">Changing Log Levels Dynamically</h2>

	<div class="body conbody">
	<p class="p">For debugging purposes you may be adjusting the logging configuration for Catalog and
	impalad servers. This required restarting the services. Impala supports adjusting the log
	levels dynamically without the need to restart the server. There is a
	<code class="ph codeph">/log_level</code> tab in the debug page of all Impala servers. You can query the
	<code class="ph codeph">log4j</code> log level of <code class="ph codeph">root</code> or
	<code class="ph codeph">org.apache.impala</code> by using the <code class="ph codeph">Get Java Log Level</code>
	button. Also you can change the <code class="ph codeph">vlog/log4j</code> levels to any supported levels
	of logging. You can select the log level using the <code class="ph codeph">LOG LEVEL</code> drop down box.
	You also have an option to restore the log levels to their original configuration by using
	the <code class="ph codeph">RESET</code> button.</p>

	<p class="p">Here is the format of a Glog:</p>

	<pre class="pre codeblock"><code>${level}${month}${day} HH:MM:SS.${us} ${thread-id} ${source-file}:${line}] ${query-id}] ${message}</code></pre>
	<p class="p">where</p>

	<ul class="ul" id="dynamic_log_levels__ul_yqb_ynv_plb">
	<li class="li">${level} — Log Levels; displays the levels as <code class="ph codeph">I</code> for
	<code class="ph codeph">INFO</code>, <code class="ph codeph">W</code> for <code class="ph codeph">WARNING</code>,
	<code class="ph codeph">E</code> for <code class="ph codeph">ERROR</code>, <code class="ph codeph">F</code> for
	<code class="ph codeph">FATAL.</code>
	</li>

	<li class="li">${month}${day} — Month and Date. </li>

	<li class="li">HH:MM:SS — Hours, Minutes, Seconds. </li>

	<li class="li">${us} — Microseconds. </li>

	<li class="li">${thread-id} — TID of the thread. </li>

	<li class="li">${source-file}:${line}] — File name and line number. </li>

	<li class="li">${query-id}] — An unique id for each and every query that is run in Impala. </li>

	<li class="li">${message} — Actual log message.</li>

	</ul>

	</div>

	</div>


	<div class="topic concept nested1" aria-labelledby="ariaid-title5" id="logs_debug">

	<h2 class="title topictitle2" id="ariaid-title5">Reviewing Impala Logs</h2>


	<div class="body conbody">

	<p class="p"> By default, the Impala log is stored at <code class="ph codeph">/var/log/impalad/</code>. The most
	comprehensive log, showing informational, warning, and error messages, is in the file name
	<span class="ph filepath">impalad.INFO</span>. View log file contents by using the web interface or by
	examining the contents of the log file. (When you examine the logs through the file system,
	you can troubleshoot problems by reading the <span class="ph filepath">impalad.WARNING</span> and/or
	<span class="ph filepath">impalad.ERROR</span> files, which contain the subsets of messages indicating
	potential problems.) </p>


	<div class="note note"><span class="notetitle">Note:</span>
	<p class="p"> The web interface limits the amount of logging information displayed. To view every log
	entry, access the log files directly through the file system. </p>

	</div>


	<p class="p"> You can view the contents of the <code class="ph codeph">impalad.INFO</code> log file in the file
	system. With the default configuration settings, the start of the log file appears as
	follows: </p>


	<pre class="pre codeblock"><code>[user@example impalad]$ pwd
	/var/log/impalad
	[user@example impalad]$ more impalad.INFO
	Log file created at: 2013/01/07 08:42:12
	Running on machine: impala.example.com
	Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
	I0107 08:42:12.292155 14876 daemon.cc:34] impalad version 0.4 RELEASE (build 9d7fadca0461ab40b9e9df8cdb47107ec6b27cff)
	Built on Fri, 21 Dec 2012 12:55:19 PST
	I0107 08:42:12.292484 14876 daemon.cc:35] Using hostname: impala.example.com
	I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug webserver):
	--dump_ir=false
	--module_output=
	--be_port=22000
	--classpath=
	--hostname=impala.example.com</code></pre>

	<div class="note note"><span class="notetitle">Note:</span> The preceding example shows only a small part of the log file. Impala log files are
	often several megabytes in size. </div>


	</div>


	</div>


	<div class="topic concept nested1" aria-labelledby="ariaid-title6" id="log_format">

	<h2 class="title topictitle2" id="ariaid-title6">Understanding Impala Log Contents</h2>


	<div class="body conbody">

	<p class="p"> The logs store information about Impala startup options. This information appears once for
	each time Impala is started and may include: </p>


	<ul class="ul">
	<li class="li"> Machine name. </li>


	<li class="li"> Impala version number. </li>


	<li class="li"> Flags used to start Impala. </li>


	<li class="li"> CPU information. </li>


	<li class="li"> The number of available disks. </li>

	</ul>


	</div>


	</div>


	<div class="topic concept nested1" aria-labelledby="ariaid-title7" id="log_levels">

	<h2 class="title topictitle2" id="ariaid-title7">Setting Logging Levels</h2>


	<div class="body conbody">

	<p class="p"> Impala uses the GLOG system, which supports three logging levels. You can adjust logging
	levels by exporting variable settings. To change logging settings manually, use a command
	similar to the following on each node before starting <code class="ph codeph">impalad</code>: </p>


	<pre class="pre codeblock"><code>export GLOG_v=1</code></pre>

	<div class="note note"><span class="notetitle">Note:</span> For performance reasons, do not enable the most verbose logging level of 3 unless there
	is no other alternative for troubleshooting. </div>


	<p class="p"> For more information on how to configure GLOG, including how to set variable logging
	levels for different system components, see <a class="xref" href="https://github.com/google/glog" target="_blank">documentation for the
	glog project on github</a>. </p>


	<div class="section" id="log_levels__loglevels_details"><h3 class="title sectiontitle">Understanding What is Logged at Different Logging Levels</h3>

	<p class="p"> As logging levels increase, the categories of information logged are cumulative. For
	example, GLOG_v=2 records everything GLOG_v=1 records, as well as additional information. </p>

	<p class="p"> Increasing logging levels imposes performance overhead and increases log size. Where
	practical, use GLOG_v=1 for most cases: this level has minimal performance impact but
	still captures useful troubleshooting information. </p>

	<p class="p"> Additional information logged at each level is as follows: </p>

	<ul class="ul">
	<li class="li"> GLOG_v=1 - The default level. Logs information about each connection and query that
	is initiated to an <code class="ph codeph">impalad</code> instance, including runtime profiles. </li>


	<li class="li"> GLOG_v=2 - Everything from the previous level plus information for each RPC
	initiated. This level also records query execution progress information, including
	details on each file that is read. </li>


	<li class="li"> GLOG_v=3 - Everything from the previous level plus logging of every row that is read.
	This level is only applicable for the most serious troubleshooting and tuning scenarios,
	because it can produce exceptionally large and detailed log files, potentially leading
	to its own set of performance and capacity problems. </li>

	</ul>

	</div>


	</div>


	</div>


	<div class="topic concept nested1" aria-labelledby="ariaid-title8" id="redaction">

	<h2 class="title topictitle2" id="ariaid-title8">Redacting Sensitive Information from Impala Log Files</h2>




	<div class="body conbody">

	<p class="p">
	<dfn class="term">Log redaction</dfn> is a security feature that prevents sensitive information from
	being displayed in locations used by administrators for monitoring and troubleshooting, such
	as log files and the Impala debug web user interface. You configure regular expressions that
	match sensitive types of information processed by your system, such as credit card numbers
	or tax IDs, and literals matching these patterns are obfuscated wherever they would normally
	be recorded in log files or displayed in administration or debugging user interfaces. </p>


	<p class="p"> In a security context, the log redaction feature is complementary to the Ranger
	authorization framework. Ranger prevents unauthorized users from being able to directly
	access table data. Redaction prevents administrators or support personnel from seeing the
	smaller amounts of sensitive or personally identifying information (PII) that might appear
	in queries issued by those authorized users. </p>


	<p class="p"> See <span class="xref">the documentation for your Apache Hadoop distribution</span> for details about how to enable this feature and set up
	the regular expressions to detect and redact sensitive information within SQL statement
	text. </p>


	</div>


	</div>


	</body>
	</html>