why-do-kahadb-log-files-remain-after-cleanup.html - activemq-web - Git at Google

 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
 <!--

     Licensed to the Apache Software Foundation (ASF) under one or more
     contributor license agreements.  See the NOTICE file distributed with
     this work for additional information regarding copyright ownership.
     The ASF licenses this file to You under the Apache License, Version 2.0
     (the "License"); you may not use this file except in compliance with
     the License.  You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

     Unless required by applicable law or agreed to in writing, software
     distributed under the License is distributed on an "AS IS" BASIS,
     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     See the License for the specific language governing permissions and
     limitations under the License.
 -->
 <html>
 <head>
     <link href="http://activemq.apache.org/styles/site.css" rel="stylesheet" type="text/css"/>
     <link href="http://activemq.apache.org/styles/type-settings.css" rel="stylesheet" type="text/css"/>
     <script src="http://activemq.apache.org/styles/prototype.js" type="text/javascript"></script>
     <script src="http://activemq.apache.org/styles/rico.js" type="text/javascript"></script>
     <script src="http://activemq.apache.org/styles/site.js" type="text/javascript"></script>
     <style type="text/css">
       .maincontent { overflow:hidden; }
     </style>
     <!--[if IE]>
     <style type="text/css">
       .maincontent { width:100%; }
     </style>
     <![endif]-->

           <link href='http://activemq.apache.org/styles/highlighter/styles/shCore.css' rel='stylesheet' type='text/css' />
       <link href='http://activemq.apache.org/styles/highlighter/styles/shThemeEclipse.css' rel='stylesheet' type='text/css' />
       <script src='http://activemq.apache.org/styles/highlighter/scripts/shCore.js' type='text/javascript'></script>
               <script src='http://activemq.apache.org/styles/highlighter/scripts/shBrushJava.js' type='text/javascript'></script>
               <script src='http://activemq.apache.org/styles/highlighter/scripts/shBrushBash.js' type='text/javascript'></script>

       <script type="text/javascript">
         SyntaxHighlighter.defaults['toolbar'] = false;
         SyntaxHighlighter.all();
       </script>

     <title>
     Apache ActiveMQ &#8482; -- Why do KahaDB log files remain after cleanup
     </title>
 </head>
 <body>
 <div class="white_box">
 <div class="header">
   <div class="header_l">
     <div class="header_r">
     </div>
   </div>
 </div>
 <div class="content">
   <div class="content_l">
     <div class="content_r">
       <div>

 <!-- Banner -->
 <div id="asf_logo">
 	<div id="activemq_logo">
      <a shape="rect" style="float:left; width:280px;display:block;text-indent:-5000px;text-decoration:none;line-height:60px; margin-top:10px; margin-left:100px;" href="http://activemq.apache.org" title="The most popular and powerful open source Message Broker">ActiveMQ</a>
             <a shape="rect" style="float:right; width:210px;display:block;text-indent:-5000px;text-decoration:none;line-height:60px; margin-top:15px; margin-right:10px;" href="http://www.apache.org" title="The Apache Software Foundation">ASF</a>
 	</div>
 </div>

         <div class="top_red_bar">
           <div id="site-breadcrumbs">
 <a href="community.html">Community</a>&nbsp;&gt;&nbsp;<a href="faq.html">FAQ</a>&nbsp;&gt;&nbsp;<a href="errors.html">Errors</a>&nbsp;&gt;&nbsp;<a href="why-do-kahadb-log-files-remain-after-cleanup.html">Why do KahaDB log files remain after cleanup</a>
           </div>
           <div id="site-quicklinks">
 <p><a shape="rect" href="download.html">Download</a> | <a shape="rect" class="external-link" href="http://activemq.apache.org/maven/apidocs/index.html">JavaDocs</a> <a shape="rect" href="javadocs.html">More...</a> | <a shape="rect" href="source.html">Source</a> | <a shape="rect" href="discussion-forums.html">Forums</a> | <a shape="rect" href="support.html">Support</a></p>
           </div>
         </div>

   <table border="0">
   <tbody>
         <tr>
         <td valign="top" width="100%">
 <div class="wiki-content maincontent"><p>Clean-up of un-referenced KahaDB journal log files&#160;<strong><code>data-&lt;id&gt;.log</code></strong> will occur every 30 seconds by default. If a data file is in-use it will not be cleaned up.</p><p>A data file may be in-use because:</p><ol><li>It contains a&#160;pending message for a destination or durable topic subscription</li><li>It contains an ACK for a message which is in an in-use data file - the ACK cannot be removed as a recovery would then mark the message for redelivery</li><li>The journal references a pending transaction</li><li>It is a journal file, and there may be a pending write to it</li></ol><p><span style="line-height: 1.4285715;">The&#160;<strong><code>TRACE</code></strong> level logging of the&#160;<strong><code>org.apache.activemq.store.kahadb.MessageDatabase</code></strong> class provides insight into the cleanup process and will allow you to determine why a given data file is considered in-use and as a result, not a candidate for cleanup.</span></p><p>To debug, add the following (or similar) to your&#160;<strong><code>log4j.properties</code></strong> file (if needed):</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
 <pre class="brush: java; gutter: false; theme: Default" style="font-size:12px;">log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
 log4j.appender.kahadb.file=${activemq.base}/data/kahadb.log
 log4j.appender.kahadb.maxFileSize=1024KB
 log4j.appender.kahadb.maxBackupIndex=5
 log4j.appender.kahadb.append=true
 log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
 log4j.appender.kahadb.layout.ConversionPattern=%d [%-15.15t] %-5p %-30.30c{1} - %m%n
 log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=TRACE, kahadb</pre>
 </div></div><p>Either restart ActiveMQ and let the cleanup process run (give it a minute or two for example) or alternatively apply this logging configuration to a running broker via JMX. The&#160;<strong><code>Broker</code></strong> MBean exposes an operation called&#160;<strong><code>reloadLog4jProperties</code></strong> in JMX that can be used to tell the broker to reload its <strong><code>log4j.properties</code></strong>. Often its enough to apply this logging configuration for 2-5 minutes and then analyze the broker's log file.</p><p>Examine the log file and look for cleanup of the data files. The process starts with the complete set of known data files and queries the index on a per destination basis to prune this list. Anything that remains is a candidate for cleanup. The trace logging gives the destination and the log file numbers that remain candidates for removal as it iterates through the index.</p><p>At some point you'll hit a destination and the number of data file ids will suddenly drop because that destination references them. It could be a DLQ or an offline durable subscriber. In any event, the logging will help you pinpoint the destinations that are hogging disk space.</p><p>Here is a quick sample:</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
 <pre class="brush: java; gutter: false; theme: Default" style="font-size:12px;"> TRACE | Last update: 164:41712, full gc candidates set: [86, 87, 163, 164] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates after first tx:164:41712, [86, 87, 163] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates after dest:0:A, [86, 87, 163] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates after dest:1:B, [86, 87, 163] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates after dest:0:D, [86, 87, 163] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates after dest:0:E, [86, 87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates after dest:0:H, [86, 87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates after dest:0:I, [86, 87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates after dest:0:J, [87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  TRACE | gc candidates: [87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
  DEBUG | Cleanup removing the data files: [87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker</pre>
 </div></div><p>We get one candidate,&#160;<strong><code>data-87.log</code></strong> from the existing set of journal data files <strong><code>[86, 87, 163, 164]</code></strong>. There is a current transaction using <strong><code>164</code></strong>, destination (Queue named&#160;<strong><code>E</code></strong>) <code>'<strong>0:E</strong>'</code> has some messages in <strong><code>163</code></strong>, destination <code>'<strong>0:I</strong>'</code> has messages in&#160;<strong><code>86</code></strong> and&#160;<strong><code>87</code></strong> is un-referenced. In this case, there must be some long standing un-acknowledged messages or a very slow consumer on destination <code>'<strong>0:I</strong>'</code>.</p><div class="confluence-information-macro confluence-information-macro-information"><span class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span><div class="confluence-information-macro-body"><p>The <code>'<strong>0:</strong>'</code> prefix is shorthand for a queue, <code>'<strong>1:</strong>'</code> for a topic. Example: <strong><code>dest:1:B</code></strong> refers to a topic named <strong><code>B</code></strong>.</p></div></div><p>&#160;</p><p>&#160;</p><hr><h3 id="WhydoKahaDBlogfilesremainaftercleanup-Non-persistentmessages">Non-persistent messages</h3><p>Similar for non-persistent messages that are not stored in your configured KahaDB persistence adapter but get swapped to temp storage once they exceed the broker's configured&#160;<strong><code>memoryUsage</code></strong> limit. A similar logging configuration can show details of the cleanup of temp storage.</p><p>&#160;</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
 <pre class="brush: java; gutter: false; theme: Default" style="font-size:12px;">log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
 log4j.appender.kahadb.file=${activemq.base}/data/kahadb.log
 log4j.appender.kahadb.maxFileSize=1024KB
 log4j.appender.kahadb.maxBackupIndex=5
 log4j.appender.kahadb.append=true
 log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
 log4j.appender.kahadb.layout.ConversionPattern=%d [%-15.15t] %-5p %-30.30c{1} - %m%n
 log4j.logger.org.apache.activemq.store.kahadb=TRACE, kahadb
 log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=INFO, kahadb
 &#160;</pre>
 </div></div><p>&#160;</p><p>Note the last line of above logging configuration disables the verbose logging of the KahaDB cleanup task. If that line gets removed, the cleanup details of both KahaDB and temp storage will be logged to the same file but you need to be careful not to mix the logging output.</p></div>
         </td>
         <td valign="top">
           <div class="navigation">
             <div class="navigation_top">
               <div class="navigation_bottom">
 <h3 id="Navigation-Overviewhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=35985"><a shape="rect" href="overview.html">Overview</a></h3><ul class="alternate"><li><a shape="rect" href="index.html">Index</a></li><li><a shape="rect" href="news.html">News</a></li><li><a shape="rect" href="new-features.html">New Features</a></li><li><a shape="rect" href="getting-started.html">Getting Started</a></li><li><a shape="rect" href="faq.html">FAQ</a></li><li><a shape="rect" href="articles.html">Articles</a></li><li><a shape="rect" href="books.html">Books</a></li><li><a shape="rect" href="download.html">Download</a></li><li><a shape="rect" class="external-link" href="http://www.apache.org/licenses/">License</a></li></ul><h3 id="Navigation-Search">Search</h3><div>
 <form enctype="application/x-www-form-urlencoded" method="get" action="http://www.google.com/search" style="font-size: 10px;">
 <input type="hidden" name="ie" value="UTF-8">
 <input type="hidden" name="oe" value="UTF-8">
   <input maxlength="255" type="text" name="q" size="15" value="value"><br clear="none">
   <input type="submit" name="btnG" value="Search">
   <input type="hidden" name="domains" value="activemq.apache.org">
   <input type="hidden" name="sitesearch" value="activemq.apache.org">
 </form>
 </div>
 <h3 id="Navigation-SubProjects">Sub Projects</h3><ul class="alternate"><li><a shape="rect" class="external-link" href="http://activemq.apache.org/artemis/">Artemis</a></li><li><a shape="rect" class="external-link" href="http://activemq.apache.org/apollo" title="ActiveMQ Apollo">Apollo</a></li><li><a shape="rect" class="external-link" href="http://activemq.apache.org/cms/">CMS</a></li><li><a shape="rect" class="external-link" href="http://activemq.apache.org/nms/" title="NMS is the .Net Messaging API">NMS</a></li></ul><h3 id="Navigation-Communityhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=36130"><a shape="rect" href="community.html">Community</a></h3><ul class="alternate"><li><a shape="rect" href="support.html">Support</a></li><li><a shape="rect" href="contributing.html">Contributing</a></li><li><a shape="rect" href="discussion-forums.html">Discussion Forums</a></li><li><a shape="rect" href="mailing-lists.html">Mailing Lists</a></li><li><a shape="rect" href="irc.html">IRC</a></li><li><a shape="rect" class="external-link" href="http://javabot.evanchooly.com/logs/%23apache-activemq/today" rel="nofollow">IRC Log</a></li><li><a shape="rect" href="security-advisories.html">Security Advisories</a></li><li><a shape="rect" href="site.html">Site</a></li><li><a shape="rect" class="external-link" href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li><li><a shape="rect" href="projects-using-activemq.html">Projects Using ActiveMQ</a></li><li><a shape="rect" href="users.html">Users</a></li><li><a shape="rect" href="team.html">Team</a></li><li><a shape="rect" href="thanks.html">Thanks</a></li></ul><h3 id="Navigation-Featureshttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=35883"><a shape="rect" href="features.html">Features</a></h3><ul class="alternate"><li><a shape="rect" href="advisory-message.html">Advisory Message</a></li><li><a shape="rect" href="clustering.html">Clustering</a></li><li><a shape="rect" href="cross-language-clients.html">Cross Language Clients</a></li><li><a shape="rect" href="enterprise-integration-patterns.html">Enterprise Integration Patterns</a></li><li><a shape="rect" href="jmx.html">JMX</a></li><li><a shape="rect" href="jms-to-jms-bridge.html">JMS to JMS Bridge</a></li><li><a shape="rect" href="masterslave.html">MasterSlave</a></li><li><a shape="rect" href="message-groups.html">Message Groups</a></li><li><a shape="rect" href="networks-of-brokers.html">Networks of Brokers</a></li><li><a shape="rect" href="performance.html">Performance</a></li><li><a shape="rect" href="persistence.html">Persistence</a></li><li><a shape="rect" href="security.html">Security</a></li><li><a shape="rect" href="virtual-destinations.html">Virtual Destinations</a></li><li><a shape="rect" href="visualisation.html">Visualisation</a></li><li><a shape="rect" href="features.html">More ...</a></li></ul><h3 id="Navigation-Connectivityhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=36167"><a shape="rect" href="connectivity.html">Connectivity</a></h3><ul class="alternate"><li><a shape="rect" href="ajax.html">Ajax</a></li><li><a shape="rect" href="amqp.html">AMQP</a></li><li><a shape="rect" href="axis-and-cxf-support.html">Axis and CXF Support</a></li><li><a shape="rect" href="c-integration.html">C Integration</a></li><li><a shape="rect" href="activemq-c-clients.html">C++</a></li><li><a shape="rect" class="external-link" href="http://activemq.apache.org/nms/">C# and .Net Integration</a></li><li><a shape="rect" class="external-link" href="http://activemq.apache.org/cms/">CMS</a></li><li><a shape="rect" href="j2ee.html">J2EE</a></li><li><a shape="rect" href="jboss-integration.html">JBoss Integration</a></li><li><a shape="rect" class="external-link" href="http://docs.codehaus.org/display/JETTY/Integrating+with+ActiveMQ" rel="nofollow">Jetty</a></li><li><a shape="rect" href="jndi-support.html">JNDI Support</a></li><li><a shape="rect" class="external-link" href="http://activemq.apache.org/nms/" title="NMS is the .Net Messaging API">NMS</a></li><li><a shape="rect" href="rest.html">REST</a></li><li><a shape="rect" href="rss-and-atom.html">RSS and Atom</a></li><li><a shape="rect" href="spring-support.html">Spring Support</a></li><li><a shape="rect" href="stomp.html">Stomp</a></li><li><a shape="rect" href="tomcat.html">Tomcat</a></li><li><a shape="rect" href="unix-service.html">Unix Service</a></li><li><a shape="rect" href="weblogic-integration.html">WebLogic Integration</a></li><li><a shape="rect" href="xmpp.html">XMPP</a></li><li><a shape="rect" href="connectivity.html">More ...</a></li></ul><h3 id="Navigation-UsingActiveMQ5https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71176"><a shape="rect" href="using-activemq-5.html">Using ActiveMQ 5</a></h3><ul class="alternate"><li><a shape="rect" href="version-5-getting-started.html">Getting Started</a></li><li><a shape="rect" href="version-5-initial-configuration.html">Initial Configuration</a></li><li><a shape="rect" href="version-5-run-broker.html">Running a Broker</a></li><li><a shape="rect" href="how-do-i-embed-a-broker-inside-a-connection.html">Embedded Brokers</a></li><li><a shape="rect" href="activemq-command-line-tools-reference.html">Command Line Tools</a></li><li><a shape="rect" href="configuring-version-5-transports.html">Configuring Transports</a></li><li><a shape="rect" href="version-5-examples.html">Examples</a></li><li><a shape="rect" href="version-5-web-samples.html">Web Samples</a></li><li><a shape="rect" href="how-can-i-monitor-activemq.html">Monitoring the Broker</a></li><li><a shape="rect" href="version-5-xml-configuration.html">Xml Configuration</a></li><li><a shape="rect" href="xml-reference.html">Xml Reference</a></li><li><a shape="rect" href="using-activemq-5.html">More ...</a></li></ul><h3 id="Navigation-Toolshttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=35912"><a shape="rect" href="tools.html">Tools</a></h3><ul class="alternate"><li><a shape="rect" href="web-console.html">Web Console</a></li><li><a shape="rect" href="activemq-performance-module-users-manual.html">Maven2 Performance Plugin</a></li></ul><h3 id="Navigation-Supporthttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=35919"><a shape="rect" href="support.html">Support</a></h3><ul class="alternate"><li><a shape="rect" class="external-link" href="http://issues.apache.org/jira/browse/AMQ">Issues</a></li><li><a shape="rect" class="external-link" href="http://issues.apache.org/activemq/browse/AMQ?report=com.atlassian.jira.plugin.system.project:roadmap-panel">Roadmap</a></li><li><a shape="rect" class="external-link" href="http://issues.apache.org/activemq/browse/AMQ?report=com.atlassian.jira.plugin.system.project:changelog-panel">Change log</a></li></ul><h3 id="Navigation-Developershttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=35903"><a shape="rect" href="developers.html">Developers</a></h3><ul class="alternate"><li><a shape="rect" href="source.html">Source</a></li><li><a shape="rect" href="building.html">Building</a></li><li><a shape="rect" href="developer-guide.html">Developer Guide</a></li><li><a shape="rect" href="becoming-a-committer.html">Becoming a committer</a></li><li><a shape="rect" href="code-overview.html">Code Overview</a></li><li><a shape="rect" href="wire-protocol.html">Wire Protocol</a></li><li><a shape="rect" href="release-guide.html">Release Guide</a></li></ul><h3 id="Navigation-Tests">Tests</h3><ul class="alternate"><li><a shape="rect" href="activemq-performance-module-users-manual.html">Maven2 Performance Plugin</a></li><li><a shape="rect" href="benchmark-tests.html">Benchmark Tests</a></li><li><a shape="rect" href="jmeter-system-tests.html">JMeter System Tests</a></li><li><a shape="rect" href="jmeter-performance-tests.html">JMeter Performance Tests</a></li><li><a shape="rect" href="integration-tests.html">Integration Tests</a></li></ul><h3 id="Navigation-ProjectReports">Project Reports</h3><ul class="alternate"><li><a shape="rect" href="junit-reports.html">JUnit Reports</a></li><li><a shape="rect" href="source-xref.html">Source XRef</a></li><li><a shape="rect" href="test-source-xref.html">Test Source XRef</a></li><li><a shape="rect" href="xml-reference.html">Xml Reference</a></li></ul>
               </div>
             </div>
           </div>
         </td>
         </tr>
   </tbody>
         </table>
         <div class="bottom_red_bar"></div>
       </div>
     </div>
   </div>
 </div>
 <div class="black_box">
 <div class="footer">
   <div class="footer_l">
     <div class="footer_r">
       <div>
         <a href="http://activemq.apache.org/privacy-policy.html">Privacy Policy</a> -
         (<a href="https://cwiki.apache.org/confluence/pages/editpage.action?pageId=27824400">edit this page</a>)
       </div>
     </div>
   </div>
 </div>
 </div>
 </div>
 <div class="design_attribution">
 &copy; 2004-2011 The Apache Software Foundation.
 <br/>
 Apache ActiveMQ, ActiveMQ, Apache, the Apache feather logo, and the Apache ActiveMQ project logo are trademarks of The Apache Software Foundation.  All other marks mentioned may be trademarks or registered trademarks of their respective owners.
 <br/>
 <a href="http://hiramchirino.com">Graphic Design By Hiram</a>
 </div>

 <!-- delay the loading of large javascript files to the end so that they don't interfere with the loading of page content -->
 <span style="display: none">
   <script type="text/javascript">
     var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
     document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
   </script>
   <script type="text/javascript">
     var pageTracker = _gat._getTracker("UA-1347593-1");
     pageTracker._initData();
     pageTracker._trackPageview();
   </script>
 </span>
 </body>
 </html>