doc/java-broker/src/docbkx/runtime/Java-Broker-Runtime-Flow-To-Disk.xml

<?xml version="1.0"?>
<!--

 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing,
 software distributed under the License is distributed on an
 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 KIND, either express or implied.  See the License for the
 specific language governing permissions and limitations
 under the License.

-->

<section xmlns="http://docbook.org/ns/docbook" version="5.0" xml:id="Java-Broker-Runtime-Flow-To-Disk">
  <title>Flow to Disk</title>
  <para>Flow to disk limits the amount of <link linkend="Java-Broker-Runtime-Memory">direct and heap memory</link>
    that can be occupied by messages. Once this
    limit is reached any new transient messages and all existing transient messages will be
    transferred to disk. Newly arriving transient messages will continue to go to the disk until the
    cumulative size of all messages falls below the limit once again.</para>
  <para>By default the Broker makes 75% of the max direct available memory for messages. This memory is
    divided between all the queues across all virtual hosts defined on the Broker with a percentage
    calculated according to their current queue size. These calculations are refreshed periodically
    by the housekeeping cycle.</para>
  <para>For example if there are two queues, one containing 75MB and the second 100MB messages
    respectively and the Broker has 1GB direct memory with the default of 75% available for messages.
    The first queue will have a target size of 320MB and the second 430MB. Once 750MB is taken by
    messages, messages will begin to flow to disk. New messages will cease to flow to disk when
    their cumulative size falls beneath 750MB.</para>
  <para>Flow to disk is configured by Broker context variable
      <literal>broker.flowToDiskThreshold</literal>. It is expressed as a size in bytes and defaults
    to 75% of the JVM maximum heap size.</para>
  <section role="h2" xml:id="Java-Broker-Runtime-Flow-To-Disk-Monitoring">
    <title>Flow to Disk Monitoring</title>
    <para>A number of statistics attributes are available on the <literal>Broker</literal> to allow monitoring
      of the amount of utilized direct memory by the enqueued messages.
    </para>
    <para>The total amount of allocated direct memory by the <literal>Broker</literal> can be determined by
      checking <literal>Broker</literal> statistics <literal>usedDirectMemorySize</literal>. There is another
      <literal>Broker</literal> level statistics <literal>directMemoryTotalCapacity</literal> to get the total amount
      of allocated direct memory. Usually, the values reported by both statistics attributes
      <literal>usedDirectMemorySize</literal> and <literal>directMemoryTotalCapacity</literal> are the same
      or do not differ much.
    </para>
    <para>The direct memory consumed by the <literal>VirtualHost</literal> messages is reported as
      <literal>VirtualHost</literal> statistics <literal>inMemoryMessageSize</literal>. The current value of
      <literal>VirtualHost</literal> direct memory threshold is exposed with statistics attribute
      <literal>inMemoryMessageThreshold</literal>. When the value of <literal>inMemoryMessageSize</literal> is
      greater than <literal>inMemoryMessageThreshold</literal>, the flow to disk is triggered to bring the amount of
      direct memory consumed by the <literal>VirtualHost</literal> messages in-line with the
      <literal>inMemoryMessageThreshold</literal>.
    </para>
  </section>
  <section role="h2" xml:id="Java-Broker-Runtime-Flow-To-Disk-Logging">
    <title>Flow to Disk Logging</title>
    <para>The <literal>Flow to Disk</literal> events are not reported as operational logs or
      <literal>INFO</literal> logs due to quite frequent triggering of <literal>Flow to Disk</literal>
      for messaging use cases requiring holding messages on the <literal>Broker</literal> side for some time.
      As result, the <literal>Flow to Disk</literal> logs can quickly dominate the broker logs and cause unnecessary
      disk consumption.
    </para>
    <para>Though, if required, the <literal>Flow to Disk</literal> DEBUG logs can be enabled by adding
      the following logging rule into the corresponding <literal>Broker</literal> logger.
      <example>
        <title>Flow to Disk logging rule</title>
        <programlisting>
          {
            "name" : "DirectMemory",
            "type" : "NameAndLevel",
            "level" : "DEBUG",
            "loggerName" : "org.apache.qpid.server.directMemory.*"
          }
        </programlisting>
      </example>
    </para>
    <para>Please note, that the logger <literal>org.apache.qpid.server.directMemory.broker</literal>
      is used by the <literal>Broker</literal> to report conditions when direct memory utilization exceeds the pred-defined
      <literal>Broker</literal> threshold, whilst the logger <literal>org.apache.qpid.server.directMemory.virtualhost</literal>
      is used to report conditions when direct memory utilization by the <literal>VirtualHost</literal>
      messages exceeds the current value of the <literal>VirtualHost</literal> threshold.
    </para>
  </section>
</section>