doc/book/src/producer-flow-control.xml

<?xml version="1.0" encoding="utf-8"?>
<!--
 
 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at
 
   http://www.apache.org/licenses/LICENSE-2.0
 
 Unless required by applicable law or agreed to in writing,
 software distributed under the License is distributed on an
 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 KIND, either express or implied.  See the License for the
 specific language governing permissions and limitations
 under the License.
 
-->

<section id="producer-flow-control">
  <title>
    Producer Flow Control
  </title>

  <section role="h2" id="producerflowcontrol-Overview">
    <title>
      Overview
    </title>
      <para>
	As of release 0.10, the C++ broker supports the use of flow control to
	throttle back message producers that are at risk of overflowing a
	destination queue.
      </para>

      <para>
        Each queue in the C++ broker has two threshold values associated with it:
      </para>

      <para>
        Flow Stop Threshold: this is the level of queue resource
        utilization above which flow control will be enabled.  Once this
        threshold is crossed, the queue is considered in danger of overflow.
      </para>

      <para>
        Flow Resume Threshold - this is the level of queue resource utilization
        below which flow control will be disabled.  Once this threshold is
        crossed, the queue is no longer considered in danger of overflow.
      </para>

      <para>
        In the above description, queue resource utilization may be
        defined as the total count of messages currently enqueued, or the total
        sum of all message content in bytes.
      </para>

      <para>
        The value for a queue's Flow Stop Threshold must be greater than or
        equal to the value of the queue's Flow Resume Threshold.
      </para>

    <section role="h3" id="producerflowcontrol-QueueThresholdsExample">
      <title>
	Example
      </title>

      <para>
        Let's consider a queue with a maximum limit set on the total number of
        messages that may be enqueued to that queue.  Assume this maximum
        message limit is 1000 messages.  Assume also that the user configures a
        Flow Stop Threshold of 900 messages, and a Flow Resume Threshold of 500
        messages.  Then the following holds:
      </para>

      <para>
        The queue's initial flow control state is "OFF".
      </para>

      <para>
        While the total number of enqueued messages is less than or equal to
        900, the queue's flow control state remains "OFF".
      </para>

      <para>
        When the total number of enqueued messages is greater than 900, the
        queue's flow control state transitions to "ON".
      </para>

      <para>
        When the queue's flow control state is "ON", it remains "ON" until the
        total number of enqueued messages is less than 500.  At that point, the queue's
        flow control state transitions to "OFF".
      </para>

      <para>
        A similar example using total enqueued content bytes as the threshold
        units are permitted.
      </para>
    </section>

    <para>
      Thresholds may be set using both total message counts and total byte
      counts.  In this case, the following rules apply:
    </para>

    <para>
      1) Flow control is "ON" when either stop threshold value is crossed.
    </para>
    <para>
      2) Flow control remains "ON" until both resume thresholds are satisfied.
    </para>

    <section role="h3" id="producerflowcontro-MultiThresholdExample">
      <title>
	Example
      </title>

    <para>
      Let's consider a queue with a maximum size limit of 10K bytes, and 5000
      messages.  A user may assign a Flow Stop Threshold based on a total
      message count of 4000 messages.  They may also assigne a Flow Stop
      Threshold of 8K bytes.  The queue's flow control state transitions to
      "ON" if either threshold is crossed: (total-msgs greater-than 4000 OR total-bytes
      greater-than 8K).
    </para>

    <para>
      Assume the user has assigned Flow Resume threshold's of 3000 messages and
      6K bytes.  Then the queue's flow control will remain active until both
      thresholds are satified: (total-msg less-than 3000 AND total-bytes less-than 6K).
    </para>
    </section>

    <para>
      The Broker enforces flow control by delaying the completion of the
      Message.Transfer command that causes a message to be delivered to a queue
      with active flow control.  The completion of the Message.Transfer command
      is held off until flow control state transitions to "OFF" for all queues
      that are a destination for that command.
    </para>

    <para>
      A message producing client is permitted to have a finite number of
      commands pending completion.  When the total number of these outstanding
      commands reaches the limit, the client must not issue further commands
      until one or more of the outstanding commands have completed.  This
      window of outstanding commands is considered the sender's "capacity".
      This allows any given producer to have a "capacity's" worth of messages
      blocked due to flow control before the sender must stop sending further
      messages.
      </para>

      <para>
        This capacity window must be considered when determining a suitable
        flow stop threshold for a given queue, as a producer may send its
        capacity worth of messages _after_ a queue has reached the flow stop
        threshold.  Therefore, a flow stop threshould should be set such that
        the queue can accomodate more messages without overflowing.
      </para>

      <para>
        For example, assume two clients, C1 and C2, are producing messages to
        one particular destination queue.  Assume client C1 has a configured
        capacity of 50 messages, and client C2's capacity is 15 messages.  In
        this example, assume C1 and C2 are the only clients queuing messages to
        a given queue.  If this queue has a Flow Stop Threshold of 100
        messages, then, worst-case, the queue may receive up to 165 messages
        before clients C1 and C2 are blocked from sending further messages.
        This is due to the fact that the queue will enable flow control on
        receipt of its 101'st message - preventing the completion of the
        Message.Transfer command that carried the 101'st message.  However, C1
        and C2 are allowed to have a total of 65 (50 for C1 and 15 for C2)
        messages pending completion of Message.Transfer before they will stop
        producing messages.  Thus, up to 65 messages may be enqueued beyond the
        flow stop threshold before the producers will be blocked.
      </para>
    </section>

    <section role="h2" id="producerflowcontrol-UserInterface">
      <title>
        User Interface
      </title>

      <para>
        By default, the C++ broker assigns a queue's flow stop and flow resume
        thresholds when the queue is created.  The C++ broker also allows the
        user to manually specify the flow control thresholds on a per queue
        basis.
      </para>

      <para>
        However, queues that have been configured with a Limit Policy of type
        RING or RING-STRICT do NOT have queue flow thresholds enabled by
        default.  The nature of a RING queue defines its behavior when its
        capacity is reach: replace the oldest message.
      </para>

      <para>
        The flow control state of a queue can be determined by the "flowState"
        boolean in the queue's QMF management object.  The queue's management
        object also contains a counter that increments each time flow control
        becomes active for the queue.
      </para>

      <para>
        The broker applies a threshold ratio to compute a queue's default flow
        control configuration.  These thresholds are expressed as a percentage
        of a queue's maximum capacity.  There is one value for determining the
        stop threshold, and another for determining the resume threshold.  The
        user may configure these percentages using the following broker
        configuration options:
      </para>

      <programlisting>
        --default-flow-stop-threshold ("Queue capacity level at which flow control is activated.")
        --default-flow-resume-threshold ("Queue capacity level at which flow control is de-activated.")
      </programlisting>

      <para>
        For example:
      </para>

      <programlisting>
        qpidd --default-flow-stop-threshold=90 --default-flow-resume-threshold=75
      </programlisting>

      <para>
        Sets the default flow stop threshold to 90% of a queue's maximum
        capacity and the flow resume threshold to 75% of the maximum capacity.
        If a queue is created with a default-queue-limit of 10000 bytes, then
        the default flow stop threshold would be 90% of 10000 = 9000 bytes and
        the flow resume threshold would be 75% of 10000 = 7500.  The same
        computation is performed should a queue be created with a maximum size
        expressed as a message count instead of a byte count.
      </para>

      <para>
        If not overridden by the user, the value of the
        default-flow-stop-threshold is 80% and the value of the
        default-flow-resume-threshold is 70%.
      </para>

      <para>
        The user may disable default queue flow control broker-wide by
        specifying the value 0 for both of these configuration options.  Note
        that flow control may still be applied manually on a per-queue basis in
        this case.
      </para>

      <para>
        The user may manually set the flow thresholds when creating a queue.
        The following options may be provided when adding a queue using the
        <command>qpid-config</command> command line tool:
      </para>

      <programlisting>
        --flow-stop-size=<replaceable>N</replaceable>  Sets the queue's flow stop threshold to <replaceable>N</replaceable> total bytes.
        --flow-resume-size=<replaceable>N</replaceable>  Sets the queue's flow resume threshold to <replaceable>N</replaceable> total bytes.
        --flow-stop-count=<replaceable>N</replaceable> Sets the queue's flow stop threshold to <replaceable>N</replaceable> total messages.
        --flow-resume-count=<replaceable>N</replaceable> Sets the queue's flow resume threshold to <replaceable>N</replaceable> total messages.
      </programlisting>

      <para>
        Flow thresholds may also be specified in the
        <command>queue.declare</command> method, via the
        <command>arguments</command> parameter map.  The following keys can be
        provided in the arguments map for setting flow thresholds:
      </para>

      <table>
        <title>Queue Declare Method Flow Control Arguments</title>
        <tgroup cols="2">
          <thead>
            <row>
              <entry>Key</entry>
              <entry>Value</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>qpid.flow_stop_size</entry>
              <entry>integer - queue's flow stop threshold value in bytes</entry>
            </row>
            <row>
              <entry>qpid.flow_resume_size</entry>
              <entry>integer - queue's flow resume threshold value in bytes</entry>
            </row>
            <row>
              <entry>qpid.flow_stop_count</entry>
              <entry>integer - queue's flow stop threshold value as a message count</entry>
            </row>
            <row>
              <entry>qpid.flow_resume_count</entry>
              <entry>integer - queue's flow resume threshold value as a message count</entry>
            </row>
          </tbody>
        </tgroup>
      </table>

      <para>
        The user may disable flow control on a per queue basis by setting
        the flow-stop-size and flow-stop-count to zero for the queue.
      </para>

      <para>
        The current state of flow control for a given queue can be
        determined by the "flowStopped" statistic.  This statistic is
        available in the queue's QMF management object. The value of
        flowStopped is True when the queue's capacity has exceeded the
        flow stop threshold.  The value of flowStopped is False when the
        queue is no longer blocking due to flow control.
      </para>

      <para>
        A queue will also track the number of times flow control has been
        activated.  The "flowStoppedCount" statistic is incremented each time
        the queue's capacity exceeds a flow stop threshold.  This statistic can
        be used to monitor the activity of flow control for any given queue
        over time.
      </para>

      <table>
        <title>Flow Control Statistics available in Queue's QMF Class</title>
        <tgroup cols="3">
          <thead>
            <row>
              <entry>Statistic Name</entry>
              <entry>Type</entry>
              <entry>Description</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>flowStopped</entry>
              <entry>Boolean</entry>
              <entry>If true, producers are blocked by flow control.</entry>
            </row>
            <row>
              <entry>flowStoppedCount</entry>
              <entry>count32</entry>
              <entry>Number of times flow control was activated for this queue</entry>
            </row>
          </tbody>
        </tgroup>
      </table>
    </section>
    <!--h2-->
  </section>