<div class="docbook"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">4.7. Queues</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="Java-Broker-Concepts-Exchanges.html">Prev</a> </td><th width="60%" align="center">Chapter 4. Concepts</th><td width="20%" align="right"> <a accesskey="n" href="Java-Broker-Concepts-Ports.html">Next</a></td></tr></table><hr /></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="Java-Broker-Concepts-Queues"></a>4.7. Queues</h2></div></div></div><p><span class="emphasis"><em>Queue</em></span>s are named entities within a <a class="link" href="Java-Broker-Concepts-Virtualhosts.html" title="4.5. Virtualhosts">Virtualhost</a> that
     hold/buffer messages for later delivery to consumer applications.</p><p>Messages arrive on queues either from <a class="link" href="Java-Broker-Concepts-Exchanges.html" title="4.6. Exchanges">Exchanges</a>, or when
     using the AMQP 1.0 protocol, the producing application can direct messages straight to the queue.  For
     AMQP 0-8, 0-9, 0-9-1,  or 0-10, the exchange is the only way ingressing a message into a queue.</p><p>Consumers subscribe to a queue in order to receive messages from it.</p><p>The Broker supports different queue types, each with different delivery semantics.  Queues also have a range of other
   features such as the ability to group messages together for delivery to a single consumer.  These additional features
   are described below too.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queues-Types"></a>4.7.1. Types</h3></div></div></div><p>The Broker supports four different queue types, each with different delivery semantics.</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p><a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-Standard" title="4.7.1.1. Standard">Standard</a> - a simple First-In-First-Out (FIFO) queue</p></li><li class="listitem"><p><a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-Priority" title="4.7.1.2. Priority">Priority</a> - delivery order depends on the priority of each message</p></li><li class="listitem"><p><a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-Sorted" title="4.7.1.3. Sorted Queues">Sorted</a> -
            delivery order depends on the value of the sorting key property in each message</p></li><li class="listitem"><p><a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-LVQ" title="4.7.1.4. Last Value Queues (LVQ)">Last Value
              Queue</a> - also known as an LVQ, retains only the last (newest) message received
            with a given LVQ key value</p></li></ul></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-Types-Standard"></a>4.7.1.1. Standard</h4></div></div></div><p>A simple First-In-First-Out (FIFO) queue</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-Types-Priority"></a>4.7.1.2. Priority</h4></div></div></div><p>In a priority queue, messages on the queue are delivered in an order determined by the
          <a class="link" href="http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSPriority()" target="_top">JMS priority message
          header</a> within the message. By default Qpid supports the 10 priority levels
        mandated by JMS, with priority value 0 as the lowest priority and 9 as the highest. </p><p>It is possible to reduce the effective number of priorities if desired.</p><p>JMS defines the <a class="link" href="http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#DEFAULT_PRIORITY" target="_top">
          default message priority</a> as 4. Messages sent without a specified priority use this
        default. </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-Types-Sorted"></a>4.7.1.3. Sorted Queues</h4></div></div></div><p>Sorted queues allow the message delivery order to be determined by value of an arbitrary
          <a class="link" href="http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getStringProperty()" target="_top">JMS message
          property</a>. Sort order is alpha-numeric and the property value must have a type
        java.lang.String.</p><p>Messages sent to a sorted queue without the specified JMS message property will be
        put at the head of the queue.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-Types-LVQ"></a>4.7.1.4. Last Value Queues (LVQ)</h4></div></div></div><p>LVQs (or conflation queues) are special queues that automatically discard any message
        when a newer message arrives with the same key value. The key is specified by arbitrary
          <a class="link" href="http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getPropertyNames()" target="_top">JMS message
          property</a>.</p><p>An example of an LVQ might be where a queue represents prices on a stock exchange: when
        you first consume from the queue you get the latest quote for each stock, and then as new
        prices come in you are sent only these updates. </p><p>Like other queues, LVQs can either be browsed or consumed from. When browsing an
        individual subscriber does not remove the message from the queue when receiving it. This
        allows for many subscriptions to browse the same LVQ (i.e. you do not need to create and
        bind a separate LVQ for each subscriber who wishes to receive the contents of the
        LVQ).</p><p>Messages sent to an LVQ without the specified property will be delivered as normal and
        will never be "replaced".</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queues-Message-Grouping"></a>4.7.2. Messaging Grouping</h3></div></div></div><p> The broker allows messaging applications to classify a set of related messages as
      belonging to a group. This allows a message producer to indicate to the consumer that a group
      of messages should be considered a single logical operation with respect to the application. </p><p> The broker can use this group identification to enforce policies controlling how messages
      from a given group can be distributed to consumers. For instance, the broker can be configured
      to guarantee all the messages from a particular group are processed in order across multiple
      consumers. </p><p> For example, assume we have a shopping application that manages items in a virtual
      shopping cart. A user may add an item to their shopping cart, then change their mind and
      remove it. If the application sends an <span class="emphasis"><em>add</em></span> message to the broker,
      immediately followed by a <span class="emphasis"><em>remove</em></span> message, they will be queued in the
      proper order - <span class="emphasis"><em>add</em></span>, followed by <span class="emphasis"><em>remove</em></span>. </p><p> However, if there are multiple consumers, it is possible that once a consumer acquires
      the <span class="emphasis"><em>add</em></span> message, a different consumer may acquire the
        <span class="emphasis"><em>remove</em></span> message. This allows both messages to be processed in parallel,
      which could result in a "race" where the <span class="emphasis"><em>remove</em></span> operation is incorrectly
      performed before the <span class="emphasis"><em>add</em></span> operation. </p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-GroupingMessages"></a>4.7.2.1. Grouping Messages</h4></div></div></div><p>In order to group messages, JMS applications can set the JMS standard header
            <code class="literal">JMSXGroupId</code> to specify the <span class="emphasis"><em>group identifier</em></span>
            when publishing messages.
        </p><p>Alternatively, the application may designate a particular message header
            as containing a message's <span class="emphasis"><em>group identifier</em></span>. The group identifier stored
            in that header field would be a string value set by the message producer. Messages from the
            same group would have the same group identifier value. The key that identifies the header
            must also be known to the message consumers. This allows the consumers to determine a
            message's assigned group.
        </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-BrokerRole"></a>4.7.2.2.  The Role of the Broker in Message Grouping </h4></div></div></div><p> The broker will apply the following processing on each grouped message: </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>Enqueue a received message on the destination queue.</p></li><li class="listitem"><p>Determine the message's group by examining the message's group identifier
              header.</p></li><li class="listitem"><p>Enforce <span class="emphasis"><em>consumption ordering</em></span> among messages belonging to the
              same group. <span class="emphasis"><em>Consumption ordering</em></span> means one of two things
              depending on how the queue has been configured. </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p> In default mode, a group gets assigned to a single consumer for the lifetime
                  of that consumer, and the broker will pass all subsequent messages in the group to
                  that consumer. </p></li><li class="listitem"><p>In 'shared groups' mode (which gives the same behaviour as the Qpid C++
                  Broker) the broker enforces a looser guarantee, namely that all the
                    <span class="emphasis"><em>currently unacknowledged messages</em></span> in a group are sent to
                  the same consumer, but the consumer used may change over time even if the
                  consumers do not. This means that only one consumer can be processing messages
                  from a particular group at any given time, however if the consumer acknowledges
                  all of its acquired messages then the broker <span class="emphasis"><em>may</em></span> pass the
                  next pending message in that group to a different consumer. </p></li></ul></div></li></ul></div><p>
      </p><p> The absence of a value in the designated group header field of a message is treated as
        follows: </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> In default mode, failure for a message to specify a group is treated as a desire
              for the message not to be grouped at all. Such messages will be distributed to any
              available consumer, without the ordering guarantees imposed by grouping. </p></li><li class="listitem"><p> In 'shared groups' mode (which gives the same behaviour as the Qpid C++ Broker)
              the broker assigns messages without a group value to a 'default group'. Therefore, all
              such "unidentified" messages are considered by the broker as part of the same group,
              which will handled like any other group. The name of this default group is
              "qpid.no-group", although it can be customised as detailed below. </p></li></ul></div><p>
      </p><p> Note that message grouping has no effect on queue browsers.</p><p> Note well that distinct message groups would not block each other from delivery. For
        example, assume a queue contains messages from two different message groups - say group "A"
        and group "B" - and they are enqueued such that "A"'s messages are in front of "B". If the
        first message of group "A" is in the process of being consumed by a client, then the
        remaining "A" messages are blocked, but the messages of the "B" group are available for
        consumption by other consumers - even though it is "behind" group "A" in the queue. </p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queue-EnsureNonDestructiveConsumers"></a>4.7.3. Forcing all consumers to be non-destructive</h3></div></div></div><p>When a consumer attaches to a queue, the normal behaviour is that messages are
          sent to that consumer are acquired exclusively by that consumer, and when the consumer
          acknowledges them, the messages are removed from the queue.</p><p>Another common pattern is to have queue "browsers" which send all messages to the
          browser, but do not prevent other consumers from receiving the messages, and do not
          remove them from the queue when the browser is done with them.  Such a browser is an
          instance of a "non-destructive" consumer.</p><p>If every consumer on a queue is non destructive then we can obtain some interesting
          behaviours. In the case of a LVQ
           then the queue will always contain the most up to date value for every key. For
          a standard queue, if every consumer is non-destructive then we have something that
          behaves like a topic (every consumer receives every message) except that instead of
          only seeing messages that arrive after the point at which the consumer is created, all
          messages which have not been removed due to TTL expiry (or, in the case of LVQs,
          overwirtten by newer values for the same key).</p><p>A queue can be created to enforce all consumers are non-destructive.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="d0e1068"></a>4.7.3.1. Bounding size using min/max TTL</h4></div></div></div><p>For queues other than LVQs, having only non-destructive consumers could mean that
            messages would never get deleted, leaving the queue to grow unconstrainedly. To
            prevent this you can use the ability to set the maximum TTL of the queue. To ensure
            all messages have the same TTL you could also set the minimum TTL to the same value.
        </p><p>Minimum/Maximum TTL for a queue can be set though the HTTP Management UI, using the
            REST API. The attribute names are <code class="varname">minimumMessageTtl</code> and
            <code class="varname">maximumMessageTtl</code> and the TTL value is given in milliseconds.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="d0e1081"></a>4.7.3.2. Choosing to receive messages based on arrival time</h4></div></div></div><p>A queue with no destructive consumers will retain all messages until they expire
            due to TTL. It may be the case that a consumer only wishes to receive messages
            that have been sent in the last 60 minutes, and any new messages that arrive, or
            alternatively it may wish only to receive newly arriving messages and not any that
            are already in the queue. This can be achieved by using a filter on the arrival
            time.</p><p>A special parameter <code class="varname">x-qpid-replay-period</code> can be used in the
            consumer declaration to control the messages the consumer wishes to receive. The
            value of <code class="varname">x-qpid-replay-period</code> is the time, in seconds, for which
            the consumer wishes to see messages. A replay period of 0 indicates only newly
            arriving messages should be sent. A replay period of 3600 indicates that only
            messages sent in the last hour - along with any newly arriving messages - should be
            sent.</p><p>When using the Qpid JMS AMQP 0-x, the consumer declaration can be hinted using the
            address.</p><div class="table"><a id="d0e1096"></a><p class="title"><strong>Table 4.1. Setting the replay period using a Qpid JMS AMQP 0-x address</strong></p><div class="table-contents"><table summary="Setting the replay period using a Qpid JMS AMQP 0-x address" border="1"><colgroup><col class="syntax" /><col class="example" /></colgroup><thead><tr><th>Syntax</th><th>Example</th></tr></thead><tbody><tr><td>Addressing</td><td>myqueue : { link : { x-subscribe: { arguments : { x-qpid-replay-period : '3600' } } } }</td></tr><tr><td>Binding URL</td><td>direct://amq.direct/myqueue/myqueue?x-qpid-replay-period='3600'</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="d0e1119"></a>4.7.3.3. Setting a default filter</h4></div></div></div><p>A common case might be that the desired default behaviour is that newly attached consumers
            see only newly arriving messages (i.e. standard topic-like behaviour) but other consumers
            may wish to start their message stream from some point in the past. This can be achieved by
            setting a default filter on the queue so that consumers which do not explicitly set a replay
            period get a default (in this case the desired default would be 0).</p><p>The default filter set for a queue can be set via the REST API using the attribute named
            <code class="varname">defaultFilters</code>. This value is a map from filter name to type and arguments.
            To set the default behaviour for the queue to be that consumers only receive newly arrived
            messages, then you should set this attribute to the value:</p><pre class="screen">
            { "x-qpid-replay-period" : { "x-qpid-replay-period" : [ "0" ] } }
        </pre><p>
            If the desired default behaviour is that each consumer should see all messages arriving in
            the last minute, as well as all new messages then the value would need to be:</p><pre class="screen">
            { "x-qpid-replay-period" : { "x-qpid-replay-period" : [ "60" ] } }
        </pre></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queue-HoldingEntries"></a>4.7.4. Holding messages on a Queue</h3></div></div></div><p>Sometimes it is required that while a message has been placed on a queue, it is not released to consumers
            until some external condition is met. </p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="d0e1140"></a>4.7.4.1. Hold until valid</h4></div></div></div><p>Currently Queues support the "holding" of messages until a (per-message) provided point in time.
                By default this support is not enabled (since it requires extra work to be performed against every
                message entering the queue.  To enable support, the attribute <code class="varname">holdOnPublishEnabled</code>
                must evaluate to true for the Queue.  When enabled messages on the queue will be checked for the header
                (for AMQP 0-8, 0-9, 0-9-1 and 0-10 messages) or message annotation (for AMQP 1.0 messages)
                <code class="varname">x-qpid-not-valid-before</code>. If this header/annotation exists and contains a numeric value,
                it will be treated as a point in time given in milliseconds since the UNIX epoch.  The message will not
                be released from the Queue to consumers until this time has been reached.
            </p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queue-OverflowPolicy"></a>4.7.5. Controlling Queue Size</h3></div></div></div><p>
            <span class="emphasis"><em>Overflow Policy</em></span>
            can be configured on an individual <span class="emphasis"><em>Queue</em></span> to limit the queue size.  The size
            can be expressed in terms of a <span class="emphasis"><em>maximum number of bytes</em></span> and/or
            <span class="emphasis"><em>maximum number of messages</em></span>.
            The <span class="emphasis"><em>Overflow Policy</em></span> defines the Queue behaviour when any of the limits is reached.
        </p><p>
            The following <span class="emphasis"><em>Overflow Policies</em></span> are supported:
            </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
                        <span class="emphasis"><em>None</em></span>
                        - Queue is unbounded and the capacity limits are not applied.
                        This is a default policy applied implicitly when policy is not set explicitly.
                    </p></li><li class="listitem"><p>
                        <span class="emphasis"><em>Ring</em></span>
                        - If a newly arriving message takes the queue over a limit, message(s) are
                        deleted from the queue until the queue falls within its limit again.  When deleting
                        messages, the oldest messages are deleted first.  For a
                        <a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-Priority" title="4.7.1.2. Priority">Priority Queue</a>
                        the oldest messages with lowest priorities are removed.
                    </p></li><li class="listitem"><p>
                        <span class="emphasis"><em>Producer Flow Control</em></span>
                        -The producing sessions
                        are blocked until queue depth falls below the <span class="emphasis"><em>resume threshold</em></span> set as a
                        context variable <span class="emphasis"><em>${queue.queueFlowResumeLimit}</em></span>
                        (specifying the percentage from the limit values. Default is 80%).
                    </p></li><li class="listitem"><p>
                        <span class="emphasis"><em>Flow to Disk</em></span>
                        -If the queue breaches a limit, newly arriving messages are written to disk and the in-memory
                        representation of the message is minimised. The Broker will transparently retrieve messages
                        from disk as they are required by a consumer or management.  The flow to disk policy does not
                        actually restrict the overall size of the queue, merely the space occupied in memory.
                        The Broker's other <a class="link" href="Java-Broker-Runtime-Flow-To-Disk.html" title="9.6. Flow to Disk">Flow to Disk feature</a>
                        operates completely independent of this Queue Overflow Policy.
                    </p></li><li class="listitem"><p>
                        <span class="emphasis"><em>Reject</em></span>
                        -A newly arriving message is rejected when queue limit is breached.
                    </p></li></ul></div><p>
        </p><p>
            A negative value for <span class="emphasis"><em>maximum number of messages</em></span> or
            <span class="emphasis"><em>maximum number of bytes</em></span> disables the limit.
        </p><p>
            The Broker issues Operational log messages when the queue sizes are breached.  These are documented
            at <a class="xref" href="Java-Broker-Appendix-Operation-Logging.html#Java-Broker-Appendix-Operation-Logging-Message-List-Queue" title="Table C.6. Queue Log Messages">Table C.6, “Queue Log Messages”</a>.
        </p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queues-SetLowPrefetch"></a>4.7.6. Using low pre-fetch with special queue types</h3></div></div></div><p>Messaging clients may buffered messages for performance reasons.  In Qpid, this is commonly known as
            <span class="emphasis"><em>pre-fetch</em></span></p><p>When using some of the messaging features described on this section, using prefetch can give
            unexpected behaviour. Once the broker has sent a message to the client its delivery order is then fixed,
            regardless of the special behaviour of the queue. </p><p>For example, if using a priority queue and a prefetch of 100, and 100 messages arrive with
            priority 2, the broker will send these messages to the client. If then a new message arrives
            with priority 1, the broker cannot leap frog messages of lower priority. The priority 1 will
            be delivered at the front of the next batch of messages to be sent to the client.</p><p>Using pre-fetch of 1 will give exact queue-type semantics as perceived by the
            client however, this brings a performance cost. You could test with a slightly higher
            pre-fetch to trade-off between throughput and exact semantics.</p><p>See the messaging client documentation for details of how to configure prefetch.</p></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="Java-Broker-Concepts-Exchanges.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="Java-Broker-Concepts.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="Java-Broker-Concepts-Ports.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">4.6. Exchanges </td><td width="20%" align="center"><a accesskey="h" href="Apache-Qpid-Broker-J-Book.html">Home</a></td><td width="40%" align="right" valign="top"> 4.8. Ports</td></tr></table></div></div>