| <div class="docbook"><div class="navheader"><table summary="Navigation header" width="100%"><tr><th align="center" colspan="3">4.7. Queues</th></tr><tr><td align="left" width="20%"><a accesskey="p" href="Java-Broker-Concepts-Exchanges.html">Prev</a> </td><th align="center" width="60%">Chapter 4. Concepts</th><td align="right" width="20%"> <a accesskey="n" href="Java-Broker-Concepts-Ports.html">Next</a></td></tr></table><hr /></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="Java-Broker-Concepts-Queues"></a>4.7. Queues</h2></div></div></div><p><span class="emphasis"><em>Queue</em></span>s are named entities within a <a class="link" href="Java-Broker-Concepts-Virtualhosts.html" title="4.5. Virtualhosts">Virtualhost</a> that | |
| hold/buffer messages for later delivery to consumer applications.</p><p>Messages arrive on queues either from <a class="link" href="Java-Broker-Concepts-Exchanges.html" title="4.6. Exchanges">Exchanges</a>, or when | |
| using the AMQP 1.0 protocol, the producing application can direct messages straight to the queue. For | |
| AMQP 0-8, 0-9, 0-9-1, or 0-10, the exchange is the only way ingressing a message into a queue.</p><p>Consumers subscribe to a queue in order to receive messages from it.</p><p>The Broker supports different queue types, each with different delivery semantics. Queues also have a range of other | |
| features such as the ability to group messages together for delivery to a single consumer. These additional features | |
| are described below too.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queues-Types"></a>4.7.1. Types</h3></div></div></div><p>The Broker supports four different queue types, each with different delivery semantics.</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p><a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-Standard" title="4.7.1.1. Standard">Standard</a> - a simple First-In-First-Out (FIFO) queue</p></li><li class="listitem"><p><a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-Priority" title="4.7.1.2. Priority">Priority</a> - delivery order depends on the priority of each message</p></li><li class="listitem"><p><a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-Sorted" title="4.7.1.3. Sorted Queues">Sorted</a> - | |
| delivery order depends on the value of the sorting key property in each message</p></li><li class="listitem"><p><a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-LVQ" title="4.7.1.4. Last Value Queues (LVQ)">Last Value | |
| Queue</a> - also known as an LVQ, retains only the last (newest) message received | |
| with a given LVQ key value</p></li></ul></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-Types-Standard"></a>4.7.1.1. Standard</h4></div></div></div><p>A simple First-In-First-Out (FIFO) queue</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-Types-Priority"></a>4.7.1.2. Priority</h4></div></div></div><p>In a priority queue, messages on the queue are delivered in an order determined by the | |
| <a class="link" href="http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getJMSPriority()" target="_top">JMS priority message | |
| header</a> within the message. By default Qpid supports the 10 priority levels | |
| mandated by JMS, with priority value 0 as the lowest priority and 9 as the highest. </p><p>It is possible to reduce the effective number of priorities if desired.</p><p>JMS defines the <a class="link" href="http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#DEFAULT_PRIORITY" target="_top"> | |
| default message priority</a> as 4. Messages sent without a specified priority use this | |
| default. </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-Types-Sorted"></a>4.7.1.3. Sorted Queues</h4></div></div></div><p>Sorted queues allow the message delivery order to be determined by value of an arbitrary | |
| <a class="link" href="http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getStringProperty()" target="_top">JMS message | |
| property</a>. Sort order is alpha-numeric and the property value must have a type | |
| java.lang.String.</p><p>Messages sent to a sorted queue without the specified JMS message property will be | |
| put at the head of the queue.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-Types-LVQ"></a>4.7.1.4. Last Value Queues (LVQ)</h4></div></div></div><p>LVQs (or conflation queues) are special queues that automatically discard any message | |
| when a newer message arrives with the same key value. The key is specified by arbitrary | |
| <a class="link" href="http://docs.oracle.com/javaee/6/api/javax/jms/Message.html#getPropertyNames()" target="_top">JMS message | |
| property</a>.</p><p>An example of an LVQ might be where a queue represents prices on a stock exchange: when | |
| you first consume from the queue you get the latest quote for each stock, and then as new | |
| prices come in you are sent only these updates. </p><p>Like other queues, LVQs can either be browsed or consumed from. When browsing an | |
| individual subscriber does not remove the message from the queue when receiving it. This | |
| allows for many subscriptions to browse the same LVQ (i.e. you do not need to create and | |
| bind a separate LVQ for each subscriber who wishes to receive the contents of the | |
| LVQ).</p><p>Messages sent to an LVQ without the specified property will be delivered as normal and | |
| will never be "replaced".</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queues-Message-Grouping"></a>4.7.2. Messaging Grouping</h3></div></div></div><p> The broker allows messaging applications to classify a set of related messages as | |
| belonging to a group. This allows a message producer to indicate to the consumer that a group | |
| of messages should be considered a single logical operation with respect to the application. </p><p> The broker can use this group identification to enforce policies controlling how messages | |
| from a given group can be distributed to consumers. For instance, the broker can be configured | |
| to guarantee all the messages from a particular group are processed in order across multiple | |
| consumers. </p><p> For example, assume we have a shopping application that manages items in a virtual | |
| shopping cart. A user may add an item to their shopping cart, then change their mind and | |
| remove it. If the application sends an <span class="emphasis"><em>add</em></span> message to the broker, | |
| immediately followed by a <span class="emphasis"><em>remove</em></span> message, they will be queued in the | |
| proper order - <span class="emphasis"><em>add</em></span>, followed by <span class="emphasis"><em>remove</em></span>. </p><p> However, if there are multiple consumers, it is possible that once a consumer acquires | |
| the <span class="emphasis"><em>add</em></span> message, a different consumer may acquire the | |
| <span class="emphasis"><em>remove</em></span> message. This allows both messages to be processed in parallel, | |
| which could result in a "race" where the <span class="emphasis"><em>remove</em></span> operation is incorrectly | |
| performed before the <span class="emphasis"><em>add</em></span> operation. </p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-GroupingMessages"></a>4.7.2.1. Grouping Messages</h4></div></div></div><p>In order to group messages, JMS applications can set the JMS standard header | |
| <code class="literal">JMSXGroupId</code> to specify the <span class="emphasis"><em>group identifier</em></span> | |
| when publishing messages. | |
| </p><p>Alternatively, the application may designate a particular message header | |
| as containing a message's <span class="emphasis"><em>group identifier</em></span>. The group identifier stored | |
| in that header field would be a string value set by the message producer. Messages from the | |
| same group would have the same group identifier value. The key that identifies the header | |
| must also be known to the message consumers. This allows the consumers to determine a | |
| message's assigned group. | |
| </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="Java-Broker-Concepts-Queues-BrokerRole"></a>4.7.2.2.  The Role of the Broker in Message Grouping </h4></div></div></div><p> The broker will apply the following processing on each grouped message: </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>Enqueue a received message on the destination queue.</p></li><li class="listitem"><p>Determine the message's group by examining the message's group identifier | |
| header.</p></li><li class="listitem"><p>Enforce <span class="emphasis"><em>consumption ordering</em></span> among messages belonging to the | |
| same group. <span class="emphasis"><em>Consumption ordering</em></span> means one of two things | |
| depending on how the queue has been configured. </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p> In default mode, a group gets assigned to a single consumer for the lifetime | |
| of that consumer, and the broker will pass all subsequent messages in the group to | |
| that consumer. </p></li><li class="listitem"><p>In 'shared groups' mode (which gives the same behaviour as the Qpid C++ | |
| Broker) the broker enforces a looser guarantee, namely that all the | |
| <span class="emphasis"><em>currently unacknowledged messages</em></span> in a group are sent to | |
| the same consumer, but the consumer used may change over time even if the | |
| consumers do not. This means that only one consumer can be processing messages | |
| from a particular group at any given time, however if the consumer acknowledges | |
| all of its acquired messages then the broker <span class="emphasis"><em>may</em></span> pass the | |
| next pending message in that group to a different consumer. </p></li></ul></div></li></ul></div><p> | |
| </p><p> The absence of a value in the designated group header field of a message is treated as | |
| follows: </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> In default mode, failure for a message to specify a group is treated as a desire | |
| for the message not to be grouped at all. Such messages will be distributed to any | |
| available consumer, without the ordering guarantees imposed by grouping. </p></li><li class="listitem"><p> In 'shared groups' mode (which gives the same behaviour as the Qpid C++ Broker) | |
| the broker assigns messages without a group value to a 'default group'. Therefore, all | |
| such "unidentified" messages are considered by the broker as part of the same group, | |
| which will handled like any other group. The name of this default group is | |
| "qpid.no-group", although it can be customised as detailed below. </p></li></ul></div><p> | |
| </p><p> Note that message grouping has no effect on queue browsers.</p><p> Note well that distinct message groups would not block each other from delivery. For | |
| example, assume a queue contains messages from two different message groups - say group "A" | |
| and group "B" - and they are enqueued such that "A"'s messages are in front of "B". If the | |
| first message of group "A" is in the process of being consumed by a client, then the | |
| remaining "A" messages are blocked, but the messages of the "B" group are available for | |
| consumption by other consumers - even though it is "behind" group "A" in the queue. </p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queue-EnsureNonDestructiveConsumers"></a>4.7.3. Forcing all consumers to be non-destructive</h3></div></div></div><p>When a consumer attaches to a queue, the normal behaviour is that messages are | |
| sent to that consumer are acquired exclusively by that consumer, and when the consumer | |
| acknowledges them, the messages are removed from the queue.</p><p>Another common pattern is to have queue "browsers" which send all messages to the | |
| browser, but do not prevent other consumers from receiving the messages, and do not | |
| remove them from the queue when the browser is done with them. Such a browser is an | |
| instance of a "non-destructive" consumer.</p><p>If every consumer on a queue is non destructive then we can obtain some interesting | |
| behaviours. In the case of a LVQ | |
| then the queue will always contain the most up to date value for every key. For | |
| a standard queue, if every consumer is non-destructive then we have something that | |
| behaves like a topic (every consumer receives every message) except that instead of | |
| only seeing messages that arrive after the point at which the consumer is created, all | |
| messages which have not been removed due to TTL expiry (or, in the case of LVQs, | |
| overwirtten by newer values for the same key).</p><p>A queue can be created to enforce all consumers are non-destructive.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="d0e1064"></a>4.7.3.1. Bounding size using min/max TTL</h4></div></div></div><p>For queues other than LVQs, having only non-destructive consumers could mean that | |
| messages would never get deleted, leaving the queue to grow unconstrainedly. To | |
| prevent this you can use the ability to set the maximum TTL of the queue. To ensure | |
| all messages have the same TTL you could also set the minimum TTL to the same value. | |
| </p><p>Minimum/Maximum TTL for a queue can be set though the HTTP Management UI, using the | |
| REST API. The attribute names are <code class="varname">minimumMessageTtl</code> and | |
| <code class="varname">maximumMessageTtl</code> and the TTL value is given in milliseconds.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="d0e1077"></a>4.7.3.2. Choosing to receive messages based on arrival time</h4></div></div></div><p>A queue with no destructive consumers will retain all messages until they expire | |
| due to TTL. It may be the case that a consumer only wishes to receive messages | |
| that have been sent in the last 60 minutes, and any new messages that arrive, or | |
| alternatively it may wish only to receive newly arriving messages and not any that | |
| are already in the queue. This can be achieved by using a filter on the arrival | |
| time.</p><p>A special parameter <code class="varname">x-qpid-replay-period</code> can be used in the | |
| consumer declaration to control the messages the consumer wishes to receive. The | |
| value of <code class="varname">x-qpid-replay-period</code> is the time, in seconds, for which | |
| the consumer wishes to see messages. A replay period of 0 indicates only newly | |
| arriving messages should be sent. A replay period of 3600 indicates that only | |
| messages sent in the last hour - along with any newly arriving messages - should be | |
| sent.</p><p>When using the Qpid JMS AMQP 0-x, the consumer declaration can be hinted using the | |
| address.</p><div class="table"><a id="d0e1092"></a><p class="title"><strong>Table 4.1. Setting the replay period using a Qpid JMS AMQP 0-x address</strong></p><div class="table-contents"><table border="1" summary="Setting the replay period using a Qpid JMS AMQP 0-x address"><colgroup><col /><col /></colgroup><thead><tr><th>Syntax</th><th>Example</th></tr></thead><tbody><tr><td>Addressing</td><td>myqueue : { link : { x-subscribe: { arguments : { x-qpid-replay-period : '3600' } } } }</td></tr><tr><td>Binding URL</td><td>direct://amq.direct/myqueue/myqueue?x-qpid-replay-period='3600'</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="d0e1113"></a>4.7.3.3. Setting a default filter</h4></div></div></div><p>A common case might be that the desired default behaviour is that newly attached consumers | |
| see only newly arriving messages (i.e. standard topic-like behaviour) but other consumers | |
| may wish to start their message stream from some point in the past. This can be achieved by | |
| setting a default filter on the queue so that consumers which do not explicitly set a replay | |
| period get a default (in this case the desired default would be 0).</p><p>The default filter set for a queue can be set via the REST API using the attribute named | |
| <code class="varname">defaultFilters</code>. This value is a map from filter name to type and arguments. | |
| To set the default behaviour for the queue to be that consumers only receive newly arrived | |
| messages, then you should set this attribute to the value:</p><pre class="screen"> | |
| { "x-qpid-replay-period" : { "x-qpid-replay-period" : [ "0" ] } } | |
| </pre><p> | |
| If the desired default behaviour is that each consumer should see all messages arriving in | |
| the last minute, as well as all new messages then the value would need to be:</p><pre class="screen"> | |
| { "x-qpid-replay-period" : { "x-qpid-replay-period" : [ "60" ] } } | |
| </pre></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queue-HoldingEntries"></a>4.7.4. Holding messages on a Queue</h3></div></div></div><p>Sometimes it is required that while a message has been placed on a queue, it is not released to consumers | |
| until some external condition is met. </p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="d0e1134"></a>4.7.4.1. Hold until valid</h4></div></div></div><p>Currently Queues support the "holding" of messages until a (per-message) provided point in time. | |
| By default this support is not enabled (since it requires extra work to be performed against every | |
| message entering the queue. To enable support, the attribute <code class="varname">holdOnPublishEnabled</code> | |
| must evaluate to true for the Queue. When enabled messages on the queue will be checked for the header | |
| (for AMQP 0-8, 0-9, 0-9-1 and 0-10 messages) or message annotation (for AMQP 1.0 messages) | |
| <code class="varname">x-qpid-not-valid-before</code>. If this header/annotation exists and contains a numeric value, | |
| it will be treated as a point in time given in milliseconds since the UNIX epoch. The message will not | |
| be released from the Queue to consumers until this time has been reached. | |
| </p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queue-OverflowPolicy"></a>4.7.5. Controlling Queue Size</h3></div></div></div><p> | |
| <span class="emphasis"><em>Overflow Policy</em></span> | |
| can be configured on an individual <span class="emphasis"><em>Queue</em></span> to limit the queue size. The size | |
| can be expressed in terms of a <span class="emphasis"><em>maximum number of bytes</em></span> and/or | |
| <span class="emphasis"><em>maximum number of messages</em></span>. | |
| The <span class="emphasis"><em>Overflow Policy</em></span> defines the Queue behaviour when any of the limits is reached. | |
| </p><p> | |
| The following <span class="emphasis"><em>Overflow Policies</em></span> are supported: | |
| </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> | |
| <span class="emphasis"><em>None</em></span> | |
| - Queue is unbounded and the capacity limits are not applied. | |
| This is a default policy applied implicitly when policy is not set explicitly. | |
| </p></li><li class="listitem"><p> | |
| <span class="emphasis"><em>Ring</em></span> | |
| - If a newly arriving message takes the queue over a limit, message(s) are | |
| deleted from the queue until the queue falls within its limit again. When deleting | |
| messages, the oldest messages are deleted first. For a | |
| <a class="link" href="Java-Broker-Concepts-Queues.html#Java-Broker-Concepts-Queues-Types-Priority" title="4.7.1.2. Priority">Priority Queue</a> | |
| the oldest messages with lowest priorities are removed. | |
| </p></li><li class="listitem"><p> | |
| <span class="emphasis"><em>Producer Flow Control</em></span> | |
| -The producing sessions | |
| are blocked until queue depth falls below the <span class="emphasis"><em>resume threshold</em></span> set as a | |
| context variable <span class="emphasis"><em>${queue.queueFlowResumeLimit}</em></span> | |
| (specifying the percentage from the limit values. Default is 80%). | |
| </p></li><li class="listitem"><p> | |
| <span class="emphasis"><em>Flow to Disk</em></span> | |
| -If the queue breaches a limit, newly arriving messages are written to disk and the in-memory | |
| representation of the message is minimised. The Broker will transparently retrieve messages | |
| from disk as they are required by a consumer or management. The flow to disk policy does not | |
| actually restrict the overall size of the queue, merely the space occupied in memory. | |
| The Broker's other <a class="link" href="Java-Broker-Runtime-Flow-To-Disk.html" title="9.6. Flow to Disk">Flow to Disk feature</a> | |
| operates completely independent of this Queue Overflow Policy. | |
| </p></li><li class="listitem"><p> | |
| <span class="emphasis"><em>Reject</em></span> | |
| -A newly arriving message is rejected when queue limit is breached. | |
| </p></li></ul></div><p> | |
| </p><p> | |
| A negative value for <span class="emphasis"><em>maximum number of messages</em></span> or | |
| <span class="emphasis"><em>maximum number of bytes</em></span> disables the limit. | |
| </p><p> | |
| The Broker issues Operational log messages when the queue sizes are breached. These are documented | |
| at <a class="xref" href="Java-Broker-Appendix-Operation-Logging.html#Java-Broker-Appendix-Operation-Logging-Message-List-Queue" title="Table C.6. Queue Log Messages">Table C.6, “Queue Log Messages”</a>. | |
| </p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="Java-Broker-Concepts-Queues-SetLowPrefetch"></a>4.7.6. Using low pre-fetch with special queue types</h3></div></div></div><p>Messaging clients may buffered messages for performance reasons. In Qpid, this is commonly known as | |
| <span class="emphasis"><em>pre-fetch</em></span></p><p>When using some of the messaging features described on this section, using prefetch can give | |
| unexpected behaviour. Once the broker has sent a message to the client its delivery order is then fixed, | |
| regardless of the special behaviour of the queue. </p><p>For example, if using a priority queue and a prefetch of 100, and 100 messages arrive with | |
| priority 2, the broker will send these messages to the client. If then a new message arrives | |
| with priority 1, the broker cannot leap frog messages of lower priority. The priority 1 will | |
| be delivered at the front of the next batch of messages to be sent to the client.</p><p>Using pre-fetch of 1 will give exact queue-type semantics as perceived by the | |
| client however, this brings a performance cost. You could test with a slightly higher | |
| pre-fetch to trade-off between throughput and exact semantics.</p><p>See the messaging client documentation for details of how to configure prefetch.</p></div></div><div class="navfooter"><hr /><table summary="Navigation footer" width="100%"><tr><td align="left" width="40%"><a accesskey="p" href="Java-Broker-Concepts-Exchanges.html">Prev</a> </td><td align="center" width="20%"><a accesskey="u" href="Java-Broker-Concepts.html">Up</a></td><td align="right" width="40%"> <a accesskey="n" href="Java-Broker-Concepts-Ports.html">Next</a></td></tr><tr><td align="left" valign="top" width="40%">4.6. Exchanges </td><td align="center" width="20%"><a accesskey="h" href="Apache-Qpid-Broker-J-Book.html">Home</a></td><td align="right" valign="top" width="40%"> 4.8. Ports</td></tr></table></div></div> |