| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="generator" content="Asciidoctor 2.0.18"> |
| <link rel="icon" type="image/png" href="images/favicon.png"> |
| <title>Message Redelivery and Undelivered Messages</title> |
| <link rel="stylesheet" href="css/asciidoctor.css"> |
| <link rel="stylesheet" href="css/font-awesome.css"> |
| <link rel="stylesheet" href="css/rouge-github.css"> |
| </head> |
| <body class="book toc2 toc-left"> |
| <div id="header"> |
| <h1>Message Redelivery and Undelivered Messages</h1> |
| <div id="toc" class="toc2"> |
| <div id="toctitle"><a href="index.html">User Manual for 2.32.0</a></div> |
| <ul class="sectlevel1"> |
| <li><a href="#delayed-redelivery">1. Delayed Redelivery</a> |
| <ul class="sectlevel2"> |
| <li><a href="#configuring-delayed-redelivery">1.1. Configuring Delayed Redelivery</a></li> |
| <li><a href="#example">1.2. Example</a></li> |
| </ul> |
| </li> |
| <li><a href="#dead-letter-addresses">2. Dead Letter Addresses</a> |
| <ul class="sectlevel2"> |
| <li><a href="#configuring-dead-letter-addresses">2.1. Configuring Dead Letter Addresses</a></li> |
| <li><a href="#dead-letter-properties">2.2. Dead Letter Properties</a></li> |
| <li><a href="#automatically-creating-dead-letter-resources">2.3. Automatically Creating Dead Letter Resources</a></li> |
| <li><a href="#example-2">2.4. Example</a></li> |
| </ul> |
| </li> |
| <li><a href="#delivery-count-persistence">3. Delivery Count Persistence</a></li> |
| </ul> |
| </div> |
| </div> |
| <div id="content"> |
| <div id="preamble"> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Messages can be delivered unsuccessfully (e.g. if the transacted session used to consume them is rolled back). |
| Such a message goes back to its queue ready to be redelivered. |
| However, this means it is possible for a message to be delivered again and again without success thus remaining in the queue indefinitely, clogging the system.</p> |
| </div> |
| <div class="paragraph"> |
| <p>There are 2 ways to deal with these undelivered messages:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Delayed redelivery.</p> |
| <div class="paragraph"> |
| <p>It is possible to delay messages redelivery. |
| This gives the client some time to recover from any transient failures and to prevent overloading its network or CPU resources.</p> |
| </div> |
| </li> |
| <li> |
| <p>Dead Letter Address.</p> |
| <div class="paragraph"> |
| <p>It is also possible to configure a dead letter address so that after a specified number of unsuccessful deliveries, messages are removed from their queue and sent to the dead letter address. |
| These messages will not be delivered again from this queue.</p> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Both options can be combined for maximum flexibility.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="delayed-redelivery"><a class="anchor" href="#delayed-redelivery"></a><a class="link" href="#delayed-redelivery">1. Delayed Redelivery</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Delaying redelivery can often be useful in cases where clients regularly fail or rollback. |
| Without a delayed redelivery, the system can get into a "thrashing" state, with delivery being attempted, the client rolling back, and delivery being re-attempted ad infinitum in quick succession, consuming valuable CPU and network resources.</p> |
| </div> |
| <div class="paragraph"> |
| <p>#Persist Redelivery</p> |
| </div> |
| <div class="paragraph"> |
| <p>Two Journal update records are stored every time a redelivery happens. |
| One for the number of deliveries that happened, and one in case a scheduled redelivery is being used.</p> |
| </div> |
| <div class="paragraph"> |
| <p>It is recommended to keep max-redelivery-records=1 in situations where you are operating with very short redelivery delays as you will be creating unecessary records on the journal.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="configuring-delayed-redelivery"><a class="anchor" href="#configuring-delayed-redelivery"></a><a class="link" href="#configuring-delayed-redelivery">1.1. Configuring Delayed Redelivery</a></h3> |
| <div class="paragraph"> |
| <p>Delayed redelivery is defined in the address-setting configuration:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="c"><!-- delay redelivery of messages for 5s --></span> |
| <span class="nt"><address-setting</span> <span class="na">match=</span><span class="s">"exampleQueue"</span><span class="nt">></span> |
| <span class="c"><!-- default is 1.0 --></span> |
| <span class="nt"><redelivery-delay-multiplier></span>1.5<span class="nt"></redelivery-delay-multiplier></span> |
| <span class="c"><!-- default is 0 (no delay) --></span> |
| <span class="nt"><redelivery-delay></span>5000<span class="nt"></redelivery-delay></span> |
| <span class="c"><!-- default is 0.0) --></span> |
| <span class="nt"><redelivery-collision-avoidance-factor></span>0.15<span class="nt"></redelivery-collision-avoidance-factor></span> |
| <span class="c"><!-- default is redelivery-delay * 10 --></span> |
| <span class="nt"><max-redelivery-delay></span>50000<span class="nt"></max-redelivery-delay></span> |
| <span class="nt"></address-setting></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If a <code>redelivery-delay</code> is specified, Apache ActiveMQ Artemis will wait this delay before redelivering the messages.</p> |
| </div> |
| <div class="paragraph"> |
| <p>By default, there is no redelivery delay (<code>redelivery-delay</code>is set to 0).</p> |
| </div> |
| <div class="paragraph"> |
| <p>Other subsequent messages will be delivery regularly, only the cancelled message will be sent asynchronously back to the queue after the delay.</p> |
| </div> |
| <div class="paragraph"> |
| <p>You can specify a multiplier (the <code>redelivery-delay-multiplier</code>) that will take effect on top of the <code>redelivery-delay</code>. |
| Each time a message is redelivered the delay period will be equal to the previous delay * <code>redelivery-delay-multiplier</code>. |
| A <code>max-redelivery-delay</code> can be set to prevent the delay from becoming too large. |
| The <code>max-redelivery-delay</code> is defaulted to <code>redelivery-delay</code> * 10.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Example:</strong></p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>redelivery-delay=5000, redelivery-delay-multiplier=2, max-redelivery-delay=15000, redelivery-collision-avoidance-factor=0.0</p> |
| <div class="olist arabic"> |
| <ol class="arabic"> |
| <li> |
| <p>Delivery Attempt 1. |
| (Unsuccessful)</p> |
| </li> |
| <li> |
| <p>Wait Delay Period: 5000</p> |
| </li> |
| <li> |
| <p>Delivery Attempt 2. |
| (Unsuccessful)</p> |
| </li> |
| <li> |
| <p>Wait Delay Period: 10000 // (5000 * 2) < max-delay-period. |
| Use 10000</p> |
| </li> |
| <li> |
| <p>Delivery Attempt 3: (Unsuccessful)</p> |
| </li> |
| <li> |
| <p>Wait Delay Period: 15000 // (10000 * 2) > max-delay-period: Use max-delay-delivery</p> |
| </li> |
| </ol> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Address wildcards can be used to configure redelivery delay for a set of addresses (see <a href="wildcard-syntax.html#wildcard-syntax">Understanding the Wildcard Syntax</a>), so you don’t have to specify redelivery delay individually for each address.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The <code>redelivery-delay</code> can be also be modified by configuring the <code>redelivery-collision-avoidance-factor</code>. |
| This factor will be made either positive or negative at random to control whether the ultimate value will increase or decrease the <code>redelivery-delay</code>. |
| Then it’s multiplied by a random number between 0.0 and 1.0. |
| This result is then multiplied by the <code>redelivery-delay</code> and then added to the <code>redelivery-delay</code> to arrive at the final value.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The algorithm may sound complicated but the bottom line is quite simple: the larger <code>redelivery-collision-avoidance-factor</code> you choose the larger the variance of the <code>redelivery-delay</code> will be. |
| The <code>redelivery-collision-avoidance-factor</code> must be between 0.0 and 1.0.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Example:</strong></p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>redelivery-delay=1000, redelivery-delay-multiplier=1, max-redelivery-delay=15000, redelivery-collision-avoidance-factor=0.5, (bold values chosen using <code>java.util.Random</code>)</p> |
| <div class="olist arabic"> |
| <ol class="arabic"> |
| <li> |
| <p>Delivery Attempt 1. |
| (Unsuccessful)</p> |
| </li> |
| <li> |
| <p>Wait Delay Period: 875 // 1000 + (1000 * ((0.5 * <strong>-1</strong>) * <strong>.25</strong>)</p> |
| </li> |
| <li> |
| <p>Delivery Attempt 2. |
| (Unsuccessful)</p> |
| </li> |
| <li> |
| <p>Wait Delay Period: 1375 // 1000 + (1000 * ((0.5 * <strong>1</strong>) * <strong>.75</strong>)</p> |
| </li> |
| <li> |
| <p>Delivery Attempt 3: (Unsuccessful)</p> |
| </li> |
| <li> |
| <p>Wait Delay Period: 975 // 1000 + (1000 * ((0.5 * <strong>-1</strong>) * <strong>.05</strong>)</p> |
| </li> |
| </ol> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>This feature can be particularly useful in environments where there are multiple consumers on the same queue all interacting transactionally with the same external system (e.g. a database). |
| If there is overlapping data in messages which are consumed concurrently then one transaction can succeed while all the rest fail. |
| If those failed messages are redelivered at the same time then this process where one consumer succeeds and the rest fail will continue. |
| By randomly padding the redelivery-delay by a small, configurable amount these redelivery "collisions" can be avoided.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="example"><a class="anchor" href="#example"></a><a class="link" href="#example">1.2. Example</a></h3> |
| <div class="paragraph"> |
| <p>See <a href="examples.html#examples">the examples chapter</a> for an example which shows how delayed redelivery is configured and used with JMS.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="dead-letter-addresses"><a class="anchor" href="#dead-letter-addresses"></a><a class="link" href="#dead-letter-addresses">2. Dead Letter Addresses</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>To prevent a client infinitely receiving the same undelivered message (regardless of what is causing the unsuccessful deliveries), messaging systems define <em>dead letter addresses</em>: after a specified unsuccessful delivery attempts, the message is removed from its queue and sent to a dead letter address.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Any such messages can then be diverted to queue(s) where they can later be perused by the system administrator for action to be taken.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Apache ActiveMQ Artemis’s addresses can be assigned a dead letter address. |
| Once the messages have been unsuccessfully delivered for a given number of attempts, they are removed from their queue and sent to the relevant dead letter address. |
| These <em>dead letter</em> messages can later be consumed from the dead letter address for further inspection.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="configuring-dead-letter-addresses"><a class="anchor" href="#configuring-dead-letter-addresses"></a><a class="link" href="#configuring-dead-letter-addresses">2.1. Configuring Dead Letter Addresses</a></h3> |
| <div class="paragraph"> |
| <p>Dead letter address is defined in the address-setting configuration:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="c"><!-- undelivered messages in exampleQueue will be sent to the dead letter address |
| deadLetterQueue after 3 unsuccessful delivery attempts --></span> |
| <span class="nt"><address-setting</span> <span class="na">match=</span><span class="s">"exampleQueue"</span><span class="nt">></span> |
| <span class="nt"><dead-letter-address></span>deadLetterAddress<span class="nt"></dead-letter-address></span> |
| <span class="nt"><max-delivery-attempts></span>3<span class="nt"></max-delivery-attempts></span> |
| <span class="nt"></address-setting></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If a <code>dead-letter-address</code> is not specified, messages will be removed after <code>max-delivery-attempts</code> unsuccessful attempts.</p> |
| </div> |
| <div class="paragraph"> |
| <p>By default, messages are redelivered 10 times at the maximum. |
| Set <code>max-delivery-attempts</code> to -1 for infinite redeliveries.</p> |
| </div> |
| <div class="paragraph"> |
| <p>A <code>dead letter address</code> can be set globally for a set of matching addresses and you can set <code>max-delivery-attempts</code> to -1 for a specific address setting to allow infinite redeliveries only for this address.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Address wildcards can be used to configure dead letter settings for a set of addresses (see <a href="wildcard-syntax.html#wildcard-syntax">Understanding the Wildcard Syntax</a>).</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="dead-letter-properties"><a class="anchor" href="#dead-letter-properties"></a><a class="link" href="#dead-letter-properties">2.2. Dead Letter Properties</a></h3> |
| <div class="paragraph"> |
| <p>Dead letter messages get <a href="copied-message-properties.html#properties-for-copied-messages">special properties</a>.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="automatically-creating-dead-letter-resources"><a class="anchor" href="#automatically-creating-dead-letter-resources"></a><a class="link" href="#automatically-creating-dead-letter-resources">2.3. Automatically Creating Dead Letter Resources</a></h3> |
| <div class="paragraph"> |
| <p>It’s common to segregate undelivered messages by their original address. |
| For example, a message sent to the <code>stocks</code> address that couldn’t be delivered for some reason might be ultimately routed to the <code>DLQ.stocks</code> queue, and likewise a message sent to the <code>orders</code> address that couldn’t be delivered might be routed to the <code>DLQ.orders</code> queue.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Using this pattern can make it easy to track and administrate undelivered messages. |
| However, it can pose a challenge in environments which predominantly use auto-created addresses and queues. |
| Typically administrators in those environments don’t want to manually create an <code>address-setting</code> to configure the <code>dead-letter-address</code> much less the actual <code>address</code> and <code>queue</code> to hold the undelivered messages.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The solution to this problem is to set the <code>auto-create-dead-letter-resources</code> <code>address-setting</code> to <code>true</code> (it’s <code>false</code> by default) so that the broker will create the <code>address</code> and <code>queue</code> to deal with the undelivered messages automatically. |
| The <code>address</code> created will be the one defined by the <code>dead-letter-address</code>. |
| A <code>MULTICAST</code> <code>queue</code> will be created on that <code>address</code>. |
| It will be named by the <code>address</code> to which the message was previously sent, and it will have a filter defined using the property <code>_AMQ_ORIG_ADDRESS</code> so that it will only receive messages sent to the relevant <code>address</code>. |
| The <code>queue</code> name can be configured with a prefix and suffix. |
| See the relevant settings in the table below:</p> |
| </div> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 50%;"> |
| <col style="width: 50%;"> |
| </colgroup> |
| <thead> |
| <tr> |
| <th class="tableblock halign-left valign-top"><code>address-setting</code></th> |
| <th class="tableblock halign-left valign-top">default</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>dead-letter-queue-prefix</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>DLQ.</code></p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>dead-letter-queue-suffix</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">(empty string)</p></td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="paragraph"> |
| <p>Here is an example configuration:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><address-setting</span> <span class="na">match=</span><span class="s">"#"</span><span class="nt">></span> |
| <span class="nt"><dead-letter-address></span>DLA<span class="nt"></dead-letter-address></span> |
| <span class="nt"><max-delivery-attempts></span>3<span class="nt"></max-delivery-attempts></span> |
| <span class="nt"><auto-create-dead-letter-resources></span>true<span class="nt"></auto-create-dead-letter-resources></span> |
| <span class="nt"><dead-letter-queue-prefix></dead-letter-queue-prefix></span> <span class="c"><!-- override the default --></span> |
| <span class="nt"><dead-letter-queue-suffix></span>.DLQ<span class="nt"></dead-letter-queue-suffix></span> |
| <span class="nt"></address-setting></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The queue holding the undeliverable messages can be accessed directly either by using the queue’s name by itself (e.g. when using the core client) or by using the fully qualified queue name (e.g. when using a JMS client) just like any other queue. |
| Also, note that the queue is auto-created which means it will be auto-deleted as per the relevant <code>address-settings</code>.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="example-2"><a class="anchor" href="#example-2"></a><a class="link" href="#example-2">2.4. Example</a></h3> |
| <div class="paragraph"> |
| <p>See: Dead Letter section of the <a href="examples.html#examples">Examples</a> for an example that shows how dead letter resources can be statically configured and used with JMS.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="delivery-count-persistence"><a class="anchor" href="#delivery-count-persistence"></a><a class="link" href="#delivery-count-persistence">3. Delivery Count Persistence</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>In normal use, Apache ActiveMQ Artemis does not update delivery count <em>persistently</em> until a message is rolled back (i.e. the delivery count is not updated <em>before</em> the message is delivered to the consumer). |
| In most messaging use cases, the messages are consumed, acknowledged and forgotten as soon as they are consumed. |
| In these cases, updating the delivery count persistently before delivering the message would add an extra persistent step <em>for each message delivered</em>, implying a significant performance penalty.</p> |
| </div> |
| <div class="paragraph"> |
| <p>However, if the delivery count is not updated persistently before the message delivery happens, in the event of a server crash, messages might have been delivered but that will not have been reflected in the delivery count. |
| During the recovery phase, the server will not have knowledge of that and will deliver the message with <code>redelivered</code> set to <code>false</code> while it should be <code>true</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>As this behavior breaks strict JMS semantics, Apache ActiveMQ Artemis allows to persist delivery count before message delivery but this feature is disabled by default due to performance implications.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To enable it, set <code>persist-delivery-count-before-delivery</code> to <code>true</code> in <code>broker.xml</code>:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><persist-delivery-count-before-delivery></span>true<span class="nt"></persist-delivery-count-before-delivery></span></code></pre> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </body> |
| </html> |
