src/components/artemis/documentation/previous/2.50.0/duplicate-detection.html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 2.0.23">
<link rel="icon" type="image/png" href="images/favicon.png">
<title>Duplicate Message Detection</title>
<link rel="stylesheet" href="css/asciidoctor.css">
<link rel="stylesheet" href="css/font-awesome.css">
<link rel="stylesheet" href="css/rouge-github.css">
<script>
    document.addEventListener("DOMContentLoaded", function() {
        const pathSegments = window.location.pathname.split('/');
        if (window.location.hostname == "artemis.apache.org" && pathSegments[pathSegments.length - 2] != "latest") {
            var message = document.createElement("div");
            message.style.margin = "20px";
            message.style.textAlign = "center";
            message.style.backgroundColor = "#FFFFE0";
            message.textContent = "Please be aware that this documentation is out of date. ";

            var link = document.createElement("a");
            link.href = "../../latest";
            link.textContent = "Here is the latest documentation.";
            message.appendChild(link);

            document.body.insertBefore(message, document.body.firstChild);
        }
    });
</script>
</head>
<body class="book toc2 toc-left">
<div id="header">
<h1>Duplicate Message Detection</h1>
<div id="toc" class="toc2">
<div id="toctitle"><a href="index.html">User Manual for 2.50.0</a></div>
<ul class="sectlevel1">
<li><a href="#using-duplicate-detection-for-message-sending">1. Using Duplicate Detection for Message Sending</a></li>
<li><a href="#duplicate-detection-semantics">2. Duplicate Detection Semantics</a></li>
<li><a href="#configuring-the-duplicate-id-cache">3. Configuring the Duplicate ID Cache</a>
<ul class="sectlevel2">
<li><a href="#global-configuration">3.1. Global Configuration</a></li>
<li><a href="#address-specific-configuration">3.2. Address-Specific Configuration</a></li>
<li><a href="#persisting-the-cache-to-storage">3.3. Persisting the Cache to Storage</a></li>
</ul>
</li>
<li><a href="#duplicate-detection-and-bridges">4. Duplicate Detection and Bridges</a></li>
<li><a href="#duplicate-detection-and-cluster-connections">5. Duplicate Detection and Cluster Connections</a></li>
<li><a href="#performance-considerations">6. Performance Considerations</a></li>
</ul>
</div>
</div>
<div id="content">
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p>The broker includes powerful automatic duplicate message detection, filtering out duplicate messages without you having to code your own fiddly duplicate detection logic at the application level.
This chapter will explain what duplicate detection is, how the broker uses it, and how to configure it.</p>
</div>
<div class="paragraph">
<p>When sending messages from a client to a server, or indeed from a server to another server, if the target server or connection fails sometime after sending the message, but before the sender receives a response that the send (or commit) was processed successfully, then the sender cannot know for sure if the message was sent successfully to the address.</p>
</div>
<div class="paragraph">
<p>If the target server or connection failed after the send was received and processed, but before the response was sent back then the message will have been sent to the address successfully, but if the target server or connection failed before the send was received and finished processing, then it will not have been sent to the address successfully.
From the senders' point of view, it&#8217;s not possible to distinguish these two cases.</p>
</div>
<div class="paragraph">
<p>When the server recovers, this leaves the client in a difficult situation.
It knows the target server failed, but it does not know if the last message reached its destination successfully.
If it decides to resend the last message then that could result in a duplicate message being sent to the address.
If each message was an order or a trade then this could result in the order being fulfilled twice or the trade being double-booked.
This is clearly not a desirable situation.</p>
</div>
<div class="paragraph">
<p>Sending the message(s) in a transaction does not help either.
If the server or connection fails while the transaction commit is being processed, it is also indeterminate whether the transaction was successfully committed or not!</p>
</div>
<div class="paragraph">
<p>Automatic duplicate messages detection solves these problems.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="using-duplicate-detection-for-message-sending"><a class="anchor" href="#using-duplicate-detection-for-message-sending"></a><a class="link" href="#using-duplicate-detection-for-message-sending">1. Using Duplicate Detection for Message Sending</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>To enable duplicate message detection for sent messages you just need to set a special duplicate ID property on the message to a <strong>unique</strong> value.
You can create the value however you like, as long as it is unique.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Using duplicate detection to move messages between nodes can give you the same <em>once and only once</em> delivery guarantees as if you were using an XA transaction to consume messages from source and send them to the target, but with less overhead and much easier configuration than using XA.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>If you&#8217;re sending messages in a transaction then you don&#8217;t have to set the property for <em>every</em> message you send in that transaction.
You only need to set it once in the transaction.
If the server detects a duplicate message for any message in the transaction then it will ignore the entire transaction.</p>
</div>
<div class="paragraph">
<p>The name of the duplicate ID property is <code>_AMQ_DUPL_ID</code>. As a convenience for Java-based applications using the Core client <code>org.apache.activemq.artemis.api.core.Message.HDR_DUPLICATE_DETECTION_ID</code> can be used.</p>
</div>
<div class="paragraph">
<p>When using JMS the property&#8217;s value must be a <code>String</code>, and similarly a string type would be used in other client APIs or protocols used with the broker.</p>
</div>
<div class="paragraph">
<p>Here&#8217;s an example of setting the property using the JMS API:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight nowrap"><code data-lang="java"><span class="nc">Message</span> <span class="n">jmsMessage</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="na">createMessage</span><span class="o">();</span>

<span class="nc">String</span> <span class="n">myUniqueID</span> <span class="o">=</span> <span class="s">"This is my unique id"</span><span class="o">;</span>   <span class="c1">// Could use a UUID for this</span>

<span class="n">message</span><span class="o">.</span><span class="na">setStringProperty</span><span class="o">(</span><span class="no">HDR_DUPLICATE_DETECTION_ID</span><span class="o">.</span><span class="na">toString</span><span class="o">(),</span> <span class="n">myUniqueID</span><span class="o">);</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>If using the Core client the value of the property can be of type <code>String</code>, <code>SimpleString</code>, or <code>byte[]</code>.</p>
</div>
<div class="paragraph">
<p>Here&#8217;s an example of setting the property using the Core API:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight nowrap"><code data-lang="java"><span class="nc">ClientMessage</span> <span class="n">message</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="na">createMessage</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>

<span class="nc">SimpleString</span> <span class="n">myUniqueID</span> <span class="o">=</span> <span class="nc">SimpleString</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"This is my unique id"</span><span class="o">);</span>   <span class="c1">// Could use a UUID for this</span>

<span class="n">message</span><span class="o">.</span><span class="na">putStringProperty</span><span class="o">(</span><span class="no">HDR_DUPLICATE_DETECTION_ID</span><span class="o">,</span> <span class="n">myUniqueID</span><span class="o">);</span></code></pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="duplicate-detection-semantics"><a class="anchor" href="#duplicate-detection-semantics"></a><a class="link" href="#duplicate-detection-semantics">2. Duplicate Detection Semantics</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>The server maintains a <strong>circular</strong>, fixed-size, per-address cache of duplicate IDs from messages it receives.</p>
</div>
<div class="paragraph">
<p>When the server receives the message it will check if the duplicate ID property is set.
If it is then it will check to see if its cache for the correspond address already contains that duplicate ID.
If the cache already contains that duplicate ID then the message will not be routed to any queues, and the server will log a <code>WARN</code> message, e.g.:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight nowrap"><code data-lang="console"><span class="go">WARN  [org.apache.activemq.artemis.core.server] AMQ222059: Duplicate message detected - message will not be routed. Message information:
CoreMessage[messageID=15, durable=false, userID=null, priority=4, timestamp=Thu Jan 01 00:00:00 UTC 1970, expiration=0, durable=false, address=myAddress, size=166, properties=TypedProperties[_AMQ_DUPL_ID=[6100 6200 6300 6400 6500 6600 6700]]]@1034478028</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>If the cache does not contain that duplicate ID then it is added to the cache and the message is routed to any applicable queues.</p>
</div>
<div class="paragraph">
<p>Since the cache is circular then if it has a maximum size of <code>n</code> elements the <code>n + 1</code>th id stored will overwrite the <code>0</code>th element in the cache.
Duplicate IDs are <em>only</em> removed from the cache when they are overwritten or cleared administratively (e.g. using the <code>clearDuplicateIdCache</code> operation on the corresponding <a href="management.html#address-management"><code>AddressControl</code></a> from the web console).
Even if a message is acknowledged or expires its duplicate ID is not removed from the cache because another message with that same duplicate ID may still be sent.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="configuring-the-duplicate-id-cache"><a class="anchor" href="#configuring-the-duplicate-id-cache"></a><a class="link" href="#configuring-the-duplicate-id-cache">3. Configuring the Duplicate ID Cache</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>The size of the duplicate ID cache can be configured globally for all addresses or on a per-address basis.</p>
</div>
<div class="paragraph">
<p>Whether the cache is persisted to storage is also configurable.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>When choosing a size of the duplicate id cache be sure to set it to a larger enough size so if you resend messages all the previously sent ones are in the cache not having been overwritten.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="global-configuration"><a class="anchor" href="#global-configuration"></a><a class="link" href="#global-configuration">3.1. Global Configuration</a></h3>
<div class="paragraph">
<p>The maximum size of the cache is configured by the parameter <code>id-cache-size</code> in <code>broker.xml</code>, e.g.:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt">&lt;core&gt;</span>
   ...
   <span class="nt">&lt;id-cache-size&gt;</span>5000<span class="nt">&lt;/id-cache-size&gt;</span>
   ...
<span class="nt">&lt;/core&gt;</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>The default value for the global <code>id-cache-size</code> is <code>20000</code>. A value of <code>0</code> disables caching.</p>
</div>
</div>
<div class="sect2">
<h3 id="address-specific-configuration"><a class="anchor" href="#address-specific-configuration"></a><a class="link" href="#address-specific-configuration">3.2. Address-Specific Configuration</a></h3>
<div class="paragraph">
<p>To configure the cache size on a per-address basis use the <code>id-cache-size</code> <code>address-settings</code> section in <code>broker.xml</code>, e.g.:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt">&lt;address-setting</span> <span class="na">match=</span><span class="s">"myAddress"</span><span class="nt">&gt;</span>
   ...
   <span class="nt">&lt;id-cache-size&gt;</span>1000<span class="nt">&lt;/id-cache-size&gt;</span>
   ...
<span class="nt">&lt;/address-setting&gt;</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>When a message is sent to an address with a specific <code>id-cache-size</code> configured it will take precedence over the global <code>id-cache-size</code> value.
This allows for greater flexibility and optimization of duplicate ID caches.</p>
</div>
<div class="paragraph">
<p>The default value for the per-address <code>id-cache-size</code> is <code>20000</code>. A value of <code>0</code> disables caching.</p>
</div>
</div>
<div class="sect2">
<h3 id="persisting-the-cache-to-storage"><a class="anchor" href="#persisting-the-cache-to-storage"></a><a class="link" href="#persisting-the-cache-to-storage">3.3. Persisting the Cache to Storage</a></h3>
<div class="paragraph">
<p>Duplicate ID caches are persisted to storage by default.
The benefit to persisting the cache to storage is that if the broker is stopped for any reason then when it restarts the data will be read from storage back into the cache so duplicate messages can still be detected even if they were sent before the broker restarted.
However, there is a cost in terms of performance since it takes longer to persist the data.</p>
</div>
<div class="paragraph">
<p>Duplicate ID cache persistence is configured by the parameter <code>persist-id-cache</code> in <code>broker.xml</code>, e.g.:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt">&lt;core&gt;</span>
   ...
   <span class="nt">&lt;persist-id-cache&gt;</span>false<span class="nt">&lt;/id-cache-size&gt;</span>
   ...
<span class="nt">&lt;/core&gt;</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>If <code>persist-id-cache</code> is set to <code>true</code> then each ID will be persisted to storage as it is received.
This is configured globally.
It can&#8217;t be configured on a per-address basis.</p>
</div>
<div class="paragraph">
<p>The default value for <code>persist-id-cache</code> is <code>true</code>.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="duplicate-detection-and-bridges"><a class="anchor" href="#duplicate-detection-and-bridges"></a><a class="link" href="#duplicate-detection-and-bridges">4. Duplicate Detection and Bridges</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Core bridges can be configured to automatically add a unique duplicate id value (if there isn&#8217;t already one in the message) before forwarding the message to its target.
This ensures that if the target server crashes or the connection is interrupted and the bridge resends the message, then if it has already been received by the target server, it will be ignored.</p>
</div>
<div class="paragraph">
<p>To configure a core bridge to add the duplicate id header, simply set the <code>use-duplicate-detection</code> to <code>true</code> when configuring a bridge in <code>broker.xml</code>.</p>
</div>
<div class="paragraph">
<p>The default value for this parameter is <code>true</code>.</p>
</div>
<div class="paragraph">
<p>For more information on core bridges and how to configure them, please see <a href="core-bridges.html#core-bridges">Core Bridges</a>.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="duplicate-detection-and-cluster-connections"><a class="anchor" href="#duplicate-detection-and-cluster-connections"></a><a class="link" href="#duplicate-detection-and-cluster-connections">5. Duplicate Detection and Cluster Connections</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Cluster connections internally use core bridges to move messages reliable between nodes of the cluster.
Consequently they can also be configured to insert the duplicate id header for each message they move using their internal bridges.</p>
</div>
<div class="paragraph">
<p>To configure a cluster connection to add the duplicate id header, simply set the <code>use-duplicate-detection</code> to <code>true</code> when configuring a cluster connection in <code>broker.xml</code>.</p>
</div>
<div class="paragraph">
<p>The default value for this parameter is <code>true</code>.</p>
</div>
<div class="paragraph">
<p>For more information on cluster connections and how to configure them, please see <a href="clusters.html#clusters">Clusters</a>.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="performance-considerations"><a class="anchor" href="#performance-considerations"></a><a class="link" href="#performance-considerations">6. Performance Considerations</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>If you <strong>do not need</strong> duplicate detection at all or only for certain addresses it is best to set the global <code>id-cache-size</code> to <code>0</code> to prevent the server from pre-allocating internal cache-related objects, e.g.:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt">&lt;core&gt;</span>
   ...
   <span class="nt">&lt;id-cache-size&gt;</span>0<span class="nt">&lt;/id-cache-size&gt;</span>
   ...
<span class="nt">&lt;/core&gt;</span></code></pre>
</div>
</div>
<div class="paragraph">
<p>This will prevent needless consumption of heap memory so it is available to the broker for other uses.</p>
</div>
</div>
</div>
</div>
</body>
</html>