| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="generator" content="Asciidoctor 2.0.18"> |
| <link rel="icon" type="image/png" href="images/favicon.png"> |
| <title>Large Messages</title> |
| <link rel="stylesheet" href="css/asciidoctor.css"> |
| <link rel="stylesheet" href="css/font-awesome.css"> |
| <link rel="stylesheet" href="css/rouge-github.css"> |
| </head> |
| <body class="book toc2 toc-left"> |
| <div id="header"> |
| <h1>Large Messages</h1> |
| <div id="toc" class="toc2"> |
| <div id="toctitle"><a href="index.html">User Manual for 2.31.2</a></div> |
| <ul class="sectlevel1"> |
| <li><a href="#configuring-the-server">1. Configuring the server</a></li> |
| <li><a href="#configuring-the-core-client">2. Configuring the Core Client</a></li> |
| <li><a href="#compressed-large-messages-on-core-protocol">3. Compressed Large Messages on Core Protocol</a></li> |
| <li><a href="#streaming-large-messages-from-core-protocol">4. Streaming large messages from Core Protocol</a> |
| <ul class="sectlevel2"> |
| <li><a href="#streaming-over-core-api">4.1. Streaming over Core API</a></li> |
| <li><a href="#streaming-over-jms">4.2. Streaming over JMS</a></li> |
| <li><a href="#streaming-alternative-on-core-protocol">4.3. Streaming Alternative on Core Protocol</a></li> |
| </ul> |
| </li> |
| <li><a href="#configuring-amqp-acceptor">5. Configuring AMQP Acceptor</a></li> |
| <li><a href="#large-message-example">6. Large message example</a></li> |
| </ul> |
| </div> |
| </div> |
| <div id="content"> |
| <div id="preamble"> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Apache ActiveMQ Artemis can be configured to store messages as files when these messages are beyond a configured value.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Instead of keeping these messages in memory ActiveMQ Artemis will hold just a thin object on the queues with a reference to a file into a specific folder configured as large-messages-directory.</p> |
| </div> |
| <div class="paragraph"> |
| <p>This is supported on Core Protocol and on the AMQP Protocol.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="configuring-the-server"><a class="anchor" href="#configuring-the-server"></a><a class="link" href="#configuring-the-server">1. Configuring the server</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Large messages are stored on a disk directory on the server side, as configured on the main configuration file.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The configuration property <code>large-messages-directory</code> specifies where large messages are stored. |
| For JDBC persistence the <code>large-message-table</code> should be configured.</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><configuration</span> <span class="na">xmlns=</span><span class="s">"urn:activemq"</span> |
| <span class="na">xmlns:xsi=</span><span class="s">"http://www.w3.org/2001/XMLSchema-instance"</span> |
| <span class="na">xsi:schemaLocation=</span><span class="s">"urn:activemq /schema/artemis-server.xsd"</span><span class="nt">></span> |
| <span class="nt"><core</span> <span class="na">xmlns=</span><span class="s">"urn:activemq:core"</span> <span class="na">xmlns:xsi=</span><span class="s">"http://www.w3.org/2001/XMLSchema-instance"</span> <span class="na">xsi:schemaLocation=</span><span class="s">"urn:activemq:core"</span><span class="nt">></span> |
| ... |
| <span class="nt"><large-messages-directory></span>/data/large-messages<span class="nt"></large-messages-directory></span> |
| ... |
| <span class="nt"></core></span> |
| <span class="nt"></configuration></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>By default the large message directory is <code>data/largemessages</code> and <code>large-message-table</code> is configured as "LARGE_MESSAGE_TABLE".</p> |
| </div> |
| <div class="paragraph"> |
| <p>For the best performance we recommend using file store with large messages directory stored on a different physical volume to the message journal or paging directory.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="configuring-the-core-client"><a class="anchor" href="#configuring-the-core-client"></a><a class="link" href="#configuring-the-core-client">2. Configuring the Core Client</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Any message larger than a certain size is considered a large message. |
| Large messages will be split up and sent in fragments. |
| This is determined by the URL parameter <code>minLargeMessageSize</code></p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>Apache ActiveMQ Artemis messages are encoded using 2 bytes per character so if the message data is filled with ASCII characters (which are 1 byte) the size of the resulting Apache ActiveMQ Artemis message would roughly double. |
| This is important when calculating the size of a "large" message as it may appear to be less than the <code>minLargeMessageSize</code> before it is sent, but it then turns into a "large" message once it is encoded.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>The default value is 100KiB.</p> |
| </div> |
| <div class="paragraph"> |
| <p><a href="configuring-transports.html#configuring-the-transport-directly-from-the-client">Configuring the transport directly from the client side</a> will provide more information on how to instantiate the core session factory or JMS connection factory.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="compressed-large-messages-on-core-protocol"><a class="anchor" href="#compressed-large-messages-on-core-protocol"></a><a class="link" href="#compressed-large-messages-on-core-protocol">3. Compressed Large Messages on Core Protocol</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>You can choose to send large messages in compressed form using <code>compressLargeMessage</code> URL parameter.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If you specify the boolean URL parameter <code>compressLargeMessage</code> as true, the system will use the ZIP algorithm to compress the message body as the message is transferred to the server’s side. |
| Notice that there’s no special treatment at the server’s side, all the compressing and uncompressing is done at the client.</p> |
| </div> |
| <div class="paragraph"> |
| <p>This behavior can be tuned further by setting an optional parameter: <code>compressionLevel</code>. |
| This will decide how much the message body should be compressed. |
| <code>compressionLevel</code> accepts an integer of <code>-1</code> or a value between <code>0-9</code>. |
| The default value is <code>-1</code> which corresponds to around level 6-7.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If the compressed size of a large message is below <code>minLargeMessageSize</code>, it is sent to server as regular messages. |
| This means that the message won’t be written into the server’s large-message data directory, thus reducing the disk I/O.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| A higher <code>compressionLevel</code> means the message body will get further compressed, but this is at the cost of speed and computational overhead. |
| Make sure to tune this value according to its specific use-case. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="streaming-large-messages-from-core-protocol"><a class="anchor" href="#streaming-large-messages-from-core-protocol"></a><a class="link" href="#streaming-large-messages-from-core-protocol">4. Streaming large messages from Core Protocol</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Apache ActiveMQ Artemis supports setting the body of messages using input and output streams (<code>java.lang.io</code>)</p> |
| </div> |
| <div class="paragraph"> |
| <p>These streams are then used directly for sending (input streams) and receiving (output streams) messages.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When receiving messages there are 2 ways to deal with the output stream; |
| you may choose to block while the output stream is recovered using the method <code>ClientMessage.saveOutputStream</code> or alternatively using the method <code>ClientMessage.setOutputstream</code> which will asynchronously write the message to the stream. |
| If you choose the latter the consumer must be kept alive until the message has been fully received.</p> |
| </div> |
| <div class="paragraph"> |
| <p>You can use any kind of stream you like. |
| The most common use case is to send files stored in your disk, but you could also send things like JDBC Blobs, <code>SocketInputStream</code>, things you recovered from <code>HTTPRequests</code> etc. |
| Anything as long as it implements <code>java.io.InputStream</code> for sending messages or <code>java.io.OutputStream</code> for receiving them.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="streaming-over-core-api"><a class="anchor" href="#streaming-over-core-api"></a><a class="link" href="#streaming-over-core-api">4.1. Streaming over Core API</a></h3> |
| <div class="paragraph"> |
| <p>The following table shows a list of methods available at <code>ClientMessage</code> which are also available through JMS by the use of object properties.</p> |
| </div> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 33.3333%;"> |
| <col style="width: 33.3333%;"> |
| <col style="width: 33.3334%;"> |
| </colgroup> |
| <thead> |
| <tr> |
| <th class="tableblock halign-left valign-top">Name</th> |
| <th class="tableblock halign-left valign-top">Description</th> |
| <th class="tableblock halign-left valign-top">JMS Equivalent</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">setBodyInputStream(InputStream)</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Set the InputStream used to read a message body when sending it.</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">JMS_AMQ_InputStream</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">setOutputStream(OutputStream)</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Set the OutputStream that will receive the body of a message. |
| This method does not block.</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">JMS_AMQ_OutputStream</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">saveOutputStream(OutputStream)</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Save the body of the message to the <code>OutputStream</code>. |
| It will block until the entire content is transferred to the <code>OutputStream</code>.</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">JMS_AMQ_SaveStream</p></td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="paragraph"> |
| <p>To set the output stream when receiving a core message:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="java"><span class="nc">ClientMessage</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">consumer</span><span class="o">.</span><span class="na">receive</span><span class="o">(...);</span> |
| |
| <span class="c1">// This will block here until the stream was transferred</span> |
| <span class="n">msg</span><span class="o">.</span><span class="na">saveOutputStream</span><span class="o">(</span><span class="n">someOutputStream</span><span class="o">);</span> |
| |
| <span class="nc">ClientMessage</span> <span class="n">msg2</span> <span class="o">=</span> <span class="n">consumer</span><span class="o">.</span><span class="na">receive</span><span class="o">(...);</span> |
| |
| <span class="c1">// This will not wait the transfer to finish</span> |
| <span class="n">msg2</span><span class="o">.</span><span class="na">setOutputStream</span><span class="o">(</span><span class="n">someOtherOutputStream</span><span class="o">);</span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Set the input stream when sending a core message:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="java"><span class="nc">ClientMessage</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="na">createMessage</span><span class="o">();</span> |
| <span class="n">msg</span><span class="o">.</span><span class="na">setInputStream</span><span class="o">(</span><span class="n">dataInputStream</span><span class="o">);</span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Notice also that for messages with more than 2GiB the getBodySize() will return invalid values since this is an integer (which is also exposed to the JMS API). |
| On those cases you can use the message property _AMQ_LARGE_SIZE.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="streaming-over-jms"><a class="anchor" href="#streaming-over-jms"></a><a class="link" href="#streaming-over-jms">4.2. Streaming over JMS</a></h3> |
| <div class="paragraph"> |
| <p>When using JMS, Apache ActiveMQ Artemis maps the streaming methods on the core API (see ClientMessage API table above) by setting object properties . You can use the method <code>Message.setObjectProperty</code> to set the input and output streams.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The <code>InputStream</code> can be defined through the JMS Object Property JMS_AMQ_InputStream on messages being sent:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="java"><span class="nc">BytesMessage</span> <span class="n">message</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="na">createBytesMessage</span><span class="o">();</span> |
| |
| <span class="nc">FileInputStream</span> <span class="n">fileInputStream</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">FileInputStream</span><span class="o">(</span><span class="n">fileInput</span><span class="o">);</span> |
| |
| <span class="nc">BufferedInputStream</span> <span class="n">bufferedInput</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">BufferedInputStream</span><span class="o">(</span><span class="n">fileInputStream</span><span class="o">);</span> |
| |
| <span class="n">message</span><span class="o">.</span><span class="na">setObjectProperty</span><span class="o">(</span><span class="s">"JMS_AMQ_InputStream"</span><span class="o">,</span> <span class="n">bufferedInput</span><span class="o">);</span> |
| |
| <span class="n">someProducer</span><span class="o">.</span><span class="na">send</span><span class="o">(</span><span class="n">message</span><span class="o">);</span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The <code>OutputStream</code> can be set through the JMS Object Property JMS_AMQ_SaveStream on messages being received in a blocking way.</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="java"><span class="nc">BytesMessage</span> <span class="n">messageReceived</span> <span class="o">=</span> <span class="o">(</span><span class="nc">BytesMessage</span><span class="o">)</span><span class="n">messageConsumer</span><span class="o">.</span><span class="na">receive</span><span class="o">(</span><span class="mi">120000</span><span class="o">);</span> |
| |
| <span class="nc">File</span> <span class="n">outputFile</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">File</span><span class="o">(</span><span class="s">"huge_message_received.dat"</span><span class="o">);</span> |
| |
| <span class="nc">FileOutputStream</span> <span class="n">fileOutputStream</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">FileOutputStream</span><span class="o">(</span><span class="n">outputFile</span><span class="o">);</span> |
| |
| <span class="nc">BufferedOutputStream</span> <span class="n">bufferedOutput</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">BufferedOutputStream</span><span class="o">(</span><span class="n">fileOutputStream</span><span class="o">);</span> |
| |
| <span class="c1">// This will block until the entire content is saved on disk</span> |
| <span class="n">messageReceived</span><span class="o">.</span><span class="na">setObjectProperty</span><span class="o">(</span><span class="s">"JMS_AMQ_SaveStream"</span><span class="o">,</span> <span class="n">bufferedOutput</span><span class="o">);</span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Setting the <code>OutputStream</code> could also be done in a non blocking way using the property JMS_AMQ_OutputStream.</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="java"><span class="c1">// This won't wait the stream to finish. You need to keep the consumer active.</span> |
| <span class="n">messageReceived</span><span class="o">.</span><span class="na">setObjectProperty</span><span class="o">(</span><span class="s">"JMS_AMQ_OutputStream"</span><span class="o">,</span> <span class="n">bufferedOutput</span><span class="o">);</span></code></pre> |
| </div> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>When using JMS, Streaming large messages are only supported on <code>StreamMessage</code> and <code>BytesMessage</code>.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="streaming-alternative-on-core-protocol"><a class="anchor" href="#streaming-alternative-on-core-protocol"></a><a class="link" href="#streaming-alternative-on-core-protocol">4.3. Streaming Alternative on Core Protocol</a></h3> |
| <div class="paragraph"> |
| <p>If you choose not to use the <code>InputStream</code> or <code>OutputStream</code> capability of Apache ActiveMQ Artemis You could still access the data directly in an alternative fashion.</p> |
| </div> |
| <div class="paragraph"> |
| <p>On the Core API just get the bytes of the body as you normally would.</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="java"><span class="nc">ClientMessage</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">consumer</span><span class="o">.</span><span class="na">receive</span><span class="o">();</span> |
| |
| <span class="kt">byte</span><span class="o">[]</span> <span class="n">bytes</span> <span class="o">=</span> <span class="k">new</span> <span class="kt">byte</span><span class="o">[</span><span class="mi">1024</span><span class="o">];</span> |
| <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span> <span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">msg</span><span class="o">.</span><span class="na">getBodySize</span><span class="o">();</span> <span class="n">i</span> <span class="o">+=</span> <span class="n">bytes</span><span class="o">.</span><span class="na">length</span><span class="o">)</span> |
| <span class="o">{</span> |
| <span class="n">msg</span><span class="o">.</span><span class="na">getBody</span><span class="o">().</span><span class="na">readBytes</span><span class="o">(</span><span class="n">bytes</span><span class="o">);</span> |
| <span class="c1">// Whatever you want to do with the bytes</span> |
| <span class="o">}</span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If using JMS API, <code>BytesMessage</code> and <code>StreamMessage</code> also supports it transparently.</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="java"><span class="nc">BytesMessage</span> <span class="n">rm</span> <span class="o">=</span> <span class="o">(</span><span class="nc">BytesMessage</span><span class="o">)</span><span class="n">cons</span><span class="o">.</span><span class="na">receive</span><span class="o">(</span><span class="mi">10000</span><span class="o">);</span> |
| |
| <span class="kt">byte</span> <span class="n">data</span><span class="o">[]</span> <span class="o">=</span> <span class="k">new</span> <span class="kt">byte</span><span class="o">[</span><span class="mi">1024</span><span class="o">];</span> |
| |
| <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">rm</span><span class="o">.</span><span class="na">getBodyLength</span><span class="o">();</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">1024</span><span class="o">)</span> |
| <span class="o">{</span> |
| <span class="kt">int</span> <span class="n">numberOfBytes</span> <span class="o">=</span> <span class="n">rm</span><span class="o">.</span><span class="na">readBytes</span><span class="o">(</span><span class="n">data</span><span class="o">);</span> |
| <span class="c1">// Do whatever you want with the data</span> |
| <span class="o">}</span></code></pre> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="configuring-amqp-acceptor"><a class="anchor" href="#configuring-amqp-acceptor"></a><a class="link" href="#configuring-amqp-acceptor">5. Configuring AMQP Acceptor</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>You can configure the property <code>amqpMinLargeMessageSize</code> at the acceptor.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The default value is 102400 (100KBytes).</p> |
| </div> |
| <div class="paragraph"> |
| <p>Setting it to -1 will disable large message support.</p> |
| </div> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| setting amqpMinLargeMessageSize to -1, your AMQP message might be stored as a Core Large Message if the size of the message does not fit into the journal. |
| This is the former semantic of the broker and it is kept this way for compatibility reasons. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><acceptors></span> |
| <span class="c"><!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.--></span> |
| <span class="nt"><acceptor</span> <span class="na">name=</span><span class="s">"amqp"</span><span class="nt">></span>tcp://0.0.0.0:5672?; ..... amqpMinLargeMessageSize=102400; .... <span class="nt"></acceptor></span> |
| <span class="nt"></acceptors></span></code></pre> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="large-message-example"><a class="anchor" href="#large-message-example"></a><a class="link" href="#large-message-example">6. Large message example</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Please see the <a href="examples.html#large-message">Large Message Example</a> which shows how large messages are configured and used with JMS.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </body> |
| </html> |
