| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="generator" content="Asciidoctor 2.0.18"> |
| <link rel="icon" type="image/png" href="images/favicon.png"> |
| <title>Clusters</title> |
| <link rel="stylesheet" href="css/asciidoctor.css"> |
| <link rel="stylesheet" href="css/font-awesome.css"> |
| <link rel="stylesheet" href="css/rouge-github.css"> |
| </head> |
| <body class="book toc2 toc-left"> |
| <div id="header"> |
| <h1>Clusters</h1> |
| <div id="toc" class="toc2"> |
| <div id="toctitle"><a href="index.html">User Manual for 2.33.0</a></div> |
| <ul class="sectlevel1"> |
| <li><a href="#overview">1. Overview</a></li> |
| <li><a href="#performance-considerations">2. Performance Considerations</a></li> |
| <li><a href="#server-discovery">3. Server discovery</a> |
| <ul class="sectlevel2"> |
| <li><a href="#dynamic-discovery">3.1. Dynamic Discovery</a></li> |
| <li><a href="#discovery-using-static-connectors">3.2. Discovery using static Connectors</a></li> |
| </ul> |
| </li> |
| <li><a href="#server-side-message-load-balancing">4. Server-Side Message Load Balancing</a> |
| <ul class="sectlevel2"> |
| <li><a href="#configuring-cluster-connections">4.1. Configuring Cluster Connections</a></li> |
| <li><a href="#cluster-user-credentials">4.2. Cluster User Credentials</a></li> |
| </ul> |
| </li> |
| <li><a href="#client-side-load-balancing">5. Client-Side Load balancing</a></li> |
| <li><a href="#specifying-members-of-a-cluster-explicitly">6. Specifying Members of a Cluster Explicitly</a></li> |
| <li><a href="#message-redistribution">7. Message Redistribution</a></li> |
| <li><a href="#cluster-topologies">8. Cluster topologies</a> |
| <ul class="sectlevel2"> |
| <li><a href="#symmetric-cluster">8.1. Symmetric cluster</a></li> |
| <li><a href="#chain-cluster">8.2. Chain cluster</a></li> |
| <li><a href="#scaling-down">8.3. Scaling Down</a></li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div id="content"> |
| <div class="sect1"> |
| <h2 id="overview"><a class="anchor" href="#overview"></a><a class="link" href="#overview">1. Overview</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Apache ActiveMQ Artemis clusters allow groups of Apache ActiveMQ Artemis servers to be grouped together in order to share message processing load. |
| Each active node in the cluster is an active Apache ActiveMQ Artemis server which manages its own messages and handles its own connections.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The cluster is formed by each node declaring <em>cluster connections</em> to other nodes in the core configuration file <code>broker.xml</code>. |
| When a node forms a cluster connection to another node, internally it creates a <em>core bridge</em> (as described in <a href="core-bridges.html#core-bridges">Core Bridges</a>) connection between it and the other node, this is done transparently behind the scenes - you don’t have to declare an explicit bridge for each node. |
| These cluster connections allow messages to flow between the nodes of the cluster to balance load.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Nodes can be connected together to form a cluster in many different topologies, we will discuss a couple of the more common topologies later in this chapter.</p> |
| </div> |
| <div class="paragraph"> |
| <p>We’ll also discuss client side load balancing, where we can balance client connections across the nodes of the cluster, and we’ll consider message redistribution where Apache ActiveMQ Artemis will redistribute messages between nodes to avoid starvation.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Another important part of clustering is <em>server discovery</em> where servers can broadcast their connection details so clients or other servers can connect to them with the minimum of configuration.</p> |
| </div> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| <div id="copy-warning" class="paragraph"> |
| <p>Once a cluster node has been configured it is common to simply copy that configuration to other nodes to produce a symmetric cluster. |
| However, care must be taken when copying the Apache ActiveMQ Artemis files. |
| Do not copy the Apache ActiveMQ Artemis <em>data</em> (i.e. the <code>bindings</code>, <code>journal</code>, <code>paging</code>, and <code>large-messages</code> directories) from one node to another. |
| When a node is started for the first time and initializes its journal files it also persists a special identifier to the <code>journal</code> directory. |
| This id <em>must</em> be unique among nodes in the cluster or the cluster will not form properly.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="performance-considerations"><a class="anchor" href="#performance-considerations"></a><a class="link" href="#performance-considerations">2. Performance Considerations</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>It is important to note that while the goal of clustering is to increase overall message throughput via horizontal scaling it is not a "silver bullet." In certain situations clustering can, in fact, <em>reduce</em> message throughput so care must be taken when choosing a clustered configuration. |
| Here’s a few general guidelines:</p> |
| </div> |
| <div class="olist arabic"> |
| <ol class="arabic"> |
| <li> |
| <p><strong>Establish a clear, concrete performance goal.</strong> Performance testing & tuning are often difficult and tedious activities. |
| Small, relative gains will tempt you to keep going, and without a goal you will never know when to stop. |
| You need a goal to know "how good is good enough."</p> |
| </li> |
| <li> |
| <p><strong>Start simple.</strong> Benchmark your use-case with a single broker first. |
| A single broker can handle <em>millions</em> of messages per second in certain use-cases. |
| If you can’t meet your performance goal with a single broker only then move to a clustered configuration. |
| Only add complexity when there is a <em>clear benefit</em>.</p> |
| </li> |
| </ol> |
| </div> |
| <div class="paragraph"> |
| <p>The main way a cluster can reduce overall message throughput is if there are are not enough producers & consumers on each node leading to message build-up on some nodes and consumer starvation others. |
| The cluster has mechanisms to deal with this (i.e. message load-balancing & redistribution, which will be covered later), but you really don’t want the broker to intervene and move messages between nodes unless absolutely necessary because <strong>that adds latency</strong>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Therefore, when thinking in performance terms the main question one must answer when choosing a clustered configuration is: Do I have enough clients so that each node in the cluster has sufficient consumers to receive all the messages produced on that node? |
| If the answer to that question is "yes" then clustering may, in fact, improve overall message throughput for you. |
| If the answer to that question is "no" then you’re likely to get better performance from either a smaller cluster or just a single broker.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Also keep in mind that a <a href="connection-routers.html#connection-routers">connection router</a> may improve performance of your cluster by grouping related consumers and producers together on the same node.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="server-discovery"><a class="anchor" href="#server-discovery"></a><a class="link" href="#server-discovery">3. Server discovery</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Server discovery is a mechanism by which servers can propagate their connection details to:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Messaging clients. |
| A messaging client wants to be able to connect to the servers of the cluster without having specific knowledge of which servers in the cluster are up at any one time.</p> |
| </li> |
| <li> |
| <p>Other servers. |
| Servers in a cluster want to be able to create cluster connections to each other without having prior knowledge of all the other servers in the cluster.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>This information, let’s call it the Cluster Topology, is actually sent around normal Apache ActiveMQ Artemis connections to clients and to other servers over cluster connections. |
| This being the case we need a way of establishing the initial first connection. |
| This can be done using dynamic discovery techniques like <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a> and <a href="http://www.jgroups.org/">JGroups</a>, or by providing a list of initial connectors.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="dynamic-discovery"><a class="anchor" href="#dynamic-discovery"></a><a class="link" href="#dynamic-discovery">3.1. Dynamic Discovery</a></h3> |
| <div class="paragraph"> |
| <p>Server discovery uses <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a> multicast or <a href="http://www.jgroups.org/">JGroups</a> to broadcast server connection settings.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="broadcast-groups"><a class="anchor" href="#broadcast-groups"></a><a class="link" href="#broadcast-groups">3.1.1. Broadcast Groups</a></h4> |
| <div class="paragraph"> |
| <p>A broadcast group is the means by which a server broadcasts connectors over the network. |
| A connector defines a way in which a client (or other server) can make connections to the server. |
| For more information on what a connector is, please see <a href="configuring-transports.html#configuring-the-transport">Configuring the Transport</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The broadcast group takes a connector and broadcasts it on the network. |
| Depending on which broadcasting technique you configure, it uses either UDP or JGroups to broadcast connector information.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Broadcast groups are defined in the server configuration file <code>broker.xml</code>. |
| There can be many broadcast groups per Apache ActiveMQ Artemis server. |
| All broadcast groups must be defined in a <code>broadcast-groups</code> element.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Let’s take a look at an example broadcast group from <code>broker.xml</code> that defines a UDP broadcast group:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><broadcast-groups></span> |
| <span class="nt"><broadcast-group</span> <span class="na">name=</span><span class="s">"my-broadcast-group"</span><span class="nt">></span> |
| <span class="nt"><local-bind-address></span>172.16.9.3<span class="nt"></local-bind-address></span> |
| <span class="nt"><local-bind-port></span>5432<span class="nt"></local-bind-port></span> |
| <span class="nt"><group-address></span>231.7.7.7<span class="nt"></group-address></span> |
| <span class="nt"><group-port></span>9876<span class="nt"></group-port></span> |
| <span class="nt"><broadcast-period></span>2000<span class="nt"></broadcast-period></span> |
| <span class="nt"><connector-ref></span>netty-connector<span class="nt"></connector-ref></span> |
| <span class="nt"></broadcast-group></span> |
| <span class="nt"></broadcast-groups></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Some of the broadcast group parameters are optional and you’ll normally use the defaults, but we specify them all in the above example for clarity. |
| Let’s discuss each one in turn:</p> |
| </div> |
| <div class="dlist"> |
| <dl> |
| <dt class="hdlist1">name</dt> |
| <dd> |
| <p> attribute. |
| Each broadcast group in the server must have a unique name.</p> |
| </dd> |
| <dt class="hdlist1">local-bind-address</dt> |
| <dd> |
| <p>This is the local bind address that the datagram socket is bound to. |
| If you have multiple network interfaces on your server, you would specify which one you wish to use for broadcasts by setting this property. |
| If this property is not specified then the socket will be bound to the wildcard address, an IP address chosen by the kernel. |
| This is a UDP specific attribute.</p> |
| </dd> |
| <dt class="hdlist1">local-bind-port</dt> |
| <dd> |
| <p>If you want to specify a local port to which the datagram socket is bound you can specify it here. |
| Normally you would just use the default value of <code>-1</code> which signifies that an anonymous port should be used. |
| This parameter is always specified in conjunction with <code>local-bind-address</code>. |
| This is a UDP specific attribute.</p> |
| </dd> |
| <dt class="hdlist1">group-address</dt> |
| <dd> |
| <p>This is the multicast address to which the data will be broadcast. |
| It is a class D IP address in the range <code>224.0.0.0</code> to <code>239.255.255.255</code>, inclusive. |
| The address <code>224.0.0.0</code> is reserved and is not available for use. |
| This parameter is mandatory. |
| This is a UDP specific attribute.</p> |
| </dd> |
| <dt class="hdlist1">group-port</dt> |
| <dd> |
| <p>This is the UDP port number used for broadcasting. |
| This parameter is mandatory. |
| This is a UDP specific attribute.</p> |
| </dd> |
| <dt class="hdlist1">broadcast-period</dt> |
| <dd> |
| <p>This is the period in milliseconds between consecutive broadcasts. |
| This parameter is optional, the default value is <code>2000</code> milliseconds.</p> |
| </dd> |
| <dt class="hdlist1">connector-ref</dt> |
| <dd> |
| <p>This specifies the connector and optional backup connector that will be broadcasted (see <a href="configuring-transports.html#configuring-the-transport">Configuring the Transport</a> for more information on connectors).</p> |
| </dd> |
| </dl> |
| </div> |
| <div class="paragraph"> |
| <p>Here is another example broadcast group that defines a JGroups broadcast group:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><broadcast-groups></span> |
| <span class="nt"><broadcast-group</span> <span class="na">name=</span><span class="s">"my-broadcast-group"</span><span class="nt">></span> |
| <span class="nt"><broadcast-period></span>2000<span class="nt"></broadcast-period></span> |
| <span class="nt"><jgroups-file></span>test-jgroups-file_ping.xml<span class="nt"></jgroups-file></span> |
| <span class="nt"><jgroups-channel></span>activemq_broadcast_channel<span class="nt"></jgroups-channel></span> |
| <span class="nt"><connector-ref></span>netty-connector<span class="nt"></connector-ref></span> |
| <span class="nt"></broadcast-group></span> |
| <span class="nt"></broadcast-groups></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>To be able to use JGroups to broadcast, one must specify two attributes, i.e. <code>jgroups-file</code> and <code>jgroups-channel</code>, as discussed in details as following:</p> |
| </div> |
| <div class="dlist"> |
| <dl> |
| <dt class="hdlist1">jgroups-file</dt> |
| <dd> |
| <p> attribute. |
| This is the name of JGroups configuration file. |
| It will be used to initialize JGroups channels. |
| Make sure the file is in the java resource path so that Apache ActiveMQ Artemis can load it. |
| The typical location for the file is the <code>etc</code> directory from the broker instance.</p> |
| </dd> |
| <dt class="hdlist1">jgroups-channel</dt> |
| <dd> |
| <p> attribute. |
| The name that JGroups channels connect to for broadcasting.</p> |
| </dd> |
| </dl> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>The JGroups attributes (<code>jgroups-file</code> and <code>jgroups-channel</code>) and UDP specific attributes described above are exclusive of each other. |
| Only one set can be specified in a broadcast group configuration. |
| Don’t mix them!</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>The following is an example of a JGroups file</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><config</span> <span class="na">xmlns:xsi=</span><span class="s">"http://www.w3.org/2001/XMLSchema-instance"</span> |
| <span class="na">xmlns=</span><span class="s">"urn:org:jgroups"</span> |
| <span class="na">xsi:schemaLocation=</span><span class="s">"urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd"</span><span class="nt">></span> |
| <span class="nt"><TCP</span> <span class="na">bind_addr=</span><span class="s">"${jgroups.bind_addr:site_local}"</span> |
| <span class="na">bind_port=</span><span class="s">"${jgroups.bind_port:7800}"</span> |
| <span class="na">external_addr=</span><span class="s">"${jgroups.external_addr}"</span> |
| <span class="na">external_port=</span><span class="s">"${jgroups.external_port}"</span> |
| <span class="na">thread_pool.min_threads=</span><span class="s">"0"</span> |
| <span class="na">thread_pool.max_threads=</span><span class="s">"200"</span> |
| <span class="na">thread_pool.keep_alive_time=</span><span class="s">"30000"</span><span class="nt">/></span> |
| <span class="nt"><RED/></span> |
| |
| <span class="c"><!-- a location that can be found by both server's running --></span> |
| <span class="nt"><FILE_PING</span> <span class="na">location=</span><span class="s">"../file.ping.dir"</span><span class="nt">/></span> |
| <span class="nt"><MERGE3</span> <span class="na">min_interval=</span><span class="s">"10000"</span> |
| <span class="na">max_interval=</span><span class="s">"30000"</span><span class="nt">/></span> |
| <span class="nt"><FD_SOCK2/></span> |
| <span class="nt"><FD_ALL3</span> <span class="na">timeout=</span><span class="s">"40000"</span> <span class="na">interval=</span><span class="s">"5000"</span> <span class="nt">/></span> |
| <span class="nt"><VERIFY_SUSPECT2</span> <span class="na">timeout=</span><span class="s">"1500"</span> <span class="nt">/></span> |
| <span class="nt"><BARRIER</span> <span class="nt">/></span> |
| <span class="nt"><pbcast.NAKACK2</span> <span class="na">use_mcast_xmit=</span><span class="s">"false"</span> <span class="nt">/></span> |
| <span class="nt"><UNICAST3</span> <span class="nt">/></span> |
| <span class="nt"><pbcast.STABLE</span> <span class="na">desired_avg_gossip=</span><span class="s">"50000"</span> |
| <span class="na">max_bytes=</span><span class="s">"4M"</span><span class="nt">/></span> |
| <span class="nt"><pbcast.GMS</span> <span class="na">print_local_addr=</span><span class="s">"true"</span> <span class="na">join_timeout=</span><span class="s">"2000"</span><span class="nt">/></span> |
| <span class="nt"><UFC</span> <span class="na">max_credits=</span><span class="s">"2M"</span> |
| <span class="na">min_threshold=</span><span class="s">"0.4"</span><span class="nt">/></span> |
| <span class="nt"><MFC</span> <span class="na">max_credits=</span><span class="s">"2M"</span> |
| <span class="na">min_threshold=</span><span class="s">"0.4"</span><span class="nt">/></span> |
| <span class="nt"><FRAG2</span> <span class="na">frag_size=</span><span class="s">"60K"</span> <span class="nt">/></span> |
| <span class="c"><!--RSVP resend_interval="2000" timeout="10000"/--></span> |
| <span class="nt"><pbcast.STATE_TRANSFER/></span> |
| <span class="nt"></config></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>As it shows, the file content defines a jgroups protocol stacks. |
| If you want Apache ActiveMQ Artemis to use this stacks for channel creation, you have to make sure the value of <code>jgroups-file</code> in your broadcast-group/discovery-group configuration to be the name of this jgroups configuration file. |
| For example if the above stacks configuration is stored in a file named "jgroups-stacks.xml" then your <code>jgroups-file</code> should be like</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><jgroups-file></span>jgroups-stacks.xml<span class="nt"></jgroups-file></span></code></pre> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="discovery-groups"><a class="anchor" href="#discovery-groups"></a><a class="link" href="#discovery-groups">3.1.2. Discovery Groups</a></h4> |
| <div class="paragraph"> |
| <p>While the broadcast group defines how connector information is broadcasted from a server, a discovery group defines how connector information is received from a broadcast endpoint (a UDP multicast address or JGroup channel).</p> |
| </div> |
| <div class="paragraph"> |
| <p>A discovery group maintains a list of connector pairs - one for each broadcast by a different server. |
| As it receives broadcasts on the broadcast endpoint from a particular server it updates its entry in the list for that server.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If it has not received a broadcast from a particular server for a length of time it will remove that server’s entry from its list.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Discovery groups are used in two places in Apache ActiveMQ Artemis:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>By cluster connections so they know how to obtain an initial connection to download the topology</p> |
| </li> |
| <li> |
| <p>By messaging clients so they know how to obtain an initial connection to download the topology</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Although a discovery group will always accept broadcasts, its current list of available primary and backup servers is only ever used when an initial connection is made, from then server discovery is done over the normal Apache ActiveMQ Artemis connections.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>Each discovery group must be configured with broadcast endpoint (UDP or JGroups) that matches its broadcast group counterpart. |
| For example, if broadcast is configured using UDP, the discovery group must also use UDP, and the same multicast address.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="defining-discovery-groups-on-the-server"><a class="anchor" href="#defining-discovery-groups-on-the-server"></a><a class="link" href="#defining-discovery-groups-on-the-server">3.1.3. Defining Discovery Groups on the Server</a></h4> |
| <div class="paragraph"> |
| <p>For cluster connections, discovery groups are defined in the server side configuration file <code>broker.xml</code>. |
| All discovery groups must be defined inside a <code>discovery-groups</code> element. |
| There can be many discovery groups defined by Apache ActiveMQ Artemis server. |
| Let’s look at an example:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><discovery-groups></span> |
| <span class="nt"><discovery-group</span> <span class="na">name=</span><span class="s">"my-discovery-group"</span><span class="nt">></span> |
| <span class="nt"><local-bind-address></span>172.16.9.7<span class="nt"></local-bind-address></span> |
| <span class="nt"><group-address></span>231.7.7.7<span class="nt"></group-address></span> |
| <span class="nt"><group-port></span>9876<span class="nt"></group-port></span> |
| <span class="nt"><refresh-timeout></span>10000<span class="nt"></refresh-timeout></span> |
| <span class="nt"></discovery-group></span> |
| <span class="nt"></discovery-groups></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>We’ll consider each parameter of the discovery group:</p> |
| </div> |
| <div class="dlist"> |
| <dl> |
| <dt class="hdlist1">name</dt> |
| <dd> |
| <p> attribute. |
| Each discovery group must have a unique name per server.</p> |
| </dd> |
| <dt class="hdlist1">local-bind-address</dt> |
| <dd> |
| <p>If you are running with multiple network interfaces on the same machine, you may want to specify that the discovery group listens only a specific interface. |
| To do this you can specify the interface address with this parameter. |
| This parameter is optional. |
| This is a UDP specific attribute.</p> |
| </dd> |
| <dt class="hdlist1">group-address</dt> |
| <dd> |
| <p>This is the multicast IP address of the group to listen on. |
| It should match the <code>group-address</code> in the broadcast group that you wish to listen from. |
| This parameter is mandatory. |
| This is a UDP specific attribute.</p> |
| </dd> |
| <dt class="hdlist1">group-port</dt> |
| <dd> |
| <p>This is the UDP port of the multicast group. |
| It should match the <code>group-port</code> in the broadcast group that you wish to listen from. |
| This parameter is mandatory. |
| This is a UDP specific attribute.</p> |
| </dd> |
| <dt class="hdlist1">refresh-timeout</dt> |
| <dd> |
| <p>This is the period the discovery group waits after receiving the last broadcast from a particular server before removing that servers connector pair entry from its list. |
| You would normally set this to a value significantly higher than the <code>broadcast-period</code> on the broadcast group otherwise servers might intermittently disappear from the list even though they are still broadcasting due to slight differences in timing. |
| This parameter is optional, the default value is <code>10000</code> milliseconds (10 seconds).</p> |
| </dd> |
| </dl> |
| </div> |
| <div class="paragraph"> |
| <p>Here is another example that defines a JGroups discovery group:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><discovery-groups></span> |
| <span class="nt"><discovery-group</span> <span class="na">name=</span><span class="s">"my-broadcast-group"</span><span class="nt">></span> |
| <span class="nt"><jgroups-file></span>test-jgroups-file_ping.xml<span class="nt"></jgroups-file></span> |
| <span class="nt"><jgroups-channel></span>activemq_broadcast_channel<span class="nt"></jgroups-channel></span> |
| <span class="nt"><refresh-timeout></span>10000<span class="nt"></refresh-timeout></span> |
| <span class="nt"></discovery-group></span> |
| <span class="nt"></discovery-groups></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>To receive broadcast from JGroups channels, one must specify two attributes, <code>jgroups-file</code> and <code>jgroups-channel</code>, as discussed in details as following:</p> |
| </div> |
| <div class="dlist"> |
| <dl> |
| <dt class="hdlist1">jgroups-file</dt> |
| <dd> |
| <p> attribute. |
| This is the name of JGroups configuration file. |
| It will be used to initialize JGroups channels. |
| Make sure the file is in the java resource path so that Apache ActiveMQ Artemis can load it.</p> |
| </dd> |
| <dt class="hdlist1">jgroups-channel</dt> |
| <dd> |
| <p> attribute. |
| The name that JGroups channels connect to for receiving broadcasts.</p> |
| </dd> |
| </dl> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>The JGroups attributes (<code>jgroups-file</code> and <code>jgroups-channel</code>) and UDP specific attributes described above are exclusive of each other. |
| Only one set can be specified in a discovery group configuration. |
| Don’t mix them!</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="discovery-groups-on-the-client-side"><a class="anchor" href="#discovery-groups-on-the-client-side"></a><a class="link" href="#discovery-groups-on-the-client-side">3.1.4. Discovery Groups on the Client Side</a></h4> |
| <div class="paragraph"> |
| <p>Let’s discuss how to configure an Apache ActiveMQ Artemis client to use discovery to discover a list of servers to which it can connect. |
| The way to do this differs depending on whether you’re using JMS or the core API.</p> |
| </div> |
| <div class="sect4"> |
| <h5 id="configuring-client-discovery"><a class="anchor" href="#configuring-client-discovery"></a><a class="link" href="#configuring-client-discovery">Configuring client discovery</a></h5> |
| <div class="paragraph"> |
| <p>Use the <code>udp</code> URL scheme and a host:port combination matches the group-address and group-port from the corresponding <code>broadcast-group</code> on the server:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="nowrap">udp://231.7.7.7:9876</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The element <code>discovery-group-ref</code> specifies the name of a discovery group defined in <code>broker.xml</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Connections created using this URI will be load-balanced across the list of servers that the discovery group maintains by listening on the multicast address specified in the discovery group configuration.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The aforementioned <code>refreshTimeout</code> parameter can be set directly in the URI.</p> |
| </div> |
| <div class="paragraph"> |
| <p>There is also a URL parameter named <code>initialWaitTimeout</code>. |
| If the corresponding JMS connection factory or core session factory is used immediately after creation then it may not have had enough time to received broadcasts from all the nodes in the cluster. |
| On first usage, the connection factory will make sure it waits this long since creation before creating the first connection. |
| The default value for this parameter is <code>10000</code> milliseconds.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="discovery-using-static-connectors"><a class="anchor" href="#discovery-using-static-connectors"></a><a class="link" href="#discovery-using-static-connectors">3.2. Discovery using static Connectors</a></h3> |
| <div class="paragraph"> |
| <p>Sometimes it may be impossible to use UDP on the network you are using. |
| In this case its possible to configure a connection with an initial list of possible servers. |
| This could be just one server that you know will always be available or a list of servers where at least one will be available.</p> |
| </div> |
| <div class="paragraph"> |
| <p>This doesn’t mean that you have to know where all your servers are going to be hosted, you can configure these servers to use the reliable servers to connect to. |
| Once they are connected their connection details will be propagated via the server it connects to</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="configuring-a-cluster-connection"><a class="anchor" href="#configuring-a-cluster-connection"></a><a class="link" href="#configuring-a-cluster-connection">3.2.1. Configuring a Cluster Connection</a></h4> |
| <div class="paragraph"> |
| <p>For cluster connections there is no extra configuration needed, you just need to make sure that any connectors are defined in the usual manner, (see <a href="configuring-transports.html#configuring-the-transport">Configuring the Transport</a> for more information on connectors). |
| These are then referenced by the cluster connection configuration.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="configuring-a-client-connection"><a class="anchor" href="#configuring-a-client-connection"></a><a class="link" href="#configuring-a-client-connection">3.2.2. Configuring a Client Connection</a></h4> |
| <div class="paragraph"> |
| <p>A static list of possible servers can also be used by a normal client.</p> |
| </div> |
| <div class="sect4"> |
| <h5 id="configuring-client-discovery-2"><a class="anchor" href="#configuring-client-discovery-2"></a><a class="link" href="#configuring-client-discovery-2">Configuring client discovery</a></h5> |
| <div class="paragraph"> |
| <p>A list of servers to be used for the initial connection attempt can be specified in the connection URI using a syntax with <code>()</code>, e.g.:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="nowrap">(tcp://myhost:61616,tcp://myhost2:61616)?reconnectAttempts=5</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The brackets are expanded so the same query can be appended after the last bracket for ease.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="server-side-message-load-balancing"><a class="anchor" href="#server-side-message-load-balancing"></a><a class="link" href="#server-side-message-load-balancing">4. Server-Side Message Load Balancing</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>If cluster connections are defined between nodes of a cluster, then Apache ActiveMQ Artemis will load balance messages arriving at a particular node from a client.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Let’s take a simple example of a cluster of four nodes A, B, C, and D arranged in a <em>symmetric cluster</em> (described in Symmetrical Clusters section). |
| We have a queue called <code>OrderQueue</code> deployed on each node of the cluster.</p> |
| </div> |
| <div class="paragraph"> |
| <p>We have client Ca connected to node A, sending orders to the server. |
| We have also have order processor clients Pa, Pb, Pc, and Pd connected to each of the nodes A, B, C, D. |
| If no cluster connection was defined on node A, then as order messages arrive on node A they will all end up in the <code>OrderQueue</code> on node A, so will only get consumed by the order processor client attached to node A, Pa.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If we define a cluster connection on node A, then as ordered messages arrive on node A instead of all of them going into the local <code>OrderQueue</code> instance, they are distributed in a round-robin fashion between all the nodes of the cluster. |
| The messages are forwarded from the receiving node to other nodes of the cluster. |
| This is all done on the server side, the client maintains a single connection to node A.</p> |
| </div> |
| <div class="paragraph"> |
| <p>For example, messages arriving on node A might be distributed in the following order between the nodes: B, D, C, A, B, D, C, A, B, D. |
| The exact order depends on the order the nodes started up, but the algorithm used is round robin.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Apache ActiveMQ Artemis cluster connections can be configured to always blindly load balance messages in a round robin fashion irrespective of whether there are any matching consumers on other nodes, but they can be a bit cleverer than that and also be configured to only distribute to other nodes if they have matching consumers. |
| We’ll look at both these cases in turn with some examples, but first we’ll discuss configuring cluster connections in general.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="configuring-cluster-connections"><a class="anchor" href="#configuring-cluster-connections"></a><a class="link" href="#configuring-cluster-connections">4.1. Configuring Cluster Connections</a></h3> |
| <div class="paragraph"> |
| <p>Cluster connections group servers into clusters so that messages can be load balanced between the nodes of the cluster. |
| Let’s take a look at a typical cluster connection. |
| Cluster connections are always defined in <code>broker.xml</code> inside a <code>cluster-connection</code> element. |
| There can be zero or more cluster connections defined per Apache ActiveMQ Artemis server.</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><cluster-connections></span> |
| <span class="nt"><cluster-connection</span> <span class="na">name=</span><span class="s">"my-cluster"</span><span class="nt">></span> |
| <span class="nt"><address></address></span> |
| <span class="nt"><connector-ref></span>netty-connector<span class="nt"></connector-ref></span> |
| <span class="nt"><check-period></span>1000<span class="nt"></check-period></span> |
| <span class="nt"><connection-ttl></span>5000<span class="nt"></connection-ttl></span> |
| <span class="nt"><min-large-message-size></span>50000<span class="nt"></min-large-message-size></span> |
| <span class="nt"><call-timeout></span>5000<span class="nt"></call-timeout></span> |
| <span class="nt"><retry-interval></span>500<span class="nt"></retry-interval></span> |
| <span class="nt"><retry-interval-multiplier></span>1.0<span class="nt"></retry-interval-multiplier></span> |
| <span class="nt"><max-retry-interval></span>5000<span class="nt"></max-retry-interval></span> |
| <span class="nt"><initial-connect-attempts></span>-1<span class="nt"></initial-connect-attempts></span> |
| <span class="nt"><reconnect-attempts></span>-1<span class="nt"></reconnect-attempts></span> |
| <span class="nt"><use-duplicate-detection></span>true<span class="nt"></use-duplicate-detection></span> |
| <span class="nt"><message-load-balancing></span>ON_DEMAND<span class="nt"></message-load-balancing></span> |
| <span class="nt"><max-hops></span>1<span class="nt"></max-hops></span> |
| <span class="nt"><confirmation-window-size></span>32000<span class="nt"></confirmation-window-size></span> |
| <span class="nt"><call-failover-timeout></span>30000<span class="nt"></call-failover-timeout></span> |
| <span class="nt"><notification-interval></span>1000<span class="nt"></notification-interval></span> |
| <span class="nt"><notification-attempts></span>2<span class="nt"></notification-attempts></span> |
| <span class="nt"><discovery-group-ref</span> <span class="na">discovery-group-name=</span><span class="s">"my-discovery-group"</span><span class="nt">/></span> |
| <span class="nt"></cluster-connection></span> |
| <span class="nt"></cluster-connections></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the above cluster connection all parameters have been explicitly specified. |
| The following shows all the available configuration options</p> |
| </div> |
| <div class="dlist"> |
| <dl> |
| <dt class="hdlist1">address</dt> |
| <dd> |
| <p>Each cluster connection only applies to addresses that match the specified <code>address</code> field. |
| An address is matched on the cluster connection when it begins with the string specified in this field. |
| The <code>address</code> field on a cluster connection also supports comma separated lists and an exclude syntax <code>!</code>. |
| To prevent an address from being matched on this cluster connection, prepend a cluster connection address string with <code>!</code>.</p> |
| <div class="paragraph"> |
| <p>In the case shown above the cluster connection will load balance messages sent to all addresses (since it’s empty).</p> |
| </div> |
| <div class="paragraph"> |
| <p>The address can be any value and you can have many cluster connections with different values of <code>address</code>, simultaneously balancing messages for those addresses, potentially to different clusters of servers. |
| By having multiple cluster connections on different addresses a single Apache ActiveMQ Artemis Server can effectively take part in multiple clusters simultaneously.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Be careful not to have multiple cluster connections with overlapping values of <code>address</code>, e.g. "europe" and "europe.news" since this could result in the same messages being distributed between more than one cluster connection, possibly resulting in duplicate deliveries.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Examples:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>'eu' matches all addresses starting with 'eu'</p> |
| </li> |
| <li> |
| <p>'!eu' matches all address except for those starting with 'eu'</p> |
| </li> |
| <li> |
| <p>'eu.uk,eu.de' matches all addresses starting with either 'eu.uk' or 'eu.de'</p> |
| </li> |
| <li> |
| <p>'eu,!eu.uk' matches all addresses starting with 'eu' but not those starting with 'eu.uk'</p> |
| </li> |
| </ul> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Address exclusion will always takes precedence over address inclusion.</p> |
| </li> |
| <li> |
| <p>Address matching on cluster connections does not support wild-card matching.</p> |
| </li> |
| </ul> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| </dd> |
| <dt class="hdlist1">connector-ref</dt> |
| <dd> |
| <p>This is the connector which will be sent to other nodes in the cluster so they have the correct cluster topology.</p> |
| <div class="paragraph"> |
| <p>This parameter is mandatory.</p> |
| </div> |
| </dd> |
| <dt class="hdlist1">check-period</dt> |
| <dd> |
| <p>The period (in milliseconds) used to check if the cluster connection has failed to receive pings from another server. |
| Default is 30000.</p> |
| </dd> |
| <dt class="hdlist1">connection-ttl</dt> |
| <dd> |
| <p>This is how long a cluster connection should stay alive if it stops receiving messages from a specific node in the cluster. |
| Default is 60000.</p> |
| </dd> |
| <dt class="hdlist1">min-large-message-size</dt> |
| <dd> |
| <p>If the message size (in bytes) is larger than this value then it will be split into multiple segments when sent over the network to other cluster members. |
| Default is 102400.</p> |
| </dd> |
| <dt class="hdlist1">call-timeout</dt> |
| <dd> |
| <p>When a packet is sent via a cluster connection and is a blocking call, i.e. for acknowledgements, this is how long it will wait (in milliseconds) for the reply before throwing an exception. |
| Default is 30000.</p> |
| </dd> |
| <dt class="hdlist1">retry-interval</dt> |
| <dd> |
| <p>We mentioned before that, internally, cluster connections cause bridges to be created between the nodes of the cluster. |
| If the cluster connection is created and the target node has not been started, or say, is being rebooted, then the cluster connections from other nodes will retry connecting to the target until it comes back up, in the same way as a bridge does.</p> |
| <div class="paragraph"> |
| <p>This parameter determines the interval in milliseconds between retry attempts. |
| It has the same meaning as the <code>retry-interval</code> on a bridge (as described in <a href="core-bridges.html#core-bridges">Core Bridges</a>).</p> |
| </div> |
| <div class="paragraph"> |
| <p>This parameter is optional and its default value is <code>500</code> milliseconds.</p> |
| </div> |
| </dd> |
| <dt class="hdlist1">retry-interval-multiplier</dt> |
| <dd> |
| <p>This is a multiplier used to increase the <code>retry-interval</code> after each reconnect attempt, default is 1.</p> |
| </dd> |
| <dt class="hdlist1">max-retry-interval</dt> |
| <dd> |
| <p>The maximum delay (in milliseconds) for retries. |
| Default is 2000.</p> |
| </dd> |
| <dt class="hdlist1">initial-connect-attempts</dt> |
| <dd> |
| <p>The number of times the system will try to connect a node in the cluster initially. |
| If the max-retry is achieved this node will be considered permanently down and the system will not route messages to this node. |
| Default is -1 (infinite retries).</p> |
| </dd> |
| <dt class="hdlist1">reconnect-attempts</dt> |
| <dd> |
| <p>The number of times the system will try to reconnect to a node in the cluster. |
| If the max-retry is achieved this node will be considered permanently down and the system will stop routing messages to this node. |
| Default is -1 (infinite retries).</p> |
| </dd> |
| <dt class="hdlist1">use-duplicate-detection</dt> |
| <dd> |
| <p>Internally cluster connections use bridges to link the nodes, and bridges can be configured to add a duplicate id property in each message that is forwarded. |
| If the target node of the bridge crashes and then recovers, messages might be resent from the source node. |
| By enabling duplicate detection any duplicate messages will be filtered out and ignored on receipt at the target node.</p> |
| <div class="paragraph"> |
| <p>This parameter has the same meaning as <code>use-duplicate-detection</code> on a bridge. |
| For more information on duplicate detection, please see <a href="duplicate-detection.html#duplicate-message-detection">Duplicate Detection</a>. |
| Default is <code>true</code>.</p> |
| </div> |
| </dd> |
| <dt class="hdlist1">message-load-balancing</dt> |
| <dd> |
| <p>This parameter determines if/how messages will be distributed between other nodes of the cluster. |
| It can be one of four values - <code>OFF</code>, <code>STRICT</code>, <code>OFF_WITH_REDISTRIBUTION</code> or <code>ON_DEMAND</code> (default). |
| This parameter replaces the deprecated <code>forward-when-no-consumers</code> parameter.</p> |
| <div class="paragraph"> |
| <p>If this is set to <code>OFF</code> then messages will never be forwarded to another node in the cluster</p> |
| </div> |
| <div class="paragraph"> |
| <p>If this is set to <code>STRICT</code> then each incoming message will be round robin’d even though the same queues on the other nodes of the cluster may have no consumers at all, or they may have consumers that have non matching message filters (selectors). |
| Note that Apache ActiveMQ Artemis will <em>not</em> forward messages to other nodes if there are no <em>queues</em> of the same name on the other nodes, even if this parameter is set to <code>STRICT</code>. |
| Using <code>STRICT</code> is like setting the legacy <code>forward-when-no-consumers</code> parameter to <code>true</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If this is set to <code>ON_DEMAND</code> then Apache ActiveMQ Artemis will only forward messages to other nodes of the cluster if the address to which they are being forwarded has queues which have consumers, and if those consumers have message filters (selectors) at least one of those selectors must match the message. |
| Using <code>ON_DEMAND</code> is like setting the legacy <code>forward-when-no-consumers</code> parameter to <code>false</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If this is set to <code>OFF_WITH_REDISTRIBUTION</code> then like with 'OFF' messages won’t be initially routed to other nodes in the cluster. |
| However, if <a href="#message-redistribution">redistribution</a> is configured, it can forward messages in the normal way. |
| In this way local consumers will always have priority.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Keep in mind that this message forwarding/balancing is what we call "initial distribution." It is different than <em>redistribution</em> which is <a href="#message-redistribution">discussed below</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Default is <code>ON_DEMAND</code>.</p> |
| </div> |
| </dd> |
| <dt class="hdlist1">max-hops</dt> |
| <dd> |
| <p>When a cluster connection decides the set of nodes to which it might load balance a message, those nodes do not have to be directly connected to it via a cluster connection. |
| Apache ActiveMQ Artemis can be configured to also load balance messages to nodes which might be connected to it only indirectly with other Apache ActiveMQ Artemis servers as intermediates in a chain.</p> |
| <div class="paragraph"> |
| <p>This allows Apache ActiveMQ Artemis to be configured in more complex topologies and still provide message load balancing. |
| We’ll discuss this more later in this chapter.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The default value for this parameter is <code>1</code>, which means messages are only load balanced to other Apache ActiveMQ Artemis serves which are directly connected to this server. |
| This parameter is optional.</p> |
| </div> |
| </dd> |
| <dt class="hdlist1">confirmation-window-size</dt> |
| <dd> |
| <p>The size (in bytes) of the window used for sending confirmations from the server connected to. |
| So once the server has received <code>confirmation-window-size</code> bytes it notifies its client, default is 1048576. |
| A value of -1 means no window.</p> |
| </dd> |
| <dt class="hdlist1">producer-window-size</dt> |
| <dd> |
| <p>The size for producer flow control over cluster connection. |
| it’s by default is 1MB.</p> |
| </dd> |
| <dt class="hdlist1">call-failover-timeout</dt> |
| <dd> |
| <p>Similar to <code>call-timeout</code> but used when a call is made during a failover attempt. |
| Default is -1 (no timeout).</p> |
| </dd> |
| <dt class="hdlist1">notification-interval</dt> |
| <dd> |
| <p>How often (in milliseconds) the cluster connection should broadcast itself when attaching to the cluster. |
| Default is 1000.</p> |
| </dd> |
| <dt class="hdlist1">notification-attempts</dt> |
| <dd> |
| <p>How many times the cluster connection should broadcast itself when connecting to the cluster. |
| Default is 2.</p> |
| </dd> |
| <dt class="hdlist1">discovery-group-ref</dt> |
| <dd> |
| <p>This parameter determines which discovery group is used to obtain the list of other servers in the cluster that this cluster connection will make connections to.</p> |
| </dd> |
| </dl> |
| </div> |
| <div class="paragraph"> |
| <p>Alternatively if you would like your cluster connections to use a static list of servers for discovery then you can do it like this.</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><cluster-connection</span> <span class="na">name=</span><span class="s">"my-cluster"</span><span class="nt">></span> |
| ... |
| <span class="nt"><static-connectors></span> |
| <span class="nt"><connector-ref></span>server0-connector<span class="nt"></connector-ref></span> |
| <span class="nt"><connector-ref></span>server1-connector<span class="nt"></connector-ref></span> |
| <span class="nt"></static-connectors></span> |
| <span class="nt"></cluster-connection></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Here we have defined 2 servers that we know for sure will that at least one will be available. |
| There may be many more servers in the cluster but these will; |
| be discovered via one of these connectors once an initial connection has been made.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="cluster-user-credentials"><a class="anchor" href="#cluster-user-credentials"></a><a class="link" href="#cluster-user-credentials">4.2. Cluster User Credentials</a></h3> |
| <div class="paragraph"> |
| <p>When creating connections between nodes of a cluster to form a cluster connection, Apache ActiveMQ Artemis uses a cluster user and cluster password which is defined in <code>broker.xml</code>:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><cluster-user></span>ACTIVEMQ.CLUSTER.ADMIN.USER<span class="nt"></cluster-user></span> |
| <span class="nt"><cluster-password></span>CHANGE ME!!<span class="nt"></cluster-password></span></code></pre> |
| </div> |
| </div> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>It is imperative that these values are changed from their default, or remote clients will be able to make connections to the server using the default values. |
| If they are not changed from the default, Apache ActiveMQ Artemis will detect this and pester you with a warning on every start-up.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="client-side-load-balancing"><a class="anchor" href="#client-side-load-balancing"></a><a class="link" href="#client-side-load-balancing">5. Client-Side Load balancing</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>With Apache ActiveMQ Artemis client-side load balancing, subsequent sessions created using a single session factory can be connected to different nodes of the cluster. |
| This allows sessions to spread smoothly across the nodes of a cluster and not be "clumped" on any particular node.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The load balancing policy to be used by the client factory is configurable. |
| Apache ActiveMQ Artemis provides four out-of-the-box load balancing policies, and you can also implement your own and use that.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The out-of-the-box policies are</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Round Robin. |
| With this policy the first node is chosen randomly then each subsequent node is chosen sequentially in the same order.</p> |
| <div class="paragraph"> |
| <p>For example nodes might be chosen in the order B, C, D, A, B, C, D, A, B or D, A, B, C, D, A, B, C, D or C, D, A, B, C, D, A, B, C.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Use <code>org.apache.activemq.artemis.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy</code> as the <code><connection-load-balancing-policy-class-name></code>.</p> |
| </div> |
| </li> |
| <li> |
| <p>Random. |
| With this policy each node is chosen randomly.</p> |
| <div class="paragraph"> |
| <p>Use <code>org.apache.activemq.artemis.api.core.client.loadbalance.RandomConnectionLoadBalancingPolicy</code> as the <code><connection-load-balancing-policy-class-name></code>.</p> |
| </div> |
| </li> |
| <li> |
| <p>Random Sticky. |
| With this policy the first node is chosen randomly and then re-used for subsequent connections.</p> |
| <div class="paragraph"> |
| <p>Use <code>org.apache.activemq.artemis.api.core.client.loadbalance.RandomStickyConnectionLoadBalancingPolicy</code> as the <code><connection-load-balancing-policy-class-name></code>.</p> |
| </div> |
| </li> |
| <li> |
| <p>First Element. |
| With this policy the "first" (i.e. 0th) node is always returned.</p> |
| <div class="paragraph"> |
| <p>Use <code>org.apache.activemq.artemis.api.core.client.loadbalance.FirstElementConnectionLoadBalancingPolicy</code> as the <code><connection-load-balancing-policy-class-name></code>.</p> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>You can also implement your own policy by implementing the interface <code>org.apache.activemq.artemis.api.core.client.loadbalance.ConnectionLoadBalancingPolicy</code></p> |
| </div> |
| <div class="paragraph"> |
| <p>Specifying which load balancing policy to use differs whether you are using JMS or the core API. |
| If you don’t specify a policy then the default will be used which is <code>org.apache.activemq.artemis.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The parameter <code>connectionLoadBalancingPolicyClassName</code> can be set on the URI to configure what load balancing policy to use:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="nowrap">tcp://localhost:61616?connectionLoadBalancingPolicyClassName=org.apache.activemq.artemis.api.core.client.loadbalance.RandomConnectionLoadBalancingPolicy</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The set of servers over which the factory load balances can be determined in one of two ways:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Specifying servers explicitly in the URL. |
| This also requires setting the <code>useTopologyForLoadBalancing</code> parameter to <code>false</code> on the URL.</p> |
| </li> |
| <li> |
| <p>Using discovery. |
| This is the default behavior.</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="specifying-members-of-a-cluster-explicitly"><a class="anchor" href="#specifying-members-of-a-cluster-explicitly"></a><a class="link" href="#specifying-members-of-a-cluster-explicitly">6. Specifying Members of a Cluster Explicitly</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Sometimes you want to explicitly define a cluster more explicitly, that is control which server connect to each other in the cluster. |
| This is typically used to form non symmetrical clusters such as chain cluster or ring clusters. |
| This can only be done using a static list of connectors and is configured as follows:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><cluster-connection</span> <span class="na">name=</span><span class="s">"my-cluster"</span><span class="nt">></span> |
| <span class="nt"><address/></span> |
| <span class="nt"><connector-ref></span>netty-connector<span class="nt"></connector-ref></span> |
| <span class="nt"><retry-interval></span>500<span class="nt"></retry-interval></span> |
| <span class="nt"><use-duplicate-detection></span>true<span class="nt"></use-duplicate-detection></span> |
| <span class="nt"><message-load-balancing></span>STRICT<span class="nt"></message-load-balancing></span> |
| <span class="nt"><max-hops></span>1<span class="nt"></max-hops></span> |
| <span class="nt"><static-connectors</span> <span class="na">allow-direct-connections-only=</span><span class="s">"true"</span><span class="nt">></span> |
| <span class="nt"><connector-ref></span>server1-connector<span class="nt"></connector-ref></span> |
| <span class="nt"></static-connectors></span> |
| <span class="nt"></cluster-connection></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In this example we have set the attribute <code>allow-direct-connections-only</code> which means that the only server that this server can create a cluster connection to is server1-connector. |
| This means you can explicitly create any cluster topology you want.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="message-redistribution"><a class="anchor" href="#message-redistribution"></a><a class="link" href="#message-redistribution">7. Message Redistribution</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Another important part of clustering is message redistribution. |
| Earlier we learned how server side message load balancing round robins messages across the cluster. |
| If <code>message-load-balancing</code> is <code>OFF</code> or <code>ON_DEMAND</code> then messages won’t be forwarded to nodes which don’t have matching consumers. |
| This is great and ensures that messages aren’t moved to a queue which has no consumers to consume them. |
| However, there is a situation it doesn’t solve: What happens if the consumers on a queue close after the messages have been sent to the node? |
| If there are no consumers on the queue the message won’t get consumed and we have a <em>starvation</em> situation.</p> |
| </div> |
| <div class="paragraph"> |
| <p>This is where message redistribution comes in. |
| With message redistribution Apache ActiveMQ Artemis can be configured to automatically <em>redistribute</em> messages from queues which have no consumers or consumers with filters that don’t match messages. |
| The messages are re-routed to other nodes in the cluster which do have matching consumers. |
| To enable this functionality <code>message-load-balancing</code> must be <code>ON_DEMAND</code> or <code>OFF_WITH_REDISTRIBUTION</code></p> |
| </div> |
| <div class="paragraph"> |
| <p>Message redistribution can be configured to kick in immediately after the need to redistribute is detected, or to wait a configurable delay before redistributing. |
| By default, message redistribution is disabled.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Message redistribution can be configured on a per address basis, by specifying the redistribution delay in the address settings. |
| For more information on configuring address settings, please see <a href="address-settings.html#address-settings">Configuring Addresses and Queues via Address Settings</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Here’s an address settings snippet from <code>broker.xml</code> showing how message redistribution is enabled for a set of queues:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><address-settings></span> |
| <span class="nt"><address-setting</span> <span class="na">match=</span><span class="s">"#"</span><span class="nt">></span> |
| <span class="nt"><redistribution-delay></span>0<span class="nt"></redistribution-delay></span> |
| <span class="nt"></address-setting></span> |
| <span class="nt"></address-settings></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The above <code>address-settings</code> block would set a <code>redistribution-delay</code> of <code>0</code> for any queue which is bound to any address. |
| So the above would enable instant (no delay) redistribution for all addresses.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The attribute <code>match</code> can be an exact match or it can be a string that conforms to the Apache ActiveMQ Artemis wildcard syntax (described in <a href="wildcard-syntax.html#wildcard-syntax">Wildcard Syntax</a>).</p> |
| </div> |
| <div class="paragraph"> |
| <p>The element <code>redistribution-delay</code> defines the delay in milliseconds between detecting the need for redistribution and actually attempting redistribution. |
| A delay of zero means the messages will be immediately redistributed. |
| A value of <code>-1</code> signifies that messages will never be redistributed. |
| The default value is <code>-1</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>It often makes sense to introduce a delay before redistributing as it’s a common case that a consumer closes but another one quickly is created on the same queue, in such a case you probably don’t want to redistribute immediately since the new consumer will arrive shortly.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="cluster-topologies"><a class="anchor" href="#cluster-topologies"></a><a class="link" href="#cluster-topologies">8. Cluster topologies</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Apache ActiveMQ Artemis clusters can be connected together in many different topologies, let’s consider the two most common ones here</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="symmetric-cluster"><a class="anchor" href="#symmetric-cluster"></a><a class="link" href="#symmetric-cluster">8.1. Symmetric cluster</a></h3> |
| <div class="paragraph"> |
| <p>A symmetric cluster is probably the most common cluster topology.</p> |
| </div> |
| <div class="paragraph"> |
| <p>With a symmetric cluster every node in the cluster is connected to every other node in the cluster. |
| In other words every node in the cluster is no more than one hop away from every other node.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To form a symmetric cluster every node in the cluster defines a cluster connection with the attribute <code>max-hops</code> set to <code>1</code>. |
| Typically the cluster connection will use server discovery in order to know what other servers in the cluster it should connect to, although it is possible to explicitly define each target server too in the cluster connection if, for example, UDP is not available on your network.</p> |
| </div> |
| <div class="paragraph"> |
| <p>With a symmetric cluster each node knows about all the queues that exist on all the other nodes and what consumers they have. |
| With this knowledge it can determine how to load balance and redistribute messages around the nodes.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Don’t forget <a href="#copy-warning">this warning</a> when creating a symmetric cluster.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="chain-cluster"><a class="anchor" href="#chain-cluster"></a><a class="link" href="#chain-cluster">8.2. Chain cluster</a></h3> |
| <div class="paragraph"> |
| <p>With a chain cluster, each node in the cluster is not connected to every node in the cluster directly, instead the nodes form a chain with a node on each end of the chain and all other nodes just connecting to the previous and next nodes in the chain.</p> |
| </div> |
| <div class="paragraph"> |
| <p>An example of this would be a three node chain consisting of nodes A, B and C. |
| Node A is hosted in one network and has many producer clients connected to it sending order messages. |
| Due to corporate policy, the order consumer clients need to be hosted in a different network, and that network is only accessible via a third network. |
| In this setup node B acts as a mediator with no producers or consumers on it. |
| Any messages arriving on node A will be forwarded to node B, which will in turn forward them to node C where they can get consumed. |
| Node A does not need to directly connect to C, but all the nodes can still act as a part of the cluster.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To set up a cluster in this way, node A would define a cluster connection that connects to node B, and node B would define a cluster connection that connects to node C. |
| In this case we only want cluster connections in one direction since we’re only moving messages from node A->B->C and never from C->B->A.</p> |
| </div> |
| <div class="paragraph"> |
| <p>For this topology we would set <code>max-hops</code> to <code>2</code>. |
| With a value of <code>2</code> the knowledge of what queues and consumers that exist on node C would be propagated from node C to node B to node A. |
| Node A would then know to distribute messages to node B when they arrive, even though node B has no consumers itself, it would know that a further hop away is node C which does have consumers.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="scaling-down"><a class="anchor" href="#scaling-down"></a><a class="link" href="#scaling-down">8.3. Scaling Down</a></h3> |
| <div class="paragraph"> |
| <p>Apache ActiveMQ Artemis supports scaling down a cluster with no message loss (even for non-durable messages). |
| This is especially useful in certain environments (e.g. the cloud) where the size of a cluster may change relatively frequently. |
| When scaling up a cluster (i.e. adding nodes) there is no risk of message loss, but when scaling down a cluster (i.e. removing nodes) the messages on those nodes would be lost unless the broker sent them to another node in the cluster. |
| Apache ActiveMQ Artemis can be configured to do just that.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To enable this behavior configure <code>scale-down</code> in the <code>live-only</code> <code>ha-policy</code>, e.g.:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><ha-policy></span> |
| <span class="nt"><live-only></span> |
| <span class="nt"><scale-down></span> |
| <span class="nt"><enabled></span>true<span class="nt"></enabled></span> |
| <span class="nt"><discovery-group-ref</span> <span class="na">discovery-group-name=</span><span class="s">"my-discovery-group"</span><span class="nt">/></span> |
| <span class="nt"></scale-down></span> |
| <span class="nt"></live-only></span> |
| <span class="nt"></ha-policy></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If <code>scale-down</code>/<code>enabled</code> is <code>true</code> then when the server is shutdown gracefully (i.e. stopped without crashing) it will find another node in the cluster and send <em>all</em> of its messages (both durable and non-durable) to that node. |
| The messages are processed in order and go to the <em>back</em> of the respective queues on the other node (just as if the messages were sent from an external client for the first time).</p> |
| </div> |
| <div class="paragraph"> |
| <p>The <em>target</em> of the scale down operation can be configured a few differnt ways. |
| The above example uses <code>discovery-group-ref</code> to reference a <code>discovery-group</code> which will be used to find the target broker. |
| This should be the same <code>discovery-group</code> referenced by your <code>cluster-connection</code>. |
| You can also specify a static list of <code>connector</code> elements, e.g.:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><connectors></span> |
| ... |
| <span class="nt"><connector</span> <span class="na">name=</span><span class="s">"server0-connector"</span><span class="nt">></span>tcp://server0:61616<span class="nt"></connector></span> |
| <span class="nt"></connectors></span> |
| ... |
| <span class="nt"><ha-policy></span> |
| <span class="nt"><live-only></span> |
| <span class="nt"><scale-down></span> |
| <span class="nt"><enabled></span>true<span class="nt"></enabled></span> |
| <span class="nt"><connectors></span> |
| <span class="nt"><connector-ref></span>server0-connector<span class="nt"></connector-ref></span> |
| <span class="nt"></connectors></span> |
| <span class="nt"></scale-down></span> |
| <span class="nt"></live-only></span> |
| <span class="nt"></ha-policy></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>It’s also possible to specify <code>group-name</code>. |
| If this is specified then messages will only be sent to another node in the cluster that uses the same <code>group-name</code> as the server being shutdown, e.g.:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="nt"><ha-policy></span> |
| <span class="nt"><live-only></span> |
| <span class="nt"><scale-down></span> |
| <span class="nt"><enabled></span>true<span class="nt"></enabled></span> |
| <span class="nt"><group-name></span>my-group<span class="nt"></group-name></span> |
| <span class="nt"><discovery-group-ref</span> <span class="na">discovery-group-name=</span><span class="s">"my-discovery-group"</span><span class="nt">/></span> |
| <span class="nt"></scale-down></span> |
| <span class="nt"></live-only></span> |
| <span class="nt"></ha-policy></span></code></pre> |
| </div> |
| </div> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>If cluster nodes are grouped together with different <code>group-name</code> values beware. |
| If all the nodes in a single group are shut down then the messages from that node/group will be lost.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </body> |
| </html> |
