| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="generator" content="Asciidoctor 2.0.18"> |
| <link rel="icon" type="image/png" href="images/favicon.png"> |
| <title>Network Isolation (Split Brain)</title> |
| <link rel="stylesheet" href="css/asciidoctor.css"> |
| <link rel="stylesheet" href="css/font-awesome.css"> |
| <link rel="stylesheet" href="css/rouge-github.css"> |
| </head> |
| <body class="book toc2 toc-left"> |
| <div id="header"> |
| <h1>Network Isolation (Split Brain)</h1> |
| <div id="toc" class="toc2"> |
| <div id="toctitle"><a href="index.html">User Manual for 2.33.0</a></div> |
| <ul class="sectlevel1"> |
| <li><a href="#pluggable-lock-manager">1. Pluggable Lock Manager</a></li> |
| <li><a href="#quorum-voting">2. Quorum Voting</a> |
| <ul class="sectlevel2"> |
| <li><a href="#voting-mechanics">2.1. Voting Mechanics</a></li> |
| </ul> |
| </li> |
| <li><a href="#pinging-the-network">3. Pinging the network</a></li> |
| </ul> |
| </div> |
| </div> |
| <div id="content"> |
| <div id="preamble"> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>A <em>split brain</em> is a condition that occurs when two different brokers are serving the same messages at the same time. |
| When this happens instead of client applications all sharing the <em>same</em> broker as they ought, they may become divided between the two split brain brokers. |
| This is problematic because it can lead to:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Duplicate messages</strong> e.g. when multiple consumers on the same JMS queue split between both brokers and receive the same message(s)</p> |
| </li> |
| <li> |
| <p><strong>Missed messages</strong> e.g. when multiple consumers on the same JMS topic split between both brokers and producers are only sending messages to one broker</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Split brain most commonly happens when a pair of brokers in an HA <strong>replication</strong> configuration lose the replication connection linking them together. |
| When this connection is lost the backup assumes that the primary has died and therefore activates. |
| At this point there are two brokers on the network which are isolated from each other and since the backup has a copy of all the messages from the primary they are each serving the same messages.</p> |
| </div> |
| <div class="admonitionblock important"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-important" title="Important"></i> |
| </td> |
| <td class="content"> |
| <div class="title">What about shared store configurations?</div> |
| <div class="paragraph"> |
| <p>While it is technically possible for split brain to happen with a pair of brokers in an HA <em>shared store</em> configuration it would require a failure in the file-locking mechanism of the storage device which the brokers are sharing.</p> |
| </div> |
| <div class="paragraph"> |
| <p>One of the benefits of using a shared store is that the storage device itself acts as an arbiter to ensure consistency and mitigate split brain.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>Recovering from a split brain may be as simple as stopping the broker which activated by mistake. |
| However, this solution is only viable <strong>if</strong> no client application connected to it and performed messaging operations. |
| The longer client applications are allowed to interact with split brain brokers the more difficult it will be to understand and remediate the resulting problems.</p> |
| </div> |
| <div class="paragraph"> |
| <p>There are several different configurations you can choose from that will help mitigate split brain.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="pluggable-lock-manager"><a class="anchor" href="#pluggable-lock-manager"></a><a class="link" href="#pluggable-lock-manager">1. Pluggable Lock Manager</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>A pluggable lock manager configuration requires a 3rd party to establish a shared lock between primary and backup brokers. |
| The shared lock ensures that either the primary or backup is active at any given point in time, similar to how the file lock functions in the shared storage use-case.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The <em>plugin</em> decides what 3rd party implementation is used. |
| It could be something as simple as a shared file on a network file system that supports locking (e.g. NFS) or it could be something more complex like <a href="https://etcd.io/">etcd</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The broker ships with a <a href="ha.html#apache-zookeeper-integration">reference plugin implementation</a> based on <a href="https://zookeeper.apache.org/">Apache ZooKeeper</a> - a common implementation used for this kind of task.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The main benefit of a pluggable lock manager is that is releases the broker from the responsibility of establishing a reliable vote. |
| This means that a <em>single</em> HA pair of brokers can be reliably protected against split-brain.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="quorum-voting"><a class="anchor" href="#quorum-voting"></a><a class="link" href="#quorum-voting">2. Quorum Voting</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Quorum voting is a process by which one node in a cluster can determine whether another node in the cluster is active without directly communicating with that node. |
| Then the broker initiating the vote can take action based on the result (e.g. shutting itself down to avoid split-brain).</p> |
| </div> |
| <div class="paragraph"> |
| <p>Quorum voting requires the participation of the other <em>active</em> brokers in the cluster. |
| Of course this requires that there are, in fact, other active brokers in the cluster which means quorum voting won’t work with a single HA pair of brokers. |
| Furthermore, it also won’t work with just two HA pairs of brokers either because that’s still not enough for a legitimate quorum. |
| There must be at least three HA pairs to establish a proper quorum with quorum voting.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="voting-mechanics"><a class="anchor" href="#voting-mechanics"></a><a class="link" href="#voting-mechanics">2.1. Voting Mechanics</a></h3> |
| <div class="paragraph"> |
| <p>When the replication connection between a primary and backup is lost the backup and/or the primary may initiate a vote.</p> |
| </div> |
| <div class="admonitionblock important"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-important" title="Important"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>For a vote to pass a <em>majority</em> of affirmative responses is required. |
| For example, in a 3 node cluster a vote will pass with 2 affirmatives. |
| For a 4 node cluster this would be 3 affirmatives and so on.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="sect3"> |
| <h4 id="backup-voting"><a class="anchor" href="#backup-voting"></a><a class="link" href="#backup-voting">2.1.1. Backup Voting</a></h4> |
| <div class="paragraph"> |
| <p>By default, if a backup loses its replication connection to its primary it will activate automatically. |
| However, it can be configured via the <code>vote-on-replication-failure</code> property to initiate a quorum vote in order to decide whether to activate or not. |
| If this is done then the backup will keep voting until it either receives a vote allowing it to start or it detects that the primary is still active. |
| In the latter case it will then restart as a backup.</p> |
| </div> |
| <div class="paragraph"> |
| <p>See the section on <a href="ha.html#replication-configuration">Replication Configuration</a> for more details on configuration.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="primary-voting"><a class="anchor" href="#primary-voting"></a><a class="link" href="#primary-voting">2.1.2. Primary Voting</a></h4> |
| <div class="paragraph"> |
| <p>By default, if the primary server loses its replication connection to the backup then it will just carry on and wait for a backup to reconnect and start replicating again. |
| However, this may mean that the primary remains active even though the backup has activated so this behavior is configurable via the <code>vote-on-replication-failure</code> property.</p> |
| </div> |
| <div class="paragraph"> |
| <p>See the section on <a href="ha.html#replication-configuration">Replication Configuration</a> for more details on configuration.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="pinging-the-network"><a class="anchor" href="#pinging-the-network"></a><a class="link" href="#pinging-the-network">3. Pinging the network</a></h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>You may configure one more addresses in <code>broker.xml</code> that that will be pinged throughout the life of the server. The server will stop itself if it can’t ping one or more of the addresses in the list.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If you execute the <code>create</code> command using the <code>--ping</code> argument you will create a default XML that is ready to be used with network checks:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>./artemis create /myDir/myServer <span class="nt">--ping</span> 10.0.0.1</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>This XML will be added to your <code>broker.xml</code>:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="rouge highlight nowrap"><code data-lang="xml"><span class="c"><!-- |
| You can verify the network health of a particular NIC by specifying the <network-check-NIC> element. |
| <network-check-NIC>theNicName</network-check-NIC> |
| --></span> |
| |
| <span class="c"><!-- |
| Use this to use an HTTP server to validate the network |
| <network-check-URL-list>http://www.apache.org</network-check-URL-list> --></span> |
| |
| <span class="nt"><network-check-period></span>10000<span class="nt"></network-check-period></span> |
| <span class="nt"><network-check-timeout></span>1000<span class="nt"></network-check-timeout></span> |
| |
| <span class="c"><!-- this is a comma separated list, no spaces, just DNS or IPs |
| it should accept IPV6 |
| |
| Warning: Make sure you understand your network topology as this is meant to check if your network is up. |
| Using IPs that could eventually disappear or be partially visible may defeat the purpose. |
| You can use a list of multiple IPs, any successful ping will make the server OK to continue running --></span> |
| <span class="nt"><network-check-list></span>10.0.0.1<span class="nt"></network-check-list></span> |
| |
| <span class="c"><!-- use this to customize the ping used for ipv4 addresses --></span> |
| <span class="nt"><network-check-ping-command></span>ping -c 1 -t %d %s<span class="nt"></network-check-ping-command></span> |
| |
| <span class="c"><!-- use this to customize the ping used for ipv6 addresses --></span> |
| <span class="nt"><network-check-ping6-command></span>ping6 -c 1 %2$s<span class="nt"></network-check-ping6-command></span></code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Once you lose connectivity towards <code>10.0.0.1</code> on the given example the broker will log something like this:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="nowrap">09:49:24,562 WARN [org.apache.activemq.artemis.core.server.NetworkHealthCheck] Ping Address /10.0.0.1 wasn't reacheable |
| 09:49:36,577 INFO [org.apache.activemq.artemis.core.server.NetworkHealthCheck] Network is unhealthy, stopping service ActiveMQServerImpl::serverUUID=04fd5dd8-b18c-11e6-9efe-6a0001921ad0 |
| 09:49:36,625 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 1.6.0 [04fd5dd8-b18c-11e6-9efe-6a0001921ad0] stopped, uptime 14.787 seconds |
| 09:50:00,653 WARN [org.apache.activemq.artemis.core.server.NetworkHealthCheck] ping: sendto: No route to host |
| 09:50:10,656 WARN [org.apache.activemq.artemis.core.server.NetworkHealthCheck] Host is down: java.net.ConnectException: Host is down |
| at java.net.Inet6AddressImpl.isReachable0(Native Method) [rt.jar:1.8.0_73] |
| at java.net.Inet6AddressImpl.isReachable(Inet6AddressImpl.java:77) [rt.jar:1.8.0_73] |
| at java.net.InetAddress.isReachable(InetAddress.java:502) [rt.jar:1.8.0_73] |
| at org.apache.activemq.artemis.core.server.NetworkHealthCheck.check(NetworkHealthCheck.java:295) [artemis-commons-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] |
| at org.apache.activemq.artemis.core.server.NetworkHealthCheck.check(NetworkHealthCheck.java:276) [artemis-commons-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] |
| at org.apache.activemq.artemis.core.server.NetworkHealthCheck.run(NetworkHealthCheck.java:244) [artemis-commons-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] |
| at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:189) [artemis-commons-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] |
| at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$3.run(ActiveMQScheduledComponent.java:199) [artemis-commons-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] |
| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_73] |
| at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_73] |
| at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_73] |
| at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [rt.jar:1.8.0_73] |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_73] |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_73] |
| at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_73]</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Once you reestablish your network connections towards the configured check-list:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="nowrap">09:53:23,461 INFO [org.apache.activemq.artemis.core.server.NetworkHealthCheck] Network is healthy, starting service ActiveMQServerImpl:: |
| 09:53:23,462 INFO [org.apache.activemq.artemis.core.server] AMQ221000: primary Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging) |
| 09:53:23,462 INFO [org.apache.activemq.artemis.core.server] AMQ221013: Using NIO Journal |
| 09:53:23,462 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE |
| 09:53:23,463 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP |
| 09:53:23,463 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ |
| 09:53:23,463 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT |
| 09:53:23,464 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE |
| 09:53:23,464 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP |
| 09:53:23,541 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue jms.queue.DLQ |
| 09:53:23,541 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue jms.queue.ExpiryQueue |
| 09:53:23,549 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE] |
| 09:53:23,550 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started Acceptor at 0.0.0.0:5445 for protocols [HORNETQ,STOMP] |
| 09:53:23,554 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started Acceptor at 0.0.0.0:5672 for protocols [AMQP] |
| 09:53:23,555 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started Acceptor at 0.0.0.0:1883 for protocols [MQTT] |
| 09:53:23,556 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started Acceptor at 0.0.0.0:61613 for protocols [STOMP] |
| 09:53:23,556 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now active |
| 09:53:23,556 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 1.6.0 [0.0.0.0, nodeID=04fd5dd8-b18c-11e6-9efe-6a0001921ad0]</pre> |
| </div> |
| </div> |
| <div class="admonitionblock important"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-important" title="Important"></i> |
| </td> |
| <td class="content"> |
| <div class="paragraph"> |
| <p>Make sure you understand your network topology as this is meant to validate your network. |
| Using IPs that could eventually disappear or be partially visible may defeat the purpose. |
| You can use a list of multiple IPs. |
| Any successful ping will make the server OK to continue running</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| </div> |
| </body> |
| </html> |
