| <!DOCTYPE HTML> |
| <html lang="en" > |
| |
| <head> |
| |
| <meta charset="UTF-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge" /> |
| <title>Clusters | ActiveMQ Artemis Documentation</title> |
| <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> |
| <meta name="description" content=""> |
| <meta name="generator" content="GitBook 2.1.0"> |
| |
| |
| <meta name="HandheldFriendly" content="true"/> |
| <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> |
| <meta name="apple-mobile-web-app-capable" content="yes"> |
| <meta name="apple-mobile-web-app-status-bar-style" content="black"> |
| <link rel="apple-touch-icon-precomposed" sizes="152x152" href="gitbook/images/apple-touch-icon-precomposed-152.png"> |
| <link rel="shortcut icon" href="gitbook/images/favicon.ico" type="image/x-icon"> |
| |
| <link rel="stylesheet" href="gitbook/style.css"> |
| |
| |
| |
| |
| |
| |
| <link rel="next" href="./ha.html" /> |
| |
| |
| <link rel="prev" href="./duplicate-detection.html" /> |
| |
| |
| |
| </head> |
| <body> |
| |
| |
| <div class="book" data-level="39" data-basepath="." data-revision="Wed Sep 23 2015 12:51:51 GMT+0100 (BST)"> |
| |
| |
| <div class="book-summary"> |
| <div class="book-search"> |
| <input type="text" placeholder="Type to search" class="form-control" /> |
| </div> |
| <ul class="summary"> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <li class="chapter " data-level="0" data-path="index.html"> |
| |
| |
| <a href="./index.html"> |
| <i class="fa fa-check"></i> |
| |
| Introduction |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="1" data-path="notice.html"> |
| |
| |
| <a href="./notice.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>1.</b> |
| |
| Legal Notice |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="2" data-path="preface.html"> |
| |
| |
| <a href="./preface.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>2.</b> |
| |
| Preface |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="3" data-path="project-info.html"> |
| |
| |
| <a href="./project-info.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>3.</b> |
| |
| Project Info |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="4" data-path="messaging-concepts.html"> |
| |
| |
| <a href="./messaging-concepts.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>4.</b> |
| |
| Messaging Concepts |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="5" data-path="architecture.html"> |
| |
| |
| <a href="./architecture.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>5.</b> |
| |
| Architecture |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="6" data-path="using-server.html"> |
| |
| |
| <a href="./using-server.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>6.</b> |
| |
| Using the Server |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="7" data-path="using-jms.html"> |
| |
| |
| <a href="./using-jms.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>7.</b> |
| |
| Using JMS |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="8" data-path="using-core.html"> |
| |
| |
| <a href="./using-core.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>8.</b> |
| |
| Using Core |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="9" data-path="jms-core-mapping.html"> |
| |
| |
| <a href="./jms-core-mapping.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>9.</b> |
| |
| Mapping JMS Concepts to the Core API |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="10" data-path="client-classpath.html"> |
| |
| |
| <a href="./client-classpath.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>10.</b> |
| |
| The Client Classpath |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="11" data-path="examples.html"> |
| |
| |
| <a href="./examples.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>11.</b> |
| |
| Examples |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="12" data-path="wildcard-routing.html"> |
| |
| |
| <a href="./wildcard-routing.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>12.</b> |
| |
| Routing Messages With Wild Cards |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="13" data-path="wildcard-syntax.html"> |
| |
| |
| <a href="./wildcard-syntax.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>13.</b> |
| |
| Understanding the Apache ActiveMQ Artemis Wildcard Syntax |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="14" data-path="filter-expressions.html"> |
| |
| |
| <a href="./filter-expressions.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>14.</b> |
| |
| Filter Expressions |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="15" data-path="persistence.html"> |
| |
| |
| <a href="./persistence.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>15.</b> |
| |
| Persistence |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="16" data-path="configuring-transports.html"> |
| |
| |
| <a href="./configuring-transports.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>16.</b> |
| |
| Configuring Transports |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="17" data-path="connection-ttl.html"> |
| |
| |
| <a href="./connection-ttl.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>17.</b> |
| |
| Detecting Dead Connections |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="18" data-path="slow-consumers.html"> |
| |
| |
| <a href="./slow-consumers.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>18.</b> |
| |
| Detecting Slow Consumers |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="19" data-path="transaction-config.html"> |
| |
| |
| <a href="./transaction-config.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>19.</b> |
| |
| Resource Manager Configuration |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="20" data-path="flow-control.html"> |
| |
| |
| <a href="./flow-control.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>20.</b> |
| |
| Flow Control |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="21" data-path="send-guarantees.html"> |
| |
| |
| <a href="./send-guarantees.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>21.</b> |
| |
| Guarantees of sends and commits |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="22" data-path="undelivered-messages.html"> |
| |
| |
| <a href="./undelivered-messages.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>22.</b> |
| |
| Message Redelivery and Undelivered Messages |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="23" data-path="message-expiry.html"> |
| |
| |
| <a href="./message-expiry.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>23.</b> |
| |
| Message Expiry |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="24" data-path="large-messages.html"> |
| |
| |
| <a href="./large-messages.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>24.</b> |
| |
| Large Messages |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="25" data-path="paging.html"> |
| |
| |
| <a href="./paging.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>25.</b> |
| |
| Paging |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="26" data-path="queue-attributes.html"> |
| |
| |
| <a href="./queue-attributes.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>26.</b> |
| |
| Queue Attributes |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="27" data-path="scheduled-messages.html"> |
| |
| |
| <a href="./scheduled-messages.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>27.</b> |
| |
| Scheduled Messages |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="28" data-path="last-value-queues.html"> |
| |
| |
| <a href="./last-value-queues.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>28.</b> |
| |
| Last-Value Queues |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="29" data-path="message-grouping.html"> |
| |
| |
| <a href="./message-grouping.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>29.</b> |
| |
| Message Grouping |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="30" data-path="pre-acknowledge.html"> |
| |
| |
| <a href="./pre-acknowledge.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>30.</b> |
| |
| Extra Acknowledge Modes |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="31" data-path="management.html"> |
| |
| |
| <a href="./management.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>31.</b> |
| |
| Management |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="32" data-path="security.html"> |
| |
| |
| <a href="./security.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>32.</b> |
| |
| Security |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="33" data-path="resource-limits.html"> |
| |
| |
| <a href="./resource-limits.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>33.</b> |
| |
| Resource Limits |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="34" data-path="jms-bridge.html"> |
| |
| |
| <a href="./jms-bridge.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>34.</b> |
| |
| The JMS Bridge |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="35" data-path="client-reconnection.html"> |
| |
| |
| <a href="./client-reconnection.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>35.</b> |
| |
| Client Reconnection and Session Reattachment |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="36" data-path="diverts.html"> |
| |
| |
| <a href="./diverts.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>36.</b> |
| |
| Diverting and Splitting Message Flows |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="37" data-path="core-bridges.html"> |
| |
| |
| <a href="./core-bridges.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>37.</b> |
| |
| Core Bridges |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="38" data-path="duplicate-detection.html"> |
| |
| |
| <a href="./duplicate-detection.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>38.</b> |
| |
| Duplicate Message Detection |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter active" data-level="39" data-path="clusters.html"> |
| |
| |
| <a href="./clusters.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>39.</b> |
| |
| Clusters |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="40" data-path="ha.html"> |
| |
| |
| <a href="./ha.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>40.</b> |
| |
| High Availability and Failover |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="41" data-path="graceful-shutdown.html"> |
| |
| |
| <a href="./graceful-shutdown.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>41.</b> |
| |
| Graceful Server Shutdown |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="42" data-path="libaio.html"> |
| |
| |
| <a href="./libaio.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>42.</b> |
| |
| Libaio Native Libraries |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="43" data-path="thread-pooling.html"> |
| |
| |
| <a href="./thread-pooling.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>43.</b> |
| |
| Thread management |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="44" data-path="logging.html"> |
| |
| |
| <a href="./logging.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>44.</b> |
| |
| Logging |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="45" data-path="rest.html"> |
| |
| |
| <a href="./rest.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>45.</b> |
| |
| REST Interface |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="46" data-path="embedding-activemq.html"> |
| |
| |
| <a href="./embedding-activemq.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>46.</b> |
| |
| Embedding Apache ActiveMQ Artemis |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="47" data-path="spring-integration.html"> |
| |
| |
| <a href="./spring-integration.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>47.</b> |
| |
| Spring Integration |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="48" data-path="aerogear-integration.html"> |
| |
| |
| <a href="./aerogear-integration.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>48.</b> |
| |
| AeroGear Integration |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="49" data-path="vertx-integration.html"> |
| |
| |
| <a href="./vertx-integration.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>49.</b> |
| |
| VertX Integration |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="50" data-path="intercepting-operations.html"> |
| |
| |
| <a href="./intercepting-operations.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>50.</b> |
| |
| Intercepting Operations |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="51" data-path="protocols-interoperability.html"> |
| |
| |
| <a href="./protocols-interoperability.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>51.</b> |
| |
| Protocols and Interoperability |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="52" data-path="tools.html"> |
| |
| |
| <a href="./tools.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>52.</b> |
| |
| Tools |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="53" data-path="maven-plugin.html"> |
| |
| |
| <a href="./maven-plugin.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>53.</b> |
| |
| Maven Plugin |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="54" data-path="perf-tuning.html"> |
| |
| |
| <a href="./perf-tuning.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>54.</b> |
| |
| Troubleshooting and Performance Tuning |
| </a> |
| |
| |
| |
| </li> |
| |
| <li class="chapter " data-level="55" data-path="configuration-index.html"> |
| |
| |
| <a href="./configuration-index.html"> |
| <i class="fa fa-check"></i> |
| |
| <b>55.</b> |
| |
| Configuration Reference |
| </a> |
| |
| |
| |
| </li> |
| |
| |
| |
| |
| <li class="divider"></li> |
| <li> |
| <a href="https://www.gitbook.com" target="blank" class="gitbook-link"> |
| Published with GitBook |
| </a> |
| </li> |
| |
| </ul> |
| </div> |
| |
| <div class="book-body"> |
| <div class="body-inner"> |
| <div class="book-header"> |
| <!-- Actions Left --> |
| <a href="#" class="btn pull-left toggle-summary" aria-label="Table of Contents"><i class="fa fa-align-justify"></i></a> |
| <a href="#" class="btn pull-left toggle-search" aria-label="Search"><i class="fa fa-search"></i></a> |
| |
| <div id="font-settings-wrapper" class="dropdown pull-left"> |
| <a href="#" class="btn toggle-dropdown" aria-label="Font Settings"><i class="fa fa-font"></i> |
| </a> |
| <div class="dropdown-menu font-settings"> |
| <div class="dropdown-caret"> |
| <span class="caret-outer"></span> |
| <span class="caret-inner"></span> |
| </div> |
| |
| <div class="buttons"> |
| <button type="button" id="reduce-font-size" class="button size-2">A</button> |
| <button type="button" id="enlarge-font-size" class="button size-2">A</button> |
| </div> |
| |
| <div class="buttons font-family-list"> |
| <button type="button" data-font="0" class="button">Serif</button> |
| <button type="button" data-font="1" class="button">Sans</button> |
| </div> |
| |
| <div class="buttons color-theme-list"> |
| <button type="button" id="color-theme-preview-0" class="button size-3" data-theme="0">White</button> |
| <button type="button" id="color-theme-preview-1" class="button size-3" data-theme="1">Sepia</button> |
| <button type="button" id="color-theme-preview-2" class="button size-3" data-theme="2">Night</button> |
| </div> |
| </div> |
| |
| </div> |
| |
| <!-- Actions Right --> |
| |
| <div class="dropdown pull-right"> |
| <a href="#" class="btn toggle-dropdown" aria-label="Share"><i class="fa fa-share-alt"></i> |
| </a> |
| <div class="dropdown-menu font-settings dropdown-left"> |
| <div class="dropdown-caret"> |
| <span class="caret-outer"></span> |
| <span class="caret-inner"></span> |
| </div> |
| <div class="buttons"> |
| <button type="button" data-sharing="twitter" class="button"> |
| Share on Twitter |
| </button> |
| <button type="button" data-sharing="google-plus" class="button"> |
| Share on Google |
| </button> |
| <button type="button" data-sharing="facebook" class="button"> |
| Share on Facebook |
| </button> |
| <button type="button" data-sharing="weibo" class="button"> |
| Share on Weibo |
| </button> |
| <button type="button" data-sharing="instapaper" class="button"> |
| Share on Instapaper |
| </button> |
| </div> |
| </div> |
| </div> |
| |
| |
| |
| <a href="#" target="_blank" class="btn pull-right google-plus-sharing-link sharing-link" data-sharing="google-plus" aria-label="Google"><i class="fa fa-google-plus"></i></a> |
| |
| |
| <a href="#" target="_blank" class="btn pull-right facebook-sharing-link sharing-link" data-sharing="facebook" aria-label="Facebook"><i class="fa fa-facebook"></i></a> |
| |
| |
| <a href="#" target="_blank" class="btn pull-right twitter-sharing-link sharing-link" data-sharing="twitter" aria-label="Twitter"><i class="fa fa-twitter"></i></a> |
| |
| |
| |
| |
| |
| <!-- Title --> |
| <h1> |
| <i class="fa fa-circle-o-notch fa-spin"></i> |
| <a href="./" >ActiveMQ Artemis Documentation</a> |
| </h1> |
| </div> |
| |
| <div class="page-wrapper" tabindex="-1"> |
| <div class="page-inner"> |
| |
| |
| <section class="normal" id="section-"> |
| |
| <h1 id="clusters">Clusters</h1> |
| <h2 id="clusters-overview">Clusters Overview</h2> |
| <p>Apache ActiveMQ Artemis clusters allow groups of Apache ActiveMQ Artemis servers to be grouped |
| together in order to share message processing load. Each active node in |
| the cluster is an active Apache ActiveMQ Artemis server which manages its own messages |
| and handles its own connections.</p> |
| <p>The cluster is formed by each node declaring <em>cluster connections</em> to |
| other nodes in the core configuration file <code>broker.xml</code>. |
| When a node forms a cluster connection to another node, internally it |
| creates a <em>core bridge</em> (as described in <a href="core-bridges.html">Core Bridges</a>) connection between it and |
| the other node, this is done transparently behind the scenes - you don't |
| have to declare an explicit bridge for each node. These cluster |
| connections allow messages to flow between the nodes of the cluster to |
| balance load.</p> |
| <p>Nodes can be connected together to form a cluster in many different |
| topologies, we will discuss a couple of the more common topologies later |
| in this chapter.</p> |
| <p>We'll also discuss client side load balancing, where we can balance |
| client connections across the nodes of the cluster, and we'll consider |
| message redistribution where Apache ActiveMQ Artemis will redistribute messages between |
| nodes to avoid starvation.</p> |
| <p>Another important part of clustering is <em>server discovery</em> where servers |
| can broadcast their connection details so clients or other servers can |
| connect to them with the minimum of configuration.</p> |
| <blockquote> |
| <p><strong>Warning</strong></p> |
| <p>Once a cluster node has been configured it is common to simply copy |
| that configuration to other nodes to produce a symmetric cluster. |
| However, care must be taken when copying the Apache ActiveMQ Artemis files. Do not |
| copy the Apache ActiveMQ Artemis <em>data</em> (i.e. the <code>bindings</code>, <code>journal</code>, and |
| <code>large-messages</code> directories) from one node to another. When a node is |
| started for the first time and initializes its journal files it also |
| persists a special identifier to the <code>journal</code> directory. This id |
| <em>must</em> be unique among nodes in the cluster or the cluster will not |
| form properly.</p> |
| </blockquote> |
| <h2 id="server-discovery">Server discovery</h2> |
| <p>Server discovery is a mechanism by which servers can propagate their |
| connection details to:</p> |
| <ul> |
| <li><p>Messaging clients. A messaging client wants to be able to connect to |
| the servers of the cluster without having specific knowledge of |
| which servers in the cluster are up at any one time.</p> |
| </li> |
| <li><p>Other servers. Servers in a cluster want to be able to create |
| cluster connections to each other without having prior knowledge of |
| all the other servers in the cluster.</p> |
| </li> |
| </ul> |
| <p>This information, let's call it the Cluster Topology, is actually sent |
| around normal Apache ActiveMQ Artemis connections to clients and to other servers over |
| cluster connections. This being the case we need a way of establishing |
| the initial first connection. This can be done using dynamic discovery |
| techniques like |
| <a href="http://en.wikipedia.org/wiki/User_Datagram_Protocol" target="_blank">UDP</a> and |
| <a href="http://www.jgroups.org/" target="_blank">JGroups</a>, or by providing a list of initial |
| connectors.</p> |
| <h3 id="dynamic-discovery">Dynamic Discovery</h3> |
| <p>Server discovery uses |
| <a href="http://en.wikipedia.org/wiki/User_Datagram_Protocol" target="_blank">UDP</a> multicast or |
| <a href="http://www.jgroups.org/" target="_blank">JGroups</a> to broadcast server connection |
| settings.</p> |
| <h4 id="broadcast-groups">Broadcast Groups</h4> |
| <p>A broadcast group is the means by which a server broadcasts connectors |
| over the network. A connector defines a way in which a client (or other |
| server) can make connections to the server. For more information on what |
| a connector is, please see <a href="configuring-transports.html">Configuring the Transport</a>.</p> |
| <p>The broadcast group takes a set of connector pairs, each connector pair |
| contains connection settings for a live and backup server (if one |
| exists) and broadcasts them on the network. Depending on which |
| broadcasting technique you configure the cluster, it uses either UDP or |
| JGroups to broadcast connector pairs information.</p> |
| <p>Broadcast groups are defined in the server configuration file |
| <code>broker.xml</code>. There can be many broadcast groups per |
| Apache ActiveMQ Artemis server. All broadcast groups must be defined in a |
| <code>broadcast-groups</code> element.</p> |
| <p>Let's take a look at an example broadcast group from |
| <code>broker.xml</code> that defines a UDP broadcast group:</p> |
| <pre><code><broadcast-groups> |
| <broadcast-group name="my-broadcast-group"> |
| <local-bind-address>172.16.9.3</local-bind-address> |
| <local-bind-port>5432</local-bind-port> |
| <group-address>231.7.7.7</group-address> |
| <group-port>9876</group-port> |
| <broadcast-period>2000</broadcast-period> |
| <connector-ref connector-name="netty-connector"/> |
| </broadcast-group> |
| </broadcast-groups> |
| </code></pre><p>Some of the broadcast group parameters are optional and you'll normally |
| use the defaults, but we specify them all in the above example for |
| clarity. Let's discuss each one in turn:</p> |
| <ul> |
| <li><p><code>name</code> attribute. Each broadcast group in the server must have a |
| unique name.</p> |
| </li> |
| <li><p><code>local-bind-address</code>. This is the local bind address that the |
| datagram socket is bound to. If you have multiple network interfaces |
| on your server, you would specify which one you wish to use for |
| broadcasts by setting this property. If this property is not |
| specified then the socket will be bound to the wildcard address, an |
| IP address chosen by the kernel. This is a UDP specific attribute.</p> |
| </li> |
| <li><p><code>local-bind-port</code>. If you want to specify a local port to which the |
| datagram socket is bound you can specify it here. Normally you would |
| just use the default value of <code>-1</code> which signifies that an anonymous |
| port should be used. This parameter is always specified in |
| conjunction with <code>local-bind-address</code>. This is a UDP specific |
| attribute.</p> |
| </li> |
| <li><p><code>group-address</code>. This is the multicast address to which the data |
| will be broadcast. It is a class D IP address in the range |
| <code>224.0.0.0</code> to <code>239.255.255.255</code>, inclusive. The address <code>224.0.0.0</code> |
| is reserved and is not available for use. This parameter is |
| mandatory. This is a UDP specific attribute.</p> |
| </li> |
| <li><p><code>group-port</code>. This is the UDP port number used for broadcasting. |
| This parameter is mandatory. This is a UDP specific attribute.</p> |
| </li> |
| <li><p><code>broadcast-period</code>. This is the period in milliseconds between |
| consecutive broadcasts. This parameter is optional, the default |
| value is <code>2000</code> milliseconds.</p> |
| </li> |
| <li><p><code>connector-ref</code>. This specifies the connector and optional backup |
| connector that will be broadcasted (see <a href="configuring-transports.html">Configuring the Transport</a> for more information on |
| connectors). The connector to be broadcasted is specified by the |
| <code>connector-name</code> attribute.</p> |
| </li> |
| </ul> |
| <p>Here is another example broadcast group that defines a JGroups broadcast |
| group:</p> |
| <pre><code><broadcast-groups> |
| <broadcast-group name="my-broadcast-group"> |
| <jgroups-file>test-jgroups-file_ping.xml</jgroups-file> |
| <jgroups-channel>activemq_broadcast_channel</jgroups-channel> |
| <broadcast-period>2000</broadcast-period> |
| <connector-ref connector-name="netty-connector"/> |
| </broadcast-group> |
| </broadcast-groups> |
| </code></pre><p>To be able to use JGroups to broadcast, one must specify two attributes, |
| i.e. <code>jgroups-file</code> and <code>jgroups-channel</code>, as discussed in details as |
| following:</p> |
| <ul> |
| <li><p><code>jgroups-file</code> attribute. This is the name of JGroups configuration |
| file. It will be used to initialize JGroups channels. Make sure the |
| file is in the java resource path so that Apache ActiveMQ Artemis can load it.</p> |
| </li> |
| <li><p><code>jgroups-channel</code> attribute. The name that JGroups channels connect |
| to for broadcasting.</p> |
| </li> |
| </ul> |
| <blockquote> |
| <p><strong>Note</strong></p> |
| <p>The JGroups attributes (<code>jgroups-file</code> and <code>jgroups-channel</code>) and UDP |
| specific attributes described above are exclusive of each other. Only |
| one set can be specified in a broadcast group configuration. Don't mix |
| them!</p> |
| </blockquote> |
| <p>The following is an example of a JGroups file</p> |
| <pre><code><config xmlns="urn:org:jgroups" |
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" |
| xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-3.0.xsd"> |
| <TCP loopback="true" |
| recv_buf_size="20000000" |
| send_buf_size="640000" |
| discard_incompatible_packets="true" |
| max_bundle_size="64000" |
| max_bundle_timeout="30" |
| enable_bundling="true" |
| use_send_queues="false" |
| sock_conn_timeout="300" |
| |
| thread_pool.enabled="true" |
| thread_pool.min_threads="1" |
| thread_pool.max_threads="10" |
| thread_pool.keep_alive_time="5000" |
| thread_pool.queue_enabled="false" |
| thread_pool.queue_max_size="100" |
| thread_pool.rejection_policy="run" |
| |
| oob_thread_pool.enabled="true" |
| oob_thread_pool.min_threads="1" |
| oob_thread_pool.max_threads="8" |
| oob_thread_pool.keep_alive_time="5000" |
| oob_thread_pool.queue_enabled="false" |
| oob_thread_pool.queue_max_size="100" |
| oob_thread_pool.rejection_policy="run"/> |
| |
| <FILE_PING location="../file.ping.dir"/> |
| <MERGE2 max_interval="30000" |
| min_interval="10000"/> |
| <FD_SOCK/> |
| <FD timeout="10000" max_tries="5" /> |
| <VERIFY_SUSPECT timeout="1500" /> |
| <BARRIER /> |
| <pbcast.NAKACK |
| use_mcast_xmit="false" |
| retransmit_timeout="300,600,1200,2400,4800" |
| discard_delivered_msgs="true"/> |
| <UNICAST timeout="300,600,1200" /> |
| <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" |
| max_bytes="400000"/> |
| <pbcast.GMS print_local_addr="true" join_timeout="3000" |
| view_bundling="true"/> |
| <FC max_credits="2000000" |
| min_threshold="0.10"/> |
| <FRAG2 frag_size="60000" /> |
| <pbcast.STATE_TRANSFER/> |
| <pbcast.FLUSH timeout="0"/> |
| </config> |
| </code></pre><p>As it shows, the file content defines a jgroups protocol stacks. If you |
| want Apache ActiveMQ Artemis to use this stacks for channel creation, you have to make |
| sure the value of <code>jgroups-file</code> in your broadcast-group/discovery-group |
| configuration to be the name of this jgroups configuration file. For |
| example if the above stacks configuration is stored in a file named |
| "jgroups-stacks.xml" then your <code>jgroups-file</code> should be like</p> |
| <pre><code><jgroups-file>jgroups-stacks.xml</jgroups-file> |
| </code></pre><h4 id="discovery-groups">Discovery Groups</h4> |
| <p>While the broadcast group defines how connector information is |
| broadcasted from a server, a discovery group defines how connector |
| information is received from a broadcast endpoint (a UDP multicast |
| address or JGroup channel).</p> |
| <p>A discovery group maintains a list of connector pairs - one for each |
| broadcast by a different server. As it receives broadcasts on the |
| broadcast endpoint from a particular server it updates its entry in the |
| list for that server.</p> |
| <p>If it has not received a broadcast from a particular server for a length |
| of time it will remove that server's entry from its list.</p> |
| <p>Discovery groups are used in two places in Apache ActiveMQ Artemis:</p> |
| <ul> |
| <li><p>By cluster connections so they know how to obtain an initial |
| connection to download the topology</p> |
| </li> |
| <li><p>By messaging clients so they know how to obtain an initial |
| connection to download the topology</p> |
| </li> |
| </ul> |
| <p>Although a discovery group will always accept broadcasts, its current |
| list of available live and backup servers is only ever used when an |
| initial connection is made, from then server discovery is done over the |
| normal Apache ActiveMQ Artemis connections.</p> |
| <blockquote> |
| <p><strong>Note</strong></p> |
| <p>Each discovery group must be configured with broadcast endpoint (UDP |
| or JGroups) that matches its broadcast group counterpart. For example, |
| if broadcast is configured using UDP, the discovery group must also |
| use UDP, and the same multicast address.</p> |
| </blockquote> |
| <h4 id="defining-discovery-groups-on-the-server">Defining Discovery Groups on the Server</h4> |
| <p>For cluster connections, discovery groups are defined in the server side |
| configuration file <code>broker.xml</code>. All discovery groups |
| must be defined inside a <code>discovery-groups</code> element. There can be many |
| discovery groups defined by Apache ActiveMQ Artemis server. Let's look at an example:</p> |
| <pre><code><discovery-groups> |
| <discovery-group name="my-discovery-group"> |
| <local-bind-address>172.16.9.7</local-bind-address> |
| <group-address>231.7.7.7</group-address> |
| <group-port>9876</group-port> |
| <refresh-timeout>10000</refresh-timeout> |
| </discovery-group> |
| </discovery-groups> |
| </code></pre><p>We'll consider each parameter of the discovery group:</p> |
| <ul> |
| <li><p><code>name</code> attribute. Each discovery group must have a unique name per |
| server.</p> |
| </li> |
| <li><p><code>local-bind-address</code>. If you are running with multiple network |
| interfaces on the same machine, you may want to specify that the |
| discovery group listens only only a specific interface. To do this |
| you can specify the interface address with this parameter. This |
| parameter is optional. This is a UDP specific attribute.</p> |
| </li> |
| <li><p><code>group-address</code>. This is the multicast IP address of the group to |
| listen on. It should match the <code>group-address</code> in the broadcast |
| group that you wish to listen from. This parameter is mandatory. |
| This is a UDP specific attribute.</p> |
| </li> |
| <li><p><code>group-port</code>. This is the UDP port of the multicast group. It should |
| match the <code>group-port</code> in the broadcast group that you wish to |
| listen from. This parameter is mandatory. This is a UDP specific |
| attribute.</p> |
| </li> |
| <li><p><code>refresh-timeout</code>. This is the period the discovery group waits |
| after receiving the last broadcast from a particular server before |
| removing that servers connector pair entry from its list. You would |
| normally set this to a value significantly higher than the |
| <code>broadcast-period</code> on the broadcast group otherwise servers might |
| intermittently disappear from the list even though they are still |
| broadcasting due to slight differences in timing. This parameter is |
| optional, the default value is <code>10000</code> milliseconds (10 seconds).</p> |
| </li> |
| </ul> |
| <p>Here is another example that defines a JGroups discovery group:</p> |
| <pre><code><discovery-groups> |
| <discovery-group name="my-broadcast-group"> |
| <jgroups-file>test-jgroups-file_ping.xml</jgroups-file> |
| <jgroups-channel>activemq_broadcast_channel</jgroups-channel> |
| <refresh-timeout>10000</refresh-timeout> |
| </discovery-group> |
| </discovery-groups> |
| </code></pre><p>To receive broadcast from JGroups channels, one must specify two |
| attributes, <code>jgroups-file</code> and <code>jgroups-channel</code>, as discussed in |
| details as following:</p> |
| <ul> |
| <li><p><code>jgroups-file</code> attribute. This is the name of JGroups configuration |
| file. It will be used to initialize JGroups channels. Make sure the |
| file is in the java resource path so that Apache ActiveMQ Artemis can load it.</p> |
| </li> |
| <li><p><code>jgroups-channel</code> attribute. The name that JGroups channels connect |
| to for receiving broadcasts.</p> |
| </li> |
| </ul> |
| <blockquote> |
| <p><strong>Note</strong></p> |
| <p>The JGroups attributes (<code>jgroups-file</code> and <code>jgroups-channel</code>) and UDP |
| specific attributes described above are exclusive of each other. Only |
| one set can be specified in a discovery group configuration. Don't mix |
| them!</p> |
| </blockquote> |
| <h4 id="discovery-groups-on-the-client-side">Discovery Groups on the Client Side</h4> |
| <p>Let's discuss how to configure an Apache ActiveMQ Artemis client to use discovery to |
| discover a list of servers to which it can connect. The way to do this |
| differs depending on whether you're using JMS or the core API.</p> |
| <h5 id="configuring-client-discovery-using-jms">Configuring client discovery using JMS</h5> |
| <p>If you're using JMS and you're using JNDI on the client to look up your |
| JMS connection factory instances then you can specify these parameters |
| in the JNDI context environment. e.g. in <code>jndi.properties</code>. Simply |
| ensure the host:port combination matches the group-address and |
| group-port from the corresponding <code>broadcast-group</code> on the server. Let's |
| take a look at an example:</p> |
| <pre><code>java.naming.factory.initial = ActiveMQInitialContextFactory |
| connectionFactory.myConnectionFactory=udp://231.7.7.7:9876 |
| </code></pre><p>The element <code>discovery-group-ref</code> specifies the name of a discovery |
| group defined in <code>broker.xml</code>.</p> |
| <p>When this connection factory is downloaded from JNDI by a client |
| application and JMS connections are created from it, those connections |
| will be load-balanced across the list of servers that the discovery |
| group maintains by listening on the multicast address specified in the |
| discovery group configuration.</p> |
| <p>If you're using JMS, but you're not using JNDI to lookup a connection |
| factory - you're instantiating the JMS connection factory directly then |
| you can specify the discovery group parameters directly when creating |
| the JMS connection factory. Here's an example:</p> |
| <pre><code class="lang-java"><span class="hljs-keyword">final</span> String groupAddress = <span class="hljs-string">"231.7.7.7"</span>; |
| |
| <span class="hljs-keyword">final</span> <span class="hljs-keyword">int</span> groupPort = <span class="hljs-number">9876</span>; |
| |
| ConnectionFactory jmsConnectionFactory = |
| ActiveMQJMSClient.createConnectionFactory(<span class="hljs-keyword">new</span> DiscoveryGroupConfiguration(groupAddress, groupPort, |
| <span class="hljs-keyword">new</span> UDPBroadcastGroupConfiguration(groupAddress, groupPort, <span class="hljs-keyword">null</span>, -<span class="hljs-number">1</span>)), JMSFactoryType.CF); |
| |
| Connection jmsConnection1 = jmsConnectionFactory.createConnection(); |
| |
| Connection jmsConnection2 = jmsConnectionFactory.createConnection(); |
| </code></pre> |
| <p>The <code>refresh-timeout</code> can be set directly on the |
| DiscoveryGroupConfiguration by using the setter method |
| <code>setDiscoveryRefreshTimeout()</code> if you want to change the default value.</p> |
| <p>There is also a further parameter settable on the |
| DiscoveryGroupConfiguration using the setter method |
| <code>setDiscoveryInitialWaitTimeout()</code>. If the connection factory is used |
| immediately after creation then it may not have had enough time to |
| received broadcasts from all the nodes in the cluster. On first usage, |
| the connection factory will make sure it waits this long since creation |
| before creating the first connection. The default value for this |
| parameter is <code>10000</code> milliseconds.</p> |
| <h5 id="configuring-client-discovery-using-core">Configuring client discovery using Core</h5> |
| <p>If you're using the core API to directly instantiate |
| <code>ClientSessionFactory</code> instances, then you can specify the discovery |
| group parameters directly when creating the session factory. Here's an |
| example:</p> |
| <pre><code class="lang-java"><span class="hljs-keyword">final</span> String groupAddress = <span class="hljs-string">"231.7.7.7"</span>; |
| <span class="hljs-keyword">final</span> <span class="hljs-keyword">int</span> groupPort = <span class="hljs-number">9876</span>; |
| ServerLocator factory = ActiveMQClient.createServerLocatorWithHA(<span class="hljs-keyword">new</span> DiscoveryGroupConfiguration(groupAddress, groupPort, |
| <span class="hljs-keyword">new</span> UDPBroadcastGroupConfiguration(groupAddress, groupPort, <span class="hljs-keyword">null</span>, -<span class="hljs-number">1</span>)))); |
| ClientSessionFactory factory = locator.createSessionFactory(); |
| ClientSession session1 = factory.createSession(); |
| ClientSession session2 = factory.createSession(); |
| </code></pre> |
| <p>The <code>refresh-timeout</code> can be set directly on the |
| DiscoveryGroupConfiguration by using the setter method |
| <code>setDiscoveryRefreshTimeout()</code> if you want to change the default value.</p> |
| <p>There is also a further parameter settable on the |
| DiscoveryGroupConfiguration using the setter method |
| <code>setDiscoveryInitialWaitTimeout()</code>. If the session factory is used |
| immediately after creation then it may not have had enough time to |
| received broadcasts from all the nodes in the cluster. On first usage, |
| the session factory will make sure it waits this long since creation |
| before creating the first session. The default value for this parameter |
| is <code>10000</code> milliseconds.</p> |
| <h3 id="discovery-using-static-connectors">Discovery using static Connectors</h3> |
| <p>Sometimes it may be impossible to use UDP on the network you are using. |
| In this case its possible to configure a connection with an initial list |
| if possible servers. This could be just one server that you know will |
| always be available or a list of servers where at least one will be |
| available.</p> |
| <p>This doesn't mean that you have to know where all your servers are going |
| to be hosted, you can configure these servers to use the reliable |
| servers to connect to. Once they are connected there connection details |
| will be propagated via the server it connects to</p> |
| <h4 id="configuring-a-cluster-connection">Configuring a Cluster Connection</h4> |
| <p>For cluster connections there is no extra configuration needed, you just |
| need to make sure that any connectors are defined in the usual manner, |
| (see <a href="configuring-transports.html">Configuring the Transport</a> for more information on connectors). These are then referenced by |
| the cluster connection configuration.</p> |
| <h4 id="configuring-a-client-connection">Configuring a Client Connection</h4> |
| <p>A static list of possible servers can also be used by a normal client.</p> |
| <h5 id="configuring-client-discovery-using-jms">Configuring client discovery using JMS</h5> |
| <p>If you're using JMS and you're using JNDI on the client to look up your |
| JMS connection factory instances then you can specify these parameters |
| in the JNDI context environment in, e.g. <code>jndi.properties</code>:</p> |
| <pre><code>java.naming.factory.initial=org.apache.activemq.artemis.jndi.ActiveMQInitialContextFactory |
| connectionFactory.myConnectionFactory=(tcp://myhost:61616,tcp://myhost2:61616) |
| </code></pre><p>The <code>connectionFactory.myConnectionFactory</code> contains a list of servers to use for the |
| connection factory. When this connection factory used client application |
| and JMS connections are created from it, those connections will be |
| load-balanced across the list of servers defined within the brackets <code>()</code>. |
| The brackets are expanded so the same query cab be appended after the last bracket for ease.</p> |
| <p>If you're using JMS, but you're not using JNDI to lookup a connection |
| factory - you're instantiating the JMS connection factory directly then |
| you can specify the connector list directly when creating the JMS |
| connection factory. Here's an example:</p> |
| <pre><code class="lang-java">HashMap<String, Object> map = <span class="hljs-keyword">new</span> HashMap<String, Object>(); |
| map.put(<span class="hljs-string">"host"</span>, <span class="hljs-string">"myhost"</span>); |
| map.put(<span class="hljs-string">"port"</span>, <span class="hljs-string">"61616"</span>); |
| TransportConfiguration server1 = <span class="hljs-keyword">new</span> TransportConfiguration(NettyConnectorFactory.class.getName(), map); |
| HashMap<String, Object> map2 = <span class="hljs-keyword">new</span> HashMap<String, Object>(); |
| map2.put(<span class="hljs-string">"host"</span>, <span class="hljs-string">"myhost2"</span>); |
| map2.put(<span class="hljs-string">"port"</span>, <span class="hljs-string">"61617"</span>); |
| TransportConfiguration server2 = <span class="hljs-keyword">new</span> TransportConfiguration(NettyConnectorFactory.class.getName(), map2); |
| |
| ActiveMQConnectionFactory cf = ActiveMQJMSClient.createConnectionFactoryWithHA(JMSFactoryType.CF, server1, server2); |
| </code></pre> |
| <h5 id="configuring-client-discovery-using-core">Configuring client discovery using Core</h5> |
| <p>If you are using the core API then the same can be done as follows:</p> |
| <pre><code class="lang-java">HashMap<String, Object> map = <span class="hljs-keyword">new</span> HashMap<String, Object>(); |
| map.put(<span class="hljs-string">"host"</span>, <span class="hljs-string">"myhost"</span>); |
| map.put(<span class="hljs-string">"port"</span>, <span class="hljs-string">"61616"</span>); |
| TransportConfiguration server1 = <span class="hljs-keyword">new</span> TransportConfiguration(NettyConnectorFactory.class.getName(), map); |
| HashMap<String, Object> map2 = <span class="hljs-keyword">new</span> HashMap<String, Object>(); |
| map2.put(<span class="hljs-string">"host"</span>, <span class="hljs-string">"myhost2"</span>); |
| map2.put(<span class="hljs-string">"port"</span>, <span class="hljs-string">"61617"</span>); |
| TransportConfiguration server2 = <span class="hljs-keyword">new</span> TransportConfiguration(NettyConnectorFactory.class.getName(), map2); |
| |
| ServerLocator locator = ActiveMQClient.createServerLocatorWithHA(server1, server2); |
| ClientSessionFactory factory = locator.createSessionFactory(); |
| ClientSession session = factory.createSession(); |
| </code></pre> |
| <h2 id="server-side-message-load-balancing">Server-Side Message Load Balancing</h2> |
| <p>If cluster connections are defined between nodes of a cluster, then |
| Apache ActiveMQ Artemis will load balance messages arriving at a particular node from a |
| client.</p> |
| <p>Let's take a simple example of a cluster of four nodes A, B, C, and D |
| arranged in a <em>symmetric cluster</em> (described in Symmetrical Clusters section). We have a queue |
| called <code>OrderQueue</code> deployed on each node of the cluster.</p> |
| <p>We have client Ca connected to node A, sending orders to the server. We |
| have also have order processor clients Pa, Pb, Pc, and Pd connected to |
| each of the nodes A, B, C, D. If no cluster connection was defined on |
| node A, then as order messages arrive on node A they will all end up in |
| the <code>OrderQueue</code> on node A, so will only get consumed by the order |
| processor client attached to node A, Pa.</p> |
| <p>If we define a cluster connection on node A, then as ordered messages |
| arrive on node A instead of all of them going into the local |
| <code>OrderQueue</code> instance, they are distributed in a round-robin fashion |
| between all the nodes of the cluster. The messages are forwarded from |
| the receiving node to other nodes of the cluster. This is all done on |
| the server side, the client maintains a single connection to node A.</p> |
| <p>For example, messages arriving on node A might be distributed in the |
| following order between the nodes: B, D, C, A, B, D, C, A, B, D. The |
| exact order depends on the order the nodes started up, but the algorithm |
| used is round robin.</p> |
| <p>Apache ActiveMQ Artemis cluster connections can be configured to always blindly load |
| balance messages in a round robin fashion irrespective of whether there |
| are any matching consumers on other nodes, but they can be a bit |
| cleverer than that and also be configured to only distribute to other |
| nodes if they have matching consumers. We'll look at both these cases in |
| turn with some examples, but first we'll discuss configuring cluster |
| connections in general.</p> |
| <h3 id="configuring-cluster-connections">Configuring Cluster Connections</h3> |
| <p>Cluster connections group servers into clusters so that messages can be |
| load balanced between the nodes of the cluster. Let's take a look at a |
| typical cluster connection. Cluster connections are always defined in |
| <code>broker.xml</code> inside a <code>cluster-connection</code> element. |
| There can be zero or more cluster connections defined per Apache ActiveMQ Artemis |
| server.</p> |
| <pre><code><cluster-connections> |
| <cluster-connection name="my-cluster"> |
| <address>jms</address> |
| <connector-ref>netty-connector</connector-ref> |
| <check-period>1000</check-period> |
| <connection-ttl>5000</connection-ttl> |
| <min-large-message-size>50000</min-large-message-size> |
| <call-timeout>5000</call-timeout> |
| <retry-interval>500</retry-interval> |
| <retry-interval-multiplier>1.0</retry-interval-multiplier> |
| <max-retry-interval>5000</max-retry-interval> |
| <initial-connect-attempts>-1</initial-connect-attempts> |
| <reconnect-attempts>-1</reconnect-attempts> |
| <use-duplicate-detection>true</use-duplicate-detection> |
| <message-load-balancing>ON_DEMAND</message-load-balancing> |
| <max-hops>1</max-hops> |
| <confirmation-window-size>32000</confirmation-window-size> |
| <call-failover-timeout>30000</call-failover-timeout> |
| <notification-interval>1000</notification-interval> |
| <notification-attempts>2</notification-attempts> |
| <discovery-group-ref discovery-group-name="my-discovery-group"/> |
| </cluster-connection> |
| </cluster-connections> |
| </code></pre><p>In the above cluster connection all parameters have been explicitly |
| specified. The following shows all the available configuration options</p> |
| <ul> |
| <li><p><code>address</code> Each cluster connection only applies to addresses that |
| match the specified address field. An address is matched on the |
| cluster connection when it begins with the string specified in this |
| field. The address field on a cluster connection also supports comma |
| separated lists and an exclude syntax '!'. To prevent an address |
| from being matched on this cluster connection, prepend a cluster |
| connection address string with '!'.</p> |
| <p>In the case shown above the cluster connection will load balance |
| messages sent to addresses that start with <code>jms</code>. This cluster |
| connection, will, in effect apply to all JMS queues and topics since |
| they map to core queues that start with the substring "jms".</p> |
| <p>The address can be any value and you can have many cluster |
| connections with different values of <code>address</code>, simultaneously |
| balancing messages for those addresses, potentially to different |
| clusters of servers. By having multiple cluster connections on |
| different addresses a single Apache ActiveMQ Artemis Server can effectively take |
| part in multiple clusters simultaneously.</p> |
| <p>Be careful not to have multiple cluster connections with overlapping |
| values of <code>address</code>, e.g. "europe" and "europe.news" since this |
| could result in the same messages being distributed between more |
| than one cluster connection, possibly resulting in duplicate |
| deliveries.</p> |
| <p>Examples:</p> |
| <ul> |
| <li>'jms.eu' |
| matches all addresses starting with 'jms.eu'</li> |
| <li>'!jms.eu' |
| matches all address except for those starting with 'jms.eu'</li> |
| <li>'jms.eu.uk,jms.eu.de' |
| matches all addresses starting with either 'jms.eu.uk' or |
| 'jms.eu.de'</li> |
| <li>'jms.eu,!jms.eu.uk' |
| matches all addresses starting with 'jms.eu' but not those |
| starting with 'jms.eu.uk'</li> |
| </ul> |
| <p>Notes:</p> |
| <ul> |
| <li>Address exclusion will always takes precedence over address |
| inclusion.</li> |
| <li>Address matching on cluster connections does not support |
| wild-card matching.</li> |
| </ul> |
| <p>This parameter is mandatory.</p> |
| </li> |
| <li><p><code>connector-ref</code>. This is the connector which will be sent to other |
| nodes in the cluster so they have the correct cluster topology.</p> |
| <p>This parameter is mandatory.</p> |
| </li> |
| <li><p><code>check-period</code>. The period (in milliseconds) used to check if the |
| cluster connection has failed to receive pings from another server. |
| Default is 30000.</p> |
| </li> |
| <li><p><code>connection-ttl</code>. This is how long a cluster connection should stay |
| alive if it stops receiving messages from a specific node in the |
| cluster. Default is 60000.</p> |
| </li> |
| <li><p><code>min-large-message-size</code>. If the message size (in bytes) is larger |
| than this value then it will be split into multiple segments when |
| sent over the network to other cluster members. Default is 102400.</p> |
| </li> |
| <li><p><code>call-timeout</code>. When a packet is sent via a cluster connection and |
| is a blocking call, i.e. for acknowledgements, this is how long it |
| will wait (in milliseconds) for the reply before throwing an |
| exception. Default is 30000.</p> |
| </li> |
| <li><p><code>retry-interval</code>. We mentioned before that, internally, cluster |
| connections cause bridges to be created between the nodes of the |
| cluster. If the cluster connection is created and the target node |
| has not been started, or say, is being rebooted, then the cluster |
| connections from other nodes will retry connecting to the target |
| until it comes back up, in the same way as a bridge does.</p> |
| <p>This parameter determines the interval in milliseconds between retry |
| attempts. It has the same meaning as the <code>retry-interval</code> on a |
| bridge (as described in <a href="core-bridges.html">Core Bridges</a>).</p> |
| <p>This parameter is optional and its default value is <code>500</code> |
| milliseconds.</p> |
| </li> |
| <li><p><code>retry-interval-multiplier</code>. This is a multiplier used to increase |
| the <code>retry-interval</code> after each reconnect attempt, default is 1.</p> |
| </li> |
| <li><p><code>max-retry-interval</code>. The maximum delay (in milliseconds) for |
| retries. Default is 2000.</p> |
| </li> |
| <li><p><code>initial-connect-attempts</code>. The number of times the system will try |
| to connect a node in the cluster initially. If the max-retry is |
| achieved this node will be considered permanently down and the |
| system will not route messages to this node. Default is -1 (infinite |
| retries).</p> |
| </li> |
| <li><p><code>reconnect-attempts</code>. The number of times the system will try to |
| reconnect to a node in the cluster. If the max-retry is achieved |
| this node will be considered permanently down and the system will |
| stop routing messages to this node. Default is -1 (infinite |
| retries).</p> |
| </li> |
| <li><p><code>use-duplicate-detection</code>. Internally cluster connections use |
| bridges to link the nodes, and bridges can be configured to add a |
| duplicate id property in each message that is forwarded. If the |
| target node of the bridge crashes and then recovers, messages might |
| be resent from the source node. By enabling duplicate detection any |
| duplicate messages will be filtered out and ignored on receipt at |
| the target node.</p> |
| <p>This parameter has the same meaning as <code>use-duplicate-detection</code> on |
| a bridge. For more information on duplicate detection, please see <a href="duplicate-detection.html">Duplicate Detection</a>. |
| Default is true.</p> |
| </li> |
| <li><p><code>message-load-balancing</code>. This parameter determines if/how |
| messages will be distributed between other nodes of the cluster. |
| It can be one of three values - <code>OFF</code>, <code>STRICT</code>, or <code>ON_DEMAND</code> |
| (default). This parameter replaces the deprecated |
| <code>forward-when-no-consumers</code> parameter.</p> |
| <p>If this is set to <code>OFF</code> then messages will never be forwarded to |
| another node in the cluster</p> |
| <p>If this is set to <code>STRICT</code> then each incoming message will be round |
| robin'd even though the same queues on the other nodes of the |
| cluster may have no consumers at all, or they may have consumers |
| that have non matching message filters (selectors). Note that |
| Apache ActiveMQ Artemis will <em>not</em> forward messages to other nodes |
| if there are no <em>queues</em> of the same name on the other nodes, even |
| if this parameter is set to <code>STRICT</code>. Using <code>STRICT</code> is like setting |
| the legacy <code>forward-when-no-consumers</code> parameter to <code>true</code>.</p> |
| <p>If this is set to <code>ON_DEMAND</code> then Apache ActiveMQ Artemis will only |
| forward messages to other nodes of the cluster if the address to which |
| they are being forwarded has queues which have consumers, and if those |
| consumers have message filters (selectors) at least one of those |
| selectors must match the message. Using <code>ON_DEMAND</code> is like setting |
| the legacy <code>forward-when-no-consumers</code> parameter to <code>false</code>.</p> |
| <p>Default is <code>ON_DEMAND</code>.</p> |
| </li> |
| <li><p><code>max-hops</code>. When a cluster connection decides the set of nodes to |
| which it might load balance a message, those nodes do not have to be |
| directly connected to it via a cluster connection. Apache ActiveMQ Artemis can be |
| configured to also load balance messages to nodes which might be |
| connected to it only indirectly with other Apache ActiveMQ Artemis servers as |
| intermediates in a chain.</p> |
| <p>This allows Apache ActiveMQ Artemis to be configured in more complex topologies and |
| still provide message load balancing. We'll discuss this more later |
| in this chapter.</p> |
| <p>The default value for this parameter is <code>1</code>, which means messages |
| are only load balanced to other Apache ActiveMQ Artemis serves which are directly |
| connected to this server. This parameter is optional.</p> |
| </li> |
| <li><p><code>confirmation-window-size</code>. The size (in bytes) of the window used |
| for sending confirmations from the server connected to. So once the |
| server has received <code>confirmation-window-size</code> bytes it notifies its |
| client, default is 1048576. A value of -1 means no window.</p> |
| </li> |
| <li><p><code>producer-window-size</code>. The size for producer flow control over cluster connection. |
| it's by default disabled through the cluster connection bridge but you may want |
| to set a value if you are using really large messages in cluster. A value of -1 means no window.</p> |
| </li> |
| <li><p><code>call-failover-timeout</code>. Similar to <code>call-timeout</code> but used when a |
| call is made during a failover attempt. Default is -1 (no timeout).</p> |
| </li> |
| <li><p><code>notification-interval</code>. How often (in milliseconds) the cluster |
| connection should broadcast itself when attaching to the cluster. |
| Default is 1000.</p> |
| </li> |
| <li><p><code>notification-attempts</code>. How many times the cluster connection |
| should broadcast itself when connecting to the cluster. Default is |
| 2.</p> |
| </li> |
| <li><p><code>discovery-group-ref</code>. This parameter determines which discovery |
| group is used to obtain the list of other servers in the cluster |
| that this cluster connection will make connections to.</p> |
| </li> |
| </ul> |
| <p>Alternatively if you would like your cluster connections to use a static |
| list of servers for discovery then you can do it like this.</p> |
| <pre><code><cluster-connection name="my-cluster"> |
| ... |
| <static-connectors> |
| <connector-ref>server0-connector</connector-ref> |
| <connector-ref>server1-connector</connector-ref> |
| </static-connectors> |
| </cluster-connection> |
| </code></pre><p>Here we have defined 2 servers that we know for sure will that at least |
| one will be available. There may be many more servers in the cluster but |
| these will; be discovered via one of these connectors once an initial |
| connection has been made.</p> |
| <h3 id="cluster-user-credentials">Cluster User Credentials</h3> |
| <p>When creating connections between nodes of a cluster to form a cluster |
| connection, Apache ActiveMQ Artemis uses a cluster user and cluster password which is |
| defined in <code>broker.xml</code>:</p> |
| <pre><code><cluster-user>ACTIVEMQ.CLUSTER.ADMIN.USER</cluster-user> |
| <cluster-password>CHANGE ME!!</cluster-password> |
| </code></pre><blockquote> |
| <p><strong>Warning</strong></p> |
| <p>It is imperative that these values are changed from their default, or |
| remote clients will be able to make connections to the server using |
| the default values. If they are not changed from the default, Apache ActiveMQ Artemis |
| will detect this and pester you with a warning on every start-up.</p> |
| </blockquote> |
| <h2 id="client-side-load-balancing">Client-Side Load balancing</h2> |
| <p>With Apache ActiveMQ Artemis client-side load balancing, subsequent sessions created |
| using a single session factory can be connected to different nodes of |
| the cluster. This allows sessions to spread smoothly across the nodes of |
| a cluster and not be "clumped" on any particular node.</p> |
| <p>The load balancing policy to be used by the client factory is |
| configurable. Apache ActiveMQ Artemis provides four out-of-the-box load balancing |
| policies, and you can also implement your own and use that.</p> |
| <p>The out-of-the-box policies are</p> |
| <ul> |
| <li><p>Round Robin. With this policy the first node is chosen randomly then |
| each subsequent node is chosen sequentially in the same order.</p> |
| <p>For example nodes might be chosen in the order B, C, D, A, B, C, D, |
| A, B or D, A, B, C, D, A, B, C, D or C, D, A, B, C, D, A, B, C.</p> |
| <p>Use |
| <code>org.apache.activemq.artemis.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy</code> |
| as the <code><connection-load-balancing-policy-class-name></code>.</p> |
| </li> |
| <li><p>Random. With this policy each node is chosen randomly.</p> |
| <p>Use |
| <code>org.apache.activemq.artemis.api.core.client.loadbalance.RandomConnectionLoadBalancingPolicy</code> |
| as the <code><connection-load-balancing-policy-class-name></code>.</p> |
| </li> |
| <li><p>Random Sticky. With this policy the first node is chosen randomly |
| and then re-used for subsequent connections.</p> |
| <p>Use |
| <code>org.apache.activemq.artemis.api.core.client.loadbalance.RandomStickyConnectionLoadBalancingPolicy</code> |
| as the <code><connection-load-balancing-policy-class-name></code>.</p> |
| </li> |
| <li><p>First Element. With this policy the "first" (i.e. 0th) node is |
| always returned.</p> |
| <p>Use |
| <code>org.apache.activemq.artemis.api.core.client.loadbalance.FirstElementConnectionLoadBalancingPolicy</code> |
| as the <code><connection-load-balancing-policy-class-name></code>.</p> |
| </li> |
| </ul> |
| <p>You can also implement your own policy by implementing the interface |
| <code>org.apache.activemq.artemis.api.core.client.loadbalance.ConnectionLoadBalancingPolicy</code></p> |
| <p>Specifying which load balancing policy to use differs whether you are |
| using JMS or the core API. If you don't specify a policy then the |
| default will be used which is |
| <code>org.apache.activemq.artemis.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy</code>.</p> |
| <p>If you're using JMS and you're using JNDI on the client to look up your |
| JMS connection factory instances then you can specify these parameters |
| in the JNDI context environment in, e.g. <code>jndi.properties</code>, to specify |
| the load balancing policy directly:</p> |
| <pre><code>java.naming.factory.initial=org.apache.activemq.artemis.jndi.ActiveMQInitialContextFactory |
| connection.myConnectionFactory=tcp://localhost:61616?loadBalancingPolicyClassName=org.apache.activemq.artemis.api.core.client.loadbalance.RandomConnectionLoadBalancingPolicy |
| </code></pre><p>The above example would instantiate a JMS connection factory that uses |
| the random connection load balancing policy.</p> |
| <p>If you're using JMS but you're instantiating your connection factory |
| directly on the client side then you can set the load balancing policy |
| using the setter on the <code>ActiveMQConnectionFactory</code> before using it:</p> |
| <pre><code class="lang-java">ConnectionFactory jmsConnectionFactory = ActiveMQJMSClient.createConnectionFactory(...); |
| jmsConnectionFactory.setLoadBalancingPolicyClassName(<span class="hljs-string">"com.acme.MyLoadBalancingPolicy"</span>); |
| </code></pre> |
| <p>If you're using the core API, you can set the load balancing policy |
| directly on the <code>ServerLocator</code> instance you are using:</p> |
| <pre><code class="lang-java">ServerLocator locator = ActiveMQClient.createServerLocatorWithHA(server1, server2); |
| locator.setLoadBalancingPolicyClassName(<span class="hljs-string">"com.acme.MyLoadBalancingPolicy"</span>); |
| </code></pre> |
| <p>The set of servers over which the factory load balances can be |
| determined in one of two ways:</p> |
| <ul> |
| <li><p>Specifying servers explicitly</p> |
| </li> |
| <li><p>Using discovery.</p> |
| </li> |
| </ul> |
| <h2 id="specifying-members-of-a-cluster-explicitly">Specifying Members of a Cluster Explicitly</h2> |
| <p>Sometimes you want to explicitly define a cluster more explicitly, that |
| is control which server connect to each other in the cluster. This is |
| typically used to form non symmetrical clusters such as chain cluster or |
| ring clusters. This can only be done using a static list of connectors |
| and is configured as follows:</p> |
| <pre><code><cluster-connection name="my-cluster"> |
| <address>jms</address> |
| <connector-ref>netty-connector</connector-ref> |
| <retry-interval>500</retry-interval> |
| <use-duplicate-detection>true</use-duplicate-detection> |
| <message-load-balancing>STRICT</message-load-balancing> |
| <max-hops>1</max-hops> |
| <static-connectors allow-direct-connections-only="true"> |
| <connector-ref>server1-connector</connector-ref> |
| </static-connectors> |
| </cluster-connection> |
| </code></pre><p>In this example we have set the attribute |
| <code>allow-direct-connections-only</code> which means that the only server that |
| this server can create a cluster connection to is server1-connector. |
| This means you can explicitly create any cluster topology you want.</p> |
| <h2 id="message-redistribution">Message Redistribution</h2> |
| <p>Another important part of clustering is message redistribution. Earlier |
| we learned how server side message load balancing round robins messages |
| across the cluster. If <code>message-load-balancing</code> is <code>OFF</code> or <code>ON_DEMAND</code> |
| then messages won't be forwarded to nodes which don't have matching |
| consumers. This is great and ensures that messages aren't moved to a |
| queue which has no consumers to consume them. However, there is a |
| situation it doesn't solve: What happens if the consumers on a queue |
| close after the messages have been sent to the node? If there are no |
| consumers on the queue the message won't get consumed and we have a |
| <em>starvation</em> situation.</p> |
| <p>This is where message redistribution comes in. With message |
| redistribution Apache ActiveMQ Artemis can be configured to automatically |
| <em>redistribute</em> messages from queues which have no consumers back to |
| other nodes in the cluster which do have matching consumers. To enable |
| this functionality <code>message-load-balancing</code> must be <code>ON_DEMAND</code>.</p> |
| <p>Message redistribution can be configured to kick in immediately after |
| the last consumer on a queue is closed, or to wait a configurable delay |
| after the last consumer on a queue is closed before redistributing. By |
| default message redistribution is disabled.</p> |
| <p>Message redistribution can be configured on a per address basis, by |
| specifying the redistribution delay in the address settings, for more |
| information on configuring address settings, please see <a href="queue-attributes.html">Queue Attributes</a>.</p> |
| <p>Here's an address settings snippet from <code>broker.xml</code> |
| showing how message redistribution is enabled for a set of queues:</p> |
| <pre><code><address-settings> |
| <address-setting match="jms.#"> |
| <redistribution-delay>0</redistribution-delay> |
| </address-setting> |
| </address-settings> |
| </code></pre><p>The above <code>address-settings</code> block would set a <code>redistribution-delay</code> of |
| <code>0</code> for any queue which is bound to an address that starts with "jms.". |
| All JMS queues and topic subscriptions are bound to addresses that start |
| with "jms.", so the above would enable instant (no delay) redistribution |
| for all JMS queues and topic subscriptions.</p> |
| <p>The attribute <code>match</code> can be an exact match or it can be a string that |
| conforms to the Apache ActiveMQ Artemis wildcard syntax (described in <a href="wildcard-syntax.html">Wildcard Syntax</a>).</p> |
| <p>The element <code>redistribution-delay</code> defines the delay in milliseconds |
| after the last consumer is closed on a queue before redistributing |
| messages from that queue to other nodes of the cluster which do have |
| matching consumers. A delay of zero means the messages will be |
| immediately redistributed. A value of <code>-1</code> signifies that messages will |
| never be redistributed. The default value is <code>-1</code>.</p> |
| <p>It often makes sense to introduce a delay before redistributing as it's |
| a common case that a consumer closes but another one quickly is created |
| on the same queue, in such a case you probably don't want to |
| redistribute immediately since the new consumer will arrive shortly.</p> |
| <h2 id="cluster-topologies">Cluster topologies</h2> |
| <p>Apache ActiveMQ Artemis clusters can be connected together in many different |
| topologies, let's consider the two most common ones here</p> |
| <h3 id="symmetric-cluster">Symmetric cluster</h3> |
| <p>A symmetric cluster is probably the most common cluster topology.</p> |
| <p>With a symmetric cluster every node in the cluster is connected to every |
| other node in the cluster. In other words every node in the cluster is |
| no more than one hop away from every other node.</p> |
| <p>To form a symmetric cluster every node in the cluster defines a cluster |
| connection with the attribute <code>max-hops</code> set to <code>1</code>. Typically the |
| cluster connection will use server discovery in order to know what other |
| servers in the cluster it should connect to, although it is possible to |
| explicitly define each target server too in the cluster connection if, |
| for example, UDP is not available on your network.</p> |
| <p>With a symmetric cluster each node knows about all the queues that exist |
| on all the other nodes and what consumers they have. With this knowledge |
| it can determine how to load balance and redistribute messages around |
| the nodes.</p> |
| <p>Don't forget <a href="#copy-warning">this warning</a> when creating a symmetric |
| cluster.</p> |
| <h3 id="chain-cluster">Chain cluster</h3> |
| <p>With a chain cluster, each node in the cluster is not connected to every |
| node in the cluster directly, instead the nodes form a chain with a node |
| on each end of the chain and all other nodes just connecting to the |
| previous and next nodes in the chain.</p> |
| <p>An example of this would be a three node chain consisting of nodes A, B |
| and C. Node A is hosted in one network and has many producer clients |
| connected to it sending order messages. Due to corporate policy, the |
| order consumer clients need to be hosted in a different network, and |
| that network is only accessible via a third network. In this setup node |
| B acts as a mediator with no producers or consumers on it. Any messages |
| arriving on node A will be forwarded to node B, which will in turn |
| forward them to node C where they can get consumed. Node A does not need |
| to directly connect to C, but all the nodes can still act as a part of |
| the cluster.</p> |
| <p>To set up a cluster in this way, node A would define a cluster |
| connection that connects to node B, and node B would define a cluster |
| connection that connects to node C. In this case we only want cluster |
| connections in one direction since we're only moving messages from node |
| A->B->C and never from C->B->A.</p> |
| <p>For this topology we would set <code>max-hops</code> to <code>2</code>. With a value of <code>2</code> |
| the knowledge of what queues and consumers that exist on node C would be |
| propagated from node C to node B to node A. Node A would then know to |
| distribute messages to node B when they arrive, even though node B has |
| no consumers itself, it would know that a further hop away is node C |
| which does have consumers.</p> |
| <h3 id="scaling-down">Scaling Down</h3> |
| <p>Apache ActiveMQ Artemis supports scaling down a cluster with no message loss (even for |
| non-durable messages). This is especially useful in certain environments |
| (e.g. the cloud) where the size of a cluster may change relatively |
| frequently. When scaling up a cluster (i.e. adding nodes) there is no |
| risk of message loss, but when scaling down a cluster (i.e. removing |
| nodes) the messages on those nodes would be lost unless the broker sent |
| them to another node in the cluster. Apache ActiveMQ Artemis can be configured to do |
| just that.</p> |
| <p>The simplest way to enable this behavior is to set <code>scale-down</code> to |
| <code>true</code>. If the server is clustered and <code>scale-down</code> is <code>true</code> then when |
| the server is shutdown gracefully (i.e. stopped without crashing) it |
| will find another node in the cluster and send <em>all</em> of its messages |
| (both durable and non-durable) to that node. The messages are processed |
| in order and go to the <em>back</em> of the respective queues on the other node |
| (just as if the messages were sent from an external client for the first |
| time).</p> |
| <p>If more control over where the messages go is required then specify |
| <code>scale-down-group-name</code>. Messages will only be sent to another node in |
| the cluster that uses the same <code>scale-down-group-name</code> as the server |
| being shutdown.</p> |
| <blockquote> |
| <p><strong>Warning</strong></p> |
| <p>If cluster nodes are grouped together with different |
| <code>scale-down-group-name</code> values beware. If all the nodes in a single |
| group are shut down then the messages from that node/group will be |
| lost.</p> |
| </blockquote> |
| <p>If the server is using multiple <code>cluster-connection</code> then use |
| <code>scale-down-clustername</code> to identify the name of the |
| <code>cluster-connection</code> which should be used for scaling down.</p> |
| |
| |
| </section> |
| |
| |
| </div> |
| </div> |
| </div> |
| |
| |
| <a href="./duplicate-detection.html" class="navigation navigation-prev " aria-label="Previous page: Duplicate Message Detection"><i class="fa fa-angle-left"></i></a> |
| |
| |
| <a href="./ha.html" class="navigation navigation-next " aria-label="Next page: High Availability and Failover"><i class="fa fa-angle-right"></i></a> |
| |
| </div> |
| </div> |
| |
| |
| <script src="gitbook/app.js"></script> |
| |
| <script> |
| require(["gitbook"], function(gitbook) { |
| var config = {"fontSettings":{"theme":null,"family":"sans","size":2}}; |
| gitbook.start(config); |
| }); |
| </script> |
| |
| |
| </body> |
| |
| </html> |