Google Git
Sign in
apache / tomee-site-pub / 43cf80e624a0cecf0f38bef5ef2614bfb2410f57 / . / tomee-9.0 / docs / admin / cluster / index.html
blob: 73a50e7db42e067d49c7e333851943296481ea3f [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Apache TomEE</title>
<meta name="description"
content="Apache TomEE is a lightweight, yet powerful, JavaEE Application server with feature rich tooling." />
<meta name="keywords" content="tomee,asf,apache,javaee,jee,shade,embedded,test,junit,applicationcomposer,maven,arquillian" />
<meta name="author" content="Luka Cvetinovic for Codrops" />
<link rel="icon" href="../../../../favicon.ico">
<link rel="icon" type="image/png" href="../../../../favicon.png">
<meta name="msapplication-TileColor" content="#80287a">
<meta name="theme-color" content="#80287a">
<link rel="stylesheet" type="text/css" href="../../../../css/normalize.css">
<link rel="stylesheet" type="text/css" href="../../../../css/bootstrap.css">
<link rel="stylesheet" type="text/css" href="../../../../css/owl.css">
<link rel="stylesheet" type="text/css" href="../../../../css/animate.css">
<link rel="stylesheet" type="text/css" href="../../../../fonts/font-awesome-4.1.0/css/font-awesome.min.css">
<link rel="stylesheet" type="text/css" href="../../../../fonts/eleganticons/et-icons.css">
<link rel="stylesheet" type="text/css" href="../../../../css/jqtree.css">
<link rel="stylesheet" type="text/css" href="../../../../css/idea.css">
<link rel="stylesheet" type="text/css" href="../../../../css/cardio.css">
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-2717626-1']);
_gaq.push(['_setDomainName', 'apache.org']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<div class="preloader">
<img src="../../../../img/loader.gif" alt="Preloader image">
</div>
<nav class="navbar">
<div class="container">
<div class="row"> <div class="col-md-12">
<!-- Brand and toggle get grouped for better mobile display -->
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="/">
<span>
<img src="../../../../img/logo-active.png">
</span>
Apache TomEE
</a>
</div>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right main-nav">
<li><a href="../../../../docs.html">Documentation</a></li>
<li><a href="../../../../community/index.html">Community</a></li>
<li><a href="../../../../security/security.html">Security</a></li>
<li><a href="../../../../download-ng.html">Downloads</a></li>
</ul>
</div>
<!-- /.navbar-collapse -->
</div></div>
</div>
<!-- /.container-fluid -->
</nav>
<div id="main-block" class="container main-block">
<div class="row title">
<div class="col-md-12">
<div class='page-header'>
<h1>Clustering and High Availability (HA)</h1>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="sect1">
<h2 id="_session_clustering">Session clustering</h2>
<div class="sectionbody">
<div class="paragraph">
<p>TomEE fully relies on Tomcat clustering: <a href="https://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html">Tomcat Clustering</a>.</p>
</div>
<div class="paragraph">
<p>The configuration is mainly in <code>conf/server.xml</code> and since TomEE 7 CDI <code>@SessionScoped</code> is transparently clustered
through the session.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_hazelcast_as_session_provider">Hazelcast as session provider</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Hazelcast did a post on this topic on <a href="https://hazelcast.com/use-cases/web-session-clustering/session-clustering-tomee/">Session Clustering With TomEE</a>.</p>
</div>
<div class="paragraph">
<p>Tomitribe also demonstrated you can distributed <code>@Stateful</code> beans easily relying on hazelcast: <a href="https://github.com/tomitribe/hazelcast-tomee-poc">Hazelcast TomEE PoC</a>.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_load_balancing">Load balancing</h2>
<div class="sectionbody">
<div class="paragraph">
<p>TomEE being a HTTP server all HTTP load balancer such as HTTPd (a.k.a. Apache2), ngnix, F5 etc&#8230;&#8203; will work.</p>
</div>
<div class="paragraph">
<p>More documentation on HTTPd link can be found on <a href="https://tomcat.apache.org/connectors-doc/webserver_howto/apache.html">Tomcat</a> website.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_ejbd">EJBd</h2>
<div class="sectionbody">
<div class="paragraph">
<p>If you use the EJBd protocol (<code>@Remote</code> EJB proprietary protocol of TomEE) you can get cluster features on the client
part.</p>
</div>
<div class="sect2">
<h3 id="_multicast">Multicast</h3>
<div class="paragraph">
<p>Multicast is the preferred way to broadcast the heartbeat on the network. The simple technique of broadcasting a non-changing service URI on the network has specific advantages to multicast. The URI itself is essentially stateless and there is no "i&#8217;m alive" URI or an "i&#8217;m dead" URI.</p>
</div>
<div class="paragraph">
<p>In this way the issues with UDP being unordered and unreliable melt away as state is no longer a concern and packet sizes are always small. Complicated libraries that ride atop UDP and attempt to offer reliability (retransmission) and ordering on UDP can be avoided. As well the advantages UDP has over TCP are retained as there are no java layers attempting to force UDP communication to be more TCP-like. The simple design means UDP/Multicast is only used for discovery and from there on out critical information is transmitted over TCP/IP which is obviously going to do a better job at ensuring reliability and ordering.</p>
</div>
<div class="sect3">
<h4 id="_server_configuration">Server Configuration</h4>
<div class="paragraph">
<p>When you boot the server there should be a conf/multicast.properties file containing:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-properties" data-lang="properties">server = org.apache.openejb.server.discovery.MulticastDiscoveryAgent
bind = 239.255.2.3
port = 6142
disabled = true
group = default</code></pre>
</div>
</div>
<div class="paragraph">
<p>Just need to enable that by setting disabled=false. All of the above settings except server can be changed. The port and bind must be valid for general multicast/udp network communication.</p>
</div>
<div class="paragraph">
<p>The group setting can be changed to further group servers that may use the same multicast channel. As shown below the client also has a group setting which can be used to select an appropriate server from the multicast channel.</p>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<i class="fa icon-important" title="Important"></i>
</td>
<td class="content">
for multicast to work you need to have ejbd activated as a normal service. This can be done setting in <code>conf/system.properties</code> the entry: <code>openejb.service.manager.class = org.apache.openejb.server.SimpleServiceManager</code>.
</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
for multicast to work you need to have ejbd activated as a normal service. This can be done setting in <code>conf/system.properties</code> the entry: <code>openejb.service.manager.class = org.apache.openejb.server.SimpleServiceManager</code>.
</td>
</tr>
</table>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
for multicast to work you need to have ejbd activated as a normal service. This can be done setting in <code>conf/system.properties</code> the entry: <code>openejb.service.manager.class = org.apache.openejb.server.SimpleServiceManager</code>.
</td>
</tr>
</table>
</div>
<div class="admonitionblock caution">
<table>
<tr>
<td class="icon">
<i class="fa icon-caution" title="Caution"></i>
</td>
<td class="content">
for multicast to work you need to have ejbd activated as a normal service. This can be done setting in <code>conf/system.properties</code> the entry: <code>openejb.service.manager.class = org.apache.openejb.server.SimpleServiceManager</code>.
</td>
</tr>
</table>
</div>
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
for multicast to work you need to have ejbd activated as a normal service. This can be done setting in <code>conf/system.properties</code> the entry: <code>openejb.service.manager.class = org.apache.openejb.server.SimpleServiceManager</code>.
</td>
</tr>
</table>
</div>
<div class="sect4">
<h5 id="_multicast_client">Multicast Client</h5>
<div class="paragraph">
<p>The multicast functionality is not just for servers to find each other in a cluster, it can also be used for EJB clients to discover a server. A special multicast:// URL can be used in the InitialContext properties to signify that multicast should be used to seed the connection process. Such as:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY,
"org.apache.openejb.client.RemoteInitialContextFactory");
p.put(Context.PROVIDER_URL, "multicast://239.255.2.3:6142?group=default");
InitialContext remoteContext = new InitialContext(p);</code></pre>
</div>
</div>
<div class="paragraph">
<p>The URL has optional query parameters such as schemes and group and timeout which allow you to zero in on a particular type of service of a particular cluster group as well as set how long you are willing to wait in the discovery process till finally giving up. The first matching service that it sees "flowing" around on the UDP stream is the one it picks and sticks to for that and subsequent requests, ensuring UDP is only used when there are no other servers to talk to.</p>
</div>
<div class="paragraph">
<p>Note that EJB clients do not need to use multicast to find a server. If the client knows the URL of a server in the cluster, it may use it and connect directly to that server, at which point that server will share the full list of its peers.</p>
</div>
</div>
<div class="sect4">
<h5 id="_multicast_servers_with_tcp_clients">Multicast Servers with TCP Clients</h5>
<div class="paragraph">
<p>Note that clients do not need to use multicast to communicate with servers. Servers can use multicast to discover each other, but clients are still free to connect to servers in the network using the server&#8217;s TCP address.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory");
p.put(Context.PROVIDER_URL, "ejbd://192.168.1.30:4201");
InitialContext remoteContext = new InitialContext(p);
When the client connects, the server will send the URLs of all the servers in the group and failover will take place normally.</code></pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="_multipulse">Multipulse</h4>
<div class="paragraph">
<p>MultiPulse is an alternative multicast lookup that does not use a regular heartbeat. Instead, servers listen for a multicast request packet (a pulse) to which a response is then sent. Multicast network traffic is effectively reduced to an absolute minimum.</p>
</div>
<div class="paragraph">
<p>MultiPulse is only useful in network scenarios where both client and server can be configured to send multicast UDP packets.</p>
</div>
<div class="sect4">
<h5 id="_server_configuration_2">Server Configuration</h5>
<div class="paragraph">
<p>After you boot the server for the first time the default configuration will create the file conf/conf.d/multipulse.properties containing:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-properties" data-lang="properties">server = org.apache.openejb.server.discovery.MulticastPulseAgent
bind = 239.255.2.3
port = 6142
disabled = true
group = default</code></pre>
</div>
</div>
<div class="paragraph">
<p>You just need to enable the agent by setting disabled = false. It is advisable to disable multicast in the multicast.properties file, or at least to use a different bind address or port should you wish to use both.</p>
</div>
<div class="paragraph">
<p>All of the above settings except server can be modified as required. The port and bind must be valid for general multicast/udp network communication.</p>
</div>
<div class="paragraph">
<p>The group setting can be changed to further group/cluster servers that may use the same multicast channel. As shown below the client also has an optional group setting which can be used to select an appropriate server cluster from the multicast channel (See MultiPulse Client).</p>
</div>
<div class="paragraph">
<p>The next step is to ensure that the advertised services are configured for discovery. Edit the ejbd.properties file (and any other enabled services such as http, etc.) and ensure that the discovery option is set to a value that remote clients will be able to resolve.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-properties" data-lang="properties">server = org.apache.openejb.server.ejbd.EjbServer
bind = 0.0.0.0
port = 4201
disabled = false
threads = 20
discovery = ejb:ejbd://{bind}:{port}</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
If either 0.0.0.0 (IPv4) or [::] (IPv6) wildcard bind addresses are used then the server will actually broadcast all of it&#8217;s known public hosts to clients. Clients will then cycle though and attempt to connect to the provided hosts until successful.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>If localhost is used then only clients on the same physical machine will actually 'see' the server response.</p>
</div>
</div>
<div class="sect4">
<h5 id="_multipulse_client">MultiPulse Client</h5>
<div class="paragraph">
<p>The multipulse functionality is not just for servers to find each other in a cluster, it can also be used for EJB clients to discover a server. A special multipulse:// URL can be used in the InitialContext properties to signify that multipulse should be used to seed the connection process. Such as:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory");
p.put(Context.PROVIDER_URL, "multipulse://239.255.2.3:6142?group=default&amp;timeout=250");
InitialContext remoteContext = new InitialContext(p);</code></pre>
</div>
</div>
<div class="paragraph">
<p>The URL has optional query parameters such as schemes and group and timeout which allow you to zero in on a particular type of service of a particular cluster group as well as set how long you are willing to wait in the discovery process till finally giving up. The first matching service that it sees "flowing" around on the UDP stream is the one it picks and sticks to for that and subsequent requests, ensuring UDP is only used when there are no other servers to talk to.</p>
</div>
<div class="paragraph">
<p>Note that EJB clients do not need to use multipulse to find a server. If the client knows the URL of a server in the cluster, it may use it and connect directly to that server, at which point that server will share the full list of its peers.</p>
</div>
<div class="paragraph">
<p>Multicast Servers with TCP Clients</p>
</div>
<div class="paragraph">
<p>Note that clients do not need to use multipulse to communicate with servers. Servers can use multicast to discover each other, but clients are still free to connect to servers in the network using the server&#8217;s TCP address.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory");
p.put(Context.PROVIDER_URL, "ejbd://192.168.1.30:4201");
InitialContext remoteContext = new InitialContext(p);</code></pre>
</div>
</div>
<div class="paragraph">
<p>When the client connects, the server will send the URLs of all the servers in the group and failover will take place normally.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="_multipoint">Multipoint</h4>
<div class="paragraph">
<p>As TCP has no real broadcast functionality to speak of, communication of who is in the network is achieved by each server having a physical connection to each other server in the network.</p>
</div>
<div class="paragraph">
<p>To join the network, the server must be configured to know the address of at least one server in the network and connect to it. When it does both servers will exchange the full list of all the other servers each knows about. Each server will then connect to any new servers they&#8217;ve just learned about and repeat the processes with those new servers. The end result is that everyone has a direct connection to everyone 100% of the time, hence the made-up term "multipoint" to describe this situation of each server having multiple point-to-point connections which create a fully connected graph.</p>
</div>
<div class="paragraph">
<p>On the client side things are similar. It needs to know the address of at least one server in the network and be able to connect to it. When it does it will get the full (and dynamically maintained) list of every server in the network. The client doesn&#8217;t connect to each of those servers immediately, but rather consults the list in the event of a failover, using it to decide who to connect to next.</p>
</div>
<div class="paragraph">
<p>The entire process is essentially the art of using a statically maintained list to bootstrap getting the more valuable dynamically maintained list.</p>
</div>
<div class="sect4">
<h5 id="_server_configuration_3">Server Configuration</h5>
<div class="paragraph">
<p>In the server this list can be specified via the conf/multipoint.properties file like so:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-properties" data-lang="properties">server = org.apache.openejb.server.discovery.MultipointDiscoveryAgent
bind = 127.0.0.1
port = 4212
disabled = false
initialServers = 192.168.1.20:4212, 192.168.1.30:4212, 192.168.1.40:4212</code></pre>
</div>
</div>
<div class="paragraph">
<p>The above configuration shows the server has an port 4212 open for connections by other servers for multipoint communication. The initialServers list should be a comma separated list of other similar servers on the network. Only one of the servers listed is required to be running when this server starts up&#8201;&#8212;&#8201;it is not required to list all servers in the network.</p>
</div>
</div>
<div class="sect4">
<h5 id="_client_configuration">Client Configuration</h5>
<div class="paragraph">
<p>Configuration in the client is similar, but note that EJB clients do not participate directly in multipoint communication and do not connect to the multipoint port. The server list is simply a list of the regular ejbd:// urls that a client normally uses to connect to a server.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory");
p.put(Context.PROVIDER_URL, "failover:ejbd://192.168.1.20:4201,ejbd://192.168.1.30:4201");
InitialContext remoteContext = new InitialContext(p);</code></pre>
</div>
</div>
<div class="paragraph">
<p>Failover can work entirely driven by the server, the client does not need to be configured to participate. A client can connect as usual to the server.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory");
p.put(Context.PROVIDER_URL, "ejbd://192.168.1.20:4201");
InitialContext remoteContext = new InitialContext(p);</code></pre>
</div>
</div>
<div class="paragraph">
<p>If the server at 192.168.1.20:4201 supports failover, so will the client.</p>
</div>
<div class="paragraph">
<p>In this scenario the list of servers used for failover is supplied entirely by the server at 192.168.1.20:4201. The server could have aquired the list via multicast or multipoint (or both), but this detail is not visible to the client.</p>
</div>
</div>
<div class="sect4">
<h5 id="_considerations">Considerations</h5>
<div class="sect5">
<h6 id="_network_size">Network size</h6>
<div class="paragraph">
<p>The general disadvantage of this topology is the number of connections required. The number of connections for the network of servers is equal to (n * n - n) / 2, where n is the number of servers. For example, with 5 servers you need 10 connections, with 10 servers you need 45 connections, and with 50 servers you need 1225 connections. This is of course the number of connections across the entire network, each individual server only needs n - 1 connections.</p>
</div>
<div class="paragraph">
<p>The handling of these sockets is all asynchronous Java NIO code which allows the server to handle many connections (all of them) with one thread. From a pure threading perspective, the option is extremely efficient with just one thread to listen and broadcast to many peers.</p>
</div>
</div>
<div class="sect5">
<h6 id="_double_connect">Double connect</h6>
<div class="paragraph">
<p>It is possible in this process that two servers learn of each other at the same time and each attempts to connect to the other simultaneously, resulting in two connections between the same two servers. When this happens both servers will detect the extra connection and one of the connections will be dropped and one will be kept. In practice this race condition rarely happens and can be avoided almost entirely by fanning out server startup by as little as 100 milliseconds.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="_recommandation">Recommandation</h5>
<div class="paragraph">
<p>As mentioned the initialServers is only used for bootstrapping the multipoint network. Once running, all servers will dynamically establish direct connections with each other and there is no single point of failure.</p>
</div>
<div class="paragraph">
<p>However to ensure that the bootstrapping process can occur successfully, the initialServers property of the conf/multipoint.properties file must be set carefully and with a specific server start order in mind. Each server consults its initialServers list exactly once in the bootstrapping phase at startup, after that time connections are made dynamically.</p>
</div>
<div class="paragraph">
<p>This means that at least one of the servers listed in initialServers must already be running when the server starts or the server might never become introduced and connected to all the other servers in the network.</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<footer>
<div class="container">
<div class="row">
<div class="col-sm-6 text-center-mobile">
<h3 class="white">Be simple. Be certified. Be Tomcat.</h3>
<h5 class="light regular light-white">"A good application in a good server"</h5>
<ul class="social-footer">
<li><a href="https://www.facebook.com/ApacheTomEE/"><i class="fa fa-facebook"></i></a></li>
<li><a href="https://twitter.com/apachetomee"><i class="fa fa-twitter"></i></a></li>
<li><a href="https://plus.google.com/communities/105208241852045684449"><i class="fa fa-google-plus"></i></a></li>
</ul>
</div>
<div class="col-sm-6 text-center-mobile">
<div class="row opening-hours">
<div class="col-sm-3 text-center-mobile">
<h5><a href="../../../../latest/docs/" class="white">Documentation</a></h5>
<ul class="list-unstyled">
<li><a href="../../../../latest/docs/admin/configuration/index.html" class="regular light-white">How to configure</a></li>
<li><a href="../../../../latest/docs/admin/file-layout.html" class="regular light-white">Dir. Structure</a></li>
<li><a href="../../../../latest/docs/developer/testing/index.html" class="regular light-white">Testing</a></li>
<li><a href="../../../../latest/docs/admin/cluster/index.html" class="regular light-white">Clustering</a></li>
</ul>
</div>
<div class="col-sm-3 text-center-mobile">
<h5><a href="../../../../latest/examples/" class="white">Examples</a></h5>
<ul class="list-unstyled">
<li><a href="../../../../latest/examples/simple-cdi-interceptor.html" class="regular light-white">CDI Interceptor</a></li>
<li><a href="../../../../latest/examples/rest-cdi.html" class="regular light-white">REST with CDI</a></li>
<li><a href="../../../../latest/examples/ejb-examples.html" class="regular light-white">EJB</a></li>
<li><a href="../../../../latest/examples/jsf-managedBean-and-ejb.html" class="regular light-white">JSF</a></li>
</ul>
</div>
<div class="col-sm-3 text-center-mobile">
<h5><a href="../../../../community/index.html" class="white">Community</a></h5>
<ul class="list-unstyled">
<li><a href="../../../../community/contributors.html" class="regular light-white">Contributors</a></li>
<li><a href="../../../../community/social.html" class="regular light-white">Social</a></li>
<li><a href="../../../../community/sources.html" class="regular light-white">Sources</a></li>
</ul>
</div>
<div class="col-sm-3 text-center-mobile">
<h5><a href="../../../../security/index.html" class="white">Security</a></h5>
<ul class="list-unstyled">
<li><a href="http://apache.org/security" target="_blank" class="regular light-white">Apache Security</a></li>
<li><a href="http://apache.org/security/projects.html" target="_blank" class="regular light-white">Security Projects</a></li>
<li><a href="http://cve.mitre.org" target="_blank" class="regular light-white">CVE</a></li>
</ul>
</div>
</div>
</div>
</div>
<div class="row bottom-footer text-center-mobile">
<div class="col-sm-12 light-white">
<p>Copyright &copy; 1999-2016 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. Apache TomEE, TomEE, Apache, the Apache feather logo, and the Apache TomEE project logo are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
</div>
</div>
</div>
</footer>
<!-- Holder for mobile navigation -->
<div class="mobile-nav">
<ul>
<li><a hef="../../../../latest/docs/admin/index.html">Administrators</a>
<li><a hef="../../../../latest/docs/developer/index.html">Developers</a>
<li><a hef="../../../../latest/docs/advanced/index.html">Advanced</a>
<li><a hef="../../../../community/index.html">Community</a>
</ul>
<a href="#" class="close-link"><i class="arrow_up"></i></a>
</div>
<!-- Scripts -->
<script src="../../../../js/jquery-1.11.1.min.js"></script>
<script src="../../../../js/owl.carousel.min.js"></script>
<script src="../../../../js/bootstrap.min.js"></script>
<script src="../../../../js/wow.min.js"></script>
<script src="../../../../js/typewriter.js"></script>
<script src="../../../../js/jquery.onepagenav.js"></script>
<script src="../../../../js/tree.jquery.js"></script>
<script src="../../../../js/highlight.pack.js"></script>
<script src="../../../../js/main.js"></script>
</body>
</html>