| |
| |
| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <meta name="description" content="Apache Ozone Documentation"> |
| |
| <title>Documentation for Apache Ozone</title> |
| |
| |
| <link href="../css/bootstrap.min.css" rel="stylesheet"> |
| |
| |
| <link href="../css/ozonedoc.css" rel="stylesheet"> |
| |
| |
| |
| <link href="../swagger-resources/swagger-ui.css" rel="stylesheet"> |
| |
| |
| <script> |
| var _paq = window._paq = window._paq || []; |
| |
| |
| |
| _paq.push(['disableCookies']); |
| |
| |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="//analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '34']); |
| var d=document, g=d.createElement('script'), |
| s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| |
| |
| </head> |
| |
| |
| <body> |
| |
| |
| <nav class="navbar navbar-inverse navbar-fixed-top"> |
| <div class="container-fluid"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a href="../index.html" class="navbar-left ozone-logo"> |
| <img src="../ozone-logo-small.png"/> |
| </a> |
| <a class="navbar-brand hidden-xs" href="../index.html"> |
| Apache Ozone/HDDS Documentation |
| </a> |
| <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a> |
| </div> |
| <div id="navbar" class="navbar-collapse collapse"> |
| <ul class="nav navbar-nav navbar-right"> |
| <li><a href="https://github.com/apache/ozone">Source</a></li> |
| <li><a href="https://ozone.apache.org">Apache Ozone</a></li> |
| <li><a href="https://apache.org">ASF</a></li> |
| </ul> |
| </div> |
| </div> |
| </nav> |
| |
| |
| <div class="wrapper"> |
| <div class="container-fluid"> |
| <div class="row"> |
| |
| <div class="col-sm-2 col-md-2 sidebar" id="sidebar"> |
| <ul class="nav nav-sidebar"> |
| |
| |
| |
| <li class=""> |
| |
| <a href="../index.html"> |
| |
| |
| |
| <span>Overview</span> |
| </a> |
| </li> |
| |
| |
| |
| <li class=""> |
| |
| <a href="../start.html"> |
| |
| |
| |
| <span>Getting Started</span> |
| </a> |
| </li> |
| |
| |
| |
| <li class=""> |
| <a href="../concept.html"> |
| |
| <span>Architecture</span> |
| </a> |
| <ul class="nav"> |
| |
| <li class=""> |
| |
| <a href="../concept/overview.html">Overview</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../concept/ozonemanager.html">Ozone Manager</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../concept/storagecontainermanager.html">Storage Container Manager</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../concept/containers.html">Containers</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../concept/datanodes.html">Datanodes</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../concept/recon.html">Recon</a> |
| |
| </li> |
| |
| </ul> |
| </li> |
| |
| |
| |
| <li class=""> |
| <a href="../feature.html"> |
| |
| <span>Features</span> |
| </a> |
| <ul class="nav"> |
| |
| <li class=""> |
| |
| <a href="../feature/decommission.html">Decommissioning</a> |
| |
| </li> |
| |
| <li class="active"> |
| |
| <a href="../feature/om-ha.html">OM High Availability</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/erasurecoding.html">Ozone Erasure Coding</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/snapshot.html">Ozone Snapshot</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/scm-ha.html">SCM High Availability</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/streaming-write-pipeline.html">Streaming Write Pipeline</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/dn-merge-rocksdb.html">Merge Container RocksDB in DN</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/prefixfso.html">Prefix based File System Optimization</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/topology.html">Topology awareness</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/quota.html">Quota in Ozone</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/recon.html">Recon Server</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/observability.html">Observability</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/s3-multi-tenancy.html"> |
| |
| <span>S3 Multi-Tenancy</span> |
| </a> |
| <ul class="nav"> |
| |
| <li class=""> |
| <a href="../feature/s3-multi-tenancy-setup.html">Setup</a> |
| </li> |
| |
| <li class=""> |
| <a href="../feature/s3-tenant-commands.html">Tenant commands</a> |
| </li> |
| |
| <li class=""> |
| <a href="../feature/s3-multi-tenancy-access-control.html">Access Control</a> |
| </li> |
| |
| </ul> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../feature/reconfigurability.html">Reconfigurability</a> |
| |
| </li> |
| |
| </ul> |
| </li> |
| |
| |
| |
| <li class=""> |
| <a href="../interface.html"> |
| |
| <span>Client Interfaces</span> |
| </a> |
| <ul class="nav"> |
| |
| <li class=""> |
| |
| <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../interface/s3.html">S3 Protocol</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../interface/cli.html">Command Line Interface</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../interface/reconapi.html">Recon API</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../interface/javaapi.html">Java API</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../interface/csi.html">CSI Protocol</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../interface/httpfs.html">HttpFS Gateway</a> |
| |
| </li> |
| |
| </ul> |
| </li> |
| |
| |
| |
| <li class=""> |
| <a href="../security.html"> |
| |
| <span>Security</span> |
| </a> |
| <ul class="nav"> |
| |
| <li class=""> |
| |
| <a href="../security/secureozone.html">Securing Ozone</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../security/securingtde.html">Transparent Data Encryption</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../security/gdpr.html">GDPR in Ozone</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../security/securingdatanodes.html">Securing Datanodes</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../security/securingozonehttp.html">Securing HTTP</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../security/securings3.html">Securing S3</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../security/securityacls.html">Ozone ACLs</a> |
| |
| </li> |
| |
| <li class=""> |
| |
| <a href="../security/securitywithranger.html">Apache Ranger</a> |
| |
| </li> |
| |
| </ul> |
| </li> |
| |
| |
| |
| <li class=""> |
| |
| <a href="../tools.html"> |
| |
| |
| |
| <span>Tools</span> |
| </a> |
| </li> |
| |
| |
| |
| <li class=""> |
| |
| <a href="../recipe.html"> |
| |
| |
| |
| <span>Recipes</span> |
| </a> |
| </li> |
| |
| |
| <li><a href="../design.html"><span><b>Design docs</b></span></a></li> |
| <li class="visible-xs"><a href="#">References</a> |
| <ul class="nav"> |
| <li><a href="https://github.com/apache/ozone"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li> |
| <li><a href="https://ozone.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Ozone</a></li> |
| <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li> |
| </ul></li> |
| </ul> |
| |
| </div> |
| |
| <div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main-content"> |
| |
| |
| |
| <div class="col-md-9"> |
| <nav aria-label="breadcrumb"> |
| <ol class="breadcrumb"> |
| <li class="breadcrumb-item"><a href="../index.html">Home</a></li> |
| <li class="breadcrumb-item" aria-current="page"><a href="../feature.html">Features</a></li> |
| <li class="breadcrumb-item active" aria-current="page">OM High Availability</li> |
| </ol> |
| </nav> |
| |
| |
| |
| <div class="pull-right"> |
| |
| |
| |
| |
| |
| <a href="../zh/feature/om-ha.html"><span class="label label-success">ä¸æ–‡</span></a> |
| |
| |
| </div> |
| |
| |
| <div class="col-md-9"> |
| <h1>OM High Availability</h1> |
| |
| <!--- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>Ozone has two metadata-manager nodes (<em>Ozone Manager</em> for key space management and <em>Storage Container Manager</em> for block space management) and multiple storage nodes (Datanode). Data is replicated between Datanodes with the help of RAFT consensus algorithm.</p> |
| <p>To avoid any single point of failure the metadata-manager nodes also should have a HA setup.</p> |
| <p>Both Ozone Manager and Storage Container Manager supports HA. In this mode the internal state is replicated via RAFT (with Apache Ratis)</p> |
| <p>This document explains the HA setup of Ozone Manager (OM) HA, please check <a href="../feature/scm-ha.html">this page</a> for SCM HA. While they can be setup for HA independently, a reliable, full HA setup requires enabling HA for both services.</p> |
| <h2 id="ozone-manager-ha">Ozone Manager HA</h2> |
| <p>A single Ozone Manager uses <a href="https://github.com/facebook/rocksdb/">RocksDB</a> to persist metadata (volumes, buckets, keys) locally. HA version of Ozone Manager does exactly the same but all the data is replicated with the help of the RAFT consensus algorithm to follower Ozone Manager instances.</p> |
| <p> |
| |
| <img src="HA-OM.png" alt='HA OM' class="img-responsive" /></p> |
| <p>Client connects to the Leader Ozone Manager which process the request and schedule the replication with RAFT. When the request is replicated to all the followers the leader can return with the response.</p> |
| <h2 id="configuration">Configuration</h2> |
| <p>HA mode of Ozone Manager can be enabled with the following settings in <code>ozone-site.xml</code>:</p> |
| <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-XML" data-lang="XML"><span style="color:#f92672"><property></span> |
| <span style="color:#f92672"><name></span>ozone.om.ratis.enable<span style="color:#f92672"></name></span> |
| <span style="color:#f92672"><value></span>true<span style="color:#f92672"></value></span> |
| <span style="color:#f92672"></property></span> |
| </code></pre></div><p>One Ozone configuration (<code>ozone-site.xml</code>) can support multiple Ozone HA cluster. To select between the available HA clusters a logical name is required for each of the clusters which can be resolved to the IP addresses (and domain names) of the Ozone Managers.</p> |
| <p>This logical name is called <code>serviceId</code> and can be configured in the <code>ozone-site.xml</code></p> |
| <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-XML" data-lang="XML"><span style="color:#f92672"><property></span> |
| <span style="color:#f92672"><name></span>ozone.om.service.ids<span style="color:#f92672"></name></span> |
| <span style="color:#f92672"><value></span>cluster1<span style="color:#f92672"></value></span> |
| <span style="color:#f92672"></property></span> |
| </code></pre></div><p>For each of the defined <code>serviceId</code> a logical configuration name should be defined for each of the servers.</p> |
| <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-XML" data-lang="XML"><span style="color:#f92672"><property></span> |
| <span style="color:#f92672"><name></span>ozone.om.nodes.cluster1<span style="color:#f92672"></name></span> |
| <span style="color:#f92672"><value></span>om1,om2,om3<span style="color:#f92672"></value></span> |
| <span style="color:#f92672"></property></span> |
| </code></pre></div><p>The defined prefixes can be used to define the address of each of the OM services:</p> |
| <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-XML" data-lang="XML"><span style="color:#f92672"><property></span> |
| <span style="color:#f92672"><name></span>ozone.om.address.cluster1.om1<span style="color:#f92672"></name></span> |
| <span style="color:#f92672"><value></span>host1<span style="color:#f92672"></value></span> |
| <span style="color:#f92672"></property></span> |
| <span style="color:#f92672"><property></span> |
| <span style="color:#f92672"><name></span>ozone.om.address.cluster1.om2<span style="color:#f92672"></name></span> |
| <span style="color:#f92672"><value></span>host2<span style="color:#f92672"></value></span> |
| <span style="color:#f92672"></property></span> |
| <span style="color:#f92672"><property></span> |
| <span style="color:#f92672"><name></span>ozone.om.address.cluster1.om3<span style="color:#f92672"></name></span> |
| <span style="color:#f92672"><value></span>host3<span style="color:#f92672"></value></span> |
| <span style="color:#f92672"></property></span> |
| </code></pre></div><p>The defined <code>serviceId</code> can be used instead of a single OM host using <a href="../interface.html">client interfaces</a></p> |
| <p>For example with <code>o3fs://</code></p> |
| <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">hdfs dfs -ls o3fs://bucket.volume.cluster1/prefix/ |
| </code></pre></div><p>Or with <code>ofs://</code>:</p> |
| <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">hdfs dfs -ls ofs://cluster1/volume/bucket/prefix/ |
| </code></pre></div><h2 id="implementation-details">Implementation details</h2> |
| <p>Raft can guarantee the replication of any request if the request is persisted to the RAFT log on the majority of the nodes. To achieve high throughput with Ozone Manager, it returns with the response even if the request is persisted only to the RAFT logs.</p> |
| <p>RocksDB instance are updated by a background thread with batching transactions (so called “double buffer” as when one of the buffers is used to commit the data the other one collects all the new requests for the next commit.) To make all data available for the next request even if the background process has not yet written them, the key data is cached in the memory.</p> |
| <p> |
| |
| <img src="HA-OM-doublebuffer.png" alt='HA - OM Double Buffer' class="img-responsive" /></p> |
| <p>The details of this approach are discussed in a separate <a href="../design/omha.html">design doc</a> but it’s an integral part of the OM HA design.</p> |
| <h2 id="om-bootstrap">OM Bootstrap</h2> |
| <p>To convert a non-HA OM to be HA or to add new OM nodes to existing HA OM ring, new OM node(s) need to be bootstrapped.</p> |
| <p>Before bootstrapping a new OM node, all the existing OM’s on-disk configuration file (ozone-site.xml) must be updated with the configuration details |
| of the new OM such as nodeId, address, port etc. Note that the existing OM’s need not be restarted. They will reload the configuration from disk when |
| they receive a bootstrap request from the bootstrapping node.</p> |
| <p>To bootstrap an OM, the following command needs to be run:</p> |
| <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">ozone om <span style="color:#f92672">[</span>global options <span style="color:#f92672">(</span>optional<span style="color:#f92672">)]</span> --bootstrap |
| </code></pre></div><p>The bootstrap command will first verify that all the OMs have the updated configuration file and fail the command otherwise. This check can be skipped |
| using the <em>force</em> option. The <em>force</em> option allows to continue with the bootstrap when one of the existing OMs is down or not responding.</p> |
| <div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">ozone om <span style="color:#f92672">[</span>global options <span style="color:#f92672">(</span>optional<span style="color:#f92672">)]</span> --bootstrap --force |
| </code></pre></div><p>Note that using the <em>force</em> option during bootstrap could crash the OM process if it does not have updated configurations.</p> |
| <h2 id="references">References</h2> |
| <ul> |
| <li>Check <a href="../design/omha.html">this page</a> for the links to the original design docs</li> |
| <li>Ozone distribution contains an example OM HA configuration, under the <code>compose/ozone-om-ha</code> directory which can be tested with the help of <a href="../start/runningviadocker.html">docker-compose</a>.</li> |
| </ul> |
| |
| |
| |
| <a class="btn btn-success btn-lg" href="../feature/erasurecoding.html">Next >></a> |
| |
| </div> |
| |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="push"></div> |
| </div> |
| |
| |
| |
| <footer class="footer"> |
| <div class="container"> |
| <span class="small text-muted"> |
| Version: 1.5.0-SNAPSHOT, Last Modified: February 26, 2024 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/1b48186a0107711235abcd2636977ae0242f6be8">1b48186</a> |
| </span> |
| </div> |
| </footer> |
| |
| |
| |
| <script src="../js/jquery-3.5.1.min.js"></script> |
| <script src="../js/ozonedoc.js"></script> |
| <script src="../js/bootstrap.min.js"></script> |
| |
| |
| </body> |
| |
| </html> |
