Google Git
Sign in
apache / ozone-site / 671b294ce8f5a79f582cb247463123c169334b14 / . / docs / 1.1.0 / interface / ofs.html
blob: 7e2c541e844772bc073b5cc09f76ee1d4da7c738 [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Apache Ozone Documentation">
<title>Documentation for Apache Ozone</title>
<link href="../css/bootstrap.min.css" rel="stylesheet">
<link href="../css/ozonedoc.css" rel="stylesheet">
</head>
<body>
<nav class="navbar navbar-inverse navbar-fixed-top">
<div class="container-fluid">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a href="../index.html" class="navbar-left ozone-logo">
<img src="../ozone-logo-small.png"/>
</a>
<a class="navbar-brand hidden-xs" href="../index.html">
Apache Ozone/HDDS documentation
</a>
<a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav navbar-right">
<li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
<li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
<li><a href="https://apache.org">ASF</a></li>
</ul>
</div>
</div>
</nav>
<div class="wrapper">
<div class="container-fluid">
<div class="row">
<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
<ul class="nav nav-sidebar">
<li class="">
<a href="../index.html">
<span>Overview</span>
</a>
</li>
<li class="">
<a href="../start.html">
<span>Getting Started</span>
</a>
</li>
<li class="">
<a href="../concept.html">
<span>Architecture</span>
</a>
<ul class="nav">
<li class="">
<a href="../concept/overview.html">Overview</a>
</li>
<li class="">
<a href="../concept/ozonemanager.html">Ozone Manager</a>
</li>
<li class="">
<a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
</li>
<li class="">
<a href="../concept/containers.html">Containers</a>
</li>
<li class="">
<a href="../concept/datanodes.html">Datanodes</a>
</li>
<li class="">
<a href="../concept/recon.html">Recon</a>
</li>
</ul>
</li>
<li class="">
<a href="../feature.html">
<span>Features</span>
</a>
<ul class="nav">
<li class="">
<a href="../feature/ha.html">High Availability</a>
</li>
<li class="">
<a href="../feature/topology.html">Topology awareness</a>
</li>
<li class="">
<a href="../feature/quota.html">Quota in Ozone</a>
</li>
<li class="">
<a href="../feature/recon.html">Recon Server</a>
</li>
<li class="">
<a href="../feature/observability.html">Observability</a>
</li>
</ul>
</li>
<li class="">
<a href="../interface.html">
<span>Client Interfaces</span>
</a>
<ul class="nav">
<li class="active">
<a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
</li>
<li class="">
<a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
</li>
<li class="">
<a href="../interface/s3.html">S3 Protocol</a>
</li>
<li class="">
<a href="../interface/cli.html">Command Line Interface</a>
</li>
<li class="">
<a href="../interface/reconapi.html">Recon API</a>
</li>
<li class="">
<a href="../interface/javaapi.html">Java API</a>
</li>
<li class="">
<a href="../interface/csi.html">CSI Protocol</a>
</li>
</ul>
</li>
<li class="">
<a href="../security.html">
<span>Security</span>
</a>
<ul class="nav">
<li class="">
<a href="../security/secureozone.html">Securing Ozone</a>
</li>
<li class="">
<a href="../security/securingtde.html">Transparent Data Encryption</a>
</li>
<li class="">
<a href="../security/gdpr.html">GDPR in Ozone</a>
</li>
<li class="">
<a href="../security/securingdatanodes.html">Securing Datanodes</a>
</li>
<li class="">
<a href="../security/securingozonehttp.html">Securing HTTP</a>
</li>
<li class="">
<a href="../security/securings3.html">Securing S3</a>
</li>
<li class="">
<a href="../security/securityacls.html">Ozone ACLs</a>
</li>
<li class="">
<a href="../security/securitywithranger.html">Apache Ranger</a>
</li>
</ul>
</li>
<li class="">
<a href="../tools.html">
<span>Tools</span>
</a>
</li>
<li class="">
<a href="../recipe.html">
<span>Recipes</span>
</a>
</li>
<li><a href="../design.html"><span><b>Design docs</b></span></a></li>
<li class="visible-xs"><a href="#">References</a>
<ul class="nav">
<li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
<li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
<li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
</ul></li>
</ul>
</div>
<div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
<div class="col-md-9">
<nav aria-label="breadcrumb">
<ol class="breadcrumb">
<li class="breadcrumb-item"><a href="../index.html">Home</a></li>
<li class="breadcrumb-item" aria-current="page"><a href="../interface.html">Client Interfaces</a></li>
<li class="breadcrumb-item active" aria-current="page">Ofs (Hadoop compatible)</li>
</ol>
</nav>
<div class="pull-right">
</div>
<div class="col-md-9">
<h1>Ofs (Hadoop compatible)</h1>
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p>The Hadoop compatible file system interface allows storage backends like Ozone
to be easily integrated into Hadoop eco-system. Ozone file system is an
Hadoop compatible file system.</p>
<div class="alert alert-warning" role="alert">
<p>Currently, Ozone supports two scheme: <code>o3fs://</code> and <code>ofs://</code>.
The biggest difference between the <code>o3fs</code> and <code>ofs</code>，is that <code>o3fs</code> supports operations
only at a <strong>single bucket</strong>, while ofs supports operations across all volumes and buckets and
provides a full view of all the volume/buckets.</p>
</div>
<h2 id="the-basics">The Basics</h2>
<p>Examples of valid OFS paths:</p>
<pre><code>ofs://om1/
ofs://om3:9862/
ofs://omservice/
ofs://omservice/volume1/
ofs://omservice/volume1/bucket1/
ofs://omservice/volume1/bucket1/dir1
ofs://omservice/volume1/bucket1/dir1/key1
ofs://omservice/tmp/
ofs://omservice/tmp/key1
</code></pre><p>Volumes and mount(s) are located at the root level of an OFS Filesystem.
Buckets are listed naturally under volumes.
Keys and directories are under each buckets.</p>
<p>Note that for mounts, only temp mount <code>/tmp</code> is supported at the moment.</p>
<h2 id="configuration">Configuration</h2>
<p>Please add the following entry to the core-site.xml.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672">&lt;property&gt;</span>
<span style="color:#f92672">&lt;name&gt;</span>fs.ofs.impl<span style="color:#f92672">&lt;/name&gt;</span>
<span style="color:#f92672">&lt;value&gt;</span>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem<span style="color:#f92672">&lt;/value&gt;</span>
<span style="color:#f92672">&lt;/property&gt;</span>
<span style="color:#f92672">&lt;property&gt;</span>
<span style="color:#f92672">&lt;name&gt;</span>fs.defaultFS<span style="color:#f92672">&lt;/name&gt;</span>
<span style="color:#f92672">&lt;value&gt;</span>ofs://om-host.example.com/<span style="color:#f92672">&lt;/value&gt;</span>
<span style="color:#f92672">&lt;/property&gt;</span></code></pre></div>
<p>This will make all the volumes and buckets to be the default Hadoop compatible file system and register the ofs file system type.</p>
<p>You also need to add the ozone-filesystem-hadoop3.jar file to the classpath:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">export HADOOP_CLASSPATH<span style="color:#f92672">=</span>/opt/ozone/share/ozonefs/lib/hadoop-ozone-filesystem-hadoop3-*.jar:$HADOOP_CLASSPATH</code></pre></div>
<p>(Note: with Hadoop 2.x, use the <code>hadoop-ozone-filesystem-hadoop2-*.jar</code>)</p>
<p>Once the default Filesystem has been setup, users can run commands like ls, put, mkdir, etc.
For example:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls /</code></pre></div>
<p>Note that ofs works on all buckets and volumes. Users can create buckets and volumes using mkdir, such as create volume named volume1 and bucket named bucket1:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -mkdir /volume1
hdfs dfs -mkdir /volume1/bucket1</code></pre></div>
<p>Or use the put command to write a file to the bucket.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -put /etc/hosts /volume1/bucket1/test</code></pre></div>
<p>For more usage, see: <a href="https://issues.apache.org/jira/secure/attachment/12987636/Design%20ofs%20v1.pdf">https://issues.apache.org/jira/secure/attachment/12987636/Design%20ofs%20v1.pdf</a></p>
<h2 id="special-note">Special note</h2>
<p>Trash is disabled even if <code>fs.trash.interval</code> is set on purpose. (HDDS-3982)</p>
<h2 id="differences-from-o3fshahahugoshortcode-s5-hbhb">Differences from <a href="../interface/o3fs.html">o3fs</a></h2>
<h3 id="creating-files">Creating files</h3>
<p>OFS doesn&rsquo;t allow creating keys(files) directly under root or volumes.
Users will receive an error message when they try to do that:</p>
<pre><code>$ ozone fs -touch /volume1/key1
touch: Cannot create file under root or volume.
</code></pre><h3 id="simplify-fsdefaultfs">Simplify fs.defaultFS</h3>
<p>With OFS, fs.defaultFS (in core-site.xml) no longer needs to have a specific
volume and bucket in its path like o3fs did.
Simply put the OM host or service ID (in case of HA):</p>
<pre><code>&lt;property&gt;
&lt;name&gt;fs.defaultFS&lt;/name&gt;
&lt;value&gt;ofs://omservice&lt;/value&gt;
&lt;/property&gt;
</code></pre><p>The client would then be able to access every volume and bucket on the cluster
without specifying the hostname or service ID.</p>
<pre><code>$ ozone fs -mkdir -p /volume1/bucket1
</code></pre><h3 id="volume-and-bucket-management-directly-from-filesystem-shell">Volume and bucket management directly from FileSystem shell</h3>
<p>Admins can create and delete volumes and buckets easily with Hadoop FS shell.
Volumes and buckets are treated similar to directories so they will be created
if they don&rsquo;t exist with <code>-p</code>:</p>
<pre><code>$ ozone fs -mkdir -p ofs://omservice/volume1/bucket1/dir1/
</code></pre><p>Note that the supported volume and bucket name character set rule still applies.
For instance, bucket and volume names don&rsquo;t take underscore(<code>_</code>):</p>
<pre><code>$ ozone fs -mkdir -p /volume_1
mkdir: Bucket or Volume name has an unsupported character : _
</code></pre><h2 id="mounts">Mounts</h2>
<p>In order to be compatible with legacy Hadoop applications that use /tmp/,
we have a special temp mount located at the root of the FS.
This feature may be expanded in the feature to support custom mount paths.</p>
<p>Important: To use it, first, an <strong>admin</strong> needs to create the volume tmp
(the volume name is hardcoded for now) and set its ACL to world ALL access.
Namely:</p>
<pre><code>$ ozone sh volume create tmp
$ ozone sh volume setacl tmp -al world::a
</code></pre><p>These commands only needs to be done <strong>once per cluster</strong>.</p>
<p>Then, <strong>each user</strong> needs to mkdir first to initialize their own temp bucket
once.</p>
<pre><code>$ ozone fs -mkdir /tmp
2020-06-04 00:00:00,050 [main] INFO rpc.RpcClient: Creating Bucket: tmp/0238 ...
</code></pre><p>After that they can write to it just like they would do to a regular
directory. e.g.:</p>
<pre><code>$ ozone fs -touch /tmp/key1
</code></pre><h2 id="delete-with-trash-enabled">Delete with trash enabled</h2>
<p>When keys are deleted with trash enabled, they are moved to a trash directory
under each bucket, because keys aren&rsquo;t allowed to be moved(renamed) between
buckets in Ozone.</p>
<pre><code>$ ozone fs -rm /volume1/bucket1/key1
2020-06-04 00:00:00,100 [main] INFO fs.TrashPolicyDefault: Moved: 'ofs://id1/volume1/bucket1/key1' to trash at: ofs://id1/volume1/bucket1/.Trash/hadoop/Current/volume1/bucket1/key1
</code></pre><p>This is very similar to how the HDFS encryption zone handles trash location.</p>
<h2 id="recursive-listing">Recursive listing</h2>
<p>OFS supports recursive volume, bucket and key listing.</p>
<p>i.e. `ozone fs -ls -R ofs://omservice/`` will recursively list all volumes,
buckets and keys the user has LIST permission to if ACL is enabled.
If ACL is disabled, the command would just list literally everything on that
cluster.</p>
<p>This feature wouldn&rsquo;t degrade server performance as the loop is on the client.
Think it as a client is issuing multiple requests to the server to get all the
information.</p>
<h2 id="special-note-1">Special note</h2>
<p>Trash is disabled even if <code>fs.trash.interval</code> is set on purpose. (HDDS-3982)</p>
<a class="btn btn-success btn-lg" href="../interface/o3fs.html">Next >></a>
</div>
</div>
</div>
</div>
</div>
<div class="push"></div>
</div>
<footer class="footer">
<div class="container">
<span class="small text-muted">
Version: 1.1.0, Last Modified: August 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/5ce6f0eab381389fd04c3130531b3ec626acbc65">5ce6f0eab</a>
</span>
</div>
</footer>
<script src="../js/jquery-3.5.1.min.js"></script>
<script src="../js/ozonedoc.js"></script>
<script src="../js/bootstrap.min.js"></script>
</body>
</html>