blob: 3339cd564d014851a2e994e315778d55fa00cf18 [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Hadoop Ozone Documentation">
<title>Documentation for Apache Hadoop Ozone</title>
<link href="../css/bootstrap.min.css" rel="stylesheet">
<link href="../css/ozonedoc.css" rel="stylesheet">
</head>
<body>
<nav class="navbar navbar-inverse navbar-fixed-top">
<div class="container-fluid">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a href="#" class="navbar-left" style="height: 50px; padding: 5px 5px 5px 0;">
<img src="../ozone-logo-small.png" width="40"/>
</a>
<a class="navbar-brand hidden-xs" href="#">
Apache Hadoop Ozone/HDDS documentation
</a>
<a class="navbar-brand visible-xs-inline" href="#">Hadoop Ozone</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav navbar-right">
<li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
<li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
<li><a href="https://apache.org">ASF</a></li>
</ul>
</div>
</div>
</nav>
<div class="container-fluid">
<div class="row">
<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
<ul class="nav nav-sidebar">
<li class="">
<a href="../index.html">
<span>Overview</span>
</a>
</li>
<li class="">
<a href="../start.html">
<span>Getting Started</span>
</a>
</li>
<li class="">
<a href="../shell.html">
<span>Command Line Interface</span>
</a>
</li>
<li class="">
<a href="../interface.html">
<span>Programming Interfaces</span>
</a>
</li>
<li class="">
<a href="../gdpr.html">
<span>GDPR</span>
</a>
</li>
<li class="">
<a href="../security.html">
<span>Security</span>
</a>
</li>
<li class="">
<a href="../concept.html">
<span>Concepts</span>
</a>
</li>
<li class="">
<a href="../beyond.html">
<span>Beyond Basics</span>
</a>
</li>
<li class="">
<a href="../tools.html">
<span>Tools</span>
</a>
</li>
<li class="">
<a href="../recipe.html">
<span>Recipes</span>
</a>
</li>
<li class="visible-xs"><a href="#">References</a>
<ul class="nav">
<li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
<li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
<li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
</ul></li>
</ul>
</div>
<div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
<div class="col-md-9">
<nav aria-label="breadcrumb">
<ol class="breadcrumb">
<li class="breadcrumb-item"><a href="../">Home</a></li>
<li class="breadcrumb-item" aria-current="page"><a href="../interface.html">Programming Interfaces</a></li>
<li class="breadcrumb-item active" aria-current="page">S3 Protocol</li>
</ol>
</nav>
<div class="pull-right">
<a href="../zh/interface/s3.html"><span class="label label-success">中文</span></a>
</div>
<div class="col-md-9">
<h1>S3 Protocol</h1>
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p>Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.</p>
<h2 id="getting-started">Getting started</h2>
<p>S3 Gateway is a separated component which provides the S3 compatible APIs. It should be started additional to the regular Ozone components.</p>
<p>You can start a docker based cluster, including the S3 gateway from the release package.</p>
<p>Go to the <code>compose/ozones3</code> directory, and start the server:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose up -d</code></pre></div>
<p>You can access the S3 gateway at <code>http://localhost:9878</code></p>
<h2 id="url-schema">URL Schema</h2>
<p>Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. <a href="http://bucketname.host:9878">http://bucketname.host:9878</a>) and the path-style addresses (eg. <a href="http://host:9878/bucketname">http://host:9878/bucketname</a>)</p>
<p>By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your <code>ozone-site.xml</code>:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672">&lt;property&gt;</span>
<span style="color:#f92672">&lt;name&gt;</span>ozone.s3g.domain.name<span style="color:#f92672">&lt;/name&gt;</span>
<span style="color:#f92672">&lt;value&gt;</span>s3g.internal<span style="color:#f92672">&lt;/value&gt;</span>
<span style="color:#f92672">&lt;/property&gt;</span></code></pre></div>
<h2 id="bucket-browser">Bucket browser</h2>
<p>Buckets could be browsed from the browser by adding <code>?browser=true</code> to the bucket URL.</p>
<p>For example the content of the &lsquo;testbucket&rsquo; could be checked from the browser using the URL <a href="http://localhost:9878/testbucket?browser=true">http://localhost:9878/testbucket?browser=true</a></p>
<h2 id="implemented-rest-endpoints">Implemented REST endpoints</h2>
<p>Operations on S3Gateway service:</p>
<table>
<thead>
<tr>
<th>Endpoint</th>
<th>Status</th>
</tr>
</thead>
<tbody>
<tr>
<td>GET service</td>
<td>implemented</td>
</tr>
</tbody>
</table>
<p>Operations on Bucket:</p>
<table>
<thead>
<tr>
<th>Endpoint</th>
<th>Status</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td>GET Bucket (List Objects) Version 2</td>
<td>implemented</td>
<td></td>
</tr>
<tr>
<td>HEAD Bucket</td>
<td>implemented</td>
<td></td>
</tr>
<tr>
<td>DELETE Bucket</td>
<td>implemented</td>
<td></td>
</tr>
<tr>
<td>PUT Bucket (Create bucket)</td>
<td>implemented</td>
<td></td>
</tr>
<tr>
<td>Delete Multiple Objects (POST)</td>
<td>implemented</td>
<td></td>
</tr>
</tbody>
</table>
<p>Operation on Objects:</p>
<table>
<thead>
<tr>
<th>Endpoint</th>
<th>Status</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td>PUT Object</td>
<td>implemented</td>
<td></td>
</tr>
<tr>
<td>GET Object</td>
<td>implemented</td>
<td></td>
</tr>
<tr>
<td>Multipart Upload</td>
<td>implemented</td>
<td>Except the listing of the current MultiPartUploads.</td>
</tr>
<tr>
<td>DELETE Object</td>
<td>implemented</td>
<td></td>
</tr>
<tr>
<td>HEAD Object</td>
<td>implemented</td>
<td></td>
</tr>
</tbody>
</table>
<h2 id="security">Security</h2>
<p>If security is not enabled, you can <em>use</em> <strong>any</strong> AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY</p>
<p>If security is enabled, you can get the key and the secret with the <code>ozone s3 getsecret</code> command (*kerberos based authentication is required).</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">/etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
ozone s3 getsecret
awsAccessKey<span style="color:#f92672">=</span>testuser/scm@EXAMPLE.COM
awsSecret<span style="color:#f92672">=</span>c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999</code></pre></div>
<p>Now, you can use the key and the secret to access the S3 endpoint:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">export AWS_ACCESS_KEY_ID<span style="color:#f92672">=</span>testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY<span style="color:#f92672">=</span>c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1</code></pre></div>
<h2 id="s3-bucket-name-mapping-to-ozone-buckets">S3 bucket name mapping to Ozone buckets</h2>
<p><strong>Note</strong>: Ozone has a notion for &lsquo;volumes&rsquo; which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are mapped to Ozone &lsquo;volume/bucket&rsquo; locations (depending on the given authentication information).</p>
<p>To show the storage location of a S3 bucket, use the <code>ozone s3 path &lt;bucketname&gt;</code> command.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket<span style="color:#f92672">=</span>bucket1
ozone s3 path bucket1
Volume name <span style="color:#66d9ef">for</span> S3Bucket is : s3thisisakey
Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey</code></pre></div>
<h2 id="clients">Clients</h2>
<h3 id="aws-cli">AWS Cli</h3>
<p><code>aws</code> CLI could be used by specifying the custom REST endpoint.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest</code></pre></div>
<p>Or</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3 ls --endpoint http://localhost:9878 s3://buckettest</code></pre></div>
<h3 id="s3-fuse-driver-goofys">S3 Fuse driver (goofys)</h3>
<p>Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1</code></pre></div>
</div>
</div>
</div>
</div>
</div>
<script src="../js/jquery-3.4.1.min.js"></script>
<script src="../js/ozonedoc.js"></script>
<script src="../js/bootstrap.min.js"></script>
</body>
</html>