blob: d4a4351f7828c6f1a1c0b9ac9f4f7674a88209e8 [file] [log] [blame]
<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Ozone Overview on Ozone</title>
<link>/</link>
<description>Recent content in Ozone Overview on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
<lastBuildDate>Tue, 18 Dec 2018 00:00:00 +0000</lastBuildDate>
<atom:link href="/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Ozone File System</title>
<link>/ozonefs.html</link>
<pubDate>Thu, 14 Sep 2017 00:00:00 +0000</pubDate>
<guid>/ozonefs.html</guid>
<description>There are many Hadoop compatible files systems under Hadoop. Hadoop compatible file systems ensures that storage backends like Ozone can easily be integrated into Hadoop eco-system.
Setting up the Ozone file system To create an ozone file system, we have to choose a bucket where the file system would live. This bucket will be used as the backend store for OzoneFileSystem. All the files and directories will be stored as keys in this bucket.</description>
</item>
<item>
<title>Alpha Cluster</title>
<link>/runningviadocker.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/runningviadocker.html</guid>
<description>This is an alpha release of Ozone. Please don&amp;rsquo;t use this release in production. Please check the road map page for features under development.
The easiest way to run ozone is to download the release tarball and launch ozone via Docker. Docker will create a small ozone cluster on your machine, including the data nodes and ozone services.
Running Ozone via Docker This assumes that you have Docker installed on the machine.</description>
</item>
<item>
<title>Building from Sources</title>
<link>/buildingsources.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/buildingsources.html</guid>
<description>This is a guide on how to build the ozone sources. If you are not planning to build sources yourself, you can safely skip this page.
If you are a Hadoop ninja, and wise in the ways of Apache, you already know that a real Apache release is a source release.
If you want to build from sources, Please untar the source tarball and run the ozone build command. This instruction assumes that you have all the dependencies to build Hadoop on your build machine.</description>
</item>
<item>
<title>Configuration</title>
<link>/settings.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/settings.html</guid>
<description>If you are feeling adventurous, you can setup ozone in a real cluster. Setting up a real cluster requires us to understand the components of Ozone. Ozone is designed to work concurrently with HDFS. However, Ozone is also capable of running independently. The components of ozone are the same in both approaches.
Ozone Components Ozone Manager - Is the server that is in charge of the namespace of Ozone.</description>
</item>
<item>
<title>Running concurrently with HDFS</title>
<link>/runningwithhdfs.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/runningwithhdfs.html</guid>
<description>Ozone is designed to work with HDFS. So it is easy to deploy ozone in an existing HDFS cluster.
Ozone does not support security today. It is a work in progress and tracked in HDDS-4. If you enable ozone in a secure HDFS cluster, for your own protection Ozone will refuse to work.
In other words, till Ozone security work is done, Ozone will not work in any secure clusters.</description>
</item>
<item>
<title>Starting an Ozone Cluster</title>
<link>/realcluster.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/realcluster.html</guid>
<description>Before we boot up the Ozone cluster, we need to initialize both SCM and Ozone Manager.
ozone scm --init This allows SCM to create the cluster Identity and initialize its state. The init command is similar to Namenode format. Init command is executed only once, that allows SCM to create all the required on-disk structures to work correctly. ozone --daemon start scm
Once we know SCM is up and running, we can create an Object Store for our use.</description>
</item>
<item>
<title>Hadoop Distributed Data Store</title>
<link>/hdds.html</link>
<pubDate>Thu, 14 Sep 2017 00:00:00 +0000</pubDate>
<guid>/hdds.html</guid>
<description>SCM Overview Storage Container Manager or SCM is a very important component of ozone. SCM offers block and container-based services to Ozone Manager. A container is a collection of unrelated blocks under ozone. SCM and data nodes work together to maintain the replication levels needed by the cluster.
It is easier to look at a putKey operation to understand the role that SCM plays.
To put a key, a client makes a call to OM with the following arguments.</description>
</item>
<item>
<title>Ozone Manager</title>
<link>/ozonemanager.html</link>
<pubDate>Thu, 14 Sep 2017 00:00:00 +0000</pubDate>
<guid>/ozonemanager.html</guid>
<description>OM Overview Ozone Manager or OM is the namespace manager for Ozone. The clients (RPC clients, Rest proxy, Ozone file system, etc.) communicate with OM to create and delete various ozone objects.
Each ozone volume is the root of a namespace under OM. This is very different from HDFS which provides a single rooted file system.
Ozone&amp;rsquo;s namespace is a collection of volumes or is a forest instead of a single rooted tree as in HDFS.</description>
</item>
<item>
<title>Ozone Security Overview</title>
<link>/ozonesecurityarchitecture.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/ozonesecurityarchitecture.html</guid>
<description>Security in Ozone Starting with badlands release (ozone-0.4.0-alpha) ozone cluster can be secured against external threats. Specifically it can be configured for following security features:
Authentication Authorization Audit Transparent Data Encryption (TDE) Authentication Kerberos Similar to hadoop, Ozone allows kerberos-based authentication. So one way to setup identities for all the daemons and clients is to create kerberos keytabs and configure it like any other service in hadoop.</description>
</item>
<item>
<title>Setup secure ozone cluster</title>
<link>/setupsecureozone.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/setupsecureozone.html</guid>
<description>Setup secure ozone cluster To enable security in ozone cluster ozone.security.enabled should be set to true.
Property Value ozone.security.enabled true Kerberos Configuration for service daemons:
Property Description hdds.scm.kerberos.principal The SCM service principal. Ex scm/HOST@REALM.COM hdds.scm.kerberos.keytab.file The keytab file used by SCM daemon to login as its service principal. ozone.</description>
</item>
<item>
<title>Generate Configurations</title>
<link>/genconf.html</link>
<pubDate>Tue, 18 Dec 2018 00:00:00 +0000</pubDate>
<guid>/genconf.html</guid>
<description>Genconf tool generates a template ozone-site.xml file at the specified path. This template file can be edited to replace with proper values.
ozone genconf &amp;lt;path&amp;gt;</description>
</item>
<item>
<title>Audit Parser</title>
<link>/auditparser.html</link>
<pubDate>Mon, 17 Dec 2018 00:00:00 +0000</pubDate>
<guid>/auditparser.html</guid>
<description>Audit Parser tool can be used for querying the ozone audit logs. This tool creates a sqllite database at the specified path. If the database already exists, it will avoid creating a database.
The database contains only one table called audit defined as:
CREATE TABLE IF NOT EXISTS audit ( datetime text, level varchar(7), logger varchar(7), user text, ip text, op text, params text, result varchar(7), exception text, UNIQUE(datetime,level,logger,user,ip,op,params,result))
Usage: ozone auditparser &amp;lt;path to db file&amp;gt; [COMMAND] [PARAM]</description>
</item>
<item>
<title>Architecture</title>
<link>/concepts.html</link>
<pubDate>Tue, 10 Oct 2017 00:00:00 +0000</pubDate>
<guid>/concepts.html</guid>
<description>Ozone is a redundant, distributed object store build by leveraging primitives present in HDFS. The primary design point of ozone is scalability, and it aims to scale to billions of objects.
Ozone consists of volumes, buckets, and keys. A volume is similar to a home directory in the ozone world. Only an administrator can create it. Volumes are used to store buckets. Once a volume is created users can create as many buckets as needed.</description>
</item>
<item>
<title>Java API</title>
<link>/javaapi.html</link>
<pubDate>Thu, 14 Sep 2017 00:00:00 +0000</pubDate>
<guid>/javaapi.html</guid>
<description>Introduction Ozone ships with its own client library that supports RPC. For generic use cases the S3 compatible REST interface also can be used instead of the Ozone client.
Creating an Ozone client The Ozone client factory creates the ozone client. To get a RPC client we can call
OzoneClient ozClient = OzoneClientFactory.getRpcClient(); If the user want to create a client based on the configuration, then they can call.</description>
</item>
<item>
<title>Freon</title>
<link>/freon.html</link>
<pubDate>Sat, 02 Sep 2017 23:58:17 -0700</pubDate>
<guid>/freon.html</guid>
<description>Overview Freon is a load-generator for Ozone. This tool is used for testing the functionality of ozone.
Random keys In randomkeys mode, the data written into ozone cluster is randomly generated. Each key will be of size 10 KB.
The number of volumes/buckets/keys can be configured. The replication type and factor (eg. replicate with ratis to 3 nodes) Also can be configured.
For more information use
bin/ozone freon --help</description>
</item>
<item>
<title>Dozone &amp; Dev Tools</title>
<link>/dozone.html</link>
<pubDate>Thu, 10 Aug 2017 00:00:00 +0000</pubDate>
<guid>/dozone.html</guid>
<description>Dozone stands for docker for ozone. Ozone supports docker to make it easy to develop and test ozone. Starting a docker-based ozone container is simple.
In the compose/ozone directory there are two files that define the docker and ozone settings.
Developers can
cd compose/ozone and simply run
docker-compose up -d to run a ozone cluster on docker.
This command will launch OM, SCM and a data node.
To access the OM UI, one can view http://localhost:9874.</description>
</item>
<item>
<title>SCMCLI</title>
<link>/scmcli.html</link>
<pubDate>Thu, 10 Aug 2017 00:00:00 +0000</pubDate>
<guid>/scmcli.html</guid>
<description>SCM is the block service for Ozone. It is also the workhorse for ozone. But user process never talks to SCM. However, being able to read the state of SCM is useful.
SCMCLI allows the developer to access SCM directly. Please note: Improper usage of this tool can destroy your cluster. Unless you know exactly what you are doing, Please do not use this tool. In other words, this is a developer only tool.</description>
</item>
<item>
<title>Bucket Commands</title>
<link>/bucketcommands.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/bucketcommands.html</guid>
<description>Ozone shell supports the following bucket commands.
create delete info list update Create The bucket create command allows users to create a bucket.
Params:
Arguments Comment Uri The name of the bucket in /volume/bucket format. ozone sh bucket create /hive/jan The above command will create a bucket called jan in the hive volume. Since no scheme was specified this command defaults to O3 (RPC) protocol.</description>
</item>
<item>
<title>Key Commands</title>
<link>/keycommands.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/keycommands.html</guid>
<description>Ozone shell supports the following key commands.
get put delete info list rename Get The key get command downloads a key from Ozone cluster to local file system.
Params:
Arguments Comment Uri The name of the key in /volume/bucket/key format. FileName Local file to download the key to. ozone sh key get /hive/jan/sales.orc sales.orc Downloads the file sales.</description>
</item>
<item>
<title>Monitoring with Prometheus</title>
<link>/prometheus.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/prometheus.html</guid>
<description>Prometheus is an open-source monitoring server developed under under the Cloud Native Computing Foundation.
Ozone supports Prometheus out of the box. The servers start a prometheus compatible metrics endpoint where all the available hadoop metrics are published in prometheus exporter format.
Prerequisites Install the and start an Ozone cluster. Download the prometheus binary. Monitoring with prometheus (1) To enable the Prometheus metrics endpoint you need to add a new configuration to the ozone-site.</description>
</item>
<item>
<title>Ozone CLI</title>
<link>/commandshell.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/commandshell.html</guid>
<description>Ozone has a set of command line tools that can be used to manage ozone.
All these commands are invoked via the ozone script.
The commands supported by ozone are:
classpath - Prints the class path needed to get the hadoop jar and the required libraries. dtutil - Operations related to delegation tokens fs - Runs a command on ozone file system. datanode - Via daemon command, the HDDS data nodes can be started or stopped.</description>
</item>
<item>
<title>S3</title>
<link>/s3.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/s3.html</guid>
<description>Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.
Getting started S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components.
You can start a docker based cluster, including the S3 gateway from the release package.
Go to the compose/ozones3 directory, and start the server:
docker-compose up -d You can access the S3 gateway at http://localhost:9878</description>
</item>
<item>
<title>Spark in Kubernetes with OzoneFS</title>
<link>/sparkozonefsk8s.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/sparkozonefsk8s.html</guid>
<description>Using Ozone from Apache Spark This recipe shows how Ozone object store can be used from Spark using:
OzoneFS (Hadoop compatible file system) Hadoop 2.7 (included in the Spark distribution) Kubernetes Spark scheduler Local spark client Requirements Download latest Spark and Ozone distribution and extract them. This method is tested with the spark-2.4.0-bin-hadoop2.7 distribution.
You also need the following:
A container repository to push and pull the spark+ozone images.</description>
</item>
<item>
<title>Volume Commands</title>
<link>/volumecommands.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/volumecommands.html</guid>
<description>Volume commands generally need administrator privileges. The ozone shell supports the following volume commands.
create delete info list update Create The volume create command allows an administrator to create a volume and assign it to a user.
Params:
Arguments Comment -q, &amp;ndash;quota Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. -u, &amp;ndash;user Required, The name of the user who owns this volume.</description>
</item>
</channel>
</rss>