Apache Hadoop Ozone

Clone this repo:
  1. 1d56bc2 HDDS-2645. Refactor MiniOzoneChaosCluster to a different package to add filesystem tests. (#292) by Mukul Kumar Singh · 7 hours ago master
  2. f3abb4c HDDS-2668. Sonar : fix issues reported in BlockManagerImpl (#317) by supratimdeka · 10 hours ago
  3. 91a6477 HDDS-2687. Fix sonar issues in package org.apache.hadoop.ozone.recon.api. (#325) by avijayanhwx · 10 hours ago
  4. 588ff1e HDDS-2689. OMException NOT_A_FILE missing space in the exception message. (#327) by avijayanhwx · 29 hours ago
  5. 1503ab8 HDDS-2692. Seek to file end throws EOF Exception. (#326) by Ayush Saxena · 35 hours ago

Apache Hadoop Ozone

Ozone is a scalable, redundant, and distributed object store for Hadoop. Apart from scaling to billions of objects of varying sizes, Ozone can function effectively in containerized environments such as Kubernetes and YARN.

  • MULTI-PROTOCOL SUPPORT: Ozone supports different protocols like S3 and Hadoop File System APIs.
  • SCALABLE: Ozone is designed to scale to tens of billions of files and blocks and, in the future, even more.
  • CONSISTENT: Ozone is a strongly consistent object store. This consistency is achieved by using protocols like RAFT.
  • CLOUD-NATIVE: Ozone is designed to work well in containerized environments like YARN and Kubernetes.
  • SECURE: Ozone integrates with Kerberos infrastructure for access control and supports TDE and on-wire encryption.
  • HIGHLY AVAILABLE: Ozone is a fully replicated system that is designed to survive multiple failures.

Documentation

The latest documentation is generated together with the releases and hosted on the apache side.

Please check the documentation page for more information.

Contact

Ozone is part of the Apache Hadoop project.

Download

Latest release artifacts (source release and binary packages) are available from the Ozone web page.

Quick start

Run Ozone from published Docker image

The easiest way to start a cluster with docker is:

docker run -p 9878:9878 apache/ozone

And you can use AWS S3 cli:

aws s3api --endpoint http://localhost:9878/ create-bucket --bucket=wordcount
aws s3 --endpoint http://localhost:9878 cp --storage-class REDUCED_REDUNDANCY  /tmp/testfile  s3://wordcount/testfile

Run Ozone from released artifact

If you need a more realistic cluster, you can download the last (binary) release package, and start a cluster with the help of docker-compose:

After you untar the binary:

cd compose/ozone
docker-compose up -d --scale datanode=3

The compose folder contains different sets of configured clusters (secure, HA, mapreduce example), you can check the various subfolders for more examples.

Run on Kubernetes

Ozone is a first class citizen of the Cloud-Native environments. The binary package contains multiple sets of K8s resource files to show how it can be deployed.

Build from source

Ozone can be built with Apache Maven:

mvn clean install -DskipTests

And can be started with the help of Docker:

cd hadoop-ozone/dist/target/ozone-*/compose/
docker-compose up -d scale datanode=3

For more information, you can check the Contribution guideline

Contribute

All contributions are welcome.

  1. Please open a Jira issue
  2. And create a pull request

For more information, you can check the Contribution guideline

License

The Apache Hadoop Ozone project is licensed under the Apache 2.0 License. See the LICENSE file for details.