Ozone is a part of the Apache Hadoop project. The bug tracking system for Ozone is under the Apache Jira project named HDDS.
If you are familiar with contributing to Apache Hadoop, then you already know everything you need to know to start filing Ozone bugs and submitting patches.
If you have never contributed to Apache Hadoop before, then you may find it useful to read How To Contribute.
This document summarize the contribution process and defines the differences.
We welcome contributions of:
ozone genesis), which can be used to test cluster and report problems.
If you have any questions, please don't hesitate to contact
Requirements to compile the code:
Additional requirements to run your first pseudo cluster:
Additional requirements to execute different type of tests:
Protobuf 2.5 can be installed from the source:
mkdir -p /usr/local/src/ cd /usr/local/src/ wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz tar xvf protobuf-2.5.0.tar.gz cd protobuf-2.5.0 ./autogen.sh ./configure --prefix=/usr make make install protoc --version
(Standard development tools such as make, gcc, etc. are required.)
After installing the requirements (especially maven and protobuf) the build is as simple as:
mvn clean install -DskipTests
And you can start your first cluster:
cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone docker-compose up -d --scale datanode=3
hadoop-ozone/dev-support/checks directory contains helper scripts to build and check your code. (Including findbugs and checkstyle). Use them if you don't know the exact maven goals / parameters.
These scripts are executed by the CI servers, so it's always good to run them locally before creating a PR.
-DskipShadeto exclude ozonefs jar file creation from the release. It‘s way more faster, but you can’t test Hadoop Compatible file system.
-DskipReconto exclude the Recon build (Web UI and monitoring) from the build. It saves about 2 additional minutes.
-Pdistto build a distribution (Without this profile you won't have the final tar file)
-Pdocker-buildto build a docker image which includes Ozone
-Ddocker.image=repo/nameto define the name of your docker image
-Pdocker-pushto push the created docker image to the docker registry
We use github pull requests instead of uploading patches to JIRA. The main contribution workflow is as follows:
apache/hadoop-ozonegithub repository (first time)
git checkout -b HDDS-1234)
We follow the code convention of Hadoop project (2 spaces instead of tabs, 80 char line width, ASF licence headers). The code checked with checkstyle, findbugs and various test frameworks.
Please don't post / commit any code with any code violations (all checks are not checking the introduced violations as checks in Hadoop but all the available violations).
The easiest way to check your contribution is using the simplified shell scripts under
hadoop-ozone/dev-support/checks. The problems will be printed out on the standard output.
hadoop-ozone/dev-support/checks/rat.sh hadoop-ozone/dev-support/checks/checkstyle.sh hadoop-ozone/dev-support/checks/findbugs.sh
Execution of rat and checkstyle are very fast. Findbugs is slightly slower. Executing unit.sh takes about 30 minutes.
The same scripts are executed by the github PR checker.
It's always good practice (and fast) to test with the related docker-compose based pseudo clusters:
cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone ./test.sh
(To test S3 use
compose/ozones3, to test security use
If you have very good reasons, you can ignore any Fingbugs warning. Your good reason can be persisted with the
@SuppressFBWarnings(value="AT_OPERATION_SEQUENCE_ON_CONCURRENT_ABSTRACTION", justification="The method is synchronized and this is the only place "+ "dnsToUuidMap is modified") private synchronized void addEntryTodnsToUuidMap( ...
As Ozone uses Apache Maven it can be developed from any IDE. As IntelliJ is a common choice, here are some suggestions to use it for Ozone development:
Ozone components depends on maven classpath. We generate classpath descriptor from the maven pom.xml files to use exactly the same classpath at runtime.
As a result, it's easy to start all the components from IDE as the right classpath (without provided scope) has already been set.
To start Ozone from IntelliJ:
Checkstyle plugin may help to detect violations directly from the IDE.
Checkstyleand Add (
+) a new
pom.xmlfor the current version of the used checkstyle and use the same version with the plugin (
Ozonerule and execute the check
IntelliJ may not pick up protoc generated classes as they can be very huge. If the protoc files can't be compiled try the following: