////
/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
////

[[developer]]
= Building and Developing Apache HBase
:doctype: book
:numbered:
:toc: left
:icons: font
:experimental:

This chapter contains information and guidelines for building and releasing HBase code and documentation.
Being familiar with these guidelines will help the HBase committers to use your contributions more easily.

[[getting.involved]]
== Getting Involved

Apache HBase gets better only when people contribute! If you are looking to contribute to Apache HBase, look for link:https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20labels%20in%20(beginner)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)[issues in JIRA tagged with the label 'beginner'].
These are issues HBase contributors have deemed worthy but not of immediate priority and a good way to ramp on HBase internals.
See link:https://lists.apache.org/thread.html/b122265f4e4054cf08f8cd38609fb06af72f398c44f9086b05ef4e21%401407246237%40%3Cdev.hbase.apache.org%3E[What label
                is used for issues that are good on ramps for new contributors?] from the dev mailing list for background.

Before you get started submitting code to HBase, please refer to <<developing,developing>>.

As Apache HBase is an Apache Software Foundation project, see <<asf,asf>>            for more information about how the ASF functions.

[[mailing.list]]
=== Mailing Lists

Sign up for the dev-list and the user-list.
See the link:https://hbase.apache.org/mail-lists.html[mailing lists] page.
Posing questions - and helping to answer other people's questions - is encouraged! There are varying levels of experience on both lists so patience and politeness are encouraged (and please stay on topic.)

[[slack]]
=== Slack
The Apache HBase project has its own link: http://apache-hbase.slack.com[Slack Channel] for real-time questions
and discussion. Mail dev@hbase.apache.org to request an invite.

[[irc]]
=== Internet Relay Chat (IRC)

(NOTE: Our IRC channel seems to have been deprecated in favor of the above Slack channel)

For real-time questions and discussions, use the `#hbase` IRC channel on the link:https://freenode.net/[FreeNode] IRC network.
FreeNode offers a web-based client, but most people prefer a native client, and several clients are available for each operating system.

=== Jira

Check for existing issues in link:https://issues.apache.org/jira/projects/HBASE/issues[Jira].
If it's either a new feature request, enhancement, or a bug, file a ticket.

We track multiple types of work in JIRA:

- Bug: Something is broken in HBase itself.
- Test: A test is needed, or a test is broken.
- New feature: You have an idea for new functionality. It's often best to bring
  these up on the mailing lists first, and then write up a design specification
  that you add to the feature request JIRA.
- Improvement: A feature exists, but could be tweaked or augmented. It's often
  best to bring these up on the mailing lists first and have a discussion, then
  summarize or link to the discussion if others seem interested in the
  improvement.
- Wish: This is like a new feature, but for something you may not have the
  background to flesh out yourself.

Bugs and tests have the highest priority and should be actionable.

==== Guidelines for reporting effective issues

- *Search for duplicates*: Your issue may have already been reported. Have a
  look, realizing that someone else might have worded the summary differently.
+
Also search the mailing lists, which may have information about your problem
and how to work around it. Don't file an issue for something that has already
been discussed and resolved on a mailing list, unless you strongly disagree
with the resolution *and* are willing to help take the issue forward.

* *Discuss in public*: Use the mailing lists to discuss what you've discovered
  and see if there is something you've missed. Avoid using back channels, so
  that you benefit from the experience and expertise of the project as a whole.

* *Don't file on behalf of others*: You might not have all the context, and you
  don't have as much motivation to see it through as the person who is actually
  experiencing the bug. It's more helpful in the long term to encourage others
  to file their own issues. Point them to this material and offer to help out
  the first time or two.

* *Write a good summary*: A good summary includes information about the problem,
  the impact on the user or developer, and the area of the code.
** Good: `Address new license dependencies from hadoop3-alpha4`
** Room for improvement: `Canary is broken`
+
If you write a bad title, someone else will rewrite it for you. This is time
they could have spent working on the issue instead.

* *Give context in the description*: It can be good to think of this in multiple
  parts:
** What happens or doesn't happen?
** How does it impact you?
** How can someone else reproduce it?
** What would "fixed" look like?
+
You don't need to know the answers for all of these, but give as much
information as you can. If you can provide technical information, such as a
Git commit SHA that you think might have caused the issue or a build failure
on builds.apache.org where you think the issue first showed up, share that
info.

* *Fill in all relevant fields*: These fields help us filter, categorize, and
  find things.

* *One bug, one issue, one patch*: To help with back-porting, don't split issues
  or fixes among multiple bugs.

* *Add value if you can*: Filing issues is great, even if you don't know how to
  fix them. But providing as much information as possible, being willing to
  triage and answer questions, and being willing to test potential fixes is even
  better! We want to fix your issue as quickly as you want it to be fixed.

* *Don't be upset if we don't fix it*: Time and resources are finite. In some
  cases, we may not be able to (or might choose not to) fix an issue, especially
  if it is an edge case or there is a workaround. Even if it doesn't get fixed,
  the JIRA is a public record of it, and will help others out if they run into
  a similar issue in the future.

==== Working on an issue

To check for existing issues which you can tackle as a beginner, search for link:https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20labels%20in%20(beginner)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)[issues in JIRA tagged with the label 'beginner'].

.JIRA Priorites
* *Blocker*: Should only be used if the issue WILL cause data loss or cluster instability reliably.
* *Critical*: The issue described can cause data loss or cluster instability in some cases.
* *Major*: Important but not tragic issues, like updates to the client API that will add a lot of much-needed functionality or significant bugs that need to be fixed but that don't cause data loss.
* *Minor*: Useful enhancements and annoying but not damaging bugs.
* *Trivial*: Useful enhancements but generally cosmetic.

.Code Blocks in Jira Comments
====
A commonly used macro in Jira is {code}. Everything inside the tags is preformatted, as in this example.

[source]
----

{code}
code snippet
{code}
----
====

[[repos]]
== Apache HBase Repositories

Apache HBase consists of multiple repositories which are hosted on https://gitbox.apache.org/[Apache GitBox].
These are the following:

* https://gitbox.apache.org/repos/asf?p=hbase.git[hbase] - main Apache HBase repository
* https://gitbox.apache.org/repos/asf?p=hbase-connectors.git[hbase-connectors] - connectors to Apache Kafka and Apache Spark
* https://gitbox.apache.org/repos/asf?p=hbase-operator-tools.git[hbase-operator-tools] - operability and supportability tools, such as <<HBCK2>>
* https://gitbox.apache.org/repos/asf?p=hbase-site.git[hbase-site] - hbase.apache.org website
* https://gitbox.apache.org/repos/asf?p=hbase-thirdparty.git[hbase-thirdparty] - relocated versions of popular third-party libraries

== IDEs

[[eclipse]]
=== Eclipse

[[eclipse.code.formatting]]
==== Code Formatting

Under the _dev-support/_ folder, you will find _hbase_eclipse_formatter.xml_.
We encourage you to have this formatter in place in eclipse when editing HBase code.

Go to `Preferences->Java->Code Style->Formatter->Import` to load the xml file.
Go to `Preferences->Java->Editor->Save Actions`, and make sure 'Format source code' and 'Format
edited lines' is selected.

In addition to the automatic formatting, make sure you follow the style guidelines explained in
<<common.patch.feedback,common.patch.feedback>>.

[[eclipse.git.plugin]]
==== Eclipse Git Plugin

If you cloned the project via git, download and install the Git plugin (EGit). Attach to your local git repo (via the [label]#Git Repositories#                    window) and you'll be able to see file revision history, generate patches, etc.

[[eclipse.maven.setup]]
==== HBase Project Setup in Eclipse using `m2eclipse`

The easiest way is to use the +m2eclipse+ plugin for Eclipse.
Eclipse Indigo or newer includes +m2eclipse+, or you can download it from http://www.eclipse.org/m2e/. It provides Maven integration for Eclipse, and even lets you use the direct Maven commands from within Eclipse to compile and test your project.

To import the project, click  and select the HBase root directory. `m2eclipse`                    locates all the hbase modules for you.

If you install +m2eclipse+ and import HBase in your workspace, do the following to fix your eclipse Build Path.

. Remove _target_ folder
. Add _target/generated-jamon_ and _target/generated-sources/java_ folders.
. Remove from your Build Path the exclusions on the _src/main/resources_ and _src/test/resources_ to avoid error message in the console, such as the following:
+
----
Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project hbase:
'An Ant BuildException has occurred: Replace: source file .../target/classes/hbase-default.xml
doesn't exist
----
+
This will also reduce the eclipse build cycles and make your life easier when developing.


[[eclipse.commandline]]
==== HBase Project Setup in Eclipse Using the Command Line

Instead of using `m2eclipse`, you can generate the Eclipse files from the command line.

. First, run the following command, which builds HBase.
  You only need to do this once.
+
[source,bourne]
----
mvn clean install -DskipTests
----

. Close Eclipse, and execute the following command from the terminal, in your local HBase project directory, to generate new _.project_ and _.classpath_                            files.
+
[source,bourne]
----
mvn eclipse:eclipse
----

. Reopen Eclipse and import the _.project_ file in the HBase directory to a workspace.

[[eclipse.maven.class]]
==== Maven Classpath Variable

The `$M2_REPO` classpath variable needs to be set up for the project.
This needs to be set to your local Maven repository, which is usually _~/.m2/repository_

If this classpath variable is not configured, you will see compile errors in Eclipse like this:

----

Description	Resource	Path	Location	Type
The project cannot be built until build path errors are resolved	hbase		Unknown	Java Problem
Unbound classpath variable: 'M2_REPO/asm/asm/3.1/asm-3.1.jar' in project 'hbase'	hbase		Build path	Build Path Problem
Unbound classpath variable: 'M2_REPO/com/google/guava/guava/r09/guava-r09.jar' in project 'hbase'	hbase		Build path	Build Path Problem
Unbound classpath variable: 'M2_REPO/com/google/protobuf/protobuf-java/2.3.0/protobuf-java-2.3.0.jar' in project 'hbase'	hbase		Build path	Build Path Problem Unbound classpath variable:
----

[[eclipse.issues]]
==== Eclipse Known Issues

Eclipse will currently complain about _Bytes.java_.
It is not possible to turn these errors off.

----

Description	Resource	Path	Location	Type
Access restriction: The method arrayBaseOffset(Class) from the type Unsafe is not accessible due to restriction on required library /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar	Bytes.java	/hbase/src/main/java/org/apache/hadoop/hbase/util	line 1061	Java Problem
Access restriction: The method arrayIndexScale(Class) from the type Unsafe is not accessible due to restriction on required library /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar	Bytes.java	/hbase/src/main/java/org/apache/hadoop/hbase/util	line 1064	Java Problem
Access restriction: The method getLong(Object, long) from the type Unsafe is not accessible due to restriction on required library /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar	Bytes.java	/hbase/src/main/java/org/apache/hadoop/hbase/util	line 1111	Java Problem
----

[[eclipse.more]]
==== Eclipse - More Information

For additional information on setting up Eclipse for HBase development on Windows, see link:http://michaelmorello.blogspot.com/2011/09/hbase-subversion-eclipse-windows.html[Michael Morello's blog] on the topic.

=== IntelliJ IDEA

A functional development environment can be setup around an IntelliJ IDEA installation that has the
plugins necessary for building Java projects with Maven.

. Use either File > New > "Project from Existing Sources..." or "Project From Version Control.."
. Depending on your version of IntelliJ, you may need to choose Maven as the "project" or "model"
type.

The following plugins are recommended:

. Maven, bundled. This allows IntelliJ to resolve dependencies and recognize the project structure.
. EditorConfig, bundled. This will apply project whitespace settings found in the the
`.editorconfig` file available on branches with
link:https://issues.apache.org/jira/browse/HBASE-23234[HBASE-23234] or later.
. link:https://plugins.jetbrains.com/plugin/1065-checkstyle-idea/[Checkstyle-IDEA]. Configure this
against the configuration file found under `hbase-checkstyle/src/main/resources/hbase/checkstyle.xml`
(If the Intellij checkstyle plugin complains parsing the volunteered hbase `checkstyle.xml`, make
sure the plugin's `version` popup menu matches the hbase checkstyle version -- see
link:https://issues.apache.org/jira/browse/HBASE-23242[HBASE-23242] for more).
This plugin will highlight style errors in the IDE, so you can fix them before they get flagged during the
pre-commit process.
. link:https://plugins.jetbrains.com/plugin/8277-protobuf-support/[Protobuf Support]. HBase uses
link:https://developers.google.com/protocol-buffers/[Protocol Buffers] in a number of places where
serialization is required. This plugin is helpful when editing these object definitions.
. link:https://plugins.jetbrains.com/plugin/7391-asciidoc/[AsciiDoc]. HBase uses
link:http://asciidoc.org[AsciiDoc] for  building it's project documentation. This plugin is helpful
when editing this book.

=== Other IDEs

If you'd have another environment with which you'd like to develop on HBase, please consider
documenting your setup process here.

[[build]]
== Building Apache HBase

[[build.basic]]
=== Basic Compile

HBase is compiled using Maven. You must use at least Maven 3.0.4. To check your Maven version, run
the command +mvn -version+.

[[build.basic.jdk_requirements]]
==== JDK Version Requirements

HBase has Java version compiler requirements that vary by release branch. At compilation time,
HBase has the same version requirements as it does for runtime. See <<java,java>> for a complete
support matrix of Java version by HBase version.

[[maven.build.commands]]
==== Maven Build Commands

All commands are executed from the local HBase project directory.

===== Package

The simplest command to compile HBase from its java source code is to use the `package` target, which builds JARs with the compiled files.

[source,bourne]
----
mvn package -DskipTests
----

Or, to clean up before compiling:

[source,bourne]
----
mvn clean package -DskipTests
----

With Eclipse set up as explained above in <<eclipse,eclipse>>, you can also use the menu:Build[] command in Eclipse.
To create the full installable HBase package takes a little bit more work, so read on.

[[maven.build.commands.compile]]
===== Compile

The `compile` target does not create the JARs with the compiled files.

[source,bourne]
----
mvn compile
----

[source,bourne]
----
mvn clean compile
----

===== Install

To install the JARs in your _~/.m2/_ directory, use the `install` target.

[source,bourne]
----
mvn install
----

[source,bourne]
----
mvn clean install
----

[source,bourne]
----
mvn clean install -DskipTests
----

[[maven.build.commands.unitall]]
==== Running all or individual Unit Tests

See the <<hbase.unittests.cmds,hbase.unittests.cmds>> section in <<hbase.unittests,hbase.unittests>>

[[maven.build.hadoop]]
==== Building against various Hadoop versions

HBase supports building against Apache Hadoop versions: 2.y and 3.y (early release artifacts).
Exactly which version of Hadoop is used by default varies by release branch. See the section
<<hadoop,Hadoop>> for the complete breakdown of supported Hadoop version by HBase release.

The mechanism for selecting a Hadoop version at build time is identical across all releases. Which
version of Hadoop is default varies. We manage Hadoop major version selection by way of Maven
profiles. Due to the peculiarities of Maven profile mutual exclusion, the profile that builds
against a particular Hadoop version is activated by setting a property, *not* the usual profile
activation. Hadoop version profile activation is summarized by the following table.

.Hadoop Profile Activation by HBase Release
[cols="3*^.^", options="header"]
|===
| | Hadoop2 Activation | Hadoop3 Activation
| HBase 1.3+ | _active by default_ | `-Dhadoop.profile=3.0`
| HBase 3.0+ | _not supported_ | _active by default_
|===

[WARNING]
====
Please note that where a profile is active by default, `hadoop.profile` must NOT be provided.
====

Once the Hadoop major version profile is activated, the exact Hadoop version can be
specified by overriding the appropriate property value. For Hadoop2 versions, the property name
is `hadoop-two.version`. With Hadoop3 versions, the property name is `hadoop-three.version`.

.Example 1, Building HBase 1.7 against Hadoop 2.10.0

For example, to build HBase 1.7 against Hadoop 2.10.0, the profile is set for Hadoop2 by default,
so only `hadoop-two.version` must be specified:

[source,bourne]
----
git checkout branch-1
mvn -Dhadoop-two.version=2.10.0 ...
----

.Example 2, Building HBase 2.3 against Hadoop 3.3.0-SNAPSHOT

This is how a developer might check the compatibility of HBase 2.3 against an unreleased Hadoop
version (currently 3.3). Both the Hadoop3 profile and version must be specified:

[source,bourne]
----
git checkout branch-2.3
mvn -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.0-SNAPSHOT ...
----

.Example 3, Building HBase 3.0 against Hadoop 3.3.0-SNAPSHOT

The same developer might want also to check the development version of HBase (currently 3.0)
against the development version of Hadoop (currently 3.3). In this case, the Hadoop3 profile is
active by default, so only `hadoop-three.version` must be specified:

[source,bourne]
----
git checkout master
mvn -Dhadoop-three.version=3.3.0-SNAPSHOT ...
----

[[maven.build.jdk11_hadoop3]]
==== Building with JDK11 and Hadoop3

HBase manages JDK-specific build settings using Maven profiles. The profile appropriate to the JDK
in use is automatically activated. Building and running on JDK8 supports both Hadoop2 and Hadoop3.
For JDK11, only Hadoop3 is supported. Thus, the Hadoop3 profile must be active when building on
JDK11, and the artifacts used when running HBase on JDK11 must be compiled against Hadoop3.
Furthermore, the JDK11 profile requires a minimum Hadoop version of 3.2.0. This value is specified
by the JDK11 profile, but it can be overridden using the `hadoop-three.version` property as normal.
For details on Hadoop profile activation by HBase branch, see
<<maven.build.hadoop,Building against various Hadoop versions>>. See <<java,java>> for a complete
support matrix of Java version by HBase version.

.Example 1, Building HBase 2.3 with JDK11

To build HBase 2.3 with JDK11, the Hadoop3 profile must be activated explicitly.

[source,bourne]
----
git checkout branch-2.3
JAVA_HOME=/usr/lib/jvm/java-11 mvn -Dhadoop.profile=3.0 ...
----

.Example 2, Building HBase 3.0 with JDK11

For HBase 3.0, the Hadoop3 profile is active by default, no additional properties need be
specified.

[source,bourne]
----
git checkout master
JAVA_HOME=/usr/lib/jvm/java-11 mvn ...
----

[[maven.build.jdk11_hadoop3_ide]]
==== Building and testing in an IDE with JDK11 and Hadoop3

Continuing the discussion from the <<maven.build.jdk11_hadoop3,earlier section>>, building and
testing with JDK11 and Hadoop3 within an IDE may require additional configuration. Specifically,
make sure the JVM version used by the IDE is a JDK11, the active JDK Maven profile is for JDK11,
and the Maven profile for JDK8 is NOT active. Likewise, ensure the Hadoop3 Maven profile is active
and the Hadoop2 Maven profile is NOT active.

[[build.protobuf]]
==== Build Protobuf

You may need to change the protobuf definitions that reside in the _hbase-protocol_ module or other modules.

Previous to hbase-2.0.0, protobuf definition files were sprinkled across all hbase modules but now all
to do with protobuf must reside in the hbase-protocol module; we are trying to contain our protobuf
use so we can freely change versions without upsetting any downstream project use of protobuf.

The protobuf files are located in _hbase-protocol/src/main/protobuf_.
For the change to be effective, you will need to regenerate the classes.

[source,bourne]
----
mvn package -pl hbase-protocol -am
----

Similarly, protobuf definitions for internal use are located in the _hbase-protocol-shaded_ module.

[source,bourne]
----
mvn package -pl hbase-protocol-shaded -am
----

Typically, protobuf code generation is done using the native `protoc` binary. In our build we use a maven plugin for
convenience; however, the plugin may not be able to retrieve appropriate binaries for all platforms. If you find yourself
on a platform where protoc fails, you will have to compile protoc from source, and run it independent of our maven build.
You can disable the inline code generation by specifying `-Dprotoc.skip` in your maven arguments, allowing your build to proceed further.

[NOTE]
If you need to manually generate your protobuf files, you should not use `clean` in subsequent maven calls, as that will delete the newly generated files.

Read the _hbase-protocol/README.txt_ for more details

[[build.thrift]]
==== Build Thrift

You may need to change the thrift definitions that reside in the _hbase-thrift_ module or other modules.

The thrift files are located in _hbase-thrift/src/main/resources_.
For the change to be effective, you will need to regenerate the classes.
You can use maven profile  `compile-thrift` to do this.

[source,bourne]
----
mvn compile -Pcompile-thrift
----

You may also want to define `thrift.path` for the thrift binary, using the following command:

[source,bourne]
----

                  mvn compile -Pcompile-thrift -Dthrift.path=/opt/local/bin/thrift
----

==== Build a Tarball

You can build a tarball without going through the release process described in <<releasing,releasing>>, by running the following command:

----
mvn -DskipTests clean install && mvn -DskipTests package assembly:single
----

The distribution tarball is built in _hbase-assembly/target/hbase-<version>-bin.tar.gz_.

You can install or deploy the tarball by having the assembly:single goal before install or deploy in the maven command:

----
mvn -DskipTests package assembly:single install
----
----
mvn -DskipTests package assembly:single deploy
----


[[build.gotchas]]
==== Build Gotchas
===== Maven Site failure
If you see `Unable to find resource 'VM_global_library.vm'`, ignore it.
It's not an error.
It is link:https://issues.apache.org/jira/browse/MSITE-286[officially ugly] though.

[[build.on.linux.aarch64]]
=== Build On Linux Aarch64
HBase runs on both Windows and UNIX-like systems, and it should run on any platform
that runs a supported version of Java. This should include JVMs on x86_64 and aarch64.
The documentation below describes how to build hbase on aarch64 platform.

==== Set Environment Variables
Manually install Java and Maven on aarch64 servers if they are not installed,
and set environment variables. For example:

[source,bourne]
----
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64
export MAVEN_HOME=/opt/maven
export PATH=${MAVEN_HOME}/bin:${JAVA_HOME}/bin:${PATH}
----

==== Use Protobuf Supported On Aarch64
Now HBase uses protobuf of two versions. Version '3.11.4' of protobuf that hbase uses
internally and version '2.5.0' as external usage.
Package protoc-2.5.0 does not work on aarch64 platform, we should add maven
profile '-Paarch64' when building. It downloads protoc-2.5.0 package from maven
repository which we made on aarch64 platform locally.

[source,bourne]
----
mvn clean install -Paarch64 -DskipTests
----

[NOTE]
Protobuf is released with aarch64 protoc since version '3.5.0', and we are planning to
upgrade protobuf later, then we don't have to add the profile '-Paarch64' anymore.


[[releasing]]
== Releasing Apache HBase

.Building against HBase 1.x
[NOTE]
====
See old refguides for how to build HBase 1.x.
The below is for building hbase2.
====

[[maven.release]]
=== Making a Release Candidate
Only committers can make releases of hbase artifacts.

.Before You Begin
Check to be sure recent builds have been passing for the branch from where you
are going to take your release. You should also have tried recent branch tips
out on a cluster under load, perhaps by running the `hbase-it` integration test
suite for a few hours to 'burn in' the near-candidate bits.

You will need a published signing key added to the hbase
link:https://dist.apache.org/repos/dist/release/hbase/KEYS[KEYS] file.
(For how to add a KEY, see _Step 1._ in link:https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease[How To Release],
the Hadoop version of this document).

Next make sure JIRA is properly primed, that all issues targeted against
the prospective release have been resolved and are present in git on the
particular branch. If any outstanding issues, move them out of the release by
adjusting the fix version to remove this pending release as a target.
Any JIRA with a fix version that matches the release candidate
target release will be included in the generated _CHANGES.md/RELEASENOTES.md_
files that ship with the release so make sure JIRA is correct before you begin.

After doing the above, you can move to the manufacture of an RC.

Building an RC is involved so we've scripted it. The script builds in a Docker
container to ensure we have a consistent environment building. It will ask you
for passwords for apache and for your gpg signing key so it can sign and commit
on your behalf. The passwords are passed to gpg-agent in the container and
purged along with the container when the build is done.

The script will:

 * Set version to the release version 
 * Updates RELEASENOTES.md and CHANGES.md
 * Tag the RC
 * Set version to next SNAPSHOT version. 
 * Builds, signs, and hashes all artifacts. 
 * Generates the api compatibility report
 * Pushes release tgzs to the dev dir in a apache dist. 
 * Pushes to repository.apache.org staging. 
 * Creates vote email template.

The _dev-support/create-release/do-release-docker.sh_ Release Candidate (RC)
Generating script is maintained in the master branch but can generate RCs
for any 2.x+ branch (The script does not work against branch-1). Check out
and update the master branch when making RCs. See
_dev-support/create-release/README.txt_ for how to configure your
environment and run the script.

[NOTE]
====
_dev-support/create-release/do-release-docker.sh_ supercedes the previous
_dev-support/make_rc.sh_ script. It is more comprehensive automating all
steps, rather than a portion, building a RC.
====

==== Release Candidate Procedure
Here we outline the steps involved generating a Release Candidate, the steps
automated by the _dev-support/create-release/do-release-docker.sh_ script
described in the previous section. Running these steps manually tends to
be error-prone so is not recommended. The below is informational only.

The process below makes use of various tools, mainly _git_ and _maven_.

.Specifying the Heap Space for Maven
[NOTE]
====
You may run into OutOfMemoryErrors building, particularly building the site and
documentation. Up the heap for Maven by setting the `MAVEN_OPTS` variable.
You can prefix the variable to the Maven command, as in the following example:

----
MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=256m" mvn package
----

You could also set this in an environment variable or alias in your shell.
====

[[maven.settings.xml]]
.Example _~/.m2/settings.xml_ File
====
Publishing to maven requires you sign the artifacts you want to upload.
For the build to sign them for you, you a properly configured _settings.xml_
in your local repository under _.m2_, such as the following.

[source,xml]
----
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <!- To publish a snapshot of some part of Maven -->
    <server>
      <id>apache.snapshots.https</id>
      <username>YOUR_APACHE_ID
      </username>
      <password>YOUR_APACHE_PASSWORD
      </password>
    </server>
    <!-- To publish a website using Maven -->
    <!-- To stage a release of some part of Maven -->
    <server>
      <id>apache.releases.https</id>
      <username>YOUR_APACHE_ID
      </username>
      <password>YOUR_APACHE_PASSWORD
      </password>
    </server>
  </servers>
  <profiles>
    <profile>
      <id>apache-release</id>
      <properties>
    <gpg.keyname>YOUR_KEYNAME</gpg.keyname>
    <!--Keyname is something like this ... 00A5F21E... do gpg --list-keys to find it-->
    <gpg.passphrase>YOUR_KEY_PASSWORD
    </gpg.passphrase>
      </properties>
    </profile>
  </profiles>
</settings>
----
====

===== Update the _CHANGES.md_  and _RELEASENOTES.md_ files and the POM files.

Update _CHANGES.md_ with the changes since the last release. Be careful with where you put
headings and license. Respect the instructions and warning you find in current
_CHANGES.md_ and _RELEASENOTES.md_ since these two files are processed by tooling that is
looking for particular string sequences. See link:https://issues.apache.org/jira/browse/HBASE-21399[HBASE-21399]
for description on how to make use of yetus generating additions to
_CHANGES.md_ and _RELEASENOTES.md_ (RECOMMENDED!).  Adding JIRA fixes, make sure the
URL to the JIRA points to the proper location which lists fixes for this release.

Next, adjust the version in all the POM files appropriately.
If you are making a release candidate, you must remove the `-SNAPSHOT` label from all versions
in all pom.xml files.
If you are running this receipe to publish a snapshot, you must keep the `-SNAPSHOT` suffix on the hbase version.
The link:http://www.mojohaus.org/versions-maven-plugin/[Versions Maven Plugin] can be of use here.
To set a version in all the many poms of the hbase multi-module project, use a command like the following:

[source,bourne]
----
$ mvn clean org.codehaus.mojo:versions-maven-plugin:2.5:set -DnewVersion=2.1.0-SNAPSHOT
----

Make sure all versions in poms are changed! Checkin the _CHANGES.md_, _RELEASENOTES.md_, and
any maven version changes.

===== Update the documentation.

Update the documentation under _src/main/asciidoc_.
This usually involves copying the latest from master branch and making version-particular
adjustments to suit this release candidate version. Commit your changes.

===== Clean the checkout dir
[source,bourne]
----
$ mvn clean
$ git clean -f -x -d
----

===== Run Apache-Rat
Check licenses are good

[source,bourne]
----
$ mvn apache-rat:check
----

If the above fails, check the rat log.

[source,bourne]
----
$ grep 'Rat check' patchprocess/mvn_apache_rat.log
----

===== Create a release tag.
Presuming you have run basic tests, the rat check, passes and all is
looking good, now is the time to tag the release candidate (You
always remove the tag if you need to redo). To tag, do
what follows substituting in the version appropriate to your build.
All tags should be signed tags; i.e. pass the _-s_ option (See
link:http://https://git-scm.com/book/id/v2/Git-Tools-Signing-Your-Work[Signing Your Work]
for how to set up your git environment for signing).

[source,bourne]
----
$ git tag -s 2.0.0-alpha4-RC0 -m "Tagging the 2.0.0-alpha4 first Releae Candidate (Candidates start at zero)"
----

Or, if you are making a release, tags should have a _rel/_ prefix to ensure
they are preserved in the Apache repo as in:

[source,bourne]
----
+$ git tag -s rel/2.0.0-alpha4 -m "Tagging the 2.0.0-alpha4 Release"
----

Push the (specific) tag (only) so others have access.

[source,bourne]
----
$ git push origin 2.0.0-alpha4-RC0
----

For how to delete tags, see
link:http://www.manikrathee.com/how-to-delete-a-tag-in-git.html[How to Delete a Tag]. Covers
deleting tags that have not yet been pushed to the remote Apache
repo as well as delete of tags pushed to Apache.

===== Build the source tarball.
Now, build the source tarball. Lets presume we are building the source
tarball for the tag _2.0.0-alpha4-RC0_ into _/tmp/hbase-2.0.0-alpha4-RC0/_
(This step requires that the mvn and git clean steps described above have just been done).

[source,bourne]
----
$ git archive --format=tar.gz --output="/tmp/hbase-2.0.0-alpha4-RC0/hbase-2.0.0-alpha4-src.tar.gz" --prefix="hbase-2.0.0-alpha4/" $git_tag
----

Above we generate the hbase-2.0.0-alpha4-src.tar.gz tarball into the
_/tmp/hbase-2.0.0-alpha4-RC0_ build output directory (We don't want the _RC0_ in the name or prefix.
These bits are currently a release candidate but if the VOTE passes, they will become the release so we do not taint
the artifact names with _RCX_).

===== Build the binary tarball.
Next, build the binary tarball. Add the `-Prelease` profile when building.
It runs the license apache-rat check among other rules that help ensure
all is wholesome. Do it in two steps.

First install into the local repository

[source,bourne]
----
$ mvn clean install -DskipTests -Prelease
----

Next, generate documentation and assemble the tarball. Be warned,
this next step can take a good while, a couple of hours generating site
documentation.

[source,bourne]
----
$ mvn install -DskipTests site assembly:single -Prelease
----

Otherwise, the build complains that hbase modules are not in the maven repository
when you try to do it all in one step, especially on a fresh repository.
It seems that you need the install goal in both steps.

Extract the generated tarball -- you'll find it under
_hbase-assembly/target_ and check it out.
Look at the documentation, see if it runs, etc.
If good, copy the tarball beside the source tarball in the
build output directory.

===== Deploy to the Maven Repository.

Next, deploy HBase to the Apache Maven repository. Add the
apache-release` profile when running the `mvn deploy` command.
This profile comes from the Apache parent pom referenced by our pom files.
It does signing of your artifacts published to Maven, as long as the
_settings.xml_ is configured correctly, as described in <<maven.settings.xml>>.
This step depends on the local repository having been populate
by the just-previous bin tarball build.


[source,bourne]
----
$ mvn deploy -DskipTests -Papache-release -Prelease
----

This command copies all artifacts up to a temporary staging Apache mvn repository in an 'open' state.
More work needs to be done on these maven artifacts to make them generally available.

We do not release HBase tarball to the Apache Maven repository. To avoid deploying the tarball, do not
include the `assembly:single` goal in your `mvn deploy` command. Check the deployed artifacts as described in the next section.

.make_rc.sh
[NOTE]
====
If you ran the old _dev-support/make_rc.sh_ script, this is as far as it takes you.
To finish the release, take up the script from here on out.
====

===== Make the Release Candidate available.

The artifacts are in the maven repository in the staging area in the 'open' state.
While in this 'open' state you can check out what you've published to make sure all is good.
To do this, log in to Apache's Nexus at link:https://repository.apache.org[repository.apache.org] using your Apache ID.
Find your artifacts in the staging repository. Click on 'Staging Repositories' and look for a new one ending in "hbase" with a status of 'Open', select it.
Use the tree view to expand the list of repository contents and inspect if the artifacts you expect are present. Check the POMs.
As long as the staging repo is open you can re-upload if something is missing or built incorrectly.

If something is seriously wrong and you would like to back out the upload, you can use the 'Drop' button to drop and delete the staging repository.
Sometimes the upload fails in the middle. This is another reason you might have to 'Drop' the upload from the staging repository.

If it checks out, close the repo using the 'Close' button. The repository must be closed before a public URL to it becomes available. It may take a few minutes for the repository to close. Once complete you'll see a public URL to the repository in the Nexus UI. You may also receive an email with the URL. Provide the URL to the temporary staging repository in the email that announces the release candidate.
(Folks will need to add this repo URL to their local poms or to their local _settings.xml_ file to pull the published release candidate artifacts.)

When the release vote concludes successfully, return here and click the 'Release' button to release the artifacts to central. The release process will automatically drop and delete the staging repository.

.hbase-downstreamer
[NOTE]
====
See the link:https://github.com/saintstack/hbase-downstreamer[hbase-downstreamer] test for a simple example of a project that is downstream of HBase an depends on it.
Check it out and run its simple test to make sure maven artifacts are properly deployed to the maven repository.
Be sure to edit the pom to point to the proper staging repository.
Make sure you are pulling from the repository when tests run and that you are not getting from your local repository, by either passing the `-U` flag or deleting your local repo content and check maven is pulling from remote out of the staging repository.
====

See link:https://www.apache.org/dev/publishing-maven-artifacts.html[Publishing Maven Artifacts] for some pointers on this maven staging process.

If the HBase version ends in `-SNAPSHOT`, the artifacts go elsewhere.
They are put into the Apache snapshots repository directly and are immediately available.
Making a SNAPSHOT release, this is what you want to happen.

At this stage, you have two tarballs in your 'build output directory' and a set of artifacts
in a staging area of the maven repository, in the 'closed' state.
Next sign, fingerprint and then 'stage' your release candiate build output directory via svnpubsub by committing
your directory to link:https://dist.apache.org/repos/dist/dev/hbase/[The dev distribution directory]
(See comments on link:https://issues.apache.org/jira/browse/HBASE-10554[HBASE-10554 Please delete old releases from mirroring system]
but in essence it is an svn checkout of link:https://dist.apache.org/repos/dist/dev/hbase[dev/hbase] -- releases are at
link:https://dist.apache.org/repos/dist/release/hbase[release/hbase]). In the _version directory_ run the following commands:

[source,bourne]
----
$ for i in *.tar.gz; do echo $i; gpg --print-md MD5 $i > $i.md5 ; done
$ for i in *.tar.gz; do echo $i; gpg --print-md SHA512 $i > $i.sha ; done
$ for i in *.tar.gz; do echo $i; gpg --armor --output $i.asc --detach-sig $i  ; done
$ cd ..
# Presuming our 'build output directory' is named 0.96.0RC0, copy it to the svn checkout of the dist dev dir
# in this case named hbase.dist.dev.svn
$ cd /Users/stack/checkouts/hbase.dist.dev.svn
$ svn info
Path: .
Working Copy Root Path: /Users/stack/checkouts/hbase.dist.dev.svn
URL: https://dist.apache.org/repos/dist/dev/hbase
Repository Root: https://dist.apache.org/repos/dist
Repository UUID: 0d268c88-bc11-4956-87df-91683dc98e59
Revision: 15087
Node Kind: directory
Schedule: normal
Last Changed Author: ndimiduk
Last Changed Rev: 15045
Last Changed Date: 2016-08-28 11:13:36 -0700 (Sun, 28 Aug 2016)
$ mv 0.96.0RC0 /Users/stack/checkouts/hbase.dist.dev.svn
$ svn add 0.96.0RC0
$ svn commit ...
----

Ensure it actually gets published by checking link:https://dist.apache.org/repos/dist/dev/hbase/[https://dist.apache.org/repos/dist/dev/hbase/].

Announce the release candidate on the mailing list and call a vote.

[[maven.snapshot]]
=== Publishing a SNAPSHOT to maven

Make sure your _settings.xml_ is set up properly (see <<maven.settings.xml>>).
Make sure the hbase version includes `-SNAPSHOT` as a suffix.
Following is an example of publishing SNAPSHOTS of a release that had an hbase version of 0.96.0 in its poms.

[source,bourne]
----

 $ mvn clean install -DskipTests  javadoc:aggregate site assembly:single -Prelease
 $ mvn -DskipTests  deploy -Papache-release
----

The _make_rc.sh_ script mentioned above (see <<maven.release,maven.release>>) can help you publish `SNAPSHOTS`.
Make sure your `hbase.version` has a `-SNAPSHOT`                suffix before running the script.
It will put a snapshot up into the apache snapshot repository for you.

[[hbase.rc.voting]]
== Voting on Release Candidates

Everyone is encouraged to try and vote on HBase release candidates.
Only the votes of PMC members are binding.
PMC members, please read this WIP doc on policy voting for a release candidate, link:https://github.com/rectang/asfrelease/blob/master/release.md[Release Policy].

[quote]
____
Before casting +1 binding votes, individuals are required to
download the signed source code package onto their own hardware, compile it as
provided, and test the resulting executable on their own platform, along with also
validating cryptographic signatures and verifying that the package meets the
requirements of the ASF policy on releases.
____

Regards the latter, run `mvn apache-rat:check` to verify all files are suitably licensed.
See link:https://mail-archives.apache.org/mod_mbox/hbase-dev/201406.mbox/%3CCA%2BRK%3D_B8EP0JMFV%2Bdt-k1g%3DBmedzyq2z1GSqrnMMiH6%3DcdoiAA%40mail.gmail.com%3E[HBase, mail # dev - On recent discussion clarifying ASF release policy]
for how we arrived at this process.

To help with the release verification, please follow the guideline below and vote based on the your verification.

=== Baseline Verifications for Voting Release Candidates

Although contributors have their own checklist for verifications, the following items are usually used for voting on release candidates.

* CHANGES.md if any
* RELEASENOTES.md (release notes) if any
* Generated API compatibility report
** For what should be compatible please refer the link:https://hbase.apache.org/book.html#hbase.versioning[versioning guideline], especially for items with marked as high severity
* Use `hbase-vote.sh`  to perform sanity checks for checksum, signatures, files are licensed, built from source, and unit tests.
** `hbase-vote.sh` shell script is available under `dev-support` directory of HBase source. Following are the usage details.

[source,bourne]
----
./dev-support/hbase-vote.sh -h
hbase-vote. A script for standard vote which verifies the following items
1. Checksum of sources and binaries
2. Signature of sources and binaries
3. Rat check
4. Built from source
5. Unit tests

Usage: hbase-vote.sh -s | --source <url> [-k | --key <signature>] [-f | --keys-file-url <url>] [-o | --output-dir </path/to/use>] [-P runSmallTests] [-D property[=value]]
       hbase-vote.sh -h | --help

  -h | --help                   Show this screen.
  -s | --source '<url>'         A URL pointing to the release candidate sources and binaries
                                e.g. https://dist.apache.org/repos/dist/dev/hbase/hbase-<version>RC0/
  -k | --key '<signature>'      A signature of the public key, e.g. 9AD2AE49
  -f | --keys-file-url '<url>'   the URL of the key file, default is
                                https://downloads.apache.org/hbase/KEYS
  -o | --output-dir '</path>'   directory which has the stdout and stderr of each verification target
  -P |                          list of maven profiles to activate for test UT/IT, i.e. <-P runSmallTests> Defaults to runAllTests
  -D |                          list of maven properties to set for the mvn invocations, i.e. <-D hadoop.profile=3.0> Defaults to unset
----
* If you see any unit test failures, please call out the solo test result and whether it's part of flaky (nightly) tests dashboard, e.g. link:https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/dashboard.html[dashboard of master branch] (please change the test branch accordingly).

=== Additional Verifications for Voting Release Candidates

Other than the common verifications, contributors may call out additional concerns, e.g. for a specific feature by running end to end tests on a distributed environment. This is optional and always encouraged.

* Start a distributed HBase cluster and call out the test result of specific workload on cluster. e.g.
** Run basic table operations, e.g. `create/put/get/scan/flush/list/disable/drop`
** Run built-in tests, e.g. `LoadTestTool` (LTT) and `IntegrationTestBigLinkedList` (ITBLL)

[[hbase.release.announcement]]
== Announcing Releases

Once an RC has passed successfully and the needed artifacts have been staged for disribution, you'll need to let everyone know about our shiny new release. It's not a requirement, but to make things easier for release managers we have a template you can start with. Be sure you replace \_version_ and other markers with the relevant version numbers. You should manually verify all links before sending.

[source,email]
----
The HBase team is happy to announce the immediate availability of HBase _version_.

Apache HBase™ is an open-source, distributed, versioned, non-relational database.
Apache HBase gives you low latency random access to billions of rows with
millions of columns atop non-specialized hardware. To learn more about HBase,
see https://hbase.apache.org/.

HBase _version_ is the _nth_ minor release in the HBase _major_.x line, which aims to
improve the stability and reliability of HBase. This release includes roughly
XXX resolved issues not covered by previous _major_.x releases.

Notable new features include:
- List text descriptions of features that fit on one line
- Including if JDK or Hadoop support versions changes
- If the "stable" pointer changes, call that out
- For those with obvious JIRA IDs, include them (HBASE-YYYYY)

The full list of issues can be found in the included CHANGES.md and RELEASENOTES.md,
or via our issue tracker:

    https://s.apache.org/hbase-_version_-jira

To download please follow the links and instructions on our website:

    https://hbase.apache.org/downloads.html


Question, comments, and problems are always welcome at: dev@hbase.apache.org.

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team
----

You should sent this message to the following lists: dev@hbase.apache.org, user@hbase.apache.org, announce@apache.org. If you'd like a spot check before sending, feel free to ask via jira or the dev list.

[[documentation]]
== Generating the HBase Reference Guide

The manual is marked up using Asciidoc.
We then use the link:http://asciidoctor.org/docs/asciidoctor-maven-plugin/[Asciidoctor maven plugin] to transform the markup to html.
This plugin is run when you specify the +site+ goal as in when you run +mvn site+.
See <<appendix_contributing_to_documentation,appendix contributing to documentation>> for more information on building the documentation.

[[hbase.org]]
== Updating link:https://hbase.apache.org[hbase.apache.org]

[[hbase.org.site.contributing]]
=== Contributing to hbase.apache.org

See <<appendix_contributing_to_documentation,appendix contributing to documentation>> for more information on contributing to the documentation or website.

[[hbase.org.site.publishing]]
=== Publishing link:https://hbase.apache.org[hbase.apache.org]

See <<website_publish>> for instructions on publishing the website and documentation.

[[hbase.tests]]
== Tests

Developers, at a minimum, should familiarize themselves with the unit test detail; unit tests in HBase have a character not usually seen in other projects.

This information is about unit tests for HBase itself.
For developing unit tests for your HBase applications, see <<unit.tests,unit.tests>>.

[[hbase.moduletests]]
=== Apache HBase Modules

As of 0.96, Apache HBase is split into multiple modules.
This creates "interesting" rules for how and where tests are written.
If you are writing code for `hbase-server`, see <<hbase.unittests,hbase.unittests>> for how to write your tests.
These tests can spin up a minicluster and will need to be categorized.
For any other module, for example `hbase-common`, the tests must be strict unit tests and just test the class under test - no use of the HBaseTestingUtility or minicluster is allowed (or even possible given the dependency tree).

[[hbase.moduletest.shell]]
==== Testing the HBase Shell

The HBase shell and its tests are predominantly written in jruby.

In order to make these tests run as a part of the standard build, there are a few JUnit test classes that take care of loading the jruby implemented tests and running them.
The tests were split into separate classes to accomodate class level timeouts (see <<hbase.unittests>> for specifics).
You can run all of these tests from the top level with:

[source,bourne]
----
      mvn clean test -Dtest=Test*Shell
----

If you have previously done a `mvn install`, then you can instruct maven to run only the tests in the hbase-shell module with:

[source,bourne]
----
      mvn clean test -pl hbase-shell
----

Alternatively, you may limit the shell tests that run using the system variable `shell.test`.
This value should specify the ruby literal equivalent of a particular test case by name.
For example, the tests that cover the shell commands for altering tables are contained in the test case `AdminAlterTableTest`        and you can run them with:

[source,bourne]
----
      mvn clean test -pl hbase-shell -Dshell.test=/AdminAlterTableTest/
----

You may also use a link:http://docs.ruby-doc.com/docs/ProgrammingRuby/html/language.html#UJ[Ruby Regular Expression
      literal] (in the `/pattern/` style) to select a set of test cases.
You can run all of the HBase admin related tests, including both the normal administration and the security administration, with the command:

[source,bourne]
----

      mvn clean test -pl hbase-shell -Dshell.test=/.*Admin.*Test/
----

In the event of a test failure, you can see details by examining the XML version of the surefire report results

[source,bourne]
----
      vim hbase-shell/target/surefire-reports/TEST-org.apache.hadoop.hbase.client.TestShell.xml
----

[[hbase.moduletest.run]]
==== Running Tests in other Modules

If the module you are developing in has no other dependencies on other HBase modules, then you can cd into that module and just run:

[source,bourne]
----
mvn test
----

which will just run the tests IN THAT MODULE.
If there are other dependencies on other modules, then you will have run the command from the ROOT HBASE DIRECTORY.
This will run the tests in the other modules, unless you specify to skip the tests in that module.
For instance, to skip the tests in the hbase-server module, you would run:

[source,bourne]
----
mvn clean test -PskipServerTests
----

from the top level directory to run all the tests in modules other than hbase-server.
Note that you can specify to skip tests in multiple modules as well as just for a single module.
For example, to skip the tests in `hbase-server` and `hbase-common`, you would run:

[source,bourne]
----
mvn clean test -PskipServerTests -PskipCommonTests
----

Also, keep in mind that if you are running tests in the `hbase-server` module you will need to apply the maven profiles discussed in <<hbase.unittests.cmds,hbase.unittests.cmds>> to get the tests to run properly.

[[hbase.unittests]]
=== Unit Tests

Apache HBase unit tests must carry a Category annotation and
as of `hbase-2.0.0`, must be stamped with the HBase `ClassRule`.
Here is an example of what a Test Class looks like with a 
Category and ClassRule included:

[source,java]
----
...
@Category(SmallTests.class)
public class TestHRegionInfo {
  @ClassRule
  public static final HBaseClassTestRule CLASS_RULE =
      HBaseClassTestRule.forClass(TestHRegionInfo.class);

  @Test
  public void testCreateHRegionInfoName() throws Exception {
    // ...
  }
}
----
Here the Test Class is `TestHRegionInfo`. The `CLASS_RULE` has
the same form in every test class only the `.class` you pass
is that of the local test; i.e. in the TestTimeout Test Class, you'd
pass `TestTimeout.class` to the `CLASS_RULE` instead of the
`TestHRegionInfo.class` we have above. The `CLASS_RULE`
is where we'll enforce timeouts (currently set at a hard-limit of
thirteen! minutes for all tests -- 780 seconds) and other cross-unit test facility.
The test is in the `SmallTest` Category.

Categories can be arbitrary and provided as a list but each test MUST
carry one from the following list of sizings: `small`, `medium`, `large`, and
`integration`. The test sizing is designated using the JUnit
link:https://github.com/junit-team/junit4/wiki/Categories[categories]: `SmallTests`, `MediumTests`, `LargeTests`, `IntegrationTests`.
JUnit Categories are denoted using java annotations (a special unit test looks
for the presence of the @Category annotation in all unit tess and will fail if it
finds a test suite missing a sizing marking).

The first three categories, `small`, `medium`, and `large`, are for test cases which run when you
type `$ mvn test`.
In other words, these three categorizations are for HBase unit tests.
The `integration` category is not for unit tests, but for integration tests.
These are normally run when you invoke `$ mvn verify`.
Integration tests are described in <<integration.tests,integration.tests>>.

Keep reading to figure which annotation of the set `small`, `medium`, and `large`
to put on your new HBase test case.

.Categorizing Tests
Small Tests (((SmallTests)))::
  _Small_ test cases are executed in a shared JVM and each test suite/test class should
   run in 15 seconds or less; i.e. a link:https://en.wikipedia.org/wiki/JUnit[junit test fixture], a java object made
   up of test methods, should finish in under 15 seconds, no matter how many or how few test methods
   it has. These test cases should not use a minicluster as a minicluster starts many services,
   most unrelated to what is being tested. Multiple start/stops may leak resources or just overwhelm
   the single JVM context.

Medium Tests (((MediumTests)))::
  _Medium_ test cases are executed in separate JVM and individual test suites or test classes or in
  junit parlance, link:https://en.wikipedia.org/wiki/JUnit[test fixture], should run in 50 seconds
   or less. These test cases can use a mini cluster. Since we start up a JVM per test fixture (and
   often a cluster too), be sure to make the startup pay by writing test fixtures that do a lot of
   testing running tens of seconds perhaps combining test rather than spin up a jvm (and cluster)
   per test method; this practice will help w/ overall test times.

Large Tests (((LargeTests)))::
  _Large_ test cases are everything else. They are typically large-scale tests, regression tests
  for specific bugs, timeout tests, or performance tests. No large test suite can take longer than
  thirteen minutes. It will be killed as timed out. Cast your test as an Integration Test if it needs
  to run longer.

Integration Tests (((IntegrationTests)))::
  _Integration_ tests are system level tests.
  See <<integration.tests,integration.tests>> for more info.
  If you invoke `$ mvn test` on integration tests, there is no timeout for the test.

[[hbase.unittests.cmds]]
=== Running tests

The state of tests on the hbase branches varies. Some branches keep good test hygiene and all tests pass
reliably with perhaps an unlucky sporadic flakey test failure. On other branches, the case may be less so with
frequent flakies and even broken tests in need of attention that fail 100% of the time. Try and figure
the state of tests on the branch you are currently interested in; the current state of nightly
link:https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/[apache jenkins builds] is a good
place to start. Tests on master branch are generally not in the best of condition as releases
are less frequent off master. This can make it hard landing patches especially given our dictum that
patches land on master branch first.

The full test suite can take from 5-6 hours on an anemic VM with 4 CPUs and minimal
parallelism to 50 minutes or less on a linux machine with dozens of CPUs and plenty of
RAM.

When you go to run the full test suite, make sure you up the test runner user nproc
(`ulimit -u` -- make sure it > 6000 or more if more parallelism) and the number of
open files (`ulimit -n` -- make sure it > 10240 or more) limits on your system.
Errors because the test run hits
limits are often only opaquely related to the constraint. You can see the current
user settings by running `ulimit -a`.

[[hbase.unittests.cmds.test]]
==== Default: small and medium category tests

Running `mvn test` will execute all small tests in a single JVM (no fork) and then medium tests in a
forked, separate JVM for each test instance (For definition of 'small' test and so on, see
<<hbase.unittests>>). Medium tests are NOT executed if there is an error in a
small test. Large tests are NOT executed.

[[hbase.unittests.cmds.test.runalltests]]
==== Running all tests

Running `mvn test -P runAllTests` will execute small tests in a single JVM, then medium and large tests
in a forked, separate JVM for each test. Medium and large tests are NOT executed if there is an error in
a small test.

[[hbase.unittests.cmds.test.localtests.mytest]]
==== Running a single test or all tests in a package

To run an individual test, e.g. `MyTest`, rum `mvn test -Dtest=MyTest` You can also pass multiple,
individual tests as a comma-delimited list:
[source,bash]
----
mvn test  -Dtest=MyTest1,MyTest2,MyTest3
----
You can also pass a package, which will run all tests under the package:
[source,bash]
----
mvn test '-Dtest=org.apache.hadoop.hbase.client.*'
----

When `-Dtest` is specified, the `localTests` profile will be used.
Each junit test is executed in a separate JVM (A fork per test class).
There is no parallelization when tests are running in this mode.
You will see a new message at the end of the -report: `"[INFO] Tests are skipped"`.
It's harmless.  However, you need to make sure the sum of
`Tests run:` in the `Results:` section of test reports matching the number of tests
you specified because no error will be reported when a non-existent test case is specified.

[[hbase.unittests.cmds.test.profiles]]
==== Other test invocation permutations

Running `mvn test -P runSmallTests` will execute "small" tests only, using a single JVM.

Running `mvn test -P runMediumTests` will execute "medium" tests only, launching a new JVM for each test-class.

Running `mvn test -P runLargeTests` will execute "large" tests only, launching a new JVM for each test-class.

For convenience, you can run `mvn test -P runDevTests` to execute both small and medium tests, using a single JVM.

[[hbase.unittests.test.faster]]
==== Running tests faster

By default, `$ mvn test -P runAllTests` runs all tests using a quarter of the CPUs available on machine
hosting the test run (see `surefire.firstPartForkCount` and `surefire.secondPartForkCount` in the top-level
hbase `pom.xml` which default to 0.25C, or 1/4 of CPU count). Up these counts to get the build to run faster.
You can also have hbase modules
run their tests in parrallel when the dependency graph allows by passing `--threads=N` when you invoke
maven, where `N` is the amount of parallelism wanted.
maven, where `N` is the amount of _module_ parallelism wanted.

For example, allowing that you want to use all cores on a machine to run tests,
you could start up the maven test run with:

----
  $ x="1.0C";  mvn -Dsurefire.firstPartForkCount=$x -Dsurefire.secondPartForkCount=$x test -PrunAllTests
----

If a 32 core machine, you should see periods during which 32 forked jvms appear in your process listing each running unit tests.
Your milage may vary. Dependent on hardware, overcommittment of CPU and/or memory can bring the test suite crashing down,
usually complaining with a spew of test system exits and incomplete test report xml files. Start gently, with the default fork
and move up gradually.

Adding the `--threads=N`, maven will run N maven modules in parallel (when module inter-dependencies allow). Be aware, if you have
set the forkcount to `1.0C`, and the `--threads` count to '2', the number of concurrent test runners can approach
2 * CPU, a count likely to overcommit the host machine (with attendant test exits failures).

You will need ~2.2GB of memory per forked JVM plus the memory used by maven itself (3-4G).

===== RAM Disk

To increase the speed, you can as well use a ramdisk. 2-3G should be sufficient. Be sure to
delete the files between each test run. The typical way to configure a ramdisk on Linux is:

----
$ sudo mkdir /ram2G
sudo mount -t tmpfs -o size=2048M tmpfs /ram2G
----

You can then use it to run all HBase tests on 2.0 with the command:

----
mvn test -PrunAllTests -Dtest.build.data.basedirectory=/ram2G
----

[[hbase.unittests.cmds.test.hbasetests]]
==== +hbasetests.sh+

It's also possible to use the script +hbasetests.sh+.
This script runs the medium and large tests in parallel with two maven instances, and provides a single report.
This script does not use the hbase version of surefire so no parallelization is being done other than the two maven instances the script sets up.
It must be executed from the directory which contains the _pom.xml_.

For example running +./dev-support/hbasetests.sh+ will execute small and medium tests.
Running +./dev-support/hbasetests.sh
                        runAllTests+ will execute all tests.
Running +./dev-support/hbasetests.sh replayFailed+ will rerun the failed tests a second time, in a separate jvm and without parallelisation.

[[hbase.unittests.timeouts]]
==== Test Timeouts(((Test Timeouts)))
The HBase unit test sizing Categorization timeouts are not strictly enforced.

Any test that runs longer than ten minutes will be timedout/killed.

As of hbase-2.0.0, we have purged all per-test-method timeouts: i.e.
[source,java]
----
...
  @Test(timeout=30000)
  public void testCreateHRegionInfoName() throws Exception {
    // ...
  }
----
They are discouraged and don't make much sense given we are timing
base of how long the whole Test Fixture/Class/Suite takes and 
that the variance in how long a test method takes varies wildly
dependent upon context (loaded Apache Infrastructure versus
developer machine with nothing else running on it).


[[hbase.unittests.resource.checker]]
==== Test Resource Checker(((Test ResourceChecker)))

A custom Maven SureFire plugin listener checks a number of resources before and after each HBase unit test runs and logs its findings at the end of the test output files which can be found in _target/surefire-reports_                    per Maven module (Tests write test reports named for the test class into this directory.
Check the _*-out.txt_ files). The resources counted are the number of threads, the number of file descriptors, etc.
If the number has increased, it adds a _LEAK?_ comment in the logs.
As you can have an HBase instance running in the background, some threads can be deleted/created without any specific action in the test.
However, if the test does not work as expected, or if the test should not impact these resources, it's worth checking these log lines [computeroutput]+...hbase.ResourceChecker(157): before...+                    and [computeroutput]+...hbase.ResourceChecker(157): after...+.
For example:

----
2012-09-26 09:22:15,315 INFO [pool-1-thread-1]
hbase.ResourceChecker(157): after:
regionserver.TestColumnSeeking#testReseeking Thread=65 (was 65),
OpenFileDescriptor=107 (was 107), MaxFileDescriptor=10240 (was 10240),
ConnectionCount=1 (was 1)
----

[[hbase.tests.writing]]
=== Writing Tests

[[hbase.tests.rules]]
==== General rules

* As much as possible, tests should be written as category small tests.
* All tests must be written to support parallel execution on the same machine, hence they should not use shared resources as fixed ports or fixed file names.
* Tests should not overlog.
  More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
* Tests can be written with `HBaseTestingUtility`.
  This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster.

[[hbase.tests.categories]]
==== Categories and execution time

* All tests must be categorized, if not they could be skipped.
* All tests should be written to be as fast as possible.
* See <<hbase.unittests,hbase.unittests>> for test case categories and corresponding timeouts.
  This should ensure a good parallelization for people using it, and ease the analysis when the test fails.

[[hbase.tests.sleeps]]
==== Sleeps in tests

Whenever possible, tests should not use [method]+Thread.sleep+, but rather waiting for the real event they need.
This is faster and clearer for the reader.
Tests should not do a [method]+Thread.sleep+ without testing an ending condition.
This allows understanding what the test is waiting for.
Moreover, the test will work whatever the machine performance is.
Sleep should be minimal to be as fast as possible.
Waiting for a variable should be done in a 40ms sleep loop.
Waiting for a socket operation should be done in a 200 ms sleep loop.

[[hbase.tests.cluster]]
==== Tests using a cluster

Tests using a HRegion do not have to start a cluster: A region can use the local file system.
Start/stopping a cluster cost around 10 seconds.
They should not be started per test method but per test class.
Started cluster must be shutdown using [method]+HBaseTestingUtility#shutdownMiniCluster+, which cleans the directories.
As most as possible, tests should use the default settings for the cluster.
When they don't, they should document it.
This will allow to share the cluster later.

[[hbase.tests.example.code]]
==== Tests Skeleton Code

Here is a test skeleton code with Categorization and a Category-based timeout rule to copy and paste and use as basis for test contribution.
[source,java]
----
/**
 * Describe what this testcase tests. Talk about resources initialized in @BeforeClass (before
 * any test is run) and before each test is run, etc.
 */
// Specify the category as explained in <<hbase.unittests,hbase.unittests>>.
@Category(SmallTests.class)
public class TestExample {
  // Replace the TestExample.class in the below with the name of your test fixture class.
  private static final Log LOG = LogFactory.getLog(TestExample.class);

  // Handy test rule that allows you subsequently get the name of the current method. See
  // down in 'testExampleFoo()' where we use it to log current test's name.
  @Rule public TestName testName = new TestName();

  // The below rule does two things. It decides the timeout based on the category
  // (small/medium/large) of the testcase. This @Rule requires that the full testcase runs
  // within this timeout irrespective of individual test methods' times. The second
  // feature is we'll dump in the log when the test is done a count of threads still
  // running.
  @Rule public static TestRule timeout = CategoryBasedTimeout.builder().
    withTimeout(this.getClass()).withLookingForStuckThread(true).build();

  @Before
  public void setUp() throws Exception {
  }

  @After
  public void tearDown() throws Exception {
  }

  @Test
  public void testExampleFoo() {
    LOG.info("Running test " + testName.getMethodName());
  }
}
----

[[integration.tests]]
=== Integration Tests

HBase integration/system tests are tests that are beyond HBase unit tests.
They are generally long-lasting, sizeable (the test can be asked to 1M rows or 1B rows), targetable (they can take configuration that will point them at the ready-made cluster they are to run against; integration tests do not include cluster start/stop code), and verifying success, integration tests rely on public APIs only; they do not attempt to examine server internals asserting success/fail.
Integration tests are what you would run when you need to more elaborate proofing of a release candidate beyond what unit tests can do.
They are not generally run on the Apache Continuous Integration build server, however, some sites opt to run integration tests as a part of their continuous testing on an actual cluster.

Integration tests currently live under the _src/test_                directory in the hbase-it submodule and will match the regex: _**/IntegrationTest*.java_.
All integration tests are also annotated with `@Category(IntegrationTests.class)`.

Integration tests can be run in two modes: using a mini cluster, or against an actual distributed cluster.
Maven failsafe is used to run the tests using the mini cluster.
IntegrationTestsDriver class is used for executing the tests against a distributed cluster.
Integration tests SHOULD NOT assume that they are running against a mini cluster, and SHOULD NOT use private API's to access cluster state.
To interact with the distributed or mini cluster uniformly, `IntegrationTestingUtility`, and `HBaseCluster` classes, and public client API's can be used.

On a distributed cluster, integration tests that use ChaosMonkey or otherwise manipulate services thru cluster manager (e.g.
restart regionservers) use SSH to do it.
To run these, test process should be able to run commands on remote end, so ssh should be configured accordingly (for example, if HBase runs under hbase user in your cluster, you can set up passwordless ssh for that user and run the test also under it). To facilitate that, `hbase.it.clustermanager.ssh.user`, `hbase.it.clustermanager.ssh.opts` and `hbase.it.clustermanager.ssh.cmd` configuration settings can be used.
"User" is the remote user that cluster manager should use to perform ssh commands.
"Opts" contains additional options that are passed to SSH (for example, "-i /tmp/my-key"). Finally, if you have some custom environment setup, "cmd" is the override format for the entire tunnel (ssh) command.
The default string is {`/usr/bin/ssh %1$s %2$s%3$s%4$s "%5$s"`} and is a good starting point.
This is a standard Java format string with 5 arguments that is used to execute the remote command.
The argument 1 (%1$s) is SSH options set the via opts setting or via environment variable, 2 is SSH user name, 3 is "@" if username is set or "" otherwise, 4 is the target host name, and 5 is the logical command to execute (that may include single quotes, so don't use them). For example, if you run the tests under non-hbase user and want to ssh as that user and change to hbase on remote machine, you can use:
[source,bash]
----
/usr/bin/ssh %1$s %2$s%3$s%4$s "su hbase - -c \"%5$s\""
----
That way, to kill RS (for example) integration tests may run:
[source,bash]
----
{/usr/bin/ssh some-hostname "su hbase - -c \"ps aux | ... | kill ...\""}
----
The command is logged in the test logs, so you can verify it is correct for your environment.

To disable the running of Integration Tests, pass the following profile on the command line `-PskipIntegrationTests`.
For example,
[source]
----
$ mvn clean install test -Dtest=TestZooKeeper  -PskipIntegrationTests
----

[[maven.build.commands.integration.tests.mini]]
==== Running integration tests against mini cluster

HBase 0.92 added a `verify` maven target.
Invoking it, for example by doing `mvn verify`, will run all the phases up to and including the verify phase via the maven link:https://maven.apache.org/plugins/maven-failsafe-plugin/[failsafe
                        plugin], running all the above mentioned HBase unit tests as well as tests that are in the HBase integration test group.
After you have completed +mvn install -DskipTests+ You can run just the integration tests by invoking:

[source,bourne]
----

cd hbase-it
mvn verify
----

If you just want to run the integration tests in top-level, you need to run two commands.
First:
----
mvn failsafe:integration-test
----

This actually runs ALL the integration tests.

NOTE: This command will always output `BUILD SUCCESS` even if there are test failures.

At this point, you could grep the output by hand looking for failed tests.
However, maven will do this for us; just use:
----
mvn failsafe:verify
----

The above command basically looks at all the test results (so don't remove the 'target' directory) for test failures and reports the results.

[[maven.build.commands.integration.tests2]]
===== Running a subset of Integration tests

This is very similar to how you specify running a subset of unit tests (see above), but use the property `it.test` instead of `test`.
To just run `IntegrationTestClassXYZ.java`, use:
----
mvn failsafe:integration-test -Dit.test=IntegrationTestClassXYZ -DfailIfNoTests=false
----
The next thing you might want to do is run groups of integration tests, say all integration tests that are named IntegrationTestClassX*.java:
----
mvn failsafe:integration-test -Dit.test=*ClassX* -DfailIfNoTests=false
----

This runs everything that is an integration test that matches *ClassX*. This means anything matching: "**/IntegrationTest*ClassX*". You can also run multiple groups of integration tests using comma-delimited lists (similar to unit tests). Using a list of matches still supports full regex matching for each of the groups. This would look something like:
----
mvn failsafe:integration-test -Dit.test=*ClassX*,*ClassY -DfailIfNoTests=false
----

[[maven.build.commands.integration.tests.distributed]]
==== Running integration tests against distributed cluster

If you have an already-setup HBase cluster, you can launch the integration tests by invoking the class `IntegrationTestsDriver`.
You may have to run test-compile first.
The configuration will be picked by the bin/hbase script.
[source,bourne]
----
mvn test-compile
----
Then launch the tests with:

[source,bourne]
----
bin/hbase [--config config_dir] org.apache.hadoop.hbase.IntegrationTestsDriver
----

Pass `-h` to get usage on this sweet tool.
Running the IntegrationTestsDriver without any argument will launch tests found under `hbase-it/src/test`, having `@Category(IntegrationTests.class)` annotation, and a name starting with `IntegrationTests`.
See the usage, by passing -h, to see how to filter test classes.
You can pass a regex which is checked against the full class name; so, part of class name can be used.
IntegrationTestsDriver uses Junit to run the tests.
Currently there is no support for running integration tests against a distributed cluster using maven (see link:https://issues.apache.org/jira/browse/HBASE-6201[HBASE-6201]).

The tests interact with the distributed cluster by using the methods in the `DistributedHBaseCluster` (implementing `HBaseCluster`) class, which in turn uses a pluggable `ClusterManager`.
Concrete implementations provide actual functionality for carrying out deployment-specific and environment-dependent tasks (SSH, etc). The default `ClusterManager` is `HBaseClusterManager`, which uses SSH to remotely execute start/stop/kill/signal commands, and assumes some posix commands (ps, etc). Also assumes the user running the test has enough "power" to start/stop servers on the remote machines.
By default, it picks up `HBASE_SSH_OPTS`, `HBASE_HOME`, `HBASE_CONF_DIR` from the env, and uses `bin/hbase-daemon.sh` to carry out the actions.
Currently tarball deployments, deployments which uses _hbase-daemons.sh_, and link:https://incubator.apache.org/ambari/[Apache Ambari]                    deployments are supported.
_/etc/init.d/_ scripts are not supported for now, but it can be easily added.
For other deployment options, a ClusterManager can be implemented and plugged in.

[[maven.build.commands.integration.tests.destructive]]
==== Destructive integration / system tests (ChaosMonkey)

HBase 0.96 introduced a tool named `ChaosMonkey`, modeled after
link:https://netflix.github.io/chaosmonkey/[same-named tool by Netflix's Chaos Monkey tool].
ChaosMonkey simulates real-world
faults in a running cluster by killing or disconnecting random servers, or injecting
other failures into the environment. You can use ChaosMonkey as a stand-alone tool
to run a policy while other tests are running. In some environments, ChaosMonkey is
always running, in order to constantly check that high availability and fault tolerance
are working as expected.

ChaosMonkey defines *Actions* and *Policies*.

Actions:: Actions are predefined sequences of events, such as the following:

* Restart active master (sleep 5 sec)
* Restart random regionserver (sleep 5 sec)
* Restart random regionserver (sleep 60 sec)
* Restart META regionserver (sleep 5 sec)
* Restart ROOT regionserver (sleep 5 sec)
* Batch restart of 50% of regionservers (sleep 5 sec)
* Rolling restart of 100% of regionservers (sleep 5 sec)

Policies:: A policy is a strategy for executing one or more actions. The default policy
executes a random action every minute based on predefined action weights.
A given policy will be executed until ChaosMonkey is interrupted.

Most ChaosMonkey actions are configured to have reasonable defaults, so you can run
ChaosMonkey against an existing cluster without any additional configuration. The
following example runs ChaosMonkey with the default configuration:

[source,bash]
----
$ bin/hbase org.apache.hadoop.hbase.util.ChaosMonkey

12/11/19 23:21:57 INFO util.ChaosMonkey: Using ChaosMonkey Policy: class org.apache.hadoop.hbase.util.ChaosMonkey$PeriodicRandomActionPolicy, period:60000
12/11/19 23:21:57 INFO util.ChaosMonkey: Sleeping for 26953 to add jitter
12/11/19 23:22:24 INFO util.ChaosMonkey: Performing action: Restart active master
12/11/19 23:22:24 INFO util.ChaosMonkey: Killing master:master.example.com,60000,1353367210440
12/11/19 23:22:24 INFO hbase.HBaseCluster: Aborting Master: master.example.com,60000,1353367210440
12/11/19 23:22:24 INFO hbase.ClusterManager: Executing remote command: ps aux | grep master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill -s SIGKILL , hostname:master.example.com
12/11/19 23:22:25 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:
12/11/19 23:22:25 INFO hbase.HBaseCluster: Waiting service:master to stop: master.example.com,60000,1353367210440
12/11/19 23:22:25 INFO hbase.ClusterManager: Executing remote command: ps aux | grep master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 , hostname:master.example.com
12/11/19 23:22:25 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:
12/11/19 23:22:25 INFO util.ChaosMonkey: Killed master server:master.example.com,60000,1353367210440
12/11/19 23:22:25 INFO util.ChaosMonkey: Sleeping for:5000
12/11/19 23:22:30 INFO util.ChaosMonkey: Starting master:master.example.com
12/11/19 23:22:30 INFO hbase.HBaseCluster: Starting Master on: master.example.com
12/11/19 23:22:30 INFO hbase.ClusterManager: Executing remote command: /homes/enis/code/hbase-0.94/bin/../bin/hbase-daemon.sh --config /homes/enis/code/hbase-0.94/bin/../conf start master , hostname:master.example.com
12/11/19 23:22:31 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:starting master, logging to /homes/enis/code/hbase-0.94/bin/../logs/hbase-enis-master-master.example.com.out
....
12/11/19 23:22:33 INFO util.ChaosMonkey: Started master: master.example.com,60000,1353367210440
12/11/19 23:22:33 INFO util.ChaosMonkey: Sleeping for:51321
12/11/19 23:23:24 INFO util.ChaosMonkey: Performing action: Restart random region server
12/11/19 23:23:24 INFO util.ChaosMonkey: Killing region server:rs3.example.com,60020,1353367027826
12/11/19 23:23:24 INFO hbase.HBaseCluster: Aborting RS: rs3.example.com,60020,1353367027826
12/11/19 23:23:24 INFO hbase.ClusterManager: Executing remote command: ps aux | grep regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill -s SIGKILL , hostname:rs3.example.com
12/11/19 23:23:25 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:
12/11/19 23:23:25 INFO hbase.HBaseCluster: Waiting service:regionserver to stop: rs3.example.com,60020,1353367027826
12/11/19 23:23:25 INFO hbase.ClusterManager: Executing remote command: ps aux | grep regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2 , hostname:rs3.example.com
12/11/19 23:23:25 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:
12/11/19 23:23:25 INFO util.ChaosMonkey: Killed region server:rs3.example.com,60020,1353367027826. Reported num of rs:6
12/11/19 23:23:25 INFO util.ChaosMonkey: Sleeping for:60000
12/11/19 23:24:25 INFO util.ChaosMonkey: Starting region server:rs3.example.com
12/11/19 23:24:25 INFO hbase.HBaseCluster: Starting RS on: rs3.example.com
12/11/19 23:24:25 INFO hbase.ClusterManager: Executing remote command: /homes/enis/code/hbase-0.94/bin/../bin/hbase-daemon.sh --config /homes/enis/code/hbase-0.94/bin/../conf start regionserver , hostname:rs3.example.com
12/11/19 23:24:26 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:starting regionserver, logging to /homes/enis/code/hbase-0.94/bin/../logs/hbase-enis-regionserver-rs3.example.com.out

12/11/19 23:24:27 INFO util.ChaosMonkey: Started region server:rs3.example.com,60020,1353367027826. Reported num of rs:6
----

The output indicates that ChaosMonkey started the default `PeriodicRandomActionPolicy`
policy, which is configured with all the available actions. It chose to run `RestartActiveMaster` and `RestartRandomRs` actions.

==== Available Policies
HBase ships with several ChaosMonkey policies, available in the
`hbase/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/policies/` directory.

[[chaos.monkey.properties]]
==== Configuring Individual ChaosMonkey Actions

ChaosMonkey integration tests can be configured per test run.
Create a Java properties file in the HBase CLASSPATH and pass it to ChaosMonkey using
the `-monkeyProps` configuration flag. Configurable properties, along with their default
values if applicable, are listed in the `org.apache.hadoop.hbase.chaos.factories.MonkeyConstants`
class. For properties that have defaults, you can override them by including them
in your properties file.

The following example uses a properties file called <<monkey.properties,monkey.properties>>.

[source,bourne]
----
$ bin/hbase org.apache.hadoop.hbase.IntegrationTestIngest -m slowDeterministic -monkeyProps monkey.properties
----

The above command will start the integration tests and chaos monkey. It will look for the
properties file _monkey.properties_ on the HBase CLASSPATH; e.g. inside the HBASE _conf_ dir.

Here is an example chaos monkey file:

[[monkey.properties]]
.Example ChaosMonkey Properties File
[source]
----
sdm.action1.period=120000
sdm.action2.period=40000
move.regions.sleep.time=80000
move.regions.max.time=1000000
move.regions.sleep.time=80000
batch.restart.rs.ratio=0.4f
----

Periods/time are expressed in milliseconds.

HBase 1.0.2 and newer adds the ability to restart HBase's underlying ZooKeeper quorum or
HDFS nodes. To use these actions, you need to configure some new properties, which
have no reasonable defaults because they are deployment-specific, in your ChaosMonkey
properties file, which may be `hbase-site.xml` or a different properties file.

[source,xml]
----
<property>
  <name>hbase.it.clustermanager.hadoop.home</name>
  <value>$HADOOP_HOME</value>
</property>
<property>
  <name>hbase.it.clustermanager.zookeeper.home</name>
  <value>$ZOOKEEPER_HOME</value>
</property>
<property>
  <name>hbase.it.clustermanager.hbase.user</name>
  <value>hbase</value>
</property>
<property>
  <name>hbase.it.clustermanager.hadoop.hdfs.user</name>
  <value>hdfs</value>
</property>
<property>
  <name>hbase.it.clustermanager.zookeeper.user</name>
  <value>zookeeper</value>
</property>
----

[[developing]]
== Developer Guidelines

=== Branches

We use Git for source code management and latest development happens on `master` branch. There are
branches for past major/minor/maintenance releases and important features and bug fixes are often
 back-ported to them.

=== Policy for Fix Version in JIRA

To determine if a given fix is in a given release purely from the release numbers following rules
are defined:

Fix version of X.Y.Z => fixed in all releases X.Y.Z' (where Z' = Z). +
Fix version of X.Y.0 => fixed in all releases X.Y'.* (where Y' = Y). +
Fix version of X.0.0 => fixed in all releases X'.\*.* (where X' = X). +

By this policy, fix version of 1.3.0 implies 1.4.0, but 1.3.2 does not imply 1.4.0 as we could not
tell purely from the numbers which release came first.

[[code.standards]]
=== Code Standards


==== Interface Classifications

Interfaces are classified both by audience and by stability level.
These labels appear at the head of a class.
The conventions followed by HBase are inherited by its parent project, Hadoop.

The following interface classifications are commonly used:

.InterfaceAudience
`@InterfaceAudience.Public`::
  APIs for users and HBase applications.
  These APIs will be deprecated through major versions of HBase.

`@InterfaceAudience.Private`::
  APIs for HBase internals developers.
  No guarantees on compatibility or availability in future versions.
  Private interfaces do not need an `@InterfaceStability` classification.

`@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)`::
  APIs for HBase coprocessor writers.

No `@InterfaceAudience` Classification::
  Packages without an `@InterfaceAudience` label are considered private.
  Mark your new packages if publicly accessible.

.Excluding Non-Public Interfaces from API Documentation
[NOTE]
====
Only interfaces classified `@InterfaceAudience.Public` should be included in API documentation (Javadoc). Committers must add new package excludes `ExcludePackageNames` section of the _pom.xml_ for new packages which do not contain public classes.
====

.@InterfaceStability
`@InterfaceStability` is important for packages marked `@InterfaceAudience.Public`.

`@InterfaceStability.Stable`::
  Public packages marked as stable cannot be changed without a deprecation path or a very good reason.

`@InterfaceStability.Unstable`::
  Public packages marked as unstable can be changed without a deprecation path.

`@InterfaceStability.Evolving`::
  Public packages marked as evolving may be changed, but it is discouraged.

No `@InterfaceStability` Label::
  Public classes with no `@InterfaceStability` label are discouraged, and should be considered implicitly unstable.

If you are unclear about how to mark packages, ask on the development list.

[[common.patch.feedback]]
==== Code Formatting Conventions

Please adhere to the following guidelines so that your patches can be reviewed more quickly.
These guidelines have been developed based upon common feedback on patches from new contributors.

See the link:http://www.oracle.com/technetwork/java/index-135089.html[Code
                    Conventions for the Java Programming Language] for more information on coding conventions in Java.
See <<eclipse.code.formatting,eclipse.code.formatting>> to setup Eclipse to check for some of
these guidelines automatically.

[[common.patch.feedback.space.invaders]]
===== Space Invaders

Do not use extra spaces around brackets.
Use the second style, rather than the first.

[source,java]
----

if ( foo.equals( bar ) ) {     // don't do this
----

[source,java]
----

if (foo.equals(bar)) {
----

[source,java]
----

foo = barArray[ i ];     // don't do this
----

[source,java]
----

foo = barArray[i];
----

[[common.patch.feedback.autogen]]
===== Auto Generated Code

Auto-generated code in Eclipse often uses bad variable names such as `arg0`.
Use more informative variable names.
Use code like the second example here.

[source,java]
----

 public void readFields(DataInput arg0) throws IOException {    // don't do this
   foo = arg0.readUTF();                                       // don't do this
----

[source,java]
----

 public void readFields(DataInput di) throws IOException {
   foo = di.readUTF();
----

[[common.patch.feedback.longlines]]
===== Long Lines

Keep lines less than 100 characters.
You can configure your IDE to do this automatically.

[source,java]
----

Bar bar = foo.veryLongMethodWithManyArguments(argument1, argument2, argument3, argument4, argument5, argument6, argument7, argument8, argument9);  // don't do this
----

[source,java]
----

Bar bar = foo.veryLongMethodWithManyArguments(
 argument1, argument2, argument3,argument4, argument5, argument6, argument7, argument8, argument9);
----

[[common.patch.feedback.trailingspaces]]
===== Trailing Spaces

Be sure there is a line break after the end of your code, and avoid lines with nothing but whitespace.
This makes diffs more meaningful.
You can configure your IDE to help with this.

[source,java]
----

Bar bar = foo.getBar();     <--- imagine there is an extra space(s) after the semicolon.
----

[[common.patch.feedback.javadoc]]
===== API Documentation (Javadoc)

Don't forget Javadoc!

Javadoc warnings are checked during precommit.
If the precommit tool gives you a '-1', please fix the javadoc issue.
Your patch won't be committed if it adds such warnings.

Also, no `@author` tags - that's a rule.

[[common.patch.feedback.findbugs]]
===== Findbugs

`Findbugs` is used to detect common bugs pattern.
It is checked during the precommit build.
If errors are found, please fix them.
You can run findbugs locally with `mvn
                            findbugs:findbugs`, which will generate the `findbugs` files locally.
Sometimes, you may have to write code smarter than `findbugs`.
You can annotate your code to tell `findbugs` you know what you're doing, by annotating your class with the following annotation:

[source,java]
----
@edu.umd.cs.findbugs.annotations.SuppressWarnings(
value="HE_EQUALS_USE_HASHCODE",
justification="I know what I'm doing")
----

It is important to use the Apache-licensed version of the annotations. That generally means using
annotations in the `edu.umd.cs.findbugs.annotations` package so that we can rely on the cleanroom
reimplementation rather than annotations in the `javax.annotations` package.

[[common.patch.feedback.javadoc.defaults]]
===== Javadoc - Useless Defaults

Don't just leave javadoc tags the way IDE generates them, or fill redundant information in them.

[source,java]
----

  /**
   * @param table                              <---- don't leave them empty!
   * @param region An HRegion object.          <---- don't fill redundant information!
   * @return Foo Object foo just created.      <---- Not useful information
   * @throws SomeException                     <---- Not useful. Function declarations already tell that!
   * @throws BarException when something went wrong  <---- really?
   */
  public Foo createFoo(Bar bar);
----

Either add something descriptive to the tags, or just remove them.
The preference is to add something descriptive and useful.

[[common.patch.feedback.onething]]
===== One Thing At A Time, Folks

If you submit a patch for one thing, don't do auto-reformatting or unrelated reformatting of code on a completely different area of code.

Likewise, don't add unrelated cleanup or refactorings outside the scope of your Jira.

[[common.patch.feedback.tests]]
===== Ambiguous Unit Tests

Make sure that you're clear about what you are testing in your unit tests and why.

==== Garbage-Collection Conserving Guidelines

The following guidelines were borrowed from http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage.
Keep them in mind to keep preventable garbage  collection to a minimum. Have a look
at the blog post for some great examples of how to refactor your code according to
these guidelines.

- Be careful with Iterators
- Estimate the size of a collection when initializing
- Defer expression evaluation
- Compile the regex patterns in advance
- Cache it if you can
- String Interns are useful but dangerous

[[design.invariants]]
=== Invariants

We don't have many but what we have we list below.
All are subject to challenge of course but until then, please hold to the rules of the road.

[[design.invariants.zk.data]]
==== No permanent state in ZooKeeper

ZooKeeper state should transient (treat it like memory). If ZooKeeper state is deleted, hbase should be able to recover and essentially be in the same state.

* .Exceptions: There are currently a few exceptions that we need to fix around whether a table is enabled or disabled.
* Replication data is currently stored only in ZooKeeper.
  Deleting ZooKeeper data related to replication may cause replication to be disabled.
  Do not delete the replication tree, _/hbase/replication/_.
+
WARNING: Replication may be disrupted and data loss may occur if you delete the replication tree (_/hbase/replication/_) from ZooKeeper.
Follow progress on this issue at link:https://issues.apache.org/jira/browse/HBASE-10295[HBASE-10295].


[[run.insitu]]
=== Running In-Situ

If you are developing Apache HBase, frequently it is useful to test your changes against a more-real cluster than what you find in unit tests.
In this case, HBase can be run directly from the source in local-mode.
All you need to do is run:

[source,bourne]
----
${HBASE_HOME}/bin/start-hbase.sh
----

This will spin up a full local-cluster, just as if you had packaged up HBase and installed it on your machine.

Keep in mind that you will need to have installed HBase into your local maven repository for the in-situ cluster to work properly.
That is, you will need to run:

[source,bourne]
----
mvn clean install -DskipTests
----

to ensure that maven can find the correct classpath and dependencies.
Generally, the above command is just a good thing to try running first, if maven is acting oddly.

[[add.metrics]]
=== Adding Metrics

After adding a new feature a developer might want to add metrics.
HBase exposes metrics using the Hadoop Metrics 2 system, so adding a new metric involves exposing that metric to the hadoop system.
Unfortunately the API of metrics2 changed from hadoop 1 to hadoop 2.
In order to get around this a set of interfaces and implementations have to be loaded at runtime.
To get an in-depth look at the reasoning and structure of these classes you can read the blog post located link:https://blogs.apache.org/hbase/entry/migration_to_the_new_metrics[here].
To add a metric to an existing MBean follow the short guide below:

==== Add Metric name and Function to Hadoop Compat Interface.

Inside of the source interface the corresponds to where the metrics are generated (eg MetricsMasterSource for things coming from HMaster) create new static strings for metric name and description.
Then add a new method that will be called to add new reading.

==== Add the Implementation to Both Hadoop 1 and Hadoop 2 Compat modules.

Inside of the implementation of the source (eg.
MetricsMasterSourceImpl in the above example) create a new histogram, counter, gauge, or stat in the init method.
Then in the method that was added to the interface wire up the parameter passed in to the histogram.

Now add tests that make sure the data is correctly exported to the metrics 2 system.
For this the MetricsAssertHelper is provided.

[[git.best.practices]]
=== Git Best Practices

Avoid git merges.::
  Use `git pull --rebase` or `git fetch` followed by `git rebase`.
Do not use `git push --force`.::
  If the push does not work, fix the problem or ask for help.

Please contribute to this document if you think of other Git best practices.

==== `rebase_all_git_branches.sh`

The _dev-support/rebase_all_git_branches.sh_ script is provided to help keep your Git repository clean.
Use the `-h`                    parameter to get usage instructions.
The script automatically refreshes your tracking branches, attempts an automatic rebase of each local branch against its remote branch, and gives you the option to delete any branch which represents a closed `HBASE-` JIRA.
The script has one optional configuration option, the location of your Git directory.
You can set a default by editing the script.
Otherwise, you can pass the git directory manually by using the `-d` parameter, followed by an absolute or relative directory name, or even '.' for the current working directory.
The script checks the directory for sub-directory called _.git/_, before proceeding.

[[submitting.patches]]
=== Submitting Patches

If you are new to submitting patches to open source or new to submitting patches to Apache, start by
 reading the link:https://commons.apache.org/patches.html[On Contributing Patches] page from
 link:https://commons.apache.org/[Apache Commons Project].
It provides a nice overview that applies equally to the Apache HBase Project.

[[submitting.patches.create]]
==== Create Patch

Make sure you review <<common.patch.feedback,common.patch.feedback>> for code style. If your
patch
was generated incorrectly or your code does not adhere to the code formatting guidelines, you may
be asked to redo some work.


.Using submit-patch.py (recommended)

[source,bourne]
----
$ dev-support/submit-patch.py -jid HBASE-xxxxx
----

Use this script to create patches, upload to jira and optionally create/update reviews on
Review Board. Patch name is automatically formatted as _(JIRA).(branch name).(patch number).patch_
 to follow Yetus' naming rules. Use `-h` flag to know detailed usage information. Most useful options
are:

* `-b BRANCH, --branch BRANCH` : Specify base branch for generating the diff. If not specified,
tracking branch is used. If there is no tracking branch, error will be thrown.
* `-jid JIRA_ID, --jira-id JIRA_ID` : If used, deduces next patch version from attachments in the
jira and uploads the new patch. Script will ask for jira username/password for authentication.
If not set, patch is named <branch>.patch.

By default, it'll also create/update review board. To skip that action, use `-srb` option. It uses
'Issue Links' in the jira to figure out if a review request already exists. If no review
request is present, then creates a new one and populates all required fields using jira summary,
patch description, etc. Also adds this review's link to the jira.

Save authentication credentials (optional)::
Since attaching patches on JIRA and creating/changing review request on ReviewBoard requires
valid user authentication, the script will prompt you for username and password. To avoid the hassle every
time, set up `~/.apache-creds` with login details and encrypt it by following the steps in footer
of script's help message.

Python dependencies:: To install required python dependencies, execute
`pip install -r dev-support/python-requirements.txt` from the master branch.

.Manually

  . Use `git rebase -i` first, to combine (squash) smaller commits into a single larger one.
  . Create patch using IDE or Git commands. `git format-patch` is preferred since it preserves patch
    author's name and commit message. Also, it handles binary files by default, whereas `git diff`
    ignores them unless you use the `--binary` option.
  . Patch name should be as follows to adhere to Yetus' naming convention: +
    `(JIRA).(branch name).(patch number).patch` +
    For eg. HBASE-11625.master.001.patch, HBASE-XXXXX.branch-1.2.0005.patch, etc.
  . Attach the patch to the JIRA using `More->Attach Files` then click on btn:[Submit Patch]
    button, which'll trigger Hudson job to check patch for validity.
  . If your patch is longer than a single screen, also create a review on Review Board  and
    add the link to JIRA. See <<reviewboard,reviewboard>>.


.Few general guidelines
* Always patch against the master branch first, even if you want to patch in another branch.
  HBase committers always apply patches first to the master branch, and backport if necessary.
* Submit one single patch for a fix. If necessary, squash local commits to merge local commits into
  a single one first. See this
  link:http://stackoverflow.com/questions/5308816/how-to-use-git-merge-squash[Stack Overflow
  question] for more information about squashing commits.
* Please understand that not every patch may get committed, and that feedback will likely be
  provided on the patch.
* If you need to revise your patch, leave the previous patch file(s) attached to the JIRA, and
  upload a new one with incremented patch number. +
  Click on btn:[Cancel Patch] and then on btn:[Submit Patch] to trigger the presubmit run.

[[submitting.patches.tests]]
==== Unit Tests
Always add and/or update relevant unit tests when making the changes.
Make sure that new/changed unit tests pass locally before submitting the patch because it is faster
than waiting for presubmit result which runs full test suite. This will save your own time and
effort.
Use <<mockito,mockito>> to make mocks which are very useful for testing failure scenarios by
injecting appropriate failures.

If you are creating a new unit test class, notice how other unit test classes have
classification/sizing annotations before class name and a static methods for setup/teardown of
testing environment. Be sure to include annotations in any new unit test files.
See <<hbase.tests,hbase.tests>> for more information on tests.

==== Integration Tests

Significant new features should provide an integration test in addition to unit tests, suitable for exercising the new feature at different points in its configuration space.

[[reviewboard]]
==== ReviewBoard

Patches larger than one screen, or patches that will be tricky to review, should go through link:https://reviews.apache.org[ReviewBoard].

.Procedure: Use ReviewBoard
. Register for an account if you don't already have one.
  It does not use the credentials from link:https://issues.apache.org[issues.apache.org].
  Log in.
. Click [label]#New Review Request#.
. Choose the `hbase-git` repository.
  Click Choose File to select the diff and optionally a parent diff.
  Click btn:[Create
  Review Request].
. Fill in the fields as required.
  At the minimum, fill in the [label]#Summary# and choose `hbase` as the [label]#Review Group#.
  If you fill in the [label]#Bugs# field, the review board links back to the relevant JIRA.
  The more fields you fill in, the better.
  Click btn:[Publish] to make your review request public.
  An email will be sent to everyone in the `hbase` group, to review the patch.
. Back in your JIRA, click , and paste in the URL of your ReviewBoard request.
  This attaches the ReviewBoard to the JIRA, for easy access.
. To cancel the request, click .

For more information on how to use ReviewBoard, see link:http://www.reviewboard.org/docs/manual/1.5/[the ReviewBoard
                        documentation].

[[github]]
==== GitHub
Submitting link:https://github.com/apache/hbase[GitHub] pull requests is another accepted form of
contributing patches. Refer to GitHub link:https://help.github.com/[documentation] for details on
how to create pull requests.

NOTE: This section is incomplete and needs to be updated. Refer to
link:https://issues.apache.org/jira/browse/HBASE-23557[HBASE-23557]

===== GitHub Tooling

====== Browser bookmarks

Following is a useful javascript based browser bookmark that redirects from GitHub pull
requests to the corresponding jira work item. This redirects based on the HBase jira  ID mentioned
in the issue title for the PR. Add the following javascript snippet as a browser bookmark to the
tool bar. Clicking on it while you are on an HBase GitHub PR page redirects you to the corresponding
jira item.

[source, javascript]
-----
javascript:location.href='https://issues.apache.org/jira/browse/'+document.getElementsByClassName("js-issue-title")[0].innerHTML.match(/HBASE-\d+/)[0];
-----

==== Guide for HBase Committers

===== Becoming a committer

Committers are responsible for reviewing and integrating code changes, testing
and voting on release candidates, weighing in on design discussions, as well as
other types of project contributions. The PMC votes to make a contributor a
committer based on an assessment of their contributions to the project. It is
expected that committers demonstrate a sustained history of high-quality
contributions to the project and community involvement.

Contributions can be made in many ways. There is no single path to becoming a
committer, nor any expected timeline. Submitting features, improvements, and bug
fixes is the most common avenue, but other methods are both recognized and
encouraged (and may be even more important to the health of HBase as a project and a
community). A non-exhaustive list of potential contributions (in no particular
order):

* <<appendix_contributing_to_documentation,Update the documentation>> for new
  changes, best practices, recipes, and other improvements.
* Keep the website up to date.
* Perform testing and report the results. For instance, scale testing and
  testing non-standard configurations is always appreciated.
* Maintain the shared Jenkins testing environment and other testing
  infrastructure.
* <<hbase.rc.voting,Vote on release candidates>> after performing validation, even if non-binding.
  A non-binding vote is a vote by a non-committer.
* Provide input for discussion threads on the link:/mail-lists.html[mailing lists] (which usually have
  `[DISCUSS]` in the subject line).
* Answer questions questions on the user or developer mailing lists and on
  Slack.
* Make sure the HBase community is a welcoming one and that we adhere to our
  link:/coc.html[Code of conduct]. Alert the PMC if you
  have concerns.
* Review other people's work (both code and non-code) and provide public
  feedback.
* Report bugs that are found, or file new feature requests.
* Triage issues and keep JIRA organized. This includes closing stale issues,
  labeling new issues, updating metadata, and other tasks as needed.
* Mentor new contributors of all sorts.
* Give talks and write blogs about HBase. Add these to the link:/[News] section
  of the website.
* Provide UX feedback about HBase, the web UI, the CLI, APIs, and the website.
* Write demo applications and scripts.
* Help attract and retain a diverse community.
* Interact with other projects in ways that benefit HBase and those other
  projects.

Not every individual is able to do all (or even any) of the items on this list.
If you think of other ways to contribute, go for it (and add them to the list).
A pleasant demeanor and willingness to contribute are all you need to make a
positive impact on the HBase project. Invitations to become a committer are the
result of steady interaction with the community over the long term, which builds
trust and recognition.

===== New committers

New committers are encouraged to first read Apache's generic committer
documentation:

* link:https://www.apache.org/dev/new-committers-guide.html[Apache New Committer Guide]
* link:https://www.apache.org/dev/committers.html[Apache Committer FAQ]

===== Review

HBase committers should, as often as possible, attempt to review patches
submitted by others. Ideally every submitted patch will get reviewed by a
committer _within a few days_. If a committer reviews a patch they have not
authored, and believe it to be of sufficient quality, then they can commit the
patch. Otherwise the patch should be cancelled with a clear explanation for why
it was rejected.

The list of submitted patches is in the
link:https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=hide&requestId=12312392[HBase Review Queue],
which is ordered by time of last modification. Committers should scan the list
from top to bottom, looking for patches that they feel qualified to review and
possibly commit. If you see a patch you think someone else is better qualified
to review, you can mention them by username in the JIRA.

For non-trivial changes, it is required that another committer review your
patches before commit. **Self-commits of non-trivial patches are not allowed.**
Use the btn:[Submit Patch] button in JIRA, just like other contributors, and
then wait for a `+1` response from another committer before committing.

===== Reject

Patches which do not adhere to the guidelines in
link:https://hbase.apache.org/book.html#developer[HowToContribute] and to the
link:https://cwiki.apache.org/confluence/display/HADOOP2/CodeReviewChecklist[code review checklist]
should be rejected. Committers should always be polite to contributors and try
to instruct and encourage them to contribute better patches. If a committer
wishes to improve an unacceptable patch, then it should first be rejected, and a
new patch should be attached by the committer for further review.

[[committing.patches]]
===== Commit

Committers commit patches to the Apache HBase GIT repository.

.Before you commit!!!!
[NOTE]
====
Make sure your local configuration is correct, especially your identity and email.
Examine the output of the +$ git config --list+ command and be sure it is correct.
See link:https://help.github.com/articles/set-up-git[Set Up Git] if you need
pointers.
====

When you commit a patch:

. Include the Jira issue ID in the commit message along with a short description
  of the change. Try to add something more than just the Jira title so that
  someone looking at `git log` output doesn't have to go to Jira to discern what
  the change is about. Be sure to get the issue ID right, because this causes
  Jira to link to the change in Git (use the issue's "All" tab to see these
  automatic links).
. Commit the patch to a new branch based off `master` or the other intended
  branch. It's a good idea to include the JIRA ID in the name of this branch.
  Check out the relevant target branch where you want to commit, and make sure
  your local branch has all remote changes, by doing a +git pull --rebase+ or
  another similar command. Next, cherry-pick the change into each relevant
  branch (such as master), and push the changes to the remote branch using
  a command such as +git push <remote-server> <remote-branch>+.
+
WARNING: If you do not have all remote changes, the push will fail.
If the push fails for any reason, fix the problem or ask for help.
Do not do a +git push --force+.
+
Before you can commit a patch, you need to determine how the patch was created.
The instructions and preferences around the way to create patches have changed,
and there will be a transition period.
+
.Determine How a Patch Was Created
* If the first few lines of the patch look like the headers of an email, with a From, Date, and
  Subject, it was created using +git format-patch+. This is the preferred way, because you can
  reuse the submitter's commit message. If the commit message is not appropriate, you can still use
  the commit, then run `git commit --amend` and reword as appropriate.
* If the first line of the patch looks similar to the following, it was created using +git diff+                                        without `--no-prefix`.
  This is acceptable too.
  Notice the `a` and `b` in front of the file names.
  This is the indication that the patch was not created with `--no-prefix`.
+
----
diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc
----

* If the first line of the patch looks similar to the following (without the `a` and `b`), the
patch was created with +git diff --no-prefix+ and you need to add `-p0` to the +git apply+ command
below.
+
----
diff --git src/main/asciidoc/_chapters/developer.adoc src/main/asciidoc/_chapters/developer.adoc
----

+
.Example of committing a Patch
====
One thing you will notice with these examples is that there are a lot of
+git pull+ commands. The only command that actually writes anything to the
remote repository is +git push+, and you need to make absolutely sure you have
the correct versions of everything and don't have any conflicts before pushing.
The extra +git pull+ commands are usually redundant, but better safe than sorry.

The first example shows how to apply a patch that was generated with +git
format-patch+ and apply it to the `master` and `branch-1` branches.

The directive to use +git format-patch+ rather than +git diff+, and not to use
`--no-prefix`, is a new one. See the second example for how to apply a patch
created with +git diff+, and educate the person who created the patch.

----
$ git checkout -b HBASE-XXXX
$ git am ~/Downloads/HBASE-XXXX-v2.patch --signoff  # If you are committing someone else's patch.
$ git checkout master
$ git pull --rebase
$ git cherry-pick <sha-from-commit>
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase          # Better safe than sorry
$ git push origin master

# Backport to branch-1
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick <sha-from-commit>
# Resolve conflicts if necessary
$ git pull --rebase          # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-XXXX
----

This example shows how to commit a patch that was created using +git diff+
without `--no-prefix`. If the patch was created with `--no-prefix`, add `-p0` to
the +git apply+ command.

----
$ git apply ~/Downloads/HBASE-XXXX-v2.patch
$ git commit -m "HBASE-XXXX Really Good Code Fix (Joe Schmo)" --author=<contributor> -a  # This and next command is needed for patches created with 'git diff'
$ git commit --amend --signoff
$ git checkout master
$ git pull --rebase
$ git cherry-pick <sha-from-commit>
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase          # Better safe than sorry
$ git push origin master

# Backport to branch-1
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick <sha-from-commit>
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase           # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-XXXX
----
====

. Resolve the issue as fixed, thanking the contributor.
  Always set the "Fix Version" at this point, but only set a single fix version
  for each branch where the change was committed, the earliest release in that
  branch in which the change will appear.

====== Commit Message Format

The commit message should contain the JIRA ID and a description of what the patch does.
The preferred commit message format is:

----
<jira-id> <jira-title> (<contributor-name-if-not-commit-author>)
----

----
HBASE-12345 Fix All The Things (jane@example.com)
----

If the contributor used +git format-patch+ to generate the patch, their commit
message is in their patch and you can use that, but be sure the JIRA ID is at
the front of the commit message, even if the contributor left it out.

[[committer.amending.author]]
====== Use GitHub's "Co-authored-by" when there are multiple authors

We've established the practice of committing to master and then cherry picking back to branches whenever possible, unless

* it's breaking compat: In which case, if it can go in minor releases, backport to branch-1 and branch-2.
* it's a new feature: No for maintenance releases, For minor releases, discuss and arrive at consensus.

There are occasions when there are multiple author for a patch.
For example when there is a minor conflict we can fix it up and just proceed with the commit.
The amending author will be different from the original committer, so you should also attribute to the original author by
adding one or more `Co-authored-by` trailers to the commit's message.
See link:https://help.github.com/en/articles/creating-a-commit-with-multiple-authors/[the GitHub documentation for "Creating a commit with multiple authors"].

In short, these are the steps to add Co-authors that will be tracked by GitHub:

. Collect the name and email address for each co-author.
. Commit the change, but after your commit description, instead of a closing quotation, add two empty lines. (Do not close the commit message with a quotation mark)
. On the next line of the commit message, type `Co-authored-by: name <name@example.com>`. After the co-author information, add a closing quotation mark.

Here is the example from the GitHub page, using 2 Co-authors:
[source,xml]
----
$ git commit -m "Refactor usability tests.
>
>
Co-authored-by: name <name@example.com>
Co-authored-by: another-name <another-name@example.com>"
----

Note: `Amending-Author: Author <committer@apache>` was used prior to this
link:https://lists.apache.org/thread.html/f00b5f9b65570e777dbb31c37d7b0ffc55c5fc567aefdb456608a042@%3Cdev.hbase.apache.org%3E[DISCUSSION].

====== Close related GitHub PRs

As a project we work to ensure there's a JIRA associated with each change, but we don't mandate any particular tool be used for reviews. Due to implementation details of the ASF's integration between hosted git repositories and GitHub, the PMC has no ability to directly close PRs on our GitHub repo. In the event that a contributor makes a Pull Request on GitHub, either because the contributor finds that easier than attaching a patch to JIRA or because a reviewer prefers that UI for examining changes, it's important to make note of the PR in the commit that goes to the master branch so that PRs are kept up to date.

To read more about the details of what kinds of commit messages will work with the GitHub "close via keyword in commit" mechanism see link:https://help.github.com/articles/closing-issues-using-keywords/[the GitHub documentation for "Closing issues using keywords"]. In summary, you should include a line with the phrase "closes #XXX", where the XXX is the pull request id. The pull request id is usually given in the GitHub UI in grey at the end of the subject heading.

[[committer.tests]]
====== Committers are responsible for making sure commits do not break the build or tests

If a committer commits a patch, it is their responsibility to make sure it passes the test suite.
It is helpful if contributors keep an eye out that their patch does not break the hbase build and/or tests, but ultimately, a contributor cannot be expected to be aware of all the particular vagaries and interconnections that occur in a project like HBase.
A committer should.

[[git.patch.flow]]
====== Patching Etiquette

In the thread link:https://lists.apache.org/thread.html/186fcd5eb71973a7b282ecdba41606d3d221efd505d533bb729e1fad%401400648690%40%3Cdev.hbase.apache.org%3E[HBase, mail # dev - ANNOUNCEMENT: Git Migration In Progress (WAS =>
                                Re: Git Migration)], it was agreed on the following patch flow

. Develop and commit the patch against master first.
. Try to cherry-pick the patch when backporting if possible.
. If this does not work, manually commit the patch to the branch.

====== Merge Commits

Avoid merge commits, as they create problems in the git history.

====== Committing Documentation

See <<appendix_contributing_to_documentation,appendix contributing to documentation>>.

====== How to re-trigger github Pull Request checks/re-build

A Pull Request (PR) submission triggers the hbase yetus checks. The checks make
sure the patch doesn't break the build or introduce test failures. The checks take
around four hours to run (They are the same set run when you submit a patch via
HBASE JIRA). When finished, they add a report to the PR as a comment. If a problem
w/ the patch -- failed compile, checkstyle violation, or an added findbugs --
the original author makes fixes and pushes a new patch. This re-runs the checks
to produce a new report.

Sometimes though, the patch is good but a flakey, unrelated test has the report vote -1
on the patch. In this case, **committers** can retrigger the check run by doing a force push of the
exact same patch. Or, click on the `Console output` link which shows toward the end
of the report (For example `https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-289/1/console`).
This will take you to `builds.apache.org`, to the build run that failed. See the
"breadcrumbs" along the top (where breadcrumbs is the listing of the directories that
gets us to this particular build page). It'll look something like
`Jenkins > HBase-PreCommit-GitHub-PR > PR-289 > #1`. Click on the
PR number -- i.e. PR-289 in our example -- and then, when you've arrived at the PR page,
find the 'Build with Parameters' menu-item (along top left-hand menu). Click here and
then `Build` leaving the JIRA_ISSUE_KEY empty. This will re-run your checks.

==== Dialog

Committers should hang out in the #hbase room on irc.freenode.net for real-time discussions.
However any substantive discussion (as with any off-list project-related discussion) should be re-iterated in Jira or on the developer list.

==== Do not edit JIRA comments

Misspellings and/or bad grammar is preferable to the disruption a JIRA comment edit.

[[thirdparty]]
=== The hbase-thirdparty dependency and shading/relocation

A new project was created for the release of hbase-2.0.0. It was called
`hbase-thirdparty`. This project exists only to provide the main hbase
project with relocated -- or shaded -- versions of popular thirdparty
libraries such as guava, netty, and protobuf. The mainline HBase project
relies on the relocated versions of these libraries gotten from hbase-thirdparty
rather than on finding these classes in their usual locations. We do this so
we can specify whatever the version we wish. If we don't relocate, we must
harmonize our version to match that which hadoop, spark, and other projects use.

For developers, this means you need to be careful referring to classes from
netty, guava, protobuf, gson, etc. (see the hbase-thirdparty pom.xml for what
it provides). Devs must refer to the hbase-thirdparty provided classes. In
practice, this is usually not an issue (though it can be a bit of a pain). You
will have to hunt for the relocated version of your particular class. You'll
find it by prepending the general relocation prefix of `org.apache.hbase.thirdparty.`.
For example if you are looking for `com.google.protobuf.Message`, the relocated
version used by HBase internals can be found at
`org.apache.hbase.thirdparty.com.google.protobuf.Message`.

For a few thirdparty libs, like protobuf (see the protobuf chapter in this book
for the why), your IDE may give you both options -- the `com.google.protobuf.*`
and the `org.apache.hbase.thirdparty.com.google.protobuf.*` -- because both
classes are on your CLASSPATH. Unless you are doing the particular juggling
required in Coprocessor Endpoint development (again see above cited protobuf
chapter), you'll want to use the shaded version, always.

The `hbase-thirdparty` project has groupid of `org.apache.hbase.thirdparty`.
As of this writing, it provides three jars; one for netty with an artifactid of
`hbase-thirdparty-netty`, one for protobuf at `hbase-thirdparty-protobuf` and then
a jar for all else -- gson, guava -- at `hbase-thirdpaty-miscellaneous`.

The hbase-thirdparty artifacts are a product produced by the Apache HBase
project under the aegis of the HBase Project Management Committee. Releases
are done via the usual voting project on the hbase dev mailing list. If issue
in the hbase-thirdparty, use the hbase JIRA and mailing lists to post notice.

[[hbase.archetypes.development]]
=== Development of HBase-related Maven archetypes

The development of HBase-related Maven archetypes was begun with
link:https://issues.apache.org/jira/browse/HBASE-14876[HBASE-14876].
For an overview of the hbase-archetypes infrastructure and instructions
for developing new HBase-related Maven archetypes, please see
`hbase/hbase-archetypes/README.md`.

ifdef::backend-docbook[]
[index]
== Index
// Generated automatically by the DocBook toolchain.
endif::backend-docbook[]