Google Git
Sign in
apache / hbase / 71c27818947aefc303a2ba16d22255a15b5f8998 / . / src / main / docbkx / configuration.xml
blob: 9e648d874b934e5248a2b751f6080c683ca6adc3 [file] [log] [blame]
<?xml version="1.0"?>
<chapter xml:id="configuration"
version="5.0" xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:m="http://www.w3.org/1998/Math/MathML"
xmlns:html="http://www.w3.org/1999/xhtml"
xmlns:db="http://docbook.org/ns/docbook">
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<title>Apache HBase Configuration</title>
<para>This chapter is the Not-So-Quick start guide to Apache HBase configuration. It goes
over system requirements, Hadoop setup, the different Apache HBase run modes, and the
various configurations in HBase. Please read this chapter carefully. At a mimimum
ensure that all <xref linkend="basic.prerequisites" /> have
been satisfied. Failure to do so will cause you (and us) grief debugging strange errors
and/or data loss.</para>
<para>
Apache HBase uses the same configuration system as Apache Hadoop.
To configure a deploy, edit a file of environment variables
in <filename>conf/hbase-env.sh</filename> -- this configuration
is used mostly by the launcher shell scripts getting the cluster
off the ground -- and then add configuration to an XML file to
do things like override HBase defaults, tell HBase what Filesystem to
use, and the location of the ZooKeeper ensemble
<footnote>
<para>
Be careful editing XML. Make sure you close all elements.
Run your file through <command>xmllint</command> or similar
to ensure well-formedness of your document after an edit session.
</para>
</footnote>
.
</para>
<para>When running in distributed mode, after you make
an edit to an HBase configuration, make sure you copy the
content of the <filename>conf</filename> directory to
all nodes of the cluster. HBase will not do this for you.
Use <command>rsync</command>. For most configuration, a restart is
needed for servers to pick up changes (caveat dynamic config. to be described later below).</para>
<section xml:id="basic.prerequisites">
<title>Basic Prerequisites</title>
<para>This section lists required services and some required system configuration.
</para>
<section xml:id="java">
<title>Java</title>
<para>Just like Hadoop, HBase requires at least Java 6 from
<link xlink:href="http://www.java.com/download/">Oracle</link>.
</para>
</section>
<section xml:id="os">
<title>Operating System</title>
<section xml:id="ssh">
<title>ssh</title>
<para><command>ssh</command> must be installed and
<command>sshd</command> must be running to use Hadoop's scripts to
manage remote Hadoop and HBase daemons. You must be able to ssh to all
nodes, including your local node, using passwordless login (Google
"ssh passwordless login"). If on mac osx, see the section,
<link xlink:href="http://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_%28Single-Node_Cluster%29">SSH: Setting up Remote Desktop and Enabling Self-Login</link>
on the hadoop wiki.</para>
</section>
<section xml:id="dns">
<title>DNS</title>
<para>HBase uses the local hostname to self-report its IP address.
Both forward and reverse DNS resolving must work in versions of
HBase previous to 0.92.0
<footnote><para>The <link xlink:href="https://github.com/sujee/hadoop-dns-checker">hadoop-dns-checker</link> tool can be used to verify
DNS is working correctly on the cluster. The project README file provides detailed instructions on usage.
</para></footnote>.</para>
<para>If your machine has multiple interfaces, HBase will use the
interface that the primary hostname resolves to.</para>
<para>If this is insufficient, you can set
<varname>hbase.regionserver.dns.interface</varname> to indicate the
primary interface. This only works if your cluster configuration is
consistent and every host has the same network interface
configuration.</para>
<para>Another alternative is setting
<varname>hbase.regionserver.dns.nameserver</varname> to choose a
different nameserver than the system wide default.</para>
</section>
<section xml:id="loopback.ip">
<title>Loopback IP</title>
<para>Previous to hbase-0.96.0, HBase expects the loopback IP address to be 127.0.0.1. See <xref linkend="loopback.ip"/></para>
</section>
<section xml:id="ntp">
<title>NTP</title>
<para>The clocks on cluster members should be in basic alignments.
Some skew is tolerable but wild skew could generate odd behaviors. Run
<link
xlink:href="http://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</link>
on your cluster, or an equivalent.</para>
<para>If you are having problems querying data, or "weird" cluster
operations, check system time!</para>
</section>
<section xml:id="ulimit">
<title>
<varname>ulimit</varname><indexterm>
<primary>ulimit</primary>
</indexterm>
and
<varname>nproc</varname><indexterm>
<primary>nproc</primary>
</indexterm>
</title>
<para>Apache HBase is a database. It uses a lot of files all at the same time.
The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems
is insufficient (On mac os x its 256). Any significant amount of loading will
lead you to <xref linkend="trouble.rs.runtime.filehandles"/>.
You may also notice errors such as... <programlisting>
2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
</programlisting> Do yourself a favor and change the upper bound on the
number of file descriptors. Set it to north of 10k. The math runs roughly as follows: per ColumnFamily
there is at least one StoreFile and possibly up to 5 or 6 if the region is under load. Multiply the
average number of StoreFiles per ColumnFamily times the number of regions per RegionServer. For example, assuming
that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily,
and there are 100 regions per RegionServer, the JVM will open 3 * 3 * 100 = 900 file descriptors
(not counting open jar files, config files, etc.)
</para>
<para>You should also up the hbase users'
<varname>nproc</varname> setting; under load, a low-nproc
setting could manifest as <classname>OutOfMemoryError</classname>
<footnote><para>See Jack Levin's <link xlink:href="">major hdfs issues</link>
note up on the user list.</para></footnote>
<footnote><para>The requirement that a database requires upping of system limits
is not peculiar to Apache HBase. See for example the section
<emphasis>Setting Shell Limits for the Oracle User</emphasis> in
<link xlink:href="http://www.akadia.com/services/ora_linux_install_10g.html">
Short Guide to install Oracle 10 on Linux</link>.</para></footnote>.
</para>
<para>To be clear, upping the file descriptors and nproc for the user who is
running the HBase process is an operating system configuration, not an
HBase configuration. Also, a common mistake is that administrators
will up the file descriptors for a particular user but for whatever
reason, HBase will be running as some one else. HBase prints in its
logs as the first line the ulimit its seeing. Ensure its correct.
<footnote>
<para>A useful read setting config on you hadoop cluster is Aaron
Kimballs' <link
xlink:ref="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/">Configuration
Parameters: What can you just ignore?</link></para>
</footnote></para>
<section xml:id="ulimit_ubuntu">
<title><varname>ulimit</varname> on Ubuntu</title>
<para>If you are on Ubuntu you will need to make the following
changes:</para>
<para>In the file <filename>/etc/security/limits.conf</filename> add
a line like: <programlisting>hadoop - nofile 32768</programlisting>
Replace <varname>hadoop</varname> with whatever user is running
Hadoop and HBase. If you have separate users, you will need 2
entries, one for each user. In the same file set nproc hard and soft
limits. For example: <programlisting>hadoop soft/hard nproc 32000</programlisting>.</para>
<para>In the file <filename>/etc/pam.d/common-session</filename> add
as the last line in the file: <programlisting>session required pam_limits.so</programlisting>
Otherwise the changes in <filename>/etc/security/limits.conf</filename> won't be
applied.</para>
<para>Don't forget to log out and back in again for the changes to
take effect!</para>
</section>
</section>
<section xml:id="windows">
<title>Windows</title>
<para>Previous to hbase-0.96.0, Apache HBase was little tested running on Windows. Running a
production install of HBase on top of Windows is not recommended.</para>
<para>If you are running HBase on Windows pre-hbase-0.96.0, you must install <link
xlink:href="http://cygwin.com/">Cygwin</link> to have a *nix-like
environment for the shell scripts. The full details are explained in
the <link xlink:href="http://hbase.apache.org/cygwin.html">Windows
Installation</link> guide. Also
<link xlink:href="http://search-hadoop.com/?q=hbase+windows&amp;fc_project=HBase&amp;fc_type=mail+_hash_+dev">search our user mailing list</link> to pick
up latest fixes figured by Windows users.</para>
<para>Post-hbase-0.96.0, hbase runs natively on windows with supporting <command>*.cmd</command> scripts bundled.
</para>
</section>
</section> <!-- OS -->
<section xml:id="hadoop">
<title><link
xlink:href="http://hadoop.apache.org">Hadoop</link><indexterm>
<primary>Hadoop</primary>
</indexterm></title>
<para>The below table shows some information about what versions of Hadoop are supported by various HBase versions.
Based on the version of HBase, you should select the most appropriate version of Hadoop.
We are not in the Hadoop distro selection business.
You can use Hadoop distributions from Apache, or learn about vendor distributions of
Hadoop at <link xlink:href="http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support"/></para>
<para>
<tip><title>Hadoop 2.x is better than Hadoop 1.x</title>
<para>Hadoop 2.x is faster, with more features such as short-circuit reads which will help improve your
HBase random read profile as well important bug fixes that will improve your overall HBase experience.
You should run Hadoop 2. rather than Hadoop 1. if you can.
</para>
</tip>
<table>
<title>Hadoop version support matrix</title>
<tgroup cols='4' align='left' colsep='1' rowsep='1'><colspec colname='c1' align='left'/><colspec colname='c2' align='center'/><colspec colname='c3' align='center'/><colspec colname='c4' align='center'/>
<thead>
<row><entry> </entry><entry>HBase-0.92.x</entry><entry>HBase-0.94.x</entry><entry>HBase-0.96.0</entry><entry>HBase-0.98.0</entry></row>
</thead><tbody>
<row><entry>Hadoop-0.20.205</entry><entry>S</entry> <entry>X</entry> <entry>X</entry><entry>X</entry></row>
<row><entry>Hadoop-0.22.x </entry><entry>S</entry> <entry>X</entry> <entry>X</entry><entry>X</entry></row>
<row><entry>Hadoop-1.0.0-1.0.2<footnote><para>HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot find KerberosUtil compiling against earlier versions of Hadoop.</para></footnote> </entry><entry>S</entry> <entry>S</entry> <entry>X</entry><entry>X</entry></row>
<row><entry>Hadoop-1.0.3+</entry><entry>S</entry> <entry>S</entry> <entry>S</entry><entry>X</entry></row>
<row><entry>Hadoop-1.1.x </entry><entry>NT</entry> <entry>S</entry> <entry>S</entry><entry>X</entry></row>
<row><entry>Hadoop-0.23.x </entry><entry>X</entry> <entry>S</entry> <entry>NT</entry><entry>X</entry></row>
<row><entry>Hadoop-2.0.x-alpha </entry><entry>X</entry> <entry>NT</entry> <entry>X</entry><entry>X</entry></row>
<row><entry>Hadoop-2.1.0-beta </entry><entry>X</entry> <entry>NT</entry> <entry>S</entry><entry>X</entry></row>
<row><entry>Hadoop-2.2.0 </entry><entry>X</entry> <entry>NT<footnote><para>To get 0.94.x to run on hadoop 2.2.0,
you need to change the hadoop 2 and protobuf versions in the <filename>pom.xml</filename> and then
build against the hadoop 2 profile by running something like the following command:
<programlisting>$ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests</programlisting>
Here is a diff with pom.xml changes:
<programlisting><![CDATA[$ svn diff pom.xml
Index: pom.xml
===================================================================
--- pom.xml (revision 1545157)
+++ pom.xml (working copy)
@@ -1034,7 +1034,7 @@
<slf4j.version>1.4.3</slf4j.version>
<log4j.version>1.2.16</log4j.version>
<mockito-all.version>1.8.5</mockito-all.version>
- <protobuf.version>2.4.0a</protobuf.version>
+ <protobuf.version>2.5.0</protobuf.version>
<stax-api.version>1.0.1</stax-api.version>
<thrift.version>0.8.0</thrift.version>
<zookeeper.version>3.4.5</zookeeper.version>
@@ -2241,7 +2241,7 @@
</property>
</activation>
<properties>
- <hadoop.version>2.0.0-alpha</hadoop.version>
+ <hadoop.version>2.2.0</hadoop.version>
<slf4j.version>1.6.1</slf4j.version>
</properties>
<dependencies>]]></programlisting>
</para></footnote></entry> <entry>S</entry><entry>S</entry></row>
<row><entry>Hadoop-2.x </entry><entry>X</entry> <entry>NT</entry> <entry>S</entry><entry>S</entry></row>
</tbody></tgroup></table>
Where
<simplelist type='vert' columns='1'>
<member>S = supported and tested,</member>
<member>X = not supported,</member>
<member>NT = it should run, but not tested enough.</member>
</simplelist>
</para>
<note xml:id="replace.hadoop"><title>Replace the Hadoop Bundled With HBase!</title>
<para>
Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its <filename>lib</filename> directory. The bundled jar is ONLY for use in standalone mode. In distributed mode, it is <emphasis>critical</emphasis> that the version of Hadoop that is out on your cluster match what is under HBase. Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up.
</para>
</note>
<section xml:id="hadoop.hbase-0.94">
<title>Apache HBase 0.92 and 0.94</title>
<para>HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x, and 1.1.x. HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have to recompile the code using the specific maven profile (see top level pom.xml)</para>
</section>
<section xml:id="hadoop.hbase-0.96">
<title>Apache HBase 0.96</title>
<para>
As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. Hadoop 2 is strongly encouraged (faster but also has fixes that help MTTR).
We will no longer run properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache HBase 0.96.x if you cannot upgrade your Hadoop<footnote><para>See <link xlink:href="http://search-hadoop.com/m/7vFVx4EsUb2">HBase, mail # dev - DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?</link></para></footnote>.</para>
</section>
<section xml:id="hadoop.older.versions">
<title>Hadoop versions 0.20.x - 1.x</title>
<para>
HBase will lose data unless it is running on an HDFS that has a durable
<code>sync</code> implementation. DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0 which DO NOT have this attribute. Currently only Hadoop versions 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have a working, durable sync
<footnote>
<para>The Cloudera blog post <link xlink:href="http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/">An update on Apache Hadoop 1.0</link>
by Charles Zedlweski has a nice exposition on how all the Hadoop versions relate.
Its worth checking out if you are having trouble making sense of the
Hadoop version morass.
</para>
</footnote>. Sync has to be explicitly enabled by setting
<varname>dfs.support.append</varname> equal
to true on both the client side -- in <filename>hbase-site.xml</filename>
-- and on the serverside in <filename>hdfs-site.xml</filename> (The sync
facility HBase needs is a subset of the append code path).
<programlisting>
&lt;property>
&lt;name>dfs.support.append&lt;/name>
&lt;value>true&lt;/value>
&lt;/property>
</programlisting>
You will have to restart your cluster after making this edit. Ignore the chicken-little
comment you'll find in the <filename>hdfs-default.xml</filename> in the
description for the <varname>dfs.support.append</varname> configuration.
</para>
</section>
<section xml:id="hadoop.security">
<title>Apache HBase on Secure Hadoop</title>
<para>Apache HBase will run on any Hadoop 0.20.x that incorporates Hadoop
security features as long as you do as
suggested above and replace the Hadoop jar that ships with HBase
with the secure version. If you want to read more about how to setup
Secure HBase, see <xref linkend="hbase.secure.configuration" />.</para>
</section>
<section xml:id="dfs.datanode.max.xcievers">
<title><varname>dfs.datanode.max.xcievers</varname><indexterm>
<primary>xcievers</primary>
</indexterm></title>
<para>An Hadoop HDFS datanode has an upper bound on the number of
files that it will serve at any one time. The upper bound parameter is
called <varname>xcievers</varname> (yes, this is misspelled). Again,
before doing any loading, make sure you have configured Hadoop's
<filename>conf/hdfs-site.xml</filename> setting the
<varname>xceivers</varname> value to at least the following:
<programlisting>
&lt;property&gt;
&lt;name&gt;dfs.datanode.max.xcievers&lt;/name&gt;
&lt;value&gt;4096&lt;/value&gt;
&lt;/property&gt;
</programlisting></para>
<para>Be sure to restart your HDFS after making the above
configuration.</para>
<para>Not having this configuration in place makes for strange looking
failures. Eventually you'll see a complain in the datanode logs
complaining about the xcievers exceeded, but on the run up to this one
manifestation is complaint about missing blocks. For example:
<code>10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block
blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node:
java.io.IOException: No live nodes contain current block. Will get new
block locations from namenode and retry...</code>
<footnote><para>See <link xlink:href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop HDFS: Deceived by Xciever</link> for an informative rant on xceivering.</para></footnote></para>
<para>See also <xref linkend="casestudies.xceivers"/>
</para>
</section>
</section> <!-- hadoop -->
</section>
<section xml:id="standalone_dist">
<title>HBase run modes: Standalone and Distributed</title>
<para>HBase has two run modes: <xref linkend="standalone" /> and <xref linkend="distributed" />. Out of the box, HBase runs in
standalone mode. Whatever your mode, you will need to configure HBase by editing files in the HBase <filename>conf</filename>
directory. At a minimum, you must edit <code>conf/hbase-env.sh</code> to tell HBase which
<command>java</command> to use. In this file you set HBase environment
variables such as the heapsize and other options for the
<application>JVM</application>, the preferred location for log files,
etc. Set <varname>JAVA_HOME</varname> to point at the root of your
<command>java</command> install.</para>
<section xml:id="standalone">
<title>Standalone HBase</title>
<para>This is the default mode. Standalone mode is what is described
in the <xref linkend="quickstart" /> section. In
standalone mode, HBase does not use HDFS -- it uses the local
filesystem instead -- and it runs all HBase daemons and a local
ZooKeeper all up in the same JVM. Zookeeper binds to a well known port
so clients may talk to HBase.</para>
</section>
<section xml:id="distributed">
<title>Distributed</title>
<para>Distributed mode can be subdivided into distributed but all
daemons run on a single node -- a.k.a
<emphasis>pseudo-distributed</emphasis>-- and
<emphasis>fully-distributed</emphasis> where the daemons are spread
across all nodes in the cluster <footnote>
<para>The pseudo-distributed vs fully-distributed nomenclature
comes from Hadoop.</para>
</footnote>.</para>
<para>Pseudo-distributed mode can run against the local filesystem or
it can run against an instance of the <emphasis>Hadoop
Distributed File System</emphasis> (HDFS). Fully-distributed mode can
ONLY run on HDFS. See the Hadoop <link
xlink:href="http://hadoop.apache.org/common/docs/r1.1.1/api/overview-summary.html#overview_description">
requirements and instructions</link> for how to set up HDFS.</para>
<para>Below we describe the different distributed setups. Starting,
verification and exploration of your install, whether a
<emphasis>pseudo-distributed</emphasis> or
<emphasis>fully-distributed</emphasis> configuration is described in a
section that follows, <xref linkend="confirm" />. The same verification script applies to both
deploy types.</para>
<section xml:id="pseudo">
<title>Pseudo-distributed</title>
<para>A pseudo-distributed mode is simply a fully-distributed mode run on
a single host. Use this configuration testing and prototyping on
HBase. Do not use this configuration for production nor for
evaluating HBase performance.</para>
<para>First, if you want to run on HDFS rather than on the local filesystem,
setup your HDFS. You can set up HDFS also in pseudo-distributed mode
(TODO: Add pointer to HOWTO doc; the hadoop site doesn't have any any more).
Ensure you have a working HDFS before proceeding.
</para>
<para>Next, configure HBase. Edit <filename>conf/hbase-site.xml</filename>.
This is the file into which you add local customizations and overrides.
At a minimum, you must tell HBase to run in (pseudo-)distributed mode rather than
in default standalone mode. To do this, set the <varname>hbase.cluster.distributed</varname>
property to true (Its default is <varname>false</varname>). The absolute bare-minimum
<filename>hbase-site.xml</filename> is therefore as follows:
<programlisting>
&lt;configuration&gt;
&lt;property&gt;
&lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;/property&gt;
&lt;/configuration&gt;
</programlisting>
With this configuration, HBase will start up an HBase Master process, a ZooKeeper server,
and a RegionServer process running against the
local filesystem writing to wherever your operating system stores temporary files into a directory
named <filename>hbase-YOUR_USER_NAME</filename>.</para>
<para>Such a setup, using the local filesystem and
writing to the operating systems's temporary directory is an ephemeral setup; the Hadoop
local filesystem -- which is what HBase uses when it is writing the local filesytem does not
support <command>sync</command> so unless the system is shutdown properly, the data will be lost. Writing to
the operating system's temporary directory can also make for data loss when the machine
is restarted as this directory is usually cleared on reboot. For a more permanent
setup, see the next example where we make use of an instance of HDFS; HBase data will
be written to the Hadoop distributed filesystem rather than to the local filesystem's
tmp directory.</para>
<para>In this <filename>conf/hbase-site.xml</filename> example, the
<varname>hbase.rootdir</varname> property points to the local HDFS instance
homed on the node <varname>h-24-30.example.com</varname>.
<note>
<title>Let HBase create <filename>${hbase.rootdir}</filename></title>
<para>Let HBase create the <varname>hbase.rootdir</varname>
directory. If you don't, you'll get warning saying HBase needs a
migration run because the directory is missing files expected by
HBase (it'll create them if you let it).</para>
</note>
<programlisting>
&lt;configuration&gt;
&lt;property&gt;
&lt;name&gt;hbase.rootdir&lt;/name&gt;
&lt;value&gt;hdfs://h-24-30.sfo.stumble.net:8020/hbase&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;/property&gt;
&lt;/configuration&gt;
</programlisting>
</para>
<para>Now skip to <xref linkend="confirm" /> for how to start and verify your
pseudo-distributed install. <footnote>
<para>See <xref linkend="pseudo.extras">Pseudo-distributed
mode extras</xref> for notes on how to start extra Masters and
RegionServers when running pseudo-distributed.</para>
</footnote></para>
<section xml:id="pseudo.extras">
<title>Pseudo-distributed Extras</title>
<section xml:id="pseudo.extras.start">
<title>Startup</title>
<para>To start up the initial HBase cluster...
<programlisting>% bin/start-hbase.sh</programlisting>
</para>
<para>To start up an extra backup master(s) on the same server run...
<programlisting>% bin/local-master-backup.sh start 1</programlisting>
... the '1' means use ports 16001 &amp; 16011, and this backup master's
logfile will be at
<filename>logs/hbase-${USER}-1-master-${HOSTNAME}.log</filename>.
</para>
<para>To startup multiple backup masters run... <programlisting>% bin/local-master-backup.sh start 2 3</programlisting> You can start up to 9 backup masters (10 total).
</para>
<para>To start up more regionservers...
<programlisting>% bin/local-regionservers.sh start 1</programlisting>
... where '1' means use ports 16201 &amp; 16301 and its logfile will be at
`<filename>logs/hbase-${USER}-1-regionserver-${HOSTNAME}.log</filename>.
</para>
<para>To add 4 more regionservers in addition to the one you just started by running... <programlisting>% bin/local-regionservers.sh start 2 3 4 5</programlisting>
This supports up to 99 extra regionservers (100 total).
</para>
</section>
<section xml:id="pseudo.options.stop">
<title>Stop</title>
<para>Assuming you want to stop master backup # 1, run...
<programlisting>% cat /tmp/hbase-${USER}-1-master.pid |xargs kill -9</programlisting>
Note that bin/local-master-backup.sh stop 1 will try to stop the cluster along with the master.
</para>
<para>To stop an individual regionserver, run...
<programlisting>% bin/local-regionservers.sh stop 1
</programlisting>
</para>
</section>
</section>
</section>
<section xml:id="fully_dist">
<title>Fully-distributed</title>
<para>For running a fully-distributed operation on more than one
host, make the following configurations. In
<filename>hbase-site.xml</filename>, add the property
<varname>hbase.cluster.distributed</varname> and set it to
<varname>true</varname> and point the HBase
<varname>hbase.rootdir</varname> at the appropriate HDFS NameNode
and location in HDFS where you would like HBase to write data. For
example, if you namenode were running at namenode.example.org on
port 8020 and you wanted to home your HBase in HDFS at
<filename>/hbase</filename>, make the following
configuration.</para>
<programlisting>
&lt;configuration&gt;
...
&lt;property&gt;
&lt;name&gt;hbase.rootdir&lt;/name&gt;
&lt;value&gt;hdfs://namenode.example.org:8020/hbase&lt;/value&gt;
&lt;description&gt;The directory shared by RegionServers.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;description&gt;The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
&lt;/description&gt;
&lt;/property&gt;
...
&lt;/configuration&gt;
</programlisting>
<section xml:id="regionserver">
<title><filename>regionservers</filename></title>
<para>In addition, a fully-distributed mode requires that you
modify <filename>conf/regionservers</filename>. The
<xref linkend="regionservers" /> file
lists all hosts that you would have running
<application>HRegionServer</application>s, one host per line (This
file in HBase is like the Hadoop <filename>slaves</filename>
file). All servers listed in this file will be started and stopped
when HBase cluster start or stop is run.</para>
</section>
<section xml:id="hbase.zookeeper">
<title>ZooKeeper and HBase</title>
<para>See section <xref linkend="zookeeper"/> for ZooKeeper setup for HBase.</para>
</section>
<section xml:id="hdfs_client_conf">
<title>HDFS Client Configuration</title>
<para>Of note, if you have made <emphasis>HDFS client
configuration</emphasis> on your Hadoop cluster -- i.e.
configuration you want HDFS clients to use as opposed to
server-side configurations -- HBase will not see this
configuration unless you do one of the following:</para>
<itemizedlist>
<listitem>
<para>Add a pointer to your <varname>HADOOP_CONF_DIR</varname>
to the <varname>HBASE_CLASSPATH</varname> environment variable
in <filename>hbase-env.sh</filename>.</para>
</listitem>
<listitem>
<para>Add a copy of <filename>hdfs-site.xml</filename> (or
<filename>hadoop-site.xml</filename>) or, better, symlinks,
under <filename>${HBASE_HOME}/conf</filename>, or</para>
</listitem>
<listitem>
<para>if only a small set of HDFS client configurations, add
them to <filename>hbase-site.xml</filename>.</para>
</listitem>
</itemizedlist>
<para>An example of such an HDFS client configuration is
<varname>dfs.replication</varname>. If for example, you want to
run with a replication factor of 5, hbase will create files with
the default of 3 unless you do the above to make the configuration
available to HBase.</para>
</section>
</section>
</section>
<section xml:id="confirm">
<title>Running and Confirming Your Installation</title>
<para>Make sure HDFS is running first. Start and stop the Hadoop HDFS
daemons by running <filename>bin/start-hdfs.sh</filename> over in the
<varname>HADOOP_HOME</varname> directory. You can ensure it started
properly by testing the <command>put</command> and
<command>get</command> of files into the Hadoop filesystem. HBase does
not normally use the mapreduce daemons. These do not need to be
started.</para>
<para><emphasis>If</emphasis> you are managing your own ZooKeeper,
start it and confirm its running else, HBase will start up ZooKeeper
for you as part of its start process.</para>
<para>Start HBase with the following command:</para>
<programlisting>bin/start-hbase.sh</programlisting>
Run the above from the
<varname>HBASE_HOME</varname>
directory.
<para>You should now have a running HBase instance. HBase logs can be
found in the <filename>logs</filename> subdirectory. Check them out
especially if HBase had trouble starting.</para>
<para>HBase also puts up a UI listing vital attributes. By default its
deployed on the Master host at port 16010 (HBase RegionServers listen
on port 16020 by default and put up an informational http server at
16030). If the Master were running on a host named
<varname>master.example.org</varname> on the default port, to see the
Master's homepage you'd point your browser at
<filename>http://master.example.org:16010</filename>.</para>
<para>Prior to HBase 0.98, the default ports the master ui was deployed
on port 16010, and the HBase RegionServers would listen
on port 16020 by default and put up an informational http server at
16030.
</para>
<para>Once HBase has started, see the <xref linkend="shell_exercises" /> for how to
create tables, add data, scan your insertions, and finally disable and
drop your tables.</para>
<para>To stop HBase after exiting the HBase shell enter
<programlisting>$ ./bin/stop-hbase.sh
stopping hbase...............</programlisting> Shutdown can take a moment to
complete. It can take longer if your cluster is comprised of many
machines. If you are running a distributed operation, be sure to wait
until HBase has shut down completely before stopping the Hadoop
daemons.</para>
</section>
</section> <!-- run modes -->
<section xml:id="config.files">
<title>Configuration Files</title>
<section xml:id="hbase.site">
<title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title>
<para>Just as in Hadoop where you add site-specific HDFS configuration
to the <filename>hdfs-site.xml</filename> file,
for HBase, site specific customizations go into
the file <filename>conf/hbase-site.xml</filename>.
For the list of configurable properties, see
<xref linkend="hbase_default_configurations" />
below or view the raw <filename>hbase-default.xml</filename>
source file in the HBase source code at
<filename>src/main/resources</filename>.
</para>
<para>
Not all configuration options make it out to
<filename>hbase-default.xml</filename>. Configuration
that it is thought rare anyone would change can exist only
in code; the only way to turn up such configurations is
via a reading of the source code itself.
</para>
<para>
Currently, changes here will require a cluster restart for HBase to notice the change.
</para>
<!--The file hbase-default.xml is generated as part of
the build of the hbase site. See the hbase pom.xml.
The generated file is a docbook section with a glossary
in it-->
<!--presumes the pre-site target has put the hbase-default.xml at this location-->
<xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="../../../target/docbkx/hbase-default.xml" />
</section>
<section xml:id="hbase.env.sh">
<title><filename>hbase-env.sh</filename></title>
<para>Set HBase environment variables in this file.
Examples include options to pass the JVM on start of
an HBase daemon such as heap size and garbarge collector configs.
You can also set configurations for HBase configuration, log directories,
niceness, ssh options, where to locate process pid files,
etc. Open the file at
<filename>conf/hbase-env.sh</filename> and peruse its content.
Each option is fairly well documented. Add your own environment
variables here if you want them read by HBase daemons on startup.</para>
<para>
Changes here will require a cluster restart for HBase to notice the change.
</para>
</section>
<section xml:id="log4j">
<title><filename>log4j.properties</filename></title>
<para>Edit this file to change rate at which HBase files
are rolled and to change the level at which HBase logs messages.
</para>
<para>
Changes here will require a cluster restart for HBase to notice the change
though log levels can be changed for particular daemons via the HBase UI.
</para>
</section>
<section xml:id="client_dependencies"><title>Client configuration and dependencies connecting to an HBase cluster</title>
<para>If you are running HBase in standalone mode, you don't need to configure anything for your client to work
provided that they are all on the same machine.</para>
<para>
Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for
current critical locations. ZooKeeper is where all these values are kept. Thus clients
require the location of the ZooKeeper ensemble information before they can do anything else.
Usually this the ensemble location is kept out in the <filename>hbase-site.xml</filename> and
is picked up by the client from the <varname>CLASSPATH</varname>.</para>
<para>If you are configuring an IDE to run a HBase client, you should
include the <filename>conf/</filename> directory on your classpath so
<filename>hbase-site.xml</filename> settings can be found (or
add <filename>src/test/resources</filename> to pick up the hbase-site.xml
used by tests).
</para>
<para>
Minimally, a client of HBase needs several libraries in its <varname>CLASSPATH</varname> when connecting to a cluster, including:
<programlisting>
commons-configuration (commons-configuration-1.6.jar)
commons-lang (commons-lang-2.5.jar)
commons-logging (commons-logging-1.1.1.jar)
hadoop-core (hadoop-core-1.0.0.jar)
hbase (hbase-0.92.0.jar)
log4j (log4j-1.2.16.jar)
slf4j-api (slf4j-api-1.5.8.jar)
slf4j-log4j (slf4j-log4j12-1.5.8.jar)
zookeeper (zookeeper-3.4.2.jar)</programlisting>
</para>
<para>
An example basic <filename>hbase-site.xml</filename> for client only
might look as follows:
<programlisting><![CDATA[
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.zookeeper.quorum</name>
<value>example1,example2,example3</value>
<description>The directory shared by region servers.
</description>
</property>
</configuration>
]]></programlisting>
</para>
<section xml:id="java.client.config">
<title>Java client configuration</title>
<para>The configuration used by a Java client is kept
in an <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HBaseConfiguration">HBaseConfiguration</link> instance.
The factory method on HBaseConfiguration, <code>HBaseConfiguration.create();</code>,
on invocation, will read in the content of the first <filename>hbase-site.xml</filename> found on
the client's <varname>CLASSPATH</varname>, if one is present
(Invocation will also factor in any <filename>hbase-default.xml</filename> found;
an hbase-default.xml ships inside the <filename>hbase.X.X.X.jar</filename>).
It is also possible to specify configuration directly without having to read from a
<filename>hbase-site.xml</filename>. For example, to set the ZooKeeper
ensemble for the cluster programmatically do as follows:
<programlisting>Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zookeeper locally</programlisting>
If multiple ZooKeeper instances make up your ZooKeeper ensemble,
they may be specified in a comma-separated list (just as in the <filename>hbase-site.xml</filename> file).
This populated <classname>Configuration</classname> instance can then be passed to an
<link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>,
and so on.
</para>
</section>
</section>
</section> <!-- config files -->
<section xml:id="example_config">
<title>Example Configurations</title>
<section>
<title>Basic Distributed HBase Install</title>
<para>Here is an example basic configuration for a distributed ten
node cluster. The nodes are named <varname>example0</varname>,
<varname>example1</varname>, etc., through node
<varname>example9</varname> in this example. The HBase Master and the
HDFS namenode are running on the node <varname>example0</varname>.
RegionServers run on nodes
<varname>example1</varname>-<varname>example9</varname>. A 3-node
ZooKeeper ensemble runs on <varname>example1</varname>,
<varname>example2</varname>, and <varname>example3</varname> on the
default ports. ZooKeeper data is persisted to the directory
<filename>/export/zookeeper</filename>. Below we show what the main
configuration files -- <filename>hbase-site.xml</filename>,
<filename>regionservers</filename>, and
<filename>hbase-env.sh</filename> -- found in the HBase
<filename>conf</filename> directory might look like.</para>
<section xml:id="hbase_site">
<title><filename>hbase-site.xml</filename></title>
<programlisting>
&lt;?xml version="1.0"?&gt;
&lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
&lt;configuration&gt;
&lt;property&gt;
&lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
&lt;value&gt;example1,example2,example3&lt;/value&gt;
&lt;description&gt;The directory shared by RegionServers.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt;
&lt;value&gt;/export/zookeeper&lt;/value&gt;
&lt;description&gt;Property from ZooKeeper's config zoo.cfg.
The directory where the snapshot is stored.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.rootdir&lt;/name&gt;
&lt;value&gt;hdfs://example0:8020/hbase&lt;/value&gt;
&lt;description&gt;The directory shared by RegionServers.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;description&gt;The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
&lt;/description&gt;
&lt;/property&gt;
&lt;/configuration&gt;
</programlisting>
</section>
<section xml:id="regionservers">
<title><filename>regionservers</filename></title>
<para>In this file you list the nodes that will run RegionServers.
In our case, these nodes are <varname>example1</varname>-<varname>example9</varname>.
</para>
<programlisting>
example1
example2
example3
example4
example5
example6
example7
example8
example9
</programlisting>
</section>
<section xml:id="hbase_env">
<title><filename>hbase-env.sh</filename></title>
<para>Below we use a <command>diff</command> to show the differences
from default in the <filename>hbase-env.sh</filename> file. Here we
are setting the HBase heap to be 4G instead of the default
1G.</para>
<programlisting>
$ git diff hbase-env.sh
diff --git a/conf/hbase-env.sh b/conf/hbase-env.sh
index e70ebc6..96f8c27 100644
--- a/conf/hbase-env.sh
+++ b/conf/hbase-env.sh
@@ -31,7 +31,7 @@ export JAVA_HOME=/usr/lib//jvm/java-6-sun/
# export HBASE_CLASSPATH=
# The maximum amount of heap to use, in MB. Default is 1000.
-# export HBASE_HEAPSIZE=1000
+export HBASE_HEAPSIZE=4096
# Extra Java runtime options.
# Below are what we set by default. May only work with SUN JVM.
</programlisting>
<para>Use <command>rsync</command> to copy the content of the
<filename>conf</filename> directory to all nodes of the
cluster.</para>
</section>
</section>
</section> <!-- example config -->
<section xml:id="important_configurations">
<title>The Important Configurations</title>
<para>Below we list what the <emphasis>important</emphasis>
Configurations. We've divided this section into
required configuration and worth-a-look recommended configs.
</para>
<section xml:id="required_configuration"><title>Required Configurations</title>
<para>Review the <xref linkend="os" /> and <xref linkend="hadoop" /> sections.
</para>
<section xml:id="big.cluster.config"><title>Big Cluster Configurations</title>
<para>If a cluster with a lot of regions, it is possible if an eager beaver
regionserver checks in soon after master start while all the rest in the
cluster are laggardly, this first server to checkin will be assigned all
regions. If lots of regions, this first server could buckle under the
load. To prevent the above scenario happening up the
<varname>hbase.master.wait.on.regionservers.mintostart</varname> from its
default value of 1. See
<link xlink:href="https://issues.apache.org/jira/browse/HBASE-6389">HBASE-6389 Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments</link>
for more detail.
</para>
</section>
<section xml:id="backup.master.fail.fast"><title>If a backup Master, making primary Master fail fast</title>
<para>If the primary Master loses its connection with ZooKeeper, it will fall into a loop where it
keeps trying to reconnect. Disable this functionality if you are running more than one Master:
i.e. a backup Master. Failing to do so, the dying Master may continue to receive RPCs though
another Master has assumed the role of primary.
See the configuration <xref linkend="fail.fast.expired.active.master" />.
</para>
</section>
</section>
<section xml:id="recommended_configurations"><title>Recommended Configurations</title>
<section xml:id="recommended_configurations.zk">
<title>ZooKeeper Configuration</title>
<section xml:id="zookeeper.session.timeout"><title><varname>zookeeper.session.timeout</varname></title>
<para>The default timeout is three minutes (specified in milliseconds). This means
that if a server crashes, it will be three minutes before the Master notices
the crash and starts recovery. You might like to tune the timeout down to
a minute or even less so the Master notices failures the sooner.
Before changing this value, be sure you have your JVM garbage collection
configuration under control otherwise, a long garbage collection that lasts
beyond the ZooKeeper session timeout will take out
your RegionServer (You might be fine with this -- you probably want recovery to start
on the server if a RegionServer has been in GC for a long period of time).</para>
<para>To change this configuration, edit <filename>hbase-site.xml</filename>,
copy the changed file around the cluster and restart.</para>
<para>We set this value high to save our having to field noob questions up on the mailing lists asking
why a RegionServer went down during a massive import. The usual cause is that their JVM is untuned and
they are running into long GC pauses. Our thinking is that
while users are getting familiar with HBase, we'd save them having to know all of its
intricacies. Later when they've built some confidence, then they can play
with configuration such as this.
</para>
</section>
<section xml:id="zookeeper.instances"><title>Number of ZooKeeper Instances</title>
<para>See <xref linkend="zookeeper"/>.
</para>
</section>
</section>
<section xml:id="recommended.configurations.hdfs">
<title>HDFS Configurations</title>
<section xml:id="dfs.datanode.failed.volumes.tolerated">
<title>dfs.datanode.failed.volumes.tolerated</title>
<para>This is the "...number of volumes that are allowed to fail before a datanode stops offering service. By default
any volume failure will cause a datanode to shutdown" from the <filename>hdfs-default.xml</filename>
description. If you have > three or four disks, you might want to set this to 1 or if you have many disks,
two or more.
</para>
</section>
</section>
<section xml:id="hbase.regionserver.handler.count"><title><varname>hbase.regionserver.handler.count</varname></title>
<para>
This setting defines the number of threads that are kept open to answer
incoming requests to user tables. The rule of thumb is to keep this
number low when the payload per request approaches the MB (big puts, scans using
a large cache) and high when the payload is small (gets, small puts, ICVs, deletes).
The total size of the queries in progress is limited by the setting
"ipc.server.max.callqueue.size".
</para>
<para>
It is safe to set that number to the
maximum number of incoming clients if their payload is small, the typical example
being a cluster that serves a website since puts aren't typically buffered
and most of the operations are gets.
</para>
<para>
The reason why it is dangerous to keep this setting high is that the aggregate
size of all the puts that are currently happening in a region server may impose
too much pressure on its memory, or even trigger an OutOfMemoryError. A region server
running on low memory will trigger its JVM's garbage collector to run more frequently
up to a point where GC pauses become noticeable (the reason being that all the memory
used to keep all the requests' payloads cannot be trashed, no matter how hard the
garbage collector tries). After some time, the overall cluster
throughput is affected since every request that hits that region server will take longer,
which exacerbates the problem even more.
</para>
<para>You can get a sense of whether you have too little or too many handlers by
<xref linkend="rpc.logging" />
on an individual RegionServer then tailing its logs (Queued requests
consume memory).
</para>
</section>
<section xml:id="big_memory">
<title>Configuration for large memory machines</title>
<para>
HBase ships with a reasonable, conservative configuration that will
work on nearly all
machine types that people might want to test with. If you have larger
machines -- HBase has 8G and larger heap -- you might the following configuration options helpful.
TODO.
</para>
</section>
<section xml:id="config.compression">
<title>Compression</title>
<para>You should consider enabling ColumnFamily compression. There are several options that are near-frictionless and in most all cases boost
performance by reducing the size of StoreFiles and thus reducing I/O.
</para>
<para>See <xref linkend="compression" /> for more information.</para>
</section>
<section xml:id="config.wals"><title>Configuring the size and number of WAL files</title>
<para>HBase uses <xref linkend="wal" /> to recover the memstore data that has not been flushed to disk in case of an RS failure. These WAL files should be configured to be slightly smaller than HDFS block (by default, HDFS block is 64Mb and WAL file is ~60Mb).</para>
<para>HBase also has a limit on number of WAL files, designed to ensure there's never too much data that needs to be replayed during recovery. This limit needs to be set according to memstore configuration, so that all the necessary data would fit. It is recommended to allocated enough WAL files to store at least that much data (when all memstores are close to full).
For example, with 16Gb RS heap, default memstore settings (0.4), and default WAL file size (~60Mb), 16Gb*0.4/60, the starting point for WAL file count is ~109.
However, as all memstores are not expected to be full all the time, less WAL files can be allocated.</para>
</section>
<section xml:id="disable.splitting">
<title>Managed Splitting</title>
<para>
Rather than let HBase auto-split your Regions, manage the splitting manually
<footnote><para>What follows is taken from the javadoc at the head of
the <classname>org.apache.hadoop.hbase.util.RegionSplitter</classname> tool
added to HBase post-0.90.0 release.
</para>
</footnote>.
With growing amounts of data, splits will continually be needed. Since
you always know exactly what regions you have, long-term debugging and
profiling is much easier with manual splits. It is hard to trace the logs to
understand region level problems if it keeps splitting and getting renamed.
Data offlining bugs + unknown number of split regions == oh crap! If an
<classname>HLog</classname> or <classname>StoreFile</classname>
was mistakenly unprocessed by HBase due to a weird bug and
you notice it a day or so later, you can be assured that the regions
specified in these files are the same as the current regions and you have
less headaches trying to restore/replay your data.
You can finely tune your compaction algorithm. With roughly uniform data
growth, it's easy to cause split / compaction storms as the regions all
roughly hit the same data size at the same time. With manual splits, you can
let staggered, time-based major compactions spread out your network IO load.
</para>
<para>
How do I turn off automatic splitting? Automatic splitting is determined by the configuration value
<code>hbase.hregion.max.filesize</code>. It is not recommended that you set this
to <varname>Long.MAX_VALUE</varname> in case you forget about manual splits. A suggested setting
is 100GB, which would result in > 1hr major compactions if reached.
</para>
<para>What's the optimal number of pre-split regions to create?
Mileage will vary depending upon your application.
You could start low with 10 pre-split regions / server and watch as data grows
over time. It's better to err on the side of too little regions and rolling split later.
A more complicated answer is that this depends upon the largest storefile
in your region. With a growing data size, this will get larger over time. You
want the largest region to be just big enough that the <classname>Store</classname> compact
selection algorithm only compacts it due to a timed major. If you don't, your
cluster can be prone to compaction storms as the algorithm decides to run
major compactions on a large series of regions all at once. Note that
compaction storms are due to the uniform data growth, not the manual split
decision.
</para>
<para> If you pre-split your regions too thin, you can increase the major compaction
interval by configuring <varname>HConstants.MAJOR_COMPACTION_PERIOD</varname>. If your data size
grows too large, use the (post-0.90.0 HBase) <classname>org.apache.hadoop.hbase.util.RegionSplitter</classname>
script to perform a network IO safe rolling split
of all regions.
</para>
</section>
<section xml:id="managed.compactions"><title>Managed Compactions</title>
<para>A common administrative technique is to manage major compactions manually, rather than letting
HBase do it. By default, <varname>HConstants.MAJOR_COMPACTION_PERIOD</varname> is one day and major compactions
may kick in when you least desire it - especially on a busy system. To turn off automatic major compactions set
the value to <varname>0</varname>.
</para>
<para>It is important to stress that major compactions are absolutely necessary for StoreFile cleanup, the only variant is when
they occur. They can be administered through the HBase shell, or via
<link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29">HBaseAdmin</link>.
</para>
<para>For more information about compactions and the compaction file selection process, see <xref linkend="compaction"/></para>
</section>
<section xml:id="spec.ex"><title>Speculative Execution</title>
<para>Speculative Execution of MapReduce tasks is on by default, and for HBase clusters it is generally advised to turn off
Speculative Execution at a system-level unless you need it for a specific case, where it can be configured per-job.
Set the properties <varname>mapred.map.tasks.speculative.execution</varname> and
<varname>mapred.reduce.tasks.speculative.execution</varname> to false.
</para>
</section>
</section>
<section xml:id="other_configuration"><title>Other Configurations</title>
<section xml:id="balancer_config"><title>Balancer</title>
<para>The balancer is a periodic operation which is run on the master to redistribute regions on the cluster. It is configured via
<varname>hbase.balancer.period</varname> and defaults to 300000 (5 minutes). </para>
<para>See <xref linkend="master.processes.loadbalancer" /> for more information on the LoadBalancer.
</para>
</section>
<section xml:id="disabling.blockcache"><title>Disabling Blockcache</title>
<para>Do not turn off block cache (You'd do it by setting <varname>hbase.block.cache.size</varname> to zero).
Currently we do not do well if you do this because the regionserver will spend all its time loading hfile
indices over and over again. If your working set it such that block cache does you no good, at least
size the block cache such that hfile indices will stay up in the cache (you can get a rough idea
on the size you need by surveying regionserver UIs; you'll see index block size accounted near the
top of the webpage).</para>
</section>
<section xml:id="nagles">
<title><link xlink:href="http://en.wikipedia.org/wiki/Nagle's_algorithm">Nagle's</link> or the small package problem</title>
<para>If a big 40ms or so occasional delay is seen in operations against HBase,
try the Nagles' setting. For example, see the user mailing list thread,
<link xlink:href="http://search-hadoop.com/m/pduLg2fydtE/Inconsistent+scan+performance+with+caching+set+&amp;subj=Re+Inconsistent+scan+performance+with+caching+set+to+1">Inconsistent scan performance with caching set to 1</link>
and the issue cited therein where setting notcpdelay improved scan speeds. You might also
see the graphs on the tail of <link xlink:href="https://issues.apache.org/jira/browse/HBASE-7008">HBASE-7008 Set scanner caching to a better default</link>
where our Lars Hofhansl tries various data sizes w/ Nagle's on and off measuring the effect.</para>
</section>
<section xml:id="mttr">
<title>Better Mean Time to Recover (MTTR)</title>
<para>This section is about configurations that will make servers come back faster after a fail.
See the Deveraj Das an Nicolas Liochon blog post
<link xlink:href="http://hortonworks.com/blog/introduction-to-hbase-mean-time-to-recover-mttr/">Introduction to HBase Mean Time to Recover (MTTR)</link>
for a brief introduction.</para>
<para>The issue <link xlink:href="https://issues.apache.org/jira/browse/HBASE-8389">HBASE-8354 forces Namenode into loop with lease recovery requests</link>
is messy but has a bunch of good discussion toward the end on low timeouts and how to effect faster recovery including citation of fixes
added to HDFS. Read the Varun Sharma comments. The below suggested configurations are Varun's suggestions distilled and tested. Make sure you are
running on a late-version HDFS so you have the fixes he refers too and himself adds to HDFS that help HBase MTTR
(e.g. HDFS-3703, HDFS-3712, and HDFS-4791 -- hadoop 2 for sure has them and late hadoop 1 has some).
Set the following in the RegionServer.
<![CDATA[<property>
<name>hbase.lease.recovery.dfs.timeout</name>
<value>23000</value>
<description>How much time we allow elapse between calls to recover lease.
Should be larger than the dfs timeout.</description>
</property>
<property>
<name>dfs.client.socket-timeout</name>
<value>10000</value>
<description>Down the DFS timeout from 60 to 10 seconds.</description>
</property>]]>
And on the namenode/datanode side, set the following to enable 'staleness' introduced in HDFS-3703, HDFS-3912.
<![CDATA[<property>
<name>dfs.client.socket-timeout</name>
<value>10000</value>
<description>Down the DFS timeout from 60 to 10 seconds.</description>
</property>
<property>
<name>dfs.datanode.socket.write.timeout</name>
<value>10000</value>
<description>Down the DFS timeout from 8 * 60 to 10 seconds.</description>
</property>
<property>
<name>ipc.client.connect.timeout</name>
<value>3000</value>
<description>Down from 60 seconds to 3.</description>
</property>
<property>
<name>ipc.client.connect.max.retries.on.timeouts</name>
<value>2</value>
<description>Down from 45 seconds to 3 (2 == 3 retries).</description>
</property>
<property>
<name>dfs.namenode.avoid.read.stale.datanode</name>
<value>true</value>
<description>Enable stale state in hdfs</description>
</property>
<property>
<name>dfs.namenode.stale.datanode.interval</name>
<value>20000</value>
<description>Down from default 30 seconds</description>
</property>
<property>
<name>dfs.namenode.avoid.write.stale.datanode</name>
<value>true</value>
<description>Enable stale state in hdfs</description>
</property>]]>
</para>
</section>
</section>
</section> <!-- important config -->
</chapter>