Google Git
Sign in
apache / hbase / refs/tags/0.92.0mvn50 / . / src / docbkx / configuration.xml
blob: 7de8a66a79a7bc4b5aff682378ada3dc256878f2 [file] [log] [blame]
<?xml version="1.0"?>
<chapter xml:id="configuration"
version="5.0" xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:m="http://www.w3.org/1998/Math/MathML"
xmlns:html="http://www.w3.org/1999/xhtml"
xmlns:db="http://docbook.org/ns/docbook">
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<title>Configuration</title>
<para>This chapter is the Not-So-Quick start guide to HBase configuration.</para>
<para>Please read this chapter carefully and ensure that all requirements have
been satisfied. Failure to do so will cause you (and us) grief debugging strange errors
and/or data loss.</para>
<para>
HBase uses the same configuration system as Hadoop.
To configure a deploy, edit a file of environment variables
in <filename>conf/hbase-env.sh</filename> -- this configuration
is used mostly by the launcher shell scripts getting the cluster
off the ground -- and then add configuration to an XML file to
do things like override HBase defaults, tell HBase what Filesystem to
use, and the location of the ZooKeeper ensemble
<footnote>
<para>
Be careful editing XML. Make sure you close all elements.
Run your file through <command>xmllint</command> or similar
to ensure well-formedness of your document after an edit session.
</para>
</footnote>
.
</para>
<para>When running in distributed mode, after you make
an edit to an HBase configuration, make sure you copy the
content of the <filename>conf</filename> directory to
all nodes of the cluster. HBase will not do this for you.
Use <command>rsync</command>.</para>
<section xml:id="java">
<title>Java</title>
<para>Just like Hadoop, HBase requires java 6 from <link
xlink:href="http://www.java.com/download/">Oracle</link>. Usually
you'll want to use the latest version available except the problematic
u18 (u24 is the latest version as of this writing).</para>
</section>
<section xml:id="os">
<title>Operating System</title>
<section xml:id="ssh">
<title>ssh</title>
<para><command>ssh</command> must be installed and
<command>sshd</command> must be running to use Hadoop's scripts to
manage remote Hadoop and HBase daemons. You must be able to ssh to all
nodes, including your local node, using passwordless login (Google
"ssh passwordless login").</para>
</section>
<section xml:id="dns">
<title>DNS</title>
<para>HBase uses the local hostname to self-report it's IP address.
Both forward and reverse DNS resolving should work.</para>
<para>If your machine has multiple interfaces, HBase will use the
interface that the primary hostname resolves to.</para>
<para>If this is insufficient, you can set
<varname>hbase.regionserver.dns.interface</varname> to indicate the
primary interface. This only works if your cluster configuration is
consistent and every host has the same network interface
configuration.</para>
<para>Another alternative is setting
<varname>hbase.regionserver.dns.nameserver</varname> to choose a
different nameserver than the system wide default.</para>
</section>
<section xml:id="ntp">
<title>NTP</title>
<para>The clocks on cluster members should be in basic alignments.
Some skew is tolerable but wild skew could generate odd behaviors. Run
<link
xlink:href="http://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</link>
on your cluster, or an equivalent.</para>
<para>If you are having problems querying data, or "weird" cluster
operations, check system time!</para>
</section>
<section xml:id="ulimit">
<title>
<varname>ulimit</varname><indexterm>
<primary>ulimit</primary>
</indexterm>
and
<varname>nproc</varname><indexterm>
<primary>nproc</primary>
</indexterm>
</title>
<para>HBase is a database. It uses a lot of files all at the same time.
The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems
is insufficient (On mac os x its 256). Any significant amount of loading will
lead you to <xref linkend="trouble.rs.runtime.filehandles"/>.
You may also notice errors such as... <programlisting>
2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
</programlisting> Do yourself a favor and change the upper bound on the
number of file descriptors. Set it to north of 10k. The math runs roughly as follows: per ColumnFamily
there is at least one StoreFile and possibly up to 5 or 6 if the region is under load. Multiply the
average number of StoreFiles per ColumnFamily times the number of regions per RegionServer. For example, assuming
that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily,
and there are 100 regions per RegionServer, the JVM will open 3 * 3 * 100 = 900 file descriptors
(not counting open jar files, config files, etc.)
</para>
<para>You should also up the hbase users'
<varname>nproc</varname> setting; under load, a low-nproc
setting could manifest as <classname>OutOfMemoryError</classname>
<footnote><para>See Jack Levin's <link xlink:href="">major hdfs issues</link>
note up on the user list.</para></footnote>
<footnote><para>The requirement that a database requires upping of system limits
is not peculiar to HBase. See for example the section
<emphasis>Setting Shell Limits for the Oracle User</emphasis> in
<link xlink:href="http://www.akadia.com/services/ora_linux_install_10g.html">
Short Guide to install Oracle 10 on Linux</link>.</para></footnote>.
</para>
<para>To be clear, upping the file descriptors and nproc for the user who is
running the HBase process is an operating system configuration, not an
HBase configuration. Also, a common mistake is that administrators
will up the file descriptors for a particular user but for whatever
reason, HBase will be running as some one else. HBase prints in its
logs as the first line the ulimit its seeing. Ensure its correct.
<footnote>
<para>A useful read setting config on you hadoop cluster is Aaron
Kimballs' <link
xlink:ref="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/">Configuration
Parameters: What can you just ignore?</link></para>
</footnote></para>
<section xml:id="ulimit_ubuntu">
<title><varname>ulimit</varname> on Ubuntu</title>
<para>If you are on Ubuntu you will need to make the following
changes:</para>
<para>In the file <filename>/etc/security/limits.conf</filename> add
a line like: <programlisting>hadoop - nofile 32768</programlisting>
Replace <varname>hadoop</varname> with whatever user is running
Hadoop and HBase. If you have separate users, you will need 2
entries, one for each user. In the same file set nproc hard and soft
limits. For example: <programlisting>hadoop soft/hard nproc 32000</programlisting>.</para>
<para>In the file <filename>/etc/pam.d/common-session</filename> add
as the last line in the file: <programlisting>session required pam_limits.so</programlisting>
Otherwise the changes in <filename>/etc/security/limits.conf</filename> won't be
applied.</para>
<para>Don't forget to log out and back in again for the changes to
take effect!</para>
</section>
</section>
<section xml:id="windows">
<title>Windows</title>
<para>HBase has been little tested running on Windows. Running a
production install of HBase on top of Windows is not
recommended.</para>
<para>If you are running HBase on Windows, you must install <link
xlink:href="http://cygwin.com/">Cygwin</link> to have a *nix-like
environment for the shell scripts. The full details are explained in
the <link xlink:href="http://hbase.apache.org/cygwin.html">Windows
Installation</link> guide. Also
<link xlink:href="http://search-hadoop.com/?q=hbase+windows&amp;fc_project=HBase&amp;fc_type=mail+_hash_+dev">search our user mailing list</link> to pick
up latest fixes figured by Windows users.</para>
</section>
</section> <!-- OS -->
<section xml:id="hadoop">
<title><link
xlink:href="http://hadoop.apache.org">Hadoop</link><indexterm>
<primary>Hadoop</primary>
</indexterm></title>
<para>
This version of HBase will only run on <link
xlink:href="http://hadoop.apache.org/common/releases.html">Hadoop
0.20.x</link>. It will not run on hadoop 0.21.x (but may run on 0.22.x/0.23.x).
HBase will lose data unless it is running on an HDFS that has a durable
<code>sync</code>. Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0
DO NOT have this attribute.
Currently only Hadoop versions 0.20.205.x or any release in excess of this
version has a durable sync. You have to explicitly enable it though by
setting <varname>dfs.support.append</varname> equal to true on both
the client side -- in <filename>hbase-site.xml</filename> though it should
be on in your <filename>base-default.xml</filename> file -- and on the
serverside in <filename>hdfs-site.xml</filename> (You will have to restart
your cluster after setting this configuration). Ignore the chicken-little
comment you'll find in the <filename>hdfs-site.xml</filename> in the
description for this configuration; it says it is not enabled because there
are <quote>... bugs in the 'append code' and is not supported in any production
cluster.</quote> because it is not true (I'm sure there are bugs but the
append code has been running in production at large scale deploys and is on
by default in the offerings of hadoop by commercial vendors)
<footnote><para>Until recently only the
<link xlink:href="http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/">branch-0.20-append</link>
branch had a working sync but no official release was ever made from this branch.
You had to build it yourself. Michael Noll wrote a detailed blog,
<link xlink:href="http://www.michael-noll.com/blog/2011/04/14/building-an-hadoop-0-20-x-version-for-hbase-0-90-2/">Building
an Hadoop 0.20.x version for HBase 0.90.2</link>, on how to build an
Hadoop from branch-0.20-append. Recommended.</para></footnote>
<footnote><para>Praveen Kumar has written
a complimentary article,
<link xlink:href="http://praveen.kumar.in/2011/06/20/building-hadoop-and-hbase-for-hbase-maven-application-development/">Building Hadoop and HBase for HBase Maven application development</link>.
</para></footnote><footnote>Cloudera have <varname>dfs.support.append</varname> set to true by default.</footnote>.</para>
<para>Or use the
<link xlink:href="http://www.cloudera.com/">Cloudera</link> or
<link xlink:href="http://www.mapr.com/">MapR</link> distributions.
Cloudera' <link xlink:href="http://archive.cloudera.com/docs/">CDH3</link>
is Apache Hadoop 0.20.x plus patches including all of the
<link xlink:href="http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/">branch-0.20-append</link>
additions needed to add a durable sync. Use the released, most recent version of CDH3.</para>
<para>
<link xlink:href="http://www.mapr.com/">MapR</link>
includes a commercial, reimplementation of HDFS.
It has a durable sync as well as some other interesting features that are not
yet in Apache Hadoop. Their <link xlink:href="http://www.mapr.com/products/mapr-editions/m3-edition">M3</link>
product is free to use and unlimited.
</para>
<para>Because HBase depends on Hadoop, it bundles an instance of the
Hadoop jar under its <filename>lib</filename> directory. The bundled jar is ONLY for use in standalone mode.
In distributed mode, it is <emphasis>critical</emphasis> that the version of Hadoop that is out
on your cluster match what is under HBase. Replace the hadoop jar found in the HBase
<filename>lib</filename> directory with the hadoop jar you are running on
your cluster to avoid version mismatch issues. Make sure you
replace the jar in HBase everywhere on your cluster. Hadoop version
mismatch issues have various manifestations but often all looks like
its hung up.</para>
<section xml:id="hadoop.security">
<title>Hadoop Security</title>
<para>HBase will run on any Hadoop 0.20.x that incorporates Hadoop
security features -- e.g. Y! 0.20S or CDH3B3 -- as long as you do as
suggested above and replace the Hadoop jar that ships with HBase
with the secure version.</para>
</section>
<section xml:id="dfs.datanode.max.xcievers">
<title><varname>dfs.datanode.max.xcievers</varname><indexterm>
<primary>xcievers</primary>
</indexterm></title>
<para>An Hadoop HDFS datanode has an upper bound on the number of
files that it will serve at any one time. The upper bound parameter is
called <varname>xcievers</varname> (yes, this is misspelled). Again,
before doing any loading, make sure you have configured Hadoop's
<filename>conf/hdfs-site.xml</filename> setting the
<varname>xceivers</varname> value to at least the following:
<programlisting>
&lt;property&gt;
&lt;name&gt;dfs.datanode.max.xcievers&lt;/name&gt;
&lt;value&gt;4096&lt;/value&gt;
&lt;/property&gt;
</programlisting></para>
<para>Be sure to restart your HDFS after making the above
configuration.</para>
<para>Not having this configuration in place makes for strange looking
failures. Eventually you'll see a complain in the datanode logs
complaining about the xcievers exceeded, but on the run up to this one
manifestation is complaint about missing blocks. For example:
<code>10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block
blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node:
java.io.IOException: No live nodes contain current block. Will get new
block locations from namenode and retry...</code>
<footnote><para>See <link xlink:href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop HDFS: Deceived by Xciever</link> for an informative rant on xceivering.</para></footnote></para>
</section>
</section> <!-- hadoop -->
<section xml:id="standalone_dist">
<title>HBase run modes: Standalone and Distributed</title>
<para>HBase has two run modes: <xref linkend="standalone" /> and <xref linkend="distributed" />. Out of the box, HBase runs in
standalone mode. To set up a distributed deploy, you will need to
configure HBase by editing files in the HBase <filename>conf</filename>
directory.</para>
<para>Whatever your mode, you will need to edit
<code>conf/hbase-env.sh</code> to tell HBase which
<command>java</command> to use. In this file you set HBase environment
variables such as the heapsize and other options for the
<application>JVM</application>, the preferred location for log files,
etc. Set <varname>JAVA_HOME</varname> to point at the root of your
<command>java</command> install.</para>
<section xml:id="standalone">
<title>Standalone HBase</title>
<para>This is the default mode. Standalone mode is what is described
in the <xref linkend="quickstart" /> section. In
standalone mode, HBase does not use HDFS -- it uses the local
filesystem instead -- and it runs all HBase daemons and a local
ZooKeeper all up in the same JVM. Zookeeper binds to a well known port
so clients may talk to HBase.</para>
</section>
<section xml:id="distributed">
<title>Distributed</title>
<para>Distributed mode can be subdivided into distributed but all
daemons run on a single node -- a.k.a
<emphasis>pseudo-distributed</emphasis>-- and
<emphasis>fully-distributed</emphasis> where the daemons are spread
across all nodes in the cluster <footnote>
<para>The pseudo-distributed vs fully-distributed nomenclature
comes from Hadoop.</para>
</footnote>.</para>
<para>Distributed modes require an instance of the <emphasis>Hadoop
Distributed File System</emphasis> (HDFS). See the Hadoop <link
xlink:href="http://hadoop.apache.org/common/docs/current/api/overview-summary.html#overview_description">
requirements and instructions</link> for how to set up a HDFS. Before
proceeding, ensure you have an appropriate, working HDFS.</para>
<para>Below we describe the different distributed setups. Starting,
verification and exploration of your install, whether a
<emphasis>pseudo-distributed</emphasis> or
<emphasis>fully-distributed</emphasis> configuration is described in a
section that follows, <xref linkend="confirm" />. The same verification script applies to both
deploy types.</para>
<section xml:id="pseudo">
<title>Pseudo-distributed</title>
<para>A pseudo-distributed mode is simply a distributed mode run on
a single host. Use this configuration testing and prototyping on
HBase. Do not use this configuration for production nor for
evaluating HBase performance.</para>
<para>Once you have confirmed your HDFS setup, edit
<filename>conf/hbase-site.xml</filename>. This is the file into
which you add local customizations and overrides for
<xreg linkend="hbase_default_configurations" /> and <xref linkend="hdfs_client_conf" />. Point HBase at the running Hadoop HDFS
instance by setting the <varname>hbase.rootdir</varname> property.
This property points HBase at the Hadoop filesystem instance to use.
For example, adding the properties below to your
<filename>hbase-site.xml</filename> says that HBase should use the
<filename>/hbase</filename> directory in the HDFS whose namenode is
at port 8020 on your local machine, and that it should run with one
replica only (recommended for pseudo-distributed mode):</para>
<programlisting>
&lt;configuration&gt;
...
&lt;property&gt;
&lt;name&gt;hbase.rootdir&lt;/name&gt;
&lt;value&gt;hdfs://localhost:8020/hbase&lt;/value&gt;
&lt;description&gt;The directory shared by RegionServers.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;dfs.replication&lt;/name&gt;
&lt;value&gt;1&lt;/value&gt;
&lt;description&gt;The replication count for HLog and HFile storage. Should not be greater than HDFS datanode count.
&lt;/description&gt;
&lt;/property&gt;
...
&lt;/configuration&gt;
</programlisting>
<note>
<para>Let HBase create the <varname>hbase.rootdir</varname>
directory. If you don't, you'll get warning saying HBase needs a
migration run because the directory is missing files expected by
HBase (it'll create them if you let it).</para>
</note>
<note>
<para>Above we bind to <varname>localhost</varname>. This means
that a remote client cannot connect. Amend accordingly, if you
want to connect from a remote location.</para>
</note>
<para>Now skip to <xref linkend="confirm" /> for how to start and verify your
pseudo-distributed install. <footnote>
<para>See <link
xlink:href="http://hbase.apache.org/pseudo-distributed.html">Pseudo-distributed
mode extras</link> for notes on how to start extra Masters and
RegionServers when running pseudo-distributed.</para>
</footnote></para>
</section>
<section xml:id="fully_dist">
<title>Fully-distributed</title>
<para>For running a fully-distributed operation on more than one
host, make the following configurations. In
<filename>hbase-site.xml</filename>, add the property
<varname>hbase.cluster.distributed</varname> and set it to
<varname>true</varname> and point the HBase
<varname>hbase.rootdir</varname> at the appropriate HDFS NameNode
and location in HDFS where you would like HBase to write data. For
example, if you namenode were running at namenode.example.org on
port 8020 and you wanted to home your HBase in HDFS at
<filename>/hbase</filename>, make the following
configuration.</para>
<programlisting>
&lt;configuration&gt;
...
&lt;property&gt;
&lt;name&gt;hbase.rootdir&lt;/name&gt;
&lt;value&gt;hdfs://namenode.example.org:8020/hbase&lt;/value&gt;
&lt;description&gt;The directory shared by RegionServers.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;description&gt;The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
&lt;/description&gt;
&lt;/property&gt;
...
&lt;/configuration&gt;
</programlisting>
<section xml:id="regionserver">
<title><filename>regionservers</filename></title>
<para>In addition, a fully-distributed mode requires that you
modify <filename>conf/regionservers</filename>. The
<xref linkend="regionservers" /> file
lists all hosts that you would have running
<application>HRegionServer</application>s, one host per line (This
file in HBase is like the Hadoop <filename>slaves</filename>
file). All servers listed in this file will be started and stopped
when HBase cluster start or stop is run.</para>
</section>
<section xml:id="hbase.zookeeper">
<title>ZooKeeper and HBase</title>
<para>See section <xref linkend="zookeeper"/> for ZooKeeper setup for HBase.</para>
</section>
<section xml:id="hdfs_client_conf">
<title>HDFS Client Configuration</title>
<para>Of note, if you have made <emphasis>HDFS client
configuration</emphasis> on your Hadoop cluster -- i.e.
configuration you want HDFS clients to use as opposed to
server-side configurations -- HBase will not see this
configuration unless you do one of the following:</para>
<itemizedlist>
<listitem>
<para>Add a pointer to your <varname>HADOOP_CONF_DIR</varname>
to the <varname>HBASE_CLASSPATH</varname> environment variable
in <filename>hbase-env.sh</filename>.</para>
</listitem>
<listitem>
<para>Add a copy of <filename>hdfs-site.xml</filename> (or
<filename>hadoop-site.xml</filename>) or, better, symlinks,
under <filename>${HBASE_HOME}/conf</filename>, or</para>
</listitem>
<listitem>
<para>if only a small set of HDFS client configurations, add
them to <filename>hbase-site.xml</filename>.</para>
</listitem>
</itemizedlist>
<para>An example of such an HDFS client configuration is
<varname>dfs.replication</varname>. If for example, you want to
run with a replication factor of 5, hbase will create files with
the default of 3 unless you do the above to make the configuration
available to HBase.</para>
</section>
</section>
</section>
<section xml:id="confirm">
<title>Running and Confirming Your Installation</title>
<para>Make sure HDFS is running first. Start and stop the Hadoop HDFS
daemons by running <filename>bin/start-hdfs.sh</filename> over in the
<varname>HADOOP_HOME</varname> directory. You can ensure it started
properly by testing the <command>put</command> and
<command>get</command> of files into the Hadoop filesystem. HBase does
not normally use the mapreduce daemons. These do not need to be
started.</para>
<para><emphasis>If</emphasis> you are managing your own ZooKeeper,
start it and confirm its running else, HBase will start up ZooKeeper
for you as part of its start process.</para>
<para>Start HBase with the following command:</para>
<programlisting>bin/start-hbase.sh</programlisting>
Run the above from the
<varname>HBASE_HOME</varname>
directory.
<para>You should now have a running HBase instance. HBase logs can be
found in the <filename>logs</filename> subdirectory. Check them out
especially if HBase had trouble starting.</para>
<para>HBase also puts up a UI listing vital attributes. By default its
deployed on the Master host at port 60010 (HBase RegionServers listen
on port 60020 by default and put up an informational http server at
60030). If the Master were running on a host named
<varname>master.example.org</varname> on the default port, to see the
Master's homepage you'd point your browser at
<filename>http://master.example.org:60010</filename>.</para>
<para>Once HBase has started, see the <xref linkend="shell_exercises" /> for how to
create tables, add data, scan your insertions, and finally disable and
drop your tables.</para>
<para>To stop HBase after exiting the HBase shell enter
<programlisting>$ ./bin/stop-hbase.sh
stopping hbase...............</programlisting> Shutdown can take a moment to
complete. It can take longer if your cluster is comprised of many
machines. If you are running a distributed operation, be sure to wait
until HBase has shut down completely before stopping the Hadoop
daemons.</para>
</section>
</section> <!-- run modes -->
<section xml:id="zookeeper">
<title>ZooKeeper<indexterm>
<primary>ZooKeeper</primary>
</indexterm></title>
<para>A distributed HBase depends on a running ZooKeeper cluster.
All participating nodes and clients need to be able to access the
running ZooKeeper ensemble. HBase by default manages a ZooKeeper
"cluster" for you. It will start and stop the ZooKeeper ensemble
as part of the HBase start/stop process. You can also manage the
ZooKeeper ensemble independent of HBase and just point HBase at
the cluster it should use. To toggle HBase management of
ZooKeeper, use the <varname>HBASE_MANAGES_ZK</varname> variable in
<filename>conf/hbase-env.sh</filename>. This variable, which
defaults to <varname>true</varname>, tells HBase whether to
start/stop the ZooKeeper ensemble servers as part of HBase
start/stop.</para>
<para>When HBase manages the ZooKeeper ensemble, you can specify
ZooKeeper configuration using its native
<filename>zoo.cfg</filename> file, or, the easier option is to
just specify ZooKeeper options directly in
<filename>conf/hbase-site.xml</filename>. A ZooKeeper
configuration option can be set as a property in the HBase
<filename>hbase-site.xml</filename> XML configuration file by
prefacing the ZooKeeper option name with
<varname>hbase.zookeeper.property</varname>. For example, the
<varname>clientPort</varname> setting in ZooKeeper can be changed
by setting the
<varname>hbase.zookeeper.property.clientPort</varname> property.
For all default values used by HBase, including ZooKeeper
configuration, see <xref linkend="hbase_default_configurations" />. Look for the
<varname>hbase.zookeeper.property</varname> prefix <footnote>
<para>For the full list of ZooKeeper configurations, see
ZooKeeper's <filename>zoo.cfg</filename>. HBase does not ship
with a <filename>zoo.cfg</filename> so you will need to browse
the <filename>conf</filename> directory in an appropriate
ZooKeeper download.</para>
</footnote></para>
<para>You must at least list the ensemble servers in
<filename>hbase-site.xml</filename> using the
<varname>hbase.zookeeper.quorum</varname> property. This property
defaults to a single ensemble member at
<varname>localhost</varname> which is not suitable for a fully
distributed HBase. (It binds to the local machine only and remote
clients will not be able to connect). <note xml:id="how_many_zks">
<title>How many ZooKeepers should I run?</title>
<para>You can run a ZooKeeper ensemble that comprises 1 node
only but in production it is recommended that you run a
ZooKeeper ensemble of 3, 5 or 7 machines; the more members an
ensemble has, the more tolerant the ensemble is of host
failures. Also, run an odd number of machines. There can be no
quorum if the number of members is an even number. Give each
ZooKeeper server around 1GB of RAM, and if possible, its own
dedicated disk (A dedicated disk is the best thing you can do
to ensure a performant ZooKeeper ensemble). For very heavily
loaded clusters, run ZooKeeper servers on separate machines
from RegionServers (DataNodes and TaskTrackers).</para>
</note></para>
<para>For example, to have HBase manage a ZooKeeper quorum on
nodes <emphasis>rs{1,2,3,4,5}.example.com</emphasis>, bound to
port 2222 (the default is 2181) ensure
<varname>HBASE_MANAGE_ZK</varname> is commented out or set to
<varname>true</varname> in <filename>conf/hbase-env.sh</filename>
and then edit <filename>conf/hbase-site.xml</filename> and set
<varname>hbase.zookeeper.property.clientPort</varname> and
<varname>hbase.zookeeper.quorum</varname>. You should also set
<varname>hbase.zookeeper.property.dataDir</varname> to other than
the default as the default has ZooKeeper persist data under
<filename>/tmp</filename> which is often cleared on system
restart. In the example below we have ZooKeeper persist to
<filename>/user/local/zookeeper</filename>. <programlisting>
&lt;configuration&gt;
...
&lt;property&gt;
&lt;name&gt;hbase.zookeeper.property.clientPort&lt;/name&gt;
&lt;value&gt;2222&lt;/value&gt;
&lt;description&gt;Property from ZooKeeper's config zoo.cfg.
The port at which the clients will connect.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
&lt;value&gt;rs1.example.com,rs2.example.com,rs3.example.com,rs4.example.com,rs5.example.com&lt;/value&gt;
&lt;description&gt;Comma separated list of servers in the ZooKeeper Quorum.
For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
By default this is set to localhost for local and pseudo-distributed modes
of operation. For a fully-distributed setup, this should be set to a full
list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh
this is the list of servers which we will start/stop ZooKeeper on.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt;
&lt;value&gt;/usr/local/zookeeper&lt;/value&gt;
&lt;description&gt;Property from ZooKeeper's config zoo.cfg.
The directory where the snapshot is stored.
&lt;/description&gt;
&lt;/property&gt;
...
&lt;/configuration&gt;</programlisting></para>
<section>
<title>Using existing ZooKeeper ensemble</title>
<para>To point HBase at an existing ZooKeeper cluster, one that
is not managed by HBase, set <varname>HBASE_MANAGES_ZK</varname>
in <filename>conf/hbase-env.sh</filename> to false
<programlisting>
...
# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=false</programlisting> Next set ensemble locations
and client port, if non-standard, in
<filename>hbase-site.xml</filename>, or add a suitably
configured <filename>zoo.cfg</filename> to HBase's
<filename>CLASSPATH</filename>. HBase will prefer the
configuration found in <filename>zoo.cfg</filename> over any
settings in <filename>hbase-site.xml</filename>.</para>
<para>When HBase manages ZooKeeper, it will start/stop the
ZooKeeper servers as a part of the regular start/stop scripts.
If you would like to run ZooKeeper yourself, independent of
HBase start/stop, you would do the following</para>
<programlisting>
${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper
</programlisting>
<para>Note that you can use HBase in this manner to spin up a
ZooKeeper cluster, unrelated to HBase. Just make sure to set
<varname>HBASE_MANAGES_ZK</varname> to <varname>false</varname>
if you want it to stay up across HBase restarts so that when
HBase shuts down, it doesn't take ZooKeeper down with it.</para>
<para>For more information about running a distinct ZooKeeper
cluster, see the ZooKeeper <link
xlink:href="http://hadoop.apache.org/zookeeper/docs/current/zookeeperStarted.html">Getting
Started Guide</link>. Additionally, see the <link xlink:href="http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7">ZooKeeper Wiki</link> or the
<link xlink:href="http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup">ZooKeeper documentation</link>
for more information on ZooKeeper sizing.
</para>
</section>
</section> <!-- zookeeper -->
<section xml:id="config.files">
<title>Configuration Files</title>
<section xml:id="hbase.site">
<title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title>
<para>Just as in Hadoop where you add site-specific HDFS configuration
to the <filename>hdfs-site.xml</filename> file,
for HBase, site specific customizations go into
the file <filename>conf/hbase-site.xml</filename>.
For the list of configurable properties, see
<xref linkend="hbase_default_configurations" />
below or view the raw <filename>hbase-default.xml</filename>
source file in the HBase source code at
<filename>src/main/resources</filename>.
</para>
<para>
Not all configuration options make it out to
<filename>hbase-default.xml</filename>. Configuration
that it is thought rare anyone would change can exist only
in code; the only way to turn up such configurations is
via a reading of the source code itself.
</para>
<para>
Currently, changes here will require a cluster restart for HBase to notice the change.
</para>
<!--The file hbase-default.xml is generated as part of
the build of the hbase site. See the hbase pom.xml.
The generated file is a docbook section with a glossary
in it-->
<xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
href="../../target/site/hbase-default.xml" />
</section>
<section xml:id="hbase.env.sh">
<title><filename>hbase-env.sh</filename></title>
<para>Set HBase environment variables in this file.
Examples include options to pass the JVM on start of
an HBase daemon such as heap size and garbarge collector configs.
You can also set configurations for HBase configuration, log directories,
niceness, ssh options, where to locate process pid files,
etc. Open the file at
<filename>conf/hbase-env.sh</filename> and peruse its content.
Each option is fairly well documented. Add your own environment
variables here if you want them read by HBase daemons on startup.</para>
<para>
Changes here will require a cluster restart for HBase to notice the change.
</para>
</section>
<section xml:id="log4j">
<title><filename>log4j.properties</filename></title>
<para>Edit this file to change rate at which HBase files
are rolled and to change the level at which HBase logs messages.
</para>
<para>
Changes here will require a cluster restart for HBase to notice the change
though log levels can be changed for particular daemons via the HBase UI.
</para>
</section>
<section xml:id="client_dependencies"><title>Client configuration and dependencies connecting to an HBase cluster</title>
<para>
Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for
current critical locations. ZooKeeper is where all these values are kept. Thus clients
require the location of the ZooKeeper ensemble information before they can do anything else.
Usually this the ensemble location is kept out in the <filename>hbase-site.xml</filename> and
is picked up by the client from the <varname>CLASSPATH</varname>.</para>
<para>If you are configuring an IDE to run a HBase client, you should
include the <filename>conf/</filename> directory on your classpath so
<filename>hbase-site.xml</filename> settings can be found (or
add <filename>src/test/resources</filename> to pick up the hbase-site.xml
used by tests).
</para>
<para>
Minimally, a client of HBase needs the hbase, hadoop, log4j, commons-logging, commons-lang,
and ZooKeeper jars in its <varname>CLASSPATH</varname> connecting to a cluster.
</para>
<para>
An example basic <filename>hbase-site.xml</filename> for client only
might look as follows:
<programlisting><![CDATA[
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.zookeeper.quorum</name>
<value>example1,example2,example3</value>
<description>The directory shared by region servers.
</description>
</property>
</configuration>
]]></programlisting>
</para>
<section xml:id="java.client.config">
<title>Java client configuration</title>
<para>The configuration used by a Java client is kept
in an <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HBaseConfiguration">HBaseConfiguration</link> instance.
The factory method on HBaseConfiguration, <code>HBaseConfiguration.create();</code>,
on invocation, will read in the content of the first <filename>hbase-site.xml</filename> found on
the client's <varname>CLASSPATH</varname>, if one is present
(Invocation will also factor in any <filename>hbase-default.xml</filename> found;
an hbase-default.xml ships inside the <filename>hbase.X.X.X.jar</filename>).
It is also possible to specify configuration directly without having to read from a
<filename>hbase-site.xml</filename>. For example, to set the ZooKeeper
ensemble for the cluster programmatically do as follows:
<programlisting>Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zookeeper locally</programlisting>
If multiple ZooKeeper instances make up your ZooKeeper ensemble,
they may be specified in a comma-separated list (just as in the <filename>hbase-site.xml</filename> file).
This populated <classname>Configuration</classname> instance can then be passed to an
<link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>,
and so on.
</para>
</section>
</section>
</section> <!-- config files -->
<section xml:id="example_config">
<title>Example Configurations</title>
<section>
<title>Basic Distributed HBase Install</title>
<para>Here is an example basic configuration for a distributed ten
node cluster. The nodes are named <varname>example0</varname>,
<varname>example1</varname>, etc., through node
<varname>example9</varname> in this example. The HBase Master and the
HDFS namenode are running on the node <varname>example0</varname>.
RegionServers run on nodes
<varname>example1</varname>-<varname>example9</varname>. A 3-node
ZooKeeper ensemble runs on <varname>example1</varname>,
<varname>example2</varname>, and <varname>example3</varname> on the
default ports. ZooKeeper data is persisted to the directory
<filename>/export/zookeeper</filename>. Below we show what the main
configuration files -- <filename>hbase-site.xml</filename>,
<filename>regionservers</filename>, and
<filename>hbase-env.sh</filename> -- found in the HBase
<filename>conf</filename> directory might look like.</para>
<section xml:id="hbase_site">
<title><filename>hbase-site.xml</filename></title>
<programlisting>
&lt;?xml version="1.0"?&gt;
&lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
&lt;configuration&gt;
&lt;property&gt;
&lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
&lt;value&gt;example1,example2,example3&lt;/value&gt;
&lt;description&gt;The directory shared by RegionServers.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt;
&lt;value&gt;/export/zookeeper&lt;/value&gt;
&lt;description&gt;Property from ZooKeeper's config zoo.cfg.
The directory where the snapshot is stored.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.rootdir&lt;/name&gt;
&lt;value&gt;hdfs://example0:8020/hbase&lt;/value&gt;
&lt;description&gt;The directory shared by RegionServers.
&lt;/description&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;description&gt;The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
&lt;/description&gt;
&lt;/property&gt;
&lt;/configuration&gt;
</programlisting>
</section>
<section xml:id="regionservers">
<title><filename>regionservers</filename></title>
<para>In this file you list the nodes that will run RegionServers.
In our case we run RegionServers on all but the head node
<varname>example1</varname> which is carrying the HBase Master and
the HDFS namenode</para>
<programlisting>
example1
example3
example4
example5
example6
example7
example8
example9
</programlisting>
</section>
<section xml:id="hbase_env">
<title><filename>hbase-env.sh</filename></title>
<para>Below we use a <command>diff</command> to show the differences
from default in the <filename>hbase-env.sh</filename> file. Here we
are setting the HBase heap to be 4G instead of the default
1G.</para>
<programlisting>
$ git diff hbase-env.sh
diff --git a/conf/hbase-env.sh b/conf/hbase-env.sh
index e70ebc6..96f8c27 100644
--- a/conf/hbase-env.sh
+++ b/conf/hbase-env.sh
@@ -31,7 +31,7 @@ export JAVA_HOME=/usr/lib//jvm/java-6-sun/
# export HBASE_CLASSPATH=
# The maximum amount of heap to use, in MB. Default is 1000.
-# export HBASE_HEAPSIZE=1000
+export HBASE_HEAPSIZE=4096
# Extra Java runtime options.
# Below are what we set by default. May only work with SUN JVM.
</programlisting>
<para>Use <command>rsync</command> to copy the content of the
<filename>conf</filename> directory to all nodes of the
cluster.</para>
</section>
</section>
</section> <!-- example config -->
<section xml:id="important_configurations">
<title>The Important Configurations</title>
<para>Below we list what the <emphasis>important</emphasis>
Configurations. We've divided this section into
required configuration and worth-a-look recommended configs.
</para>
<section xml:id="required_configuration"><title>Required Configurations</title>
<para>Review the <xref linkend="os" /> and <xref linkend="hadoop" /> sections.
</para>
</section>
<section xml:id="recommended_configurations"><title>Recommended Configuations</title>
<section xml:id="zookeeper.session.timeout"><title><varname>zookeeper.session.timeout</varname></title>
<para>The default timeout is three minutes (specified in milliseconds). This means
that if a server crashes, it will be three minutes before the Master notices
the crash and starts recovery. You might like to tune the timeout down to
a minute or even less so the Master notices failures the sooner.
Before changing this value, be sure you have your JVM garbage collection
configuration under control otherwise, a long garbage collection that lasts
beyond the ZooKeeper session timeout will take out
your RegionServer (You might be fine with this -- you probably want recovery to start
on the server if a RegionServer has been in GC for a long period of time).</para>
<para>To change this configuration, edit <filename>hbase-site.xml</filename>,
copy the changed file around the cluster and restart.</para>
<para>We set this value high to save our having to field noob questions up on the mailing lists asking
why a RegionServer went down during a massive import. The usual cause is that their JVM is untuned and
they are running into long GC pauses. Our thinking is that
while users are getting familiar with HBase, we'd save them having to know all of its
intricacies. Later when they've built some confidence, then they can play
with configuration such as this.
</para>
</section>
<section xml:id="zookeeper.instances"><title>Number of ZooKeeper Instances</title>
<para>See <xref linkend="zookeeper"/>.
</para>
</section>
<section xml:id="hbase.regionserver.handler.count"><title><varname>hbase.regionserver.handler.count</varname></title>
<para>
This setting defines the number of threads that are kept open to answer
incoming requests to user tables. The default of 10 is rather low in order to
prevent users from killing their region servers when using large write buffers
with a high number of concurrent clients. The rule of thumb is to keep this
number low when the payload per request approaches the MB (big puts, scans using
a large cache) and high when the payload is small (gets, small puts, ICVs, deletes).
</para>
<para>
It is safe to set that number to the
maximum number of incoming clients if their payload is small, the typical example
being a cluster that serves a website since puts aren't typically buffered
and most of the operations are gets.
</para>
<para>
The reason why it is dangerous to keep this setting high is that the aggregate
size of all the puts that are currently happening in a region server may impose
too much pressure on its memory, or even trigger an OutOfMemoryError. A region server
running on low memory will trigger its JVM's garbage collector to run more frequently
up to a point where GC pauses become noticeable (the reason being that all the memory
used to keep all the requests' payloads cannot be trashed, no matter how hard the
garbage collector tries). After some time, the overall cluster
throughput is affected since every request that hits that region server will take longer,
which exacerbates the problem even more.
</para>
</section>
<section xml:id="big_memory">
<title>Configuration for large memory machines</title>
<para>
HBase ships with a reasonable, conservative configuration that will
work on nearly all
machine types that people might want to test with. If you have larger
machines -- HBase has 8G and larger heap -- you might the following configuration options helpful.
TODO.
</para>
</section>
<section xml:id="config.compression">
<title>Compression</title>
<para>You should consider enabling ColumnFamily compression. There are several options that are near-frictionless and in most all cases boost
performance by reducing the size of StoreFiles and thus reducing I/O.
</para>
<para>See <xref linkend="compression" /> for more information.</para>
</section>
<section xml:id="bigger.regions">
<title>Bigger Regions</title>
<para>
Consider going to larger regions to cut down on the total number of regions
on your cluster. Generally less Regions to manage makes for a smoother running
cluster (You can always later manually split the big Regions should one prove
hot and you want to spread the request load over the cluster). By default,
regions are 256MB in size. You could run with
1G. Some run with even larger regions; 4G or even larger. Adjust
<code>hbase.hregion.max.filesize</code> in your <filename>hbase-site.xml</filename>.
</para>
</section>
<section xml:id="disable.splitting">
<title>Managed Splitting</title>
<para>
Rather than let HBase auto-split your Regions, manage the splitting manually
<footnote><para>What follows is taken from the javadoc at the head of
the <classname>org.apache.hadoop.hbase.util.RegionSplitter</classname> tool
added to HBase post-0.90.0 release.
</para>
</footnote>.
With growing amounts of data, splits will continually be needed. Since
you always know exactly what regions you have, long-term debugging and
profiling is much easier with manual splits. It is hard to trace the logs to
understand region level problems if it keeps splitting and getting renamed.
Data offlining bugs + unknown number of split regions == oh crap! If an
<classname>HLog</classname> or <classname>StoreFile</classname>
was mistakenly unprocessed by HBase due to a weird bug and
you notice it a day or so later, you can be assured that the regions
specified in these files are the same as the current regions and you have
less headaches trying to restore/replay your data.
You can finely tune your compaction algorithm. With roughly uniform data
growth, it's easy to cause split / compaction storms as the regions all
roughly hit the same data size at the same time. With manual splits, you can
let staggered, time-based major compactions spread out your network IO load.
</para>
<para>
How do I turn off automatic splitting? Automatic splitting is determined by the configuration value
<code>hbase.hregion.max.filesize</code>. It is not recommended that you set this
to <varname>Long.MAX_VALUE</varname> in case you forget about manual splits. A suggested setting
is 100GB, which would result in > 1hr major compactions if reached.
</para>
<para>What's the optimal number of pre-split regions to create?
Mileage will vary depending upon your application.
You could start low with 10 pre-split regions / server and watch as data grows
over time. It's better to err on the side of too little regions and rolling split later.
A more complicated answer is that this depends upon the largest storefile
in your region. With a growing data size, this will get larger over time. You
want the largest region to be just big enough that the <classname>Store</classname> compact
selection algorithm only compacts it due to a timed major. If you don't, your
cluster can be prone to compaction storms as the algorithm decides to run
major compactions on a large series of regions all at once. Note that
compaction storms are due to the uniform data growth, not the manual split
decision.
</para>
<para> If you pre-split your regions too thin, you can increase the major compaction
interval by configuring <varname>HConstants.MAJOR_COMPACTION_PERIOD</varname>. If your data size
grows too large, use the (post-0.90.0 HBase) <classname>org.apache.hadoop.hbase.util.RegionSplitter</classname>
script to perform a network IO safe rolling split
of all regions.
</para>
</section>
<section xml:id="managed.compactions"><title>Managed Compactions</title>
<para>A common administrative technique is to manage major compactions manually, rather than letting
HBase do it. By default, <varname>HConstants.MAJOR_COMPACTION_PERIOD</varname> is one day and major compactions
may kick in when you least desire it - especially on a busy system. To "turn off" automatic major compactions set
the value to <varname>Long.MAX_VALUE</varname>.
</para>
<para>It is important to stress that major compactions are absolutely necessary for StoreFile cleanup, the only variant is when
they occur. They can be administered through the HBase shell, or via
<link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29">HBaseAdmin</link>.
</para>
</section>
</section>
<section xml:id="other_configuration"><title>Other Configurations</title>
<section xml:id="balancer_config"><title>Balancer</title>
<para>The balancer is periodic operation run on the master to redistribute regions on the cluster. It is configured via
<varname>hbase.balancer.period</varname> and defaults to 300000 (5 minutes). </para>
<para>See <xref linkend="master.processes.loadbalancer" /> for more information on the LoadBalancer.
</para>
</section>
</section>
</section> <!-- important config -->
<section xml:id="config.bloom">
<title>Bloom Filter Configuration</title>
<section>
<title><varname>io.hfile.bloom.enabled</varname> global kill
switch</title>
<para><code>io.hfile.bloom.enabled</code> in
<classname>Configuration</classname> serves as the kill switch in case
something goes wrong. Default = <varname>true</varname>.</para>
</section>
<section>
<title><varname>io.hfile.bloom.error.rate</varname></title>
<para><varname>io.hfile.bloom.error.rate</varname> = average false
positive rate. Default = 1%. Decrease rate by ½ (e.g. to .5%) == +1
bit per bloom entry.</para>
</section>
<section>
<title><varname>io.hfile.bloom.max.fold</varname></title>
<para><varname>io.hfile.bloom.max.fold</varname> = guaranteed minimum
fold rate. Most people should leave this alone. Default = 7, or can
collapse to at least 1/128th of original size. See the
<emphasis>Development Process</emphasis> section of the document <link
xlink:href="https://issues.apache.org/jira/secure/attachment/12444007/Bloom_Filters_in_HBase.pdf">BloomFilters
in HBase</link> for more on what this option means.</para>
</section>
</section>
</chapter>