| <?xml version="1.0"?> |
| <chapter xml:id="configuration" |
| version="5.0" xmlns="http://docbook.org/ns/docbook" |
| xmlns:xlink="http://www.w3.org/1999/xlink" |
| xmlns:xi="http://www.w3.org/2001/XInclude" |
| xmlns:svg="http://www.w3.org/2000/svg" |
| xmlns:m="http://www.w3.org/1998/Math/MathML" |
| xmlns:html="http://www.w3.org/1999/xhtml" |
| xmlns:db="http://docbook.org/ns/docbook"> |
| <!-- |
| /** |
| * Licensed to the Apache Software Foundation (ASF) under one |
| * or more contributor license agreements. See the NOTICE file |
| * distributed with this work for additional information |
| * regarding copyright ownership. The ASF licenses this file |
| * to you under the Apache License, Version 2.0 (the |
| * "License"); you may not use this file except in compliance |
| * with the License. You may obtain a copy of the License at |
| * |
| * http://www.apache.org/licenses/LICENSE-2.0 |
| * |
| * Unless required by applicable law or agreed to in writing, software |
| * distributed under the License is distributed on an "AS IS" BASIS, |
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| * See the License for the specific language governing permissions and |
| * limitations under the License. |
| */ |
| --> |
| <title>Apache HBase Configuration</title> |
| <para>This chapter is the Not-So-Quick start guide to Apache HBase configuration. It goes |
| over system requirements, Hadoop setup, the different Apache HBase run modes, and the |
| various configurations in HBase. Please read this chapter carefully. At a mimimum |
| ensure that all <xref linkend="basic.prerequisites" /> have |
| been satisfied. Failure to do so will cause you (and us) grief debugging strange errors |
| and/or data loss.</para> |
| |
| <para> |
| Apache HBase uses the same configuration system as Apache Hadoop. |
| To configure a deploy, edit a file of environment variables |
| in <filename>conf/hbase-env.sh</filename> -- this configuration |
| is used mostly by the launcher shell scripts getting the cluster |
| off the ground -- and then add configuration to an XML file to |
| do things like override HBase defaults, tell HBase what Filesystem to |
| use, and the location of the ZooKeeper ensemble |
| <footnote> |
| <para> |
| Be careful editing XML. Make sure you close all elements. |
| Run your file through <command>xmllint</command> or similar |
| to ensure well-formedness of your document after an edit session. |
| </para> |
| </footnote> |
| . |
| </para> |
| |
| <para>When running in distributed mode, after you make |
| an edit to an HBase configuration, make sure you copy the |
| content of the <filename>conf</filename> directory to |
| all nodes of the cluster. HBase will not do this for you. |
| Use <command>rsync</command>. For most configuration, a restart is |
| needed for servers to pick up changes (caveat dynamic config. to be described later below).</para> |
| |
| <section xml:id="basic.prerequisites"> |
| <title>Basic Prerequisites</title> |
| <para>This section lists required services and some required system configuration. |
| </para> |
| |
| <section xml:id="java"> |
| <title>Java</title> |
| <para>Just like Hadoop, HBase requires at least Java 6 from |
| <link xlink:href="http://www.java.com/download/">Oracle</link>. |
| </para> |
| </section> |
| |
| <section xml:id="os"> |
| <title>Operating System</title> |
| <section xml:id="ssh"> |
| <title>ssh</title> |
| |
| <para><command>ssh</command> must be installed and |
| <command>sshd</command> must be running to use Hadoop's scripts to |
| manage remote Hadoop and HBase daemons. You must be able to ssh to all |
| nodes, including your local node, using passwordless login (Google |
| "ssh passwordless login"). If on mac osx, see the section, |
| <link xlink:href="http://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_%28Single-Node_Cluster%29">SSH: Setting up Remote Desktop and Enabling Self-Login</link> |
| on the hadoop wiki.</para> |
| </section> |
| |
| <section xml:id="dns"> |
| <title>DNS</title> |
| |
| <para>HBase uses the local hostname to self-report its IP address. |
| Both forward and reverse DNS resolving must work in versions of |
| HBase previous to 0.92.0 |
| <footnote><para>The <link xlink:href="https://github.com/sujee/hadoop-dns-checker">hadoop-dns-checker</link> tool can be used to verify |
| DNS is working correctly on the cluster. The project README file provides detailed instructions on usage. |
| </para></footnote>.</para> |
| |
| <para>If your machine has multiple interfaces, HBase will use the |
| interface that the primary hostname resolves to.</para> |
| |
| <para>If this is insufficient, you can set |
| <varname>hbase.regionserver.dns.interface</varname> to indicate the |
| primary interface. This only works if your cluster configuration is |
| consistent and every host has the same network interface |
| configuration.</para> |
| |
| <para>Another alternative is setting |
| <varname>hbase.regionserver.dns.nameserver</varname> to choose a |
| different nameserver than the system wide default.</para> |
| </section> |
| <section xml:id="loopback.ip"> |
| <title>Loopback IP</title> |
| <para>Previous to hbase-0.96.0, HBase expects the loopback IP address to be 127.0.0.1. See <xref linkend="loopback.ip"/></para> |
| </section> |
| |
| <section xml:id="ntp"> |
| <title>NTP</title> |
| |
| <para>The clocks on cluster members should be in basic alignments. |
| Some skew is tolerable but wild skew could generate odd behaviors. Run |
| <link |
| xlink:href="http://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</link> |
| on your cluster, or an equivalent.</para> |
| |
| <para>If you are having problems querying data, or "weird" cluster |
| operations, check system time!</para> |
| </section> |
| |
| <section xml:id="ulimit"> |
| <title> |
| <varname>ulimit</varname><indexterm> |
| <primary>ulimit</primary> |
| </indexterm> |
| and |
| <varname>nproc</varname><indexterm> |
| <primary>nproc</primary> |
| </indexterm> |
| </title> |
| |
| <para>Apache HBase is a database. It uses a lot of files all at the same time. |
| The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems |
| is insufficient (On mac os x its 256). Any significant amount of loading will |
| lead you to <xref linkend="trouble.rs.runtime.filehandles"/>. |
| You may also notice errors such as... <programlisting> |
| 2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException |
| 2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901 |
| </programlisting> Do yourself a favor and change the upper bound on the |
| number of file descriptors. Set it to north of 10k. The math runs roughly as follows: per ColumnFamily |
| there is at least one StoreFile and possibly up to 5 or 6 if the region is under load. Multiply the |
| average number of StoreFiles per ColumnFamily times the number of regions per RegionServer. For example, assuming |
| that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily, |
| and there are 100 regions per RegionServer, the JVM will open 3 * 3 * 100 = 900 file descriptors |
| (not counting open jar files, config files, etc.) |
| </para> |
| <para>You should also up the hbase users' |
| <varname>nproc</varname> setting; under load, a low-nproc |
| setting could manifest as <classname>OutOfMemoryError</classname> |
| <footnote><para>See Jack Levin's <link xlink:href="">major hdfs issues</link> |
| note up on the user list.</para></footnote> |
| <footnote><para>The requirement that a database requires upping of system limits |
| is not peculiar to Apache HBase. See for example the section |
| <emphasis>Setting Shell Limits for the Oracle User</emphasis> in |
| <link xlink:href="http://www.akadia.com/services/ora_linux_install_10g.html"> |
| Short Guide to install Oracle 10 on Linux</link>.</para></footnote>. |
| </para> |
| |
| <para>To be clear, upping the file descriptors and nproc for the user who is |
| running the HBase process is an operating system configuration, not an |
| HBase configuration. Also, a common mistake is that administrators |
| will up the file descriptors for a particular user but for whatever |
| reason, HBase will be running as some one else. HBase prints in its |
| logs as the first line the ulimit its seeing. Ensure its correct. |
| <footnote> |
| <para>A useful read setting config on you hadoop cluster is Aaron |
| Kimballs' <link |
| xlink:ref="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/">Configuration |
| Parameters: What can you just ignore?</link></para> |
| </footnote></para> |
| |
| <section xml:id="ulimit_ubuntu"> |
| <title><varname>ulimit</varname> on Ubuntu</title> |
| |
| <para>If you are on Ubuntu you will need to make the following |
| changes:</para> |
| |
| <para>In the file <filename>/etc/security/limits.conf</filename> add |
| a line like: <programlisting>hadoop - nofile 32768</programlisting> |
| Replace <varname>hadoop</varname> with whatever user is running |
| Hadoop and HBase. If you have separate users, you will need 2 |
| entries, one for each user. In the same file set nproc hard and soft |
| limits. For example: <programlisting>hadoop soft/hard nproc 32000</programlisting>.</para> |
| |
| <para>In the file <filename>/etc/pam.d/common-session</filename> add |
| as the last line in the file: <programlisting>session required pam_limits.so</programlisting> |
| Otherwise the changes in <filename>/etc/security/limits.conf</filename> won't be |
| applied.</para> |
| |
| <para>Don't forget to log out and back in again for the changes to |
| take effect!</para> |
| </section> |
| </section> |
| |
| <section xml:id="windows"> |
| <title>Windows</title> |
| |
| <para>Previous to hbase-0.96.0, Apache HBase was little tested running on Windows. Running a |
| production install of HBase on top of Windows is not recommended.</para> |
| |
| <para>If you are running HBase on Windows pre-hbase-0.96.0, you must install <link |
| xlink:href="http://cygwin.com/">Cygwin</link> to have a *nix-like |
| environment for the shell scripts. The full details are explained in |
| the <link xlink:href="http://hbase.apache.org/cygwin.html">Windows |
| Installation</link> guide. Also |
| <link xlink:href="http://search-hadoop.com/?q=hbase+windows&fc_project=HBase&fc_type=mail+_hash_+dev">search our user mailing list</link> to pick |
| up latest fixes figured by Windows users.</para> |
| <para>Post-hbase-0.96.0, hbase runs natively on windows with supporting <command>*.cmd</command> scripts bundled. |
| </para> |
| </section> |
| |
| </section> <!-- OS --> |
| |
| <section xml:id="hadoop"> |
| <title><link |
| xlink:href="http://hadoop.apache.org">Hadoop</link><indexterm> |
| <primary>Hadoop</primary> |
| </indexterm></title> |
| <para>The below table shows some information about what versions of Hadoop are supported by various HBase versions. |
| Based on the version of HBase, you should select the most appropriate version of Hadoop. |
| We are not in the Hadoop distro selection business. |
| You can use Hadoop distributions from Apache, or learn about vendor distributions of |
| Hadoop at <link xlink:href="http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support"/></para> |
| <para> |
| <tip><title>Hadoop 2.x is better than Hadoop 1.x</title> |
| <para>Hadoop 2.x is faster, with more features such as short-circuit reads which will help improve your |
| HBase random read profile as well important bug fixes that will improve your overall HBase experience. |
| You should run Hadoop 2. rather than Hadoop 1. if you can. |
| </para> |
| </tip> |
| <table> |
| <title>Hadoop version support matrix</title> |
| <tgroup cols='4' align='left' colsep='1' rowsep='1'><colspec colname='c1' align='left'/><colspec colname='c2' align='center'/><colspec colname='c3' align='center'/><colspec colname='c4' align='center'/> |
| <thead> |
| <row><entry> </entry><entry>HBase-0.92.x</entry><entry>HBase-0.94.x</entry><entry>HBase-0.96.0</entry><entry>HBase-0.98.0</entry></row> |
| </thead><tbody> |
| <row><entry>Hadoop-0.20.205</entry><entry>S</entry> <entry>X</entry> <entry>X</entry><entry>X</entry></row> |
| <row><entry>Hadoop-0.22.x </entry><entry>S</entry> <entry>X</entry> <entry>X</entry><entry>X</entry></row> |
| <row><entry>Hadoop-1.0.0-1.0.2<footnote><para>HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot find KerberosUtil compiling against earlier versions of Hadoop.</para></footnote> </entry><entry>S</entry> <entry>S</entry> <entry>X</entry><entry>X</entry></row> |
| <row><entry>Hadoop-1.0.3+</entry><entry>S</entry> <entry>S</entry> <entry>S</entry><entry>X</entry></row> |
| <row><entry>Hadoop-1.1.x </entry><entry>NT</entry> <entry>S</entry> <entry>S</entry><entry>X</entry></row> |
| <row><entry>Hadoop-0.23.x </entry><entry>X</entry> <entry>S</entry> <entry>NT</entry><entry>X</entry></row> |
| <row><entry>Hadoop-2.0.x-alpha </entry><entry>X</entry> <entry>NT</entry> <entry>X</entry><entry>X</entry></row> |
| <row><entry>Hadoop-2.1.0-beta </entry><entry>X</entry> <entry>NT</entry> <entry>S</entry><entry>X</entry></row> |
| <row><entry>Hadoop-2.2.0 </entry><entry>X</entry> <entry>NT<footnote><para>To get 0.94.x to run on hadoop 2.2.0, |
| you need to change the hadoop 2 and protobuf versions in the <filename>pom.xml</filename> and then |
| build against the hadoop 2 profile by running something like the following command: |
| <programlisting>$ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests</programlisting> |
| Here is a diff with pom.xml changes: |
| <programlisting><![CDATA[$ svn diff pom.xml |
| Index: pom.xml |
| =================================================================== |
| --- pom.xml (revision 1545157) |
| +++ pom.xml (working copy) |
| @@ -1034,7 +1034,7 @@ |
| <slf4j.version>1.4.3</slf4j.version> |
| <log4j.version>1.2.16</log4j.version> |
| <mockito-all.version>1.8.5</mockito-all.version> |
| - <protobuf.version>2.4.0a</protobuf.version> |
| + <protobuf.version>2.5.0</protobuf.version> |
| <stax-api.version>1.0.1</stax-api.version> |
| <thrift.version>0.8.0</thrift.version> |
| <zookeeper.version>3.4.5</zookeeper.version> |
| @@ -2241,7 +2241,7 @@ |
| </property> |
| </activation> |
| <properties> |
| - <hadoop.version>2.0.0-alpha</hadoop.version> |
| + <hadoop.version>2.2.0</hadoop.version> |
| <slf4j.version>1.6.1</slf4j.version> |
| </properties> |
| <dependencies>]]></programlisting> |
| </para></footnote></entry> <entry>S</entry><entry>S</entry></row> |
| <row><entry>Hadoop-2.x </entry><entry>X</entry> <entry>NT</entry> <entry>S</entry><entry>S</entry></row> |
| </tbody></tgroup></table> |
| |
| Where |
| <simplelist type='vert' columns='1'> |
| <member>S = supported and tested,</member> |
| <member>X = not supported,</member> |
| <member>NT = it should run, but not tested enough.</member> |
| </simplelist> |
| </para> |
| <note xml:id="replace.hadoop"><title>Replace the Hadoop Bundled With HBase!</title> |
| <para> |
| Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its <filename>lib</filename> directory. The bundled jar is ONLY for use in standalone mode. In distributed mode, it is <emphasis>critical</emphasis> that the version of Hadoop that is out on your cluster match what is under HBase. Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up. |
| </para> |
| </note> |
| <section xml:id="hadoop.hbase-0.94"> |
| <title>Apache HBase 0.92 and 0.94</title> |
| <para>HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x, and 1.1.x. HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have to recompile the code using the specific maven profile (see top level pom.xml)</para> |
| </section> |
| |
| <section xml:id="hadoop.hbase-0.96"> |
| <title>Apache HBase 0.96</title> |
| <para> |
| As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. Hadoop 2 is strongly encouraged (faster but also has fixes that help MTTR). |
| We will no longer run properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache HBase 0.96.x if you cannot upgrade your Hadoop<footnote><para>See <link xlink:href="http://search-hadoop.com/m/7vFVx4EsUb2">HBase, mail # dev - DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?</link></para></footnote>.</para> |
| </section> |
| |
| <section xml:id="hadoop.older.versions"> |
| <title>Hadoop versions 0.20.x - 1.x</title> |
| <para> |
| HBase will lose data unless it is running on an HDFS that has a durable |
| <code>sync</code> implementation. DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0 which DO NOT have this attribute. Currently only Hadoop versions 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have a working, durable sync |
| <footnote> |
| <para>The Cloudera blog post <link xlink:href="http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/">An update on Apache Hadoop 1.0</link> |
| by Charles Zedlweski has a nice exposition on how all the Hadoop versions relate. |
| Its worth checking out if you are having trouble making sense of the |
| Hadoop version morass. |
| </para> |
| </footnote>. Sync has to be explicitly enabled by setting |
| <varname>dfs.support.append</varname> equal |
| to true on both the client side -- in <filename>hbase-site.xml</filename> |
| -- and on the serverside in <filename>hdfs-site.xml</filename> (The sync |
| facility HBase needs is a subset of the append code path). |
| <programlisting> |
| <property> |
| <name>dfs.support.append</name> |
| <value>true</value> |
| </property> |
| </programlisting> |
| You will have to restart your cluster after making this edit. Ignore the chicken-little |
| comment you'll find in the <filename>hdfs-default.xml</filename> in the |
| description for the <varname>dfs.support.append</varname> configuration. |
| </para> |
| </section> |
| <section xml:id="hadoop.security"> |
| <title>Apache HBase on Secure Hadoop</title> |
| <para>Apache HBase will run on any Hadoop 0.20.x that incorporates Hadoop |
| security features as long as you do as |
| suggested above and replace the Hadoop jar that ships with HBase |
| with the secure version. If you want to read more about how to setup |
| Secure HBase, see <xref linkend="hbase.secure.configuration" />.</para> |
| </section> |
| |
| <section xml:id="dfs.datanode.max.xcievers"> |
| <title><varname>dfs.datanode.max.xcievers</varname><indexterm> |
| <primary>xcievers</primary> |
| </indexterm></title> |
| |
| <para>An Hadoop HDFS datanode has an upper bound on the number of |
| files that it will serve at any one time. The upper bound parameter is |
| called <varname>xcievers</varname> (yes, this is misspelled). Again, |
| before doing any loading, make sure you have configured Hadoop's |
| <filename>conf/hdfs-site.xml</filename> setting the |
| <varname>xceivers</varname> value to at least the following: |
| <programlisting> |
| <property> |
| <name>dfs.datanode.max.xcievers</name> |
| <value>4096</value> |
| </property> |
| </programlisting></para> |
| |
| <para>Be sure to restart your HDFS after making the above |
| configuration.</para> |
| |
| <para>Not having this configuration in place makes for strange looking |
| failures. Eventually you'll see a complain in the datanode logs |
| complaining about the xcievers exceeded, but on the run up to this one |
| manifestation is complaint about missing blocks. For example: |
| <code>10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block |
| blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: |
| java.io.IOException: No live nodes contain current block. Will get new |
| block locations from namenode and retry...</code> |
| <footnote><para>See <link xlink:href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop HDFS: Deceived by Xciever</link> for an informative rant on xceivering.</para></footnote></para> |
| <para>See also <xref linkend="casestudies.xceivers"/> |
| </para> |
| </section> |
| |
| </section> <!-- hadoop --> |
| </section> |
| |
| <section xml:id="standalone_dist"> |
| <title>HBase run modes: Standalone and Distributed</title> |
| |
| <para>HBase has two run modes: <xref linkend="standalone" /> and <xref linkend="distributed" />. Out of the box, HBase runs in |
| standalone mode. Whatever your mode, you will need to configure HBase by editing files in the HBase <filename>conf</filename> |
| directory. At a minimum, you must edit <code>conf/hbase-env.sh</code> to tell HBase which |
| <command>java</command> to use. In this file you set HBase environment |
| variables such as the heapsize and other options for the |
| <application>JVM</application>, the preferred location for log files, |
| etc. Set <varname>JAVA_HOME</varname> to point at the root of your |
| <command>java</command> install.</para> |
| |
| <section xml:id="standalone"> |
| <title>Standalone HBase</title> |
| |
| <para>This is the default mode. Standalone mode is what is described |
| in the <xref linkend="quickstart" /> section. In |
| standalone mode, HBase does not use HDFS -- it uses the local |
| filesystem instead -- and it runs all HBase daemons and a local |
| ZooKeeper all up in the same JVM. Zookeeper binds to a well known port |
| so clients may talk to HBase.</para> |
| </section> |
| |
| <section xml:id="distributed"> |
| <title>Distributed</title> |
| |
| <para>Distributed mode can be subdivided into distributed but all |
| daemons run on a single node -- a.k.a |
| <emphasis>pseudo-distributed</emphasis>-- and |
| <emphasis>fully-distributed</emphasis> where the daemons are spread |
| across all nodes in the cluster <footnote> |
| <para>The pseudo-distributed vs fully-distributed nomenclature |
| comes from Hadoop.</para> |
| </footnote>.</para> |
| |
| <para>Pseudo-distributed mode can run against the local filesystem or |
| it can run against an instance of the <emphasis>Hadoop |
| Distributed File System</emphasis> (HDFS). Fully-distributed mode can |
| ONLY run on HDFS. See the Hadoop <link |
| xlink:href="http://hadoop.apache.org/common/docs/r1.1.1/api/overview-summary.html#overview_description"> |
| requirements and instructions</link> for how to set up HDFS.</para> |
| |
| <para>Below we describe the different distributed setups. Starting, |
| verification and exploration of your install, whether a |
| <emphasis>pseudo-distributed</emphasis> or |
| <emphasis>fully-distributed</emphasis> configuration is described in a |
| section that follows, <xref linkend="confirm" />. The same verification script applies to both |
| deploy types.</para> |
| |
| |
| |
| |
| |
| |
| <section xml:id="pseudo"> |
| <title>Pseudo-distributed</title> |
| |
| <para>A pseudo-distributed mode is simply a fully-distributed mode run on |
| a single host. Use this configuration testing and prototyping on |
| HBase. Do not use this configuration for production nor for |
| evaluating HBase performance.</para> |
| |
| <para>First, if you want to run on HDFS rather than on the local filesystem, |
| setup your HDFS. You can set up HDFS also in pseudo-distributed mode |
| (TODO: Add pointer to HOWTO doc; the hadoop site doesn't have any any more). |
| Ensure you have a working HDFS before proceeding. |
| </para> |
| |
| <para>Next, configure HBase. Edit <filename>conf/hbase-site.xml</filename>. |
| This is the file into which you add local customizations and overrides. |
| At a minimum, you must tell HBase to run in (pseudo-)distributed mode rather than |
| in default standalone mode. To do this, set the <varname>hbase.cluster.distributed</varname> |
| property to true (Its default is <varname>false</varname>). The absolute bare-minimum |
| <filename>hbase-site.xml</filename> is therefore as follows: |
| <programlisting> |
| <configuration> |
| <property> |
| <name>hbase.cluster.distributed</name> |
| <value>true</value> |
| </property> |
| </configuration> |
| </programlisting> |
| With this configuration, HBase will start up an HBase Master process, a ZooKeeper server, |
| and a RegionServer process running against the |
| local filesystem writing to wherever your operating system stores temporary files into a directory |
| named <filename>hbase-YOUR_USER_NAME</filename>.</para> |
| |
| <para>Such a setup, using the local filesystem and |
| writing to the operating systems's temporary directory is an ephemeral setup; the Hadoop |
| local filesystem -- which is what HBase uses when it is writing the local filesytem does not |
| support <command>sync</command> so unless the system is shutdown properly, the data will be lost. Writing to |
| the operating system's temporary directory can also make for data loss when the machine |
| is restarted as this directory is usually cleared on reboot. For a more permanent |
| setup, see the next example where we make use of an instance of HDFS; HBase data will |
| be written to the Hadoop distributed filesystem rather than to the local filesystem's |
| tmp directory.</para> |
| <para>In this <filename>conf/hbase-site.xml</filename> example, the |
| <varname>hbase.rootdir</varname> property points to the local HDFS instance |
| homed on the node <varname>h-24-30.example.com</varname>. |
| <note> |
| <title>Let HBase create <filename>${hbase.rootdir}</filename></title> |
| <para>Let HBase create the <varname>hbase.rootdir</varname> |
| directory. If you don't, you'll get warning saying HBase needs a |
| migration run because the directory is missing files expected by |
| HBase (it'll create them if you let it).</para> |
| </note> |
| <programlisting> |
| <configuration> |
| <property> |
| <name>hbase.rootdir</name> |
| <value>hdfs://h-24-30.sfo.stumble.net:8020/hbase</value> |
| </property> |
| <property> |
| <name>hbase.cluster.distributed</name> |
| <value>true</value> |
| </property> |
| </configuration> |
| </programlisting> |
| </para> |
| <para>Now skip to <xref linkend="confirm" /> for how to start and verify your |
| pseudo-distributed install. <footnote> |
| <para>See <xref linkend="pseudo.extras">Pseudo-distributed |
| mode extras</xref> for notes on how to start extra Masters and |
| RegionServers when running pseudo-distributed.</para> |
| </footnote></para> |
| |
| <section xml:id="pseudo.extras"> |
| <title>Pseudo-distributed Extras</title> |
| |
| <section xml:id="pseudo.extras.start"> |
| <title>Startup</title> |
| <para>To start up the initial HBase cluster... |
| <programlisting>% bin/start-hbase.sh</programlisting> |
| </para> |
| <para>To start up an extra backup master(s) on the same server run... |
| <programlisting>% bin/local-master-backup.sh start 1</programlisting> |
| ... the '1' means use ports 16001 & 16011, and this backup master's |
| logfile will be at |
| <filename>logs/hbase-${USER}-1-master-${HOSTNAME}.log</filename>. |
| </para> |
| <para>To startup multiple backup masters run... <programlisting>% bin/local-master-backup.sh start 2 3</programlisting> You can start up to 9 backup masters (10 total). |
| </para> |
| <para>To start up more regionservers... |
| <programlisting>% bin/local-regionservers.sh start 1</programlisting> |
| ... where '1' means use ports 16201 & 16301 and its logfile will be at |
| `<filename>logs/hbase-${USER}-1-regionserver-${HOSTNAME}.log</filename>. |
| </para> |
| <para>To add 4 more regionservers in addition to the one you just started by running... <programlisting>% bin/local-regionservers.sh start 2 3 4 5</programlisting> |
| This supports up to 99 extra regionservers (100 total). |
| </para> |
| </section> |
| <section xml:id="pseudo.options.stop"> |
| <title>Stop</title> |
| <para>Assuming you want to stop master backup # 1, run... |
| <programlisting>% cat /tmp/hbase-${USER}-1-master.pid |xargs kill -9</programlisting> |
| Note that bin/local-master-backup.sh stop 1 will try to stop the cluster along with the master. |
| </para> |
| <para>To stop an individual regionserver, run... |
| <programlisting>% bin/local-regionservers.sh stop 1 |
| </programlisting> |
| </para> |
| </section> |
| |
| </section> |
| |
| </section> |
| |
| |
| |
| |
| |
| <section xml:id="fully_dist"> |
| <title>Fully-distributed</title> |
| |
| <para>For running a fully-distributed operation on more than one |
| host, make the following configurations. In |
| <filename>hbase-site.xml</filename>, add the property |
| <varname>hbase.cluster.distributed</varname> and set it to |
| <varname>true</varname> and point the HBase |
| <varname>hbase.rootdir</varname> at the appropriate HDFS NameNode |
| and location in HDFS where you would like HBase to write data. For |
| example, if you namenode were running at namenode.example.org on |
| port 8020 and you wanted to home your HBase in HDFS at |
| <filename>/hbase</filename>, make the following |
| configuration.</para> |
| |
| <programlisting> |
| <configuration> |
| ... |
| <property> |
| <name>hbase.rootdir</name> |
| <value>hdfs://namenode.example.org:8020/hbase</value> |
| <description>The directory shared by RegionServers. |
| </description> |
| </property> |
| <property> |
| <name>hbase.cluster.distributed</name> |
| <value>true</value> |
| <description>The mode the cluster will be in. Possible values are |
| false: standalone and pseudo-distributed setups with managed Zookeeper |
| true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) |
| </description> |
| </property> |
| ... |
| </configuration> |
| </programlisting> |
| |
| <section xml:id="regionserver"> |
| <title><filename>regionservers</filename></title> |
| |
| <para>In addition, a fully-distributed mode requires that you |
| modify <filename>conf/regionservers</filename>. The |
| <xref linkend="regionservers" /> file |
| lists all hosts that you would have running |
| <application>HRegionServer</application>s, one host per line (This |
| file in HBase is like the Hadoop <filename>slaves</filename> |
| file). All servers listed in this file will be started and stopped |
| when HBase cluster start or stop is run.</para> |
| </section> |
| |
| <section xml:id="hbase.zookeeper"> |
| <title>ZooKeeper and HBase</title> |
| <para>See section <xref linkend="zookeeper"/> for ZooKeeper setup for HBase.</para> |
| </section> |
| |
| <section xml:id="hdfs_client_conf"> |
| <title>HDFS Client Configuration</title> |
| |
| <para>Of note, if you have made <emphasis>HDFS client |
| configuration</emphasis> on your Hadoop cluster -- i.e. |
| configuration you want HDFS clients to use as opposed to |
| server-side configurations -- HBase will not see this |
| configuration unless you do one of the following:</para> |
| |
| <itemizedlist> |
| <listitem> |
| <para>Add a pointer to your <varname>HADOOP_CONF_DIR</varname> |
| to the <varname>HBASE_CLASSPATH</varname> environment variable |
| in <filename>hbase-env.sh</filename>.</para> |
| </listitem> |
| |
| <listitem> |
| <para>Add a copy of <filename>hdfs-site.xml</filename> (or |
| <filename>hadoop-site.xml</filename>) or, better, symlinks, |
| under <filename>${HBASE_HOME}/conf</filename>, or</para> |
| </listitem> |
| |
| <listitem> |
| <para>if only a small set of HDFS client configurations, add |
| them to <filename>hbase-site.xml</filename>.</para> |
| </listitem> |
| </itemizedlist> |
| |
| <para>An example of such an HDFS client configuration is |
| <varname>dfs.replication</varname>. If for example, you want to |
| run with a replication factor of 5, hbase will create files with |
| the default of 3 unless you do the above to make the configuration |
| available to HBase.</para> |
| </section> |
| </section> |
| </section> |
| |
| <section xml:id="confirm"> |
| <title>Running and Confirming Your Installation</title> |
| |
| |
| |
| <para>Make sure HDFS is running first. Start and stop the Hadoop HDFS |
| daemons by running <filename>bin/start-hdfs.sh</filename> over in the |
| <varname>HADOOP_HOME</varname> directory. You can ensure it started |
| properly by testing the <command>put</command> and |
| <command>get</command> of files into the Hadoop filesystem. HBase does |
| not normally use the mapreduce daemons. These do not need to be |
| started.</para> |
| |
| |
| |
| <para><emphasis>If</emphasis> you are managing your own ZooKeeper, |
| start it and confirm its running else, HBase will start up ZooKeeper |
| for you as part of its start process.</para> |
| |
| |
| |
| <para>Start HBase with the following command:</para> |
| |
| |
| |
| <programlisting>bin/start-hbase.sh</programlisting> |
| |
| Run the above from the |
| |
| <varname>HBASE_HOME</varname> |
| |
| directory. |
| |
| <para>You should now have a running HBase instance. HBase logs can be |
| found in the <filename>logs</filename> subdirectory. Check them out |
| especially if HBase had trouble starting.</para> |
| |
| |
| |
| <para>HBase also puts up a UI listing vital attributes. By default its |
| deployed on the Master host at port 16010 (HBase RegionServers listen |
| on port 16020 by default and put up an informational http server at |
| 16030). If the Master were running on a host named |
| <varname>master.example.org</varname> on the default port, to see the |
| Master's homepage you'd point your browser at |
| <filename>http://master.example.org:16010</filename>.</para> |
| |
| <para>Prior to HBase 0.98, the default ports the master ui was deployed |
| on port 16010, and the HBase RegionServers would listen |
| on port 16020 by default and put up an informational http server at |
| 16030. |
| </para> |
| |
| <para>Once HBase has started, see the <xref linkend="shell_exercises" /> for how to |
| create tables, add data, scan your insertions, and finally disable and |
| drop your tables.</para> |
| |
| |
| |
| <para>To stop HBase after exiting the HBase shell enter |
| <programlisting>$ ./bin/stop-hbase.sh |
| stopping hbase...............</programlisting> Shutdown can take a moment to |
| complete. It can take longer if your cluster is comprised of many |
| machines. If you are running a distributed operation, be sure to wait |
| until HBase has shut down completely before stopping the Hadoop |
| daemons.</para> |
| |
| |
| </section> |
| </section> <!-- run modes --> |
| |
| |
| |
| <section xml:id="config.files"> |
| <title>Configuration Files</title> |
| |
| <section xml:id="hbase.site"> |
| <title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title> |
| <para>Just as in Hadoop where you add site-specific HDFS configuration |
| to the <filename>hdfs-site.xml</filename> file, |
| for HBase, site specific customizations go into |
| the file <filename>conf/hbase-site.xml</filename>. |
| For the list of configurable properties, see |
| <xref linkend="hbase_default_configurations" /> |
| below or view the raw <filename>hbase-default.xml</filename> |
| source file in the HBase source code at |
| <filename>src/main/resources</filename>. |
| </para> |
| <para> |
| Not all configuration options make it out to |
| <filename>hbase-default.xml</filename>. Configuration |
| that it is thought rare anyone would change can exist only |
| in code; the only way to turn up such configurations is |
| via a reading of the source code itself. |
| </para> |
| <para> |
| Currently, changes here will require a cluster restart for HBase to notice the change. |
| </para> |
| <!--The file hbase-default.xml is generated as part of |
| the build of the hbase site. See the hbase pom.xml. |
| The generated file is a docbook section with a glossary |
| in it--> |
| <!--presumes the pre-site target has put the hbase-default.xml at this location--> |
| <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="../../../target/docbkx/hbase-default.xml" /> |
| </section> |
| |
| <section xml:id="hbase.env.sh"> |
| <title><filename>hbase-env.sh</filename></title> |
| <para>Set HBase environment variables in this file. |
| Examples include options to pass the JVM on start of |
| an HBase daemon such as heap size and garbarge collector configs. |
| You can also set configurations for HBase configuration, log directories, |
| niceness, ssh options, where to locate process pid files, |
| etc. Open the file at |
| <filename>conf/hbase-env.sh</filename> and peruse its content. |
| Each option is fairly well documented. Add your own environment |
| variables here if you want them read by HBase daemons on startup.</para> |
| <para> |
| Changes here will require a cluster restart for HBase to notice the change. |
| </para> |
| </section> |
| |
| <section xml:id="log4j"> |
| <title><filename>log4j.properties</filename></title> |
| <para>Edit this file to change rate at which HBase files |
| are rolled and to change the level at which HBase logs messages. |
| </para> |
| <para> |
| Changes here will require a cluster restart for HBase to notice the change |
| though log levels can be changed for particular daemons via the HBase UI. |
| </para> |
| </section> |
| |
| <section xml:id="client_dependencies"><title>Client configuration and dependencies connecting to an HBase cluster</title> |
| <para>If you are running HBase in standalone mode, you don't need to configure anything for your client to work |
| provided that they are all on the same machine.</para> |
| <para> |
| Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for |
| current critical locations. ZooKeeper is where all these values are kept. Thus clients |
| require the location of the ZooKeeper ensemble information before they can do anything else. |
| Usually this the ensemble location is kept out in the <filename>hbase-site.xml</filename> and |
| is picked up by the client from the <varname>CLASSPATH</varname>.</para> |
| |
| <para>If you are configuring an IDE to run a HBase client, you should |
| include the <filename>conf/</filename> directory on your classpath so |
| <filename>hbase-site.xml</filename> settings can be found (or |
| add <filename>src/test/resources</filename> to pick up the hbase-site.xml |
| used by tests). |
| </para> |
| <para> |
| Minimally, a client of HBase needs several libraries in its <varname>CLASSPATH</varname> when connecting to a cluster, including: |
| <programlisting> |
| commons-configuration (commons-configuration-1.6.jar) |
| commons-lang (commons-lang-2.5.jar) |
| commons-logging (commons-logging-1.1.1.jar) |
| hadoop-core (hadoop-core-1.0.0.jar) |
| hbase (hbase-0.92.0.jar) |
| log4j (log4j-1.2.16.jar) |
| slf4j-api (slf4j-api-1.5.8.jar) |
| slf4j-log4j (slf4j-log4j12-1.5.8.jar) |
| zookeeper (zookeeper-3.4.2.jar)</programlisting> |
| </para> |
| <para> |
| An example basic <filename>hbase-site.xml</filename> for client only |
| might look as follows: |
| <programlisting><![CDATA[ |
| <?xml version="1.0"?> |
| <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> |
| <configuration> |
| <property> |
| <name>hbase.zookeeper.quorum</name> |
| <value>example1,example2,example3</value> |
| <description>The directory shared by region servers. |
| </description> |
| </property> |
| </configuration> |
| ]]></programlisting> |
| </para> |
| |
| <section xml:id="java.client.config"> |
| <title>Java client configuration</title> |
| <para>The configuration used by a Java client is kept |
| in an <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HBaseConfiguration">HBaseConfiguration</link> instance. |
| The factory method on HBaseConfiguration, <code>HBaseConfiguration.create();</code>, |
| on invocation, will read in the content of the first <filename>hbase-site.xml</filename> found on |
| the client's <varname>CLASSPATH</varname>, if one is present |
| (Invocation will also factor in any <filename>hbase-default.xml</filename> found; |
| an hbase-default.xml ships inside the <filename>hbase.X.X.X.jar</filename>). |
| It is also possible to specify configuration directly without having to read from a |
| <filename>hbase-site.xml</filename>. For example, to set the ZooKeeper |
| ensemble for the cluster programmatically do as follows: |
| <programlisting>Configuration config = HBaseConfiguration.create(); |
| config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zookeeper locally</programlisting> |
| If multiple ZooKeeper instances make up your ZooKeeper ensemble, |
| they may be specified in a comma-separated list (just as in the <filename>hbase-site.xml</filename> file). |
| This populated <classname>Configuration</classname> instance can then be passed to an |
| <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>, |
| and so on. |
| </para> |
| </section> |
| </section> |
| |
| </section> <!-- config files --> |
| |
| <section xml:id="example_config"> |
| <title>Example Configurations</title> |
| |
| <section> |
| <title>Basic Distributed HBase Install</title> |
| |
| <para>Here is an example basic configuration for a distributed ten |
| node cluster. The nodes are named <varname>example0</varname>, |
| <varname>example1</varname>, etc., through node |
| <varname>example9</varname> in this example. The HBase Master and the |
| HDFS namenode are running on the node <varname>example0</varname>. |
| RegionServers run on nodes |
| <varname>example1</varname>-<varname>example9</varname>. A 3-node |
| ZooKeeper ensemble runs on <varname>example1</varname>, |
| <varname>example2</varname>, and <varname>example3</varname> on the |
| default ports. ZooKeeper data is persisted to the directory |
| <filename>/export/zookeeper</filename>. Below we show what the main |
| configuration files -- <filename>hbase-site.xml</filename>, |
| <filename>regionservers</filename>, and |
| <filename>hbase-env.sh</filename> -- found in the HBase |
| <filename>conf</filename> directory might look like.</para> |
| |
| <section xml:id="hbase_site"> |
| <title><filename>hbase-site.xml</filename></title> |
| |
| <programlisting> |
| |
| <?xml version="1.0"?> |
| <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> |
| <configuration> |
| <property> |
| <name>hbase.zookeeper.quorum</name> |
| <value>example1,example2,example3</value> |
| <description>The directory shared by RegionServers. |
| </description> |
| </property> |
| <property> |
| <name>hbase.zookeeper.property.dataDir</name> |
| <value>/export/zookeeper</value> |
| <description>Property from ZooKeeper's config zoo.cfg. |
| The directory where the snapshot is stored. |
| </description> |
| </property> |
| <property> |
| <name>hbase.rootdir</name> |
| <value>hdfs://example0:8020/hbase</value> |
| <description>The directory shared by RegionServers. |
| </description> |
| </property> |
| <property> |
| <name>hbase.cluster.distributed</name> |
| <value>true</value> |
| <description>The mode the cluster will be in. Possible values are |
| false: standalone and pseudo-distributed setups with managed Zookeeper |
| true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) |
| </description> |
| </property> |
| </configuration> |
| |
| </programlisting> |
| </section> |
| |
| <section xml:id="regionservers"> |
| <title><filename>regionservers</filename></title> |
| |
| <para>In this file you list the nodes that will run RegionServers. |
| In our case, these nodes are <varname>example1</varname>-<varname>example9</varname>. |
| </para> |
| |
| <programlisting> |
| example1 |
| example2 |
| example3 |
| example4 |
| example5 |
| example6 |
| example7 |
| example8 |
| example9 |
| </programlisting> |
| </section> |
| |
| <section xml:id="hbase_env"> |
| <title><filename>hbase-env.sh</filename></title> |
| |
| <para>Below we use a <command>diff</command> to show the differences |
| from default in the <filename>hbase-env.sh</filename> file. Here we |
| are setting the HBase heap to be 4G instead of the default |
| 1G.</para> |
| |
| <programlisting> |
| |
| $ git diff hbase-env.sh |
| diff --git a/conf/hbase-env.sh b/conf/hbase-env.sh |
| index e70ebc6..96f8c27 100644 |
| --- a/conf/hbase-env.sh |
| +++ b/conf/hbase-env.sh |
| @@ -31,7 +31,7 @@ export JAVA_HOME=/usr/lib//jvm/java-6-sun/ |
| # export HBASE_CLASSPATH= |
| |
| # The maximum amount of heap to use, in MB. Default is 1000. |
| -# export HBASE_HEAPSIZE=1000 |
| +export HBASE_HEAPSIZE=4096 |
| |
| # Extra Java runtime options. |
| # Below are what we set by default. May only work with SUN JVM. |
| |
| </programlisting> |
| |
| <para>Use <command>rsync</command> to copy the content of the |
| <filename>conf</filename> directory to all nodes of the |
| cluster.</para> |
| </section> |
| </section> |
| </section> <!-- example config --> |
| |
| |
| <section xml:id="important_configurations"> |
| <title>The Important Configurations</title> |
| <para>Below we list what the <emphasis>important</emphasis> |
| Configurations. We've divided this section into |
| required configuration and worth-a-look recommended configs. |
| </para> |
| |
| |
| <section xml:id="required_configuration"><title>Required Configurations</title> |
| <para>Review the <xref linkend="os" /> and <xref linkend="hadoop" /> sections. |
| </para> |
| <section xml:id="big.cluster.config"><title>Big Cluster Configurations</title> |
| <para>If a cluster with a lot of regions, it is possible if an eager beaver |
| regionserver checks in soon after master start while all the rest in the |
| cluster are laggardly, this first server to checkin will be assigned all |
| regions. If lots of regions, this first server could buckle under the |
| load. To prevent the above scenario happening up the |
| <varname>hbase.master.wait.on.regionservers.mintostart</varname> from its |
| default value of 1. See |
| <link xlink:href="https://issues.apache.org/jira/browse/HBASE-6389">HBASE-6389 Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments</link> |
| for more detail. |
| </para> |
| </section> |
| <section xml:id="backup.master.fail.fast"><title>If a backup Master, making primary Master fail fast</title> |
| <para>If the primary Master loses its connection with ZooKeeper, it will fall into a loop where it |
| keeps trying to reconnect. Disable this functionality if you are running more than one Master: |
| i.e. a backup Master. Failing to do so, the dying Master may continue to receive RPCs though |
| another Master has assumed the role of primary. |
| See the configuration <xref linkend="fail.fast.expired.active.master" />. |
| |
| </para> |
| </section> |
| </section> |
| |
| <section xml:id="recommended_configurations"><title>Recommended Configurations</title> |
| <section xml:id="recommended_configurations.zk"> |
| <title>ZooKeeper Configuration</title> |
| <section xml:id="zookeeper.session.timeout"><title><varname>zookeeper.session.timeout</varname></title> |
| <para>The default timeout is three minutes (specified in milliseconds). This means |
| that if a server crashes, it will be three minutes before the Master notices |
| the crash and starts recovery. You might like to tune the timeout down to |
| a minute or even less so the Master notices failures the sooner. |
| Before changing this value, be sure you have your JVM garbage collection |
| configuration under control otherwise, a long garbage collection that lasts |
| beyond the ZooKeeper session timeout will take out |
| your RegionServer (You might be fine with this -- you probably want recovery to start |
| on the server if a RegionServer has been in GC for a long period of time).</para> |
| |
| <para>To change this configuration, edit <filename>hbase-site.xml</filename>, |
| copy the changed file around the cluster and restart.</para> |
| |
| <para>We set this value high to save our having to field noob questions up on the mailing lists asking |
| why a RegionServer went down during a massive import. The usual cause is that their JVM is untuned and |
| they are running into long GC pauses. Our thinking is that |
| while users are getting familiar with HBase, we'd save them having to know all of its |
| intricacies. Later when they've built some confidence, then they can play |
| with configuration such as this. |
| </para> |
| </section> |
| <section xml:id="zookeeper.instances"><title>Number of ZooKeeper Instances</title> |
| <para>See <xref linkend="zookeeper"/>. |
| </para> |
| </section> |
| </section> |
| <section xml:id="recommended.configurations.hdfs"> |
| <title>HDFS Configurations</title> |
| <section xml:id="dfs.datanode.failed.volumes.tolerated"> |
| <title>dfs.datanode.failed.volumes.tolerated</title> |
| <para>This is the "...number of volumes that are allowed to fail before a datanode stops offering service. By default |
| any volume failure will cause a datanode to shutdown" from the <filename>hdfs-default.xml</filename> |
| description. If you have > three or four disks, you might want to set this to 1 or if you have many disks, |
| two or more. |
| </para> |
| </section> |
| </section> |
| <section xml:id="hbase.regionserver.handler.count"><title><varname>hbase.regionserver.handler.count</varname></title> |
| <para> |
| This setting defines the number of threads that are kept open to answer |
| incoming requests to user tables. The rule of thumb is to keep this |
| number low when the payload per request approaches the MB (big puts, scans using |
| a large cache) and high when the payload is small (gets, small puts, ICVs, deletes). |
| The total size of the queries in progress is limited by the setting |
| "ipc.server.max.callqueue.size". |
| </para> |
| <para> |
| It is safe to set that number to the |
| maximum number of incoming clients if their payload is small, the typical example |
| being a cluster that serves a website since puts aren't typically buffered |
| and most of the operations are gets. |
| </para> |
| <para> |
| The reason why it is dangerous to keep this setting high is that the aggregate |
| size of all the puts that are currently happening in a region server may impose |
| too much pressure on its memory, or even trigger an OutOfMemoryError. A region server |
| running on low memory will trigger its JVM's garbage collector to run more frequently |
| up to a point where GC pauses become noticeable (the reason being that all the memory |
| used to keep all the requests' payloads cannot be trashed, no matter how hard the |
| garbage collector tries). After some time, the overall cluster |
| throughput is affected since every request that hits that region server will take longer, |
| which exacerbates the problem even more. |
| </para> |
| <para>You can get a sense of whether you have too little or too many handlers by |
| <xref linkend="rpc.logging" /> |
| on an individual RegionServer then tailing its logs (Queued requests |
| consume memory). |
| </para> |
| </section> |
| <section xml:id="big_memory"> |
| <title>Configuration for large memory machines</title> |
| <para> |
| HBase ships with a reasonable, conservative configuration that will |
| work on nearly all |
| machine types that people might want to test with. If you have larger |
| machines -- HBase has 8G and larger heap -- you might the following configuration options helpful. |
| TODO. |
| </para> |
| |
| </section> |
| |
| <section xml:id="config.compression"> |
| <title>Compression</title> |
| <para>You should consider enabling ColumnFamily compression. There are several options that are near-frictionless and in most all cases boost |
| performance by reducing the size of StoreFiles and thus reducing I/O. |
| </para> |
| <para>See <xref linkend="compression" /> for more information.</para> |
| </section> |
| <section xml:id="config.wals"><title>Configuring the size and number of WAL files</title> |
| <para>HBase uses <xref linkend="wal" /> to recover the memstore data that has not been flushed to disk in case of an RS failure. These WAL files should be configured to be slightly smaller than HDFS block (by default, HDFS block is 64Mb and WAL file is ~60Mb).</para> |
| <para>HBase also has a limit on number of WAL files, designed to ensure there's never too much data that needs to be replayed during recovery. This limit needs to be set according to memstore configuration, so that all the necessary data would fit. It is recommended to allocated enough WAL files to store at least that much data (when all memstores are close to full). |
| For example, with 16Gb RS heap, default memstore settings (0.4), and default WAL file size (~60Mb), 16Gb*0.4/60, the starting point for WAL file count is ~109. |
| However, as all memstores are not expected to be full all the time, less WAL files can be allocated.</para> |
| </section> |
| <section xml:id="disable.splitting"> |
| <title>Managed Splitting</title> |
| <para> |
| Rather than let HBase auto-split your Regions, manage the splitting manually |
| <footnote><para>What follows is taken from the javadoc at the head of |
| the <classname>org.apache.hadoop.hbase.util.RegionSplitter</classname> tool |
| added to HBase post-0.90.0 release. |
| </para> |
| </footnote>. |
| With growing amounts of data, splits will continually be needed. Since |
| you always know exactly what regions you have, long-term debugging and |
| profiling is much easier with manual splits. It is hard to trace the logs to |
| understand region level problems if it keeps splitting and getting renamed. |
| Data offlining bugs + unknown number of split regions == oh crap! If an |
| <classname>HLog</classname> or <classname>StoreFile</classname> |
| was mistakenly unprocessed by HBase due to a weird bug and |
| you notice it a day or so later, you can be assured that the regions |
| specified in these files are the same as the current regions and you have |
| less headaches trying to restore/replay your data. |
| You can finely tune your compaction algorithm. With roughly uniform data |
| growth, it's easy to cause split / compaction storms as the regions all |
| roughly hit the same data size at the same time. With manual splits, you can |
| let staggered, time-based major compactions spread out your network IO load. |
| </para> |
| <para> |
| How do I turn off automatic splitting? Automatic splitting is determined by the configuration value |
| <code>hbase.hregion.max.filesize</code>. It is not recommended that you set this |
| to <varname>Long.MAX_VALUE</varname> in case you forget about manual splits. A suggested setting |
| is 100GB, which would result in > 1hr major compactions if reached. |
| </para> |
| <para>What's the optimal number of pre-split regions to create? |
| Mileage will vary depending upon your application. |
| You could start low with 10 pre-split regions / server and watch as data grows |
| over time. It's better to err on the side of too little regions and rolling split later. |
| A more complicated answer is that this depends upon the largest storefile |
| in your region. With a growing data size, this will get larger over time. You |
| want the largest region to be just big enough that the <classname>Store</classname> compact |
| selection algorithm only compacts it due to a timed major. If you don't, your |
| cluster can be prone to compaction storms as the algorithm decides to run |
| major compactions on a large series of regions all at once. Note that |
| compaction storms are due to the uniform data growth, not the manual split |
| decision. |
| </para> |
| <para> If you pre-split your regions too thin, you can increase the major compaction |
| interval by configuring <varname>HConstants.MAJOR_COMPACTION_PERIOD</varname>. If your data size |
| grows too large, use the (post-0.90.0 HBase) <classname>org.apache.hadoop.hbase.util.RegionSplitter</classname> |
| script to perform a network IO safe rolling split |
| of all regions. |
| </para> |
| </section> |
| <section xml:id="managed.compactions"><title>Managed Compactions</title> |
| <para>A common administrative technique is to manage major compactions manually, rather than letting |
| HBase do it. By default, <varname>HConstants.MAJOR_COMPACTION_PERIOD</varname> is one day and major compactions |
| may kick in when you least desire it - especially on a busy system. To turn off automatic major compactions set |
| the value to <varname>0</varname>. |
| </para> |
| <para>It is important to stress that major compactions are absolutely necessary for StoreFile cleanup, the only variant is when |
| they occur. They can be administered through the HBase shell, or via |
| <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29">HBaseAdmin</link>. |
| </para> |
| <para>For more information about compactions and the compaction file selection process, see <xref linkend="compaction"/></para> |
| </section> |
| |
| <section xml:id="spec.ex"><title>Speculative Execution</title> |
| <para>Speculative Execution of MapReduce tasks is on by default, and for HBase clusters it is generally advised to turn off |
| Speculative Execution at a system-level unless you need it for a specific case, where it can be configured per-job. |
| Set the properties <varname>mapred.map.tasks.speculative.execution</varname> and |
| <varname>mapred.reduce.tasks.speculative.execution</varname> to false. |
| </para> |
| </section> |
| </section> |
| |
| <section xml:id="other_configuration"><title>Other Configurations</title> |
| <section xml:id="balancer_config"><title>Balancer</title> |
| <para>The balancer is a periodic operation which is run on the master to redistribute regions on the cluster. It is configured via |
| <varname>hbase.balancer.period</varname> and defaults to 300000 (5 minutes). </para> |
| <para>See <xref linkend="master.processes.loadbalancer" /> for more information on the LoadBalancer. |
| </para> |
| </section> |
| <section xml:id="disabling.blockcache"><title>Disabling Blockcache</title> |
| <para>Do not turn off block cache (You'd do it by setting <varname>hbase.block.cache.size</varname> to zero). |
| Currently we do not do well if you do this because the regionserver will spend all its time loading hfile |
| indices over and over again. If your working set it such that block cache does you no good, at least |
| size the block cache such that hfile indices will stay up in the cache (you can get a rough idea |
| on the size you need by surveying regionserver UIs; you'll see index block size accounted near the |
| top of the webpage).</para> |
| </section> |
| <section xml:id="nagles"> |
| <title><link xlink:href="http://en.wikipedia.org/wiki/Nagle's_algorithm">Nagle's</link> or the small package problem</title> |
| <para>If a big 40ms or so occasional delay is seen in operations against HBase, |
| try the Nagles' setting. For example, see the user mailing list thread, |
| <link xlink:href="http://search-hadoop.com/m/pduLg2fydtE/Inconsistent+scan+performance+with+caching+set+&subj=Re+Inconsistent+scan+performance+with+caching+set+to+1">Inconsistent scan performance with caching set to 1</link> |
| and the issue cited therein where setting notcpdelay improved scan speeds. You might also |
| see the graphs on the tail of <link xlink:href="https://issues.apache.org/jira/browse/HBASE-7008">HBASE-7008 Set scanner caching to a better default</link> |
| where our Lars Hofhansl tries various data sizes w/ Nagle's on and off measuring the effect.</para> |
| </section> |
| <section xml:id="mttr"> |
| <title>Better Mean Time to Recover (MTTR)</title> |
| <para>This section is about configurations that will make servers come back faster after a fail. |
| See the Deveraj Das an Nicolas Liochon blog post |
| <link xlink:href="http://hortonworks.com/blog/introduction-to-hbase-mean-time-to-recover-mttr/">Introduction to HBase Mean Time to Recover (MTTR)</link> |
| for a brief introduction.</para> |
| <para>The issue <link xlink:href="https://issues.apache.org/jira/browse/HBASE-8389">HBASE-8354 forces Namenode into loop with lease recovery requests</link> |
| is messy but has a bunch of good discussion toward the end on low timeouts and how to effect faster recovery including citation of fixes |
| added to HDFS. Read the Varun Sharma comments. The below suggested configurations are Varun's suggestions distilled and tested. Make sure you are |
| running on a late-version HDFS so you have the fixes he refers too and himself adds to HDFS that help HBase MTTR |
| (e.g. HDFS-3703, HDFS-3712, and HDFS-4791 -- hadoop 2 for sure has them and late hadoop 1 has some). |
| Set the following in the RegionServer. |
| <![CDATA[<property> |
| <name>hbase.lease.recovery.dfs.timeout</name> |
| <value>23000</value> |
| <description>How much time we allow elapse between calls to recover lease. |
| Should be larger than the dfs timeout.</description> |
| </property> |
| <property> |
| <name>dfs.client.socket-timeout</name> |
| <value>10000</value> |
| <description>Down the DFS timeout from 60 to 10 seconds.</description> |
| </property>]]> |
| And on the namenode/datanode side, set the following to enable 'staleness' introduced in HDFS-3703, HDFS-3912. |
| <![CDATA[<property> |
| <name>dfs.client.socket-timeout</name> |
| <value>10000</value> |
| <description>Down the DFS timeout from 60 to 10 seconds.</description> |
| </property> |
| <property> |
| <name>dfs.datanode.socket.write.timeout</name> |
| <value>10000</value> |
| <description>Down the DFS timeout from 8 * 60 to 10 seconds.</description> |
| </property> |
| <property> |
| <name>ipc.client.connect.timeout</name> |
| <value>3000</value> |
| <description>Down from 60 seconds to 3.</description> |
| </property> |
| <property> |
| <name>ipc.client.connect.max.retries.on.timeouts</name> |
| <value>2</value> |
| <description>Down from 45 seconds to 3 (2 == 3 retries).</description> |
| </property> |
| <property> |
| <name>dfs.namenode.avoid.read.stale.datanode</name> |
| <value>true</value> |
| <description>Enable stale state in hdfs</description> |
| </property> |
| <property> |
| <name>dfs.namenode.stale.datanode.interval</name> |
| <value>20000</value> |
| <description>Down from default 30 seconds</description> |
| </property> |
| <property> |
| <name>dfs.namenode.avoid.write.stale.datanode</name> |
| <value>true</value> |
| <description>Enable stale state in hdfs</description> |
| </property>]]> |
| |
| </para> |
| </section> |
| |
| </section> |
| |
| </section> <!-- important config --> |
| |
| </chapter> |