<?xml version="1.0"?>
  <chapter xml:id="configuration"
      version="5.0" xmlns="http://docbook.org/ns/docbook"
      xmlns:xlink="http://www.w3.org/1999/xlink"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      xmlns:svg="http://www.w3.org/2000/svg"
      xmlns:m="http://www.w3.org/1998/Math/MathML"
      xmlns:html="http://www.w3.org/1999/xhtml"
      xmlns:db="http://docbook.org/ns/docbook">
<!--
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
    <title>Apache HBase Configuration</title>
    <para>This chapter is the Not-So-Quick start guide to Apache HBase configuration.  It goes
    over system requirements, Hadoop setup, the different Apache HBase run modes, and the
    various configurations in HBase.  Please read this chapter carefully.  At a mimimum
    ensure that all <xref linkend="basic.prerequisites" /> have
      been satisfied.  Failure to do so will cause you (and us) grief debugging strange errors
      and/or data loss.</para>

    <para>
        Apache HBase uses the same configuration system as Apache Hadoop.
        To configure a deploy, edit a file of environment variables
        in <filename>conf/hbase-env.sh</filename> -- this configuration
        is used mostly by the launcher shell scripts getting the cluster
        off the ground -- and then add configuration to an XML file to
        do things like override HBase defaults, tell HBase what Filesystem to
        use, and the location of the ZooKeeper ensemble
        <footnote>
<para>
Be careful editing XML.  Make sure you close all elements.
Run your file through <command>xmllint</command> or similar
to ensure well-formedness of your document after an edit session.
</para>
        </footnote>
        .
    </para>

    <para>When running in distributed mode, after you make
    an edit to an HBase configuration, make sure you copy the
    content of the <filename>conf</filename> directory to
    all nodes of the cluster.  HBase will not do this for you.
    Use <command>rsync</command>.  For most configuration, a restart is
needed for servers to pick up changes (caveat dynamic config. to be described later below).</para>

    <section xml:id="basic.prerequisites">
    <title>Basic Prerequisites</title>
    <para>This section lists required services and some required system configuration.
    </para>

    <section xml:id="java">
        <title>Java</title>
        <para>Just like Hadoop, HBase requires at least Java 6 from
        <link xlink:href="http://www.java.com/download/">Oracle</link>.
        </para>
    </section>

    <section xml:id="os">
        <title>Operating System</title>
      <section xml:id="ssh">
        <title>ssh</title>

        <para><command>ssh</command> must be installed and
        <command>sshd</command> must be running to use Hadoop's scripts to
        manage remote Hadoop and HBase daemons. You must be able to ssh to all
        nodes, including your local node, using passwordless login (Google
        "ssh passwordless login").  If on mac osx, see the section,
        <link xlink:href="http://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_%28Single-Node_Cluster%29">SSH: Setting up Remote Desktop and Enabling Self-Login</link>
        on the hadoop wiki.</para>
      </section>

      <section xml:id="dns">
        <title>DNS</title>

        <para>HBase uses the local hostname to self-report its IP address.
        Both forward and reverse DNS resolving must work in versions of
        HBase previous to 0.92.0
        <footnote><para>The <link xlink:href="https://github.com/sujee/hadoop-dns-checker">hadoop-dns-checker</link> tool can be used to verify
        DNS is working correctly on the cluster.  The project README file provides detailed instructions on usage.
</para></footnote>.</para>

        <para>If your machine has multiple interfaces, HBase will use the
        interface that the primary hostname resolves to.</para>

        <para>If this is insufficient, you can set
        <varname>hbase.regionserver.dns.interface</varname> to indicate the
        primary interface. This only works if your cluster configuration is
        consistent and every host has the same network interface
        configuration.</para>

        <para>Another alternative is setting
        <varname>hbase.regionserver.dns.nameserver</varname> to choose a
        different nameserver than the system wide default.</para>
      </section>
      <section xml:id="loopback.ip">
        <title>Loopback IP</title>
        <para>Previous to hbase-0.96.0, HBase expects the loopback IP address to be 127.0.0.1.  See <xref linkend="loopback.ip"/></para>
       </section>

      <section xml:id="ntp">
        <title>NTP</title>

        <para>The clocks on cluster members should be in basic alignments.
        Some skew is tolerable but wild skew could generate odd behaviors. Run
        <link
        xlink:href="http://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</link>
        on your cluster, or an equivalent.</para>

        <para>If you are having problems querying data, or "weird" cluster
        operations, check system time!</para>
      </section>

      <section xml:id="ulimit">
        <title>
          <varname>ulimit</varname><indexterm>
            <primary>ulimit</primary>
            </indexterm>
            and
          <varname>nproc</varname><indexterm>
            <primary>nproc</primary>
            </indexterm>
        </title>

        <para>Apache HBase is a database.  It uses a lot of files all at the same time.
        The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems
        is insufficient (On mac os x its 256). Any significant amount of loading will
        lead you to <xref linkend="trouble.rs.runtime.filehandles"/>.
        You may also notice errors such as... <programlisting>
      2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
      2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
      </programlisting> Do yourself a favor and change the upper bound on the
        number of file descriptors. Set it to north of 10k.  The math runs roughly as follows:  per ColumnFamily
        there is at least one StoreFile and possibly up to 5 or 6 if the region is under load.  Multiply the
        average number of StoreFiles per ColumnFamily times the number of regions per RegionServer.  For example, assuming
        that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily,
        and there are 100 regions per RegionServer, the JVM will open 3 * 3 * 100 = 900 file descriptors
        (not counting open jar files, config files, etc.)
        </para>
        <para>You should also up the hbase users'
        <varname>nproc</varname> setting; under load, a low-nproc
        setting could manifest as <classname>OutOfMemoryError</classname>
        <footnote><para>See Jack Levin's <link xlink:href="">major hdfs issues</link>
                note up on the user list.</para></footnote>
        <footnote><para>The requirement that a database requires upping of system limits
        is not peculiar to Apache HBase.  See for example the section
        <emphasis>Setting Shell Limits for the Oracle User</emphasis> in
        <link xlink:href="http://www.akadia.com/services/ora_linux_install_10g.html">
        Short Guide to install Oracle 10 on Linux</link>.</para></footnote>.
       </para>

        <para>To be clear, upping the file descriptors and nproc for the user who is
        running the HBase process is an operating system configuration, not an
        HBase configuration. Also, a common mistake is that administrators
        will up the file descriptors for a particular user but for whatever
        reason, HBase will be running as some one else. HBase prints in its
        logs as the first line the ulimit its seeing. Ensure its correct.
        <footnote>
            <para>A useful read setting config on you hadoop cluster is Aaron
            Kimballs' <link
            xlink:ref="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/">Configuration
            Parameters: What can you just ignore?</link></para>
          </footnote></para>

        <section xml:id="ulimit_ubuntu">
          <title><varname>ulimit</varname> on Ubuntu</title>

          <para>If you are on Ubuntu you will need to make the following
          changes:</para>

          <para>In the file <filename>/etc/security/limits.conf</filename> add
          a line like: <programlisting>hadoop  -       nofile  32768</programlisting>
          Replace <varname>hadoop</varname> with whatever user is running
          Hadoop and HBase. If you have separate users, you will need 2
          entries, one for each user.  In the same file set nproc hard and soft
          limits.  For example: <programlisting>hadoop soft/hard nproc 32000</programlisting>.</para>

          <para>In the file <filename>/etc/pam.d/common-session</filename> add
          as the last line in the file: <programlisting>session required  pam_limits.so</programlisting>
          Otherwise the changes in <filename>/etc/security/limits.conf</filename> won't be
          applied.</para>

          <para>Don't forget to log out and back in again for the changes to
          take effect!</para>
        </section>
      </section>

      <section xml:id="windows">
        <title>Windows</title>

        <para>Previous to hbase-0.96.0, Apache HBase was little tested running on Windows. Running a
        production install of HBase on top of Windows is not recommended.</para>

        <para>If you are running HBase on Windows pre-hbase-0.96.0, you must install <link
        xlink:href="http://cygwin.com/">Cygwin</link> to have a *nix-like
        environment for the shell scripts. The full details are explained in
        the <link xlink:href="http://hbase.apache.org/cygwin.html">Windows
        Installation</link> guide. Also
        <link xlink:href="http://search-hadoop.com/?q=hbase+windows&amp;fc_project=HBase&amp;fc_type=mail+_hash_+dev">search our user mailing list</link> to pick
        up latest fixes figured by Windows users.</para>
    <para>Post-hbase-0.96.0, hbase runs natively on windows with supporting <command>*.cmd</command> scripts bundled.
    </para>
      </section>

    </section>   <!--  OS -->

    <section xml:id="hadoop">
        <title><link
        xlink:href="http://hadoop.apache.org">Hadoop</link><indexterm>
            <primary>Hadoop</primary>
          </indexterm></title>
          <para>The below table shows some information about what versions of Hadoop are supported by various HBase versions.
              Based on the version of HBase, you should select the most appropriate version of Hadoop.
              We are not in the Hadoop distro selection business.
              You can use Hadoop distributions from Apache, or learn about vendor distributions of
              Hadoop at <link xlink:href="http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support"/></para>
         <para>
             <tip><title>Hadoop 2.x is better than Hadoop 1.x</title>
                 <para>Hadoop 2.x is faster, with more features such as short-circuit reads which will help improve your
                     HBase random read profile as well important bug fixes that will improve your overall HBase experience.
                     You should run Hadoop 2 rather than Hadoop 1. HBase 0.98 deprecates use of Hadoop1.
                     HBase 1.0 will not support Hadoop1.
             </para>
         </tip>
	     <table>
		 <title>Hadoop version support matrix</title>
		 <tgroup cols='6' align='left' colsep='1' rowsep='1'>
                   <colspec colname='c1' align='left'/>
                   <colspec colname='c2' align='center'/>
                   <colspec colname='c3' align='center'/>
                   <colspec colname='c4' align='center'/>
                   <colspec colname='c5' align='center'/>
                   <colspec colname='c6' align='center'/>
         <thead>
         <row><entry>               </entry><entry>HBase-0.92.x</entry><entry>HBase-0.94.x</entry><entry>HBase-0.96.x</entry><entry>HBase-0.98.x<footnote><para>Support for Hadoop 1.x is deprecated.</para></footnote></entry><entry>HBase-1.0.x<footnote><para>Hadoop 1.x is NOT supported</para></footnote></entry></row>
	     </thead><tbody>
          <row><entry>Hadoop-0.20.205</entry><entry>S</entry>          <entry>X</entry>           <entry>X</entry><entry>X</entry><entry>X</entry></row>
          <row><entry>Hadoop-0.22.x  </entry><entry>S</entry>          <entry>X</entry>           <entry>X</entry><entry>X</entry><entry>X</entry></row>
          <row><entry>Hadoop-1.0.0-1.0.2<footnote><para>HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot find KerberosUtil compiling against earlier versions of Hadoop.</para></footnote>   </entry><entry>X</entry>          <entry>X</entry>           <entry>X</entry><entry>X</entry><entry>X</entry></row>
          <row><entry>Hadoop-1.0.3+</entry><entry>S</entry>          <entry>S</entry>           <entry>S</entry><entry>X</entry><entry>X</entry></row>
          <row><entry>Hadoop-1.1.x   </entry><entry>NT</entry>         <entry>S</entry>           <entry>S</entry><entry>X</entry><entry>X</entry></row>
          <row><entry>Hadoop-0.23.x  </entry><entry>X</entry>          <entry>S</entry>           <entry>NT</entry><entry>X</entry><entry>X</entry></row>
          <row><entry>Hadoop-2.0.x-alpha     </entry><entry>X</entry>          <entry>NT</entry>           <entry>X</entry><entry>X</entry><entry>X</entry></row>
          <row><entry>Hadoop-2.1.0-beta     </entry><entry>X</entry>          <entry>NT</entry>           <entry>S</entry><entry>X</entry><entry>X</entry></row>
         <row><entry>Hadoop-2.2.0     </entry><entry>X</entry>          <entry>NT<footnote><para>To get 0.94.x to run on hadoop 2.2.0,
                         you need to change the hadoop 2 and protobuf versions in the <filename>pom.xml</filename> and then
                         build against the hadoop 2 profile by running something like the following command:
                         <programlisting>$  mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests</programlisting>
                         Here is a diff with pom.xml changes:
                         <programlisting><![CDATA[$ svn diff pom.xml
Index: pom.xml
===================================================================
--- pom.xml     (revision 1545157)
+++ pom.xml     (working copy)
@@ -1034,7 +1034,7 @@
     <slf4j.version>1.4.3</slf4j.version>
     <log4j.version>1.2.16</log4j.version>
     <mockito-all.version>1.8.5</mockito-all.version>
-    <protobuf.version>2.4.0a</protobuf.version>
+    <protobuf.version>2.5.0</protobuf.version>
     <stax-api.version>1.0.1</stax-api.version>
     <thrift.version>0.8.0</thrift.version>
     <zookeeper.version>3.4.5</zookeeper.version>
@@ -2241,7 +2241,7 @@
         </property>
       </activation>
       <properties>
-        <hadoop.version>2.0.0-alpha</hadoop.version>
+        <hadoop.version>2.2.0</hadoop.version>
         <slf4j.version>1.6.1</slf4j.version>
       </properties>
       <dependencies>]]></programlisting>
          </para></footnote></entry>           <entry>S</entry><entry>S</entry><entry>NT</entry></row>
          <row><entry>Hadoop-2.3.x</entry><entry>X</entry>          <entry>NT</entry>           <entry>S</entry><entry>S</entry><entry>NT</entry></row>
          <row><entry>Hadoop-2.4.x</entry><entry>X</entry>          <entry>NT</entry>           <entry>S</entry><entry>S</entry><entry>S</entry></row>
          <row><entry>Hadoop-2.5.x</entry><entry>X</entry>          <entry>NT</entry>           <entry>S</entry><entry>S</entry><entry>S</entry></row>

		 </tbody></tgroup></table>

        Where
		<simplelist type='vert' columns='1'>
		<member>S = supported and tested,</member>
		<member>X = not supported,</member>
		<member>NT = it should run, but not tested enough.</member>
		</simplelist>
        </para>
        <note xml:id="replace.hadoop"><title>Replace the Hadoop Bundled With HBase!</title>
        <para>
	Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its <filename>lib</filename> directory. The bundled jar is ONLY for use in standalone mode. In distributed mode, it is <emphasis>critical</emphasis> that the version of Hadoop that is out on your cluster match what is under HBase. Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up.
    </para>
</note>
    <section xml:id="hadoop.hbase-0.94">
	<title>Apache HBase 0.92 and 0.94</title>
	<para>HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x, and 1.1.x. HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have to recompile the code using the specific maven profile (see top level pom.xml)</para>
   </section>

    <section xml:id="hadoop.hbase-0.96">
	<title>Apache HBase 0.96</title>
	<para>
    As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required.  Hadoop 2 is strongly encouraged (faster but also has fixes that help MTTR).
    We will no longer run properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache HBase 0.96.x if you cannot upgrade your Hadoop<footnote><para>See <link xlink:href="http://search-hadoop.com/m/7vFVx4EsUb2">HBase, mail # dev - DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?</link></para></footnote>.</para>
   </section>

    <section xml:id="hadoop.older.versions">
	<title>Hadoop versions 0.20.x - 1.x</title>
	<para>
     HBase will lose data unless it is running on an HDFS that has a durable
        <code>sync</code> implementation.  DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0 which DO NOT have this attribute. Currently only Hadoop versions 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have a working, durable sync
          <footnote>
          <para>The Cloudera blog post <link xlink:href="http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/">An update on Apache Hadoop 1.0</link>
          by Charles Zedlweski has a nice exposition on how all the Hadoop versions relate.
          Its worth checking out if you are having trouble making sense of the
          Hadoop version morass.
          </para>
          </footnote>.  Sync has to be explicitly enabled by setting
        <varname>dfs.support.append</varname> equal
        to true on both the client side -- in <filename>hbase-site.xml</filename>
        -- and on the serverside in <filename>hdfs-site.xml</filename> (The sync
        facility HBase needs is a subset of the append code path).
        <programlisting>
  &lt;property>
    &lt;name>dfs.support.append&lt;/name>
    &lt;value>true&lt;/value>
  &lt;/property>
        </programlisting>
        You will have to restart your cluster after making this edit.  Ignore the chicken-little
        comment you'll find in the <filename>hdfs-default.xml</filename> in the
        description for the <varname>dfs.support.append</varname> configuration.
     </para>
     </section>
       <section xml:id="hadoop.security">
          <title>Apache HBase on Secure Hadoop</title>
          <para>Apache HBase will run on any Hadoop 0.20.x that incorporates Hadoop
          security features as long as you do as
          suggested above and replace the Hadoop jar that ships with HBase
          with the secure version.  If you want to read more about how to setup
          Secure HBase, see <xref linkend="hbase.secure.configuration" />.</para>
       </section>

       <section xml:id="dfs.datanode.max.xcievers">
        <title><varname>dfs.datanode.max.xcievers</varname><indexterm>
            <primary>xcievers</primary>
          </indexterm></title>

        <para>An Hadoop HDFS datanode has an upper bound on the number of
        files that it will serve at any one time. The upper bound parameter is
        called <varname>xcievers</varname> (yes, this is misspelled). Again,
        before doing any loading, make sure you have configured Hadoop's
        <filename>conf/hdfs-site.xml</filename> setting the
        <varname>xceivers</varname> value to at least the following:
        <programlisting>
      &lt;property&gt;
        &lt;name&gt;dfs.datanode.max.xcievers&lt;/name&gt;
        &lt;value&gt;4096&lt;/value&gt;
      &lt;/property&gt;
      </programlisting></para>

        <para>Be sure to restart your HDFS after making the above
        configuration.</para>

        <para>Not having this configuration in place makes for strange looking
        failures. Eventually you'll see a complain in the datanode logs
        complaining about the xcievers exceeded, but on the run up to this one
        manifestation is complaint about missing blocks. For example:
        <code>10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block
        blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node:
        java.io.IOException: No live nodes contain current block. Will get new
        block locations from namenode and retry...</code>
        <footnote><para>See <link xlink:href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop HDFS: Deceived by Xciever</link> for an informative rant on xceivering.</para></footnote></para>
       <para>See also <xref linkend="casestudies.xceivers"/>
       </para>
      </section>

     </section>    <!--  hadoop -->
     </section>

    <section xml:id="standalone_dist">
      <title>HBase run modes: Standalone and Distributed</title>

      <para>HBase has two run modes: <xref linkend="standalone" /> and <xref linkend="distributed" />. Out of the box, HBase runs in
          standalone mode.  Whatever your mode, you will need to configure HBase by editing files in the HBase <filename>conf</filename>
      directory.  At a minimum, you must edit <code>conf/hbase-env.sh</code> to tell HBase which
      <command>java</command> to use. In this file you set HBase environment
      variables such as the heapsize and other options for the
      <application>JVM</application>, the preferred location for log files,
      etc. Set <varname>JAVA_HOME</varname> to point at the root of your
      <command>java</command> install.</para>

      <section xml:id="standalone">
        <title>Standalone HBase</title>

        <para>This is the default mode. Standalone mode is what is described
            in the <xref linkend="quickstart" /> section. In
        standalone mode, HBase does not use HDFS -- it uses the local
        filesystem instead -- and it runs all HBase daemons and a local
        ZooKeeper all up in the same JVM. Zookeeper binds to a well known port
        so clients may talk to HBase.</para>
      </section>

      <section xml:id="distributed">
        <title>Distributed</title>

        <para>Distributed mode can be subdivided into distributed but all
        daemons run on a single node -- a.k.a
        <emphasis>pseudo-distributed</emphasis>-- and
        <emphasis>fully-distributed</emphasis> where the daemons are spread
        across all nodes in the cluster <footnote>
            <para>The pseudo-distributed vs fully-distributed nomenclature
            comes from Hadoop.</para>
          </footnote>.</para>

          <para>Pseudo-distributed mode can run against the local filesystem or
              it can run against an instance of the <emphasis>Hadoop
                  Distributed File System</emphasis> (HDFS). Fully-distributed mode can
              ONLY run on HDFS. See the Hadoop <link
        xlink:href="http://hadoop.apache.org/common/docs/r1.1.1/api/overview-summary.html#overview_description">
        requirements and instructions</link> for how to set up HDFS.</para>

        <para>Below we describe the different distributed setups. Starting,
        verification and exploration of your install, whether a
        <emphasis>pseudo-distributed</emphasis> or
        <emphasis>fully-distributed</emphasis> configuration is described in a
        section that follows, <xref linkend="confirm" />. The same verification script applies to both
        deploy types.</para>






        <section xml:id="pseudo">
          <title>Pseudo-distributed</title>

          <para>A pseudo-distributed mode is simply a fully-distributed mode run on
          a single host. Use this configuration testing and prototyping on
          HBase. Do not use this configuration for production nor for
          evaluating HBase performance.</para>

      <para>First, if you want to run on HDFS rather than on the local filesystem,
          setup your HDFS.  You can set up HDFS also in pseudo-distributed mode
          (TODO: Add pointer to HOWTO doc; the hadoop site doesn't have any any more).
          Ensure you have a working HDFS before proceeding.
   	      </para>

          <para>Next, configure HBase.  Edit <filename>conf/hbase-site.xml</filename>.
              This is the file into which you add local customizations and overrides.
          At a minimum, you must tell HBase to run in (pseudo-)distributed mode rather than
          in default standalone mode.  To do this, set the <varname>hbase.cluster.distributed</varname>
          property to true (Its default is <varname>false</varname>).  The absolute bare-minimum
          <filename>hbase-site.xml</filename> is therefore as follows:
<programlisting>
&lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
&lt;/configuration&gt;
</programlisting>
With this configuration, HBase will start up an HBase Master process, a ZooKeeper server,
and a RegionServer process running against the
local filesystem writing to wherever your operating system stores temporary files into a directory
named <filename>hbase-YOUR_USER_NAME</filename>.</para>

<para>Such a setup, using the local filesystem and
writing to the operating systems's temporary directory is an ephemeral setup; the Hadoop
local filesystem -- which is what HBase uses when it is writing the local filesytem does not
support <command>sync</command> so unless the system is shutdown properly, the data will be lost.  Writing to
the operating system's temporary directory can also make for data loss when the machine
is restarted as this directory is usually cleared on reboot.  For a more permanent
setup, see the next example where we make use of an instance of HDFS; HBase data will
be written to the Hadoop distributed filesystem rather than to the local filesystem's
tmp directory.</para>
<para>In this <filename>conf/hbase-site.xml</filename> example, the
<varname>hbase.rootdir</varname> property points to the local HDFS instance
homed on the node <varname>h-24-30.example.com</varname>.
          <note>
              <title>Let HBase create <filename>${hbase.rootdir}</filename></title>
            <para>Let HBase create the <varname>hbase.rootdir</varname>
            directory. If you don't, you'll get warning saying HBase needs a
            migration run because the directory is missing files expected by
            HBase (it'll create them if you let it).</para>
          </note>
<programlisting>
&lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;hbase.rootdir&lt;/name&gt;
    &lt;value&gt;hdfs://h-24-30.sfo.stumble.net:8020/hbase&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
&lt;/configuration&gt;
</programlisting>
</para>
          <para>Now skip to <xref linkend="confirm" /> for how to start and verify your
          pseudo-distributed install. <footnote>
              <para>See <xref linkend="pseudo.extras">Pseudo-distributed
              mode extras</xref> for notes on how to start extra Masters and
              RegionServers when running pseudo-distributed.</para>
            </footnote></para>

		  <section xml:id="pseudo.extras">
		    <title>Pseudo-distributed Extras</title>

		  <section xml:id="pseudo.extras.start">
		  	<title>Startup</title>
		    	<para>To start up the initial HBase cluster...
                   <programlisting>% bin/start-hbase.sh</programlisting>
                </para>
            	<para>To start up an extra backup master(s) on the same server run...
                       <programlisting>% bin/local-master-backup.sh start 1</programlisting>
                       ... the '1' means use ports 16001 &amp; 16011, and this backup master's
		       logfile will be at 
		       <filename>logs/hbase-${USER}-1-master-${HOSTNAME}.log</filename>.
                </para>
                <para>To startup multiple backup masters run... <programlisting>% bin/local-master-backup.sh start 2 3</programlisting> You can start up to 9 backup masters (10 total).
 				</para>
				<para>To start up more regionservers...
     			  <programlisting>% bin/local-regionservers.sh start 1</programlisting>
			... where '1' means use ports 16201 &amp; 16301 and its logfile will be at 
			`<filename>logs/hbase-${USER}-1-regionserver-${HOSTNAME}.log</filename>.
     			</para>
     			<para>To add 4 more regionservers in addition to the one you just started by running... <programlisting>% bin/local-regionservers.sh start 2 3 4 5</programlisting>
     			This supports up to 99 extra regionservers (100 total).
				</para>
			</section>
			<section xml:id="pseudo.options.stop">
		  	<title>Stop</title>
    			<para>Assuming you want to stop master backup # 1, run...
            	<programlisting>% cat /tmp/hbase-${USER}-1-master.pid |xargs kill -9</programlisting>
            	Note that bin/local-master-backup.sh stop 1 will try to stop the cluster along with the master.
            	</para>
            	<para>To stop an individual regionserver, run...
                	<programlisting>% bin/local-regionservers.sh stop 1
	                </programlisting>
				</para>
			</section>

		  </section>

        </section>





        <section xml:id="fully_dist">
          <title>Fully-distributed</title>

          <para>For running a fully-distributed operation on more than one
          host, make the following configurations. In
          <filename>hbase-site.xml</filename>, add the property
          <varname>hbase.cluster.distributed</varname> and set it to
          <varname>true</varname> and point the HBase
          <varname>hbase.rootdir</varname> at the appropriate HDFS NameNode
          and location in HDFS where you would like HBase to write data. For
          example, if you namenode were running at namenode.example.org on
          port 8020 and you wanted to home your HBase in HDFS at
          <filename>/hbase</filename>, make the following
          configuration.</para>

          <programlisting>
&lt;configuration&gt;
  ...
  &lt;property&gt;
    &lt;name&gt;hbase.rootdir&lt;/name&gt;
    &lt;value&gt;hdfs://namenode.example.org:8020/hbase&lt;/value&gt;
    &lt;description&gt;The directory shared by RegionServers.
    &lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
    &lt;description&gt;The mode the cluster will be in. Possible values are
      false: standalone and pseudo-distributed setups with managed Zookeeper
      true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
    &lt;/description&gt;
  &lt;/property&gt;
  ...
&lt;/configuration&gt;
</programlisting>

          <section xml:id="regionserver">
            <title><filename>regionservers</filename></title>

            <para>In addition, a fully-distributed mode requires that you
            modify <filename>conf/regionservers</filename>. The
            <xref linkend="regionservers" /> file
            lists all hosts that you would have running
            <application>HRegionServer</application>s, one host per line (This
            file in HBase is like the Hadoop <filename>slaves</filename>
            file). All servers listed in this file will be started and stopped
            when HBase cluster start or stop is run.</para>
          </section>

          <section xml:id="hbase.zookeeper">
            <title>ZooKeeper and HBase</title>
            <para>See section <xref linkend="zookeeper"/> for ZooKeeper setup for HBase.</para>
          </section>

          <section xml:id="hdfs_client_conf">
            <title>HDFS Client Configuration</title>

            <para>Of note, if you have made <emphasis>HDFS client
            configuration</emphasis> on your Hadoop cluster -- i.e.
            configuration you want HDFS clients to use as opposed to
            server-side configurations -- HBase will not see this
            configuration unless you do one of the following:</para>

            <itemizedlist>
              <listitem>
                <para>Add a pointer to your <varname>HADOOP_CONF_DIR</varname>
                to the <varname>HBASE_CLASSPATH</varname> environment variable
                in <filename>hbase-env.sh</filename>.</para>
              </listitem>

              <listitem>
                <para>Add a copy of <filename>hdfs-site.xml</filename> (or
                <filename>hadoop-site.xml</filename>) or, better, symlinks,
                under <filename>${HBASE_HOME}/conf</filename>, or</para>
              </listitem>

              <listitem>
                <para>if only a small set of HDFS client configurations, add
                them to <filename>hbase-site.xml</filename>.</para>
              </listitem>
            </itemizedlist>

            <para>An example of such an HDFS client configuration is
            <varname>dfs.replication</varname>. If for example, you want to
            run with a replication factor of 5, hbase will create files with
            the default of 3 unless you do the above to make the configuration
            available to HBase.</para>
          </section>
        </section>
      </section>

      <section xml:id="confirm">
        <title>Running and Confirming Your Installation</title>



        <para>Make sure HDFS is running first. Start and stop the Hadoop HDFS
        daemons by running <filename>bin/start-hdfs.sh</filename> over in the
        <varname>HADOOP_HOME</varname> directory. You can ensure it started
        properly by testing the <command>put</command> and
        <command>get</command> of files into the Hadoop filesystem. HBase does
        not normally use the mapreduce daemons. These do not need to be
        started.</para>



        <para><emphasis>If</emphasis> you are managing your own ZooKeeper,
        start it and confirm its running else, HBase will start up ZooKeeper
        for you as part of its start process.</para>



        <para>Start HBase with the following command:</para>



        <programlisting>bin/start-hbase.sh</programlisting>

         Run the above from the

        <varname>HBASE_HOME</varname>

         directory.

        <para>You should now have a running HBase instance. HBase logs can be
        found in the <filename>logs</filename> subdirectory. Check them out
        especially if HBase had trouble starting.</para>



        <para>HBase also puts up a UI listing vital attributes. By default its
        deployed on the Master host at port 16010 (HBase RegionServers listen
        on port 16020 by default and put up an informational http server at
        16030). If the Master were running on a host named
        <varname>master.example.org</varname> on the default port, to see the
        Master's homepage you'd point your browser at
        <filename>http://master.example.org:16010</filename>.</para>

	<para>Prior to HBase 0.98, the default ports the master ui was deployed
	on port 16010, and the HBase RegionServers would listen
        on port 16020 by default and put up an informational http server at
        16030.
	</para>

    <para>Once HBase has started, see the <xref linkend="shell_exercises" /> for how to
        create tables, add data, scan your insertions, and finally disable and
        drop your tables.</para>



        <para>To stop HBase after exiting the HBase shell enter
        <programlisting>$ ./bin/stop-hbase.sh
stopping hbase...............</programlisting> Shutdown can take a moment to
        complete. It can take longer if your cluster is comprised of many
        machines. If you are running a distributed operation, be sure to wait
        until HBase has shut down completely before stopping the Hadoop
        daemons.</para>


      </section>
     </section>    <!--  run modes -->



    <section xml:id="config.files">
         <title>Configuration Files</title>

    <section xml:id="hbase.site">
    <title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title>
    <para>Just as in Hadoop where you add site-specific HDFS configuration
    to the <filename>hdfs-site.xml</filename> file,
    for HBase, site specific customizations go into
    the file <filename>conf/hbase-site.xml</filename>.
    For the list of configurable properties, see
    <xref linkend="hbase_default_configurations" />
    below or view the raw <filename>hbase-default.xml</filename>
    source file in the HBase source code at
    <filename>src/main/resources</filename>.
    </para>
    <para>
    Not all configuration options make it out to
    <filename>hbase-default.xml</filename>.  Configuration
    that it is thought rare anyone would change can exist only
    in code; the only way to turn up such configurations is
    via a reading of the source code itself.
    </para>
      <para>
      Currently, changes here will require a cluster restart for HBase to notice the change.
      </para>
    <!--The file hbase-default.xml is generated as part of
    the build of the hbase site.  See the hbase pom.xml.
    The generated file is a docbook section with a glossary
    in it-->
    <!--presumes the pre-site target has put the hbase-default.xml at this location-->
    <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="../../../target/docbkx/hbase-default.xml" />
    </section>

      <section xml:id="hbase.env.sh">
      <title><filename>hbase-env.sh</filename></title>
      <para>Set HBase environment variables in this file.
      Examples include options to pass the JVM on start of
      an HBase daemon such as heap size and garbarge collector configs.
      You can also set configurations for HBase configuration, log directories,
      niceness, ssh options, where to locate process pid files,
      etc. Open the file at
      <filename>conf/hbase-env.sh</filename> and peruse its content.
      Each option is fairly well documented.  Add your own environment
      variables here if you want them read by HBase daemons on startup.</para>
      <para>
      Changes here will require a cluster restart for HBase to notice the change.
      </para>
      </section>

      <section xml:id="log4j">
      <title><filename>log4j.properties</filename></title>
      <para>Edit this file to change rate at which HBase files
      are rolled and to change the level at which HBase logs messages.
      </para>
      <para>
      Changes here will require a cluster restart for HBase to notice the change
      though log levels can be changed for particular daemons via the HBase UI.
      </para>
      </section>

      <section xml:id="client_dependencies"><title>Client configuration and dependencies connecting to an HBase cluster</title>
       <para>If you are running HBase in standalone mode, you don't need to configure anything for your client to work
             provided that they are all on the same machine.</para>
       <para>
          Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for
          current critical locations.  ZooKeeper is where all these values are kept.  Thus clients
          require the location of the ZooKeeper ensemble information before they can do anything else.
          Usually this the ensemble location is kept out in the <filename>hbase-site.xml</filename> and
          is picked up by the client from the <varname>CLASSPATH</varname>.</para>

        <para>If you are configuring an IDE to run a HBase client, you should
            include the <filename>conf/</filename> directory on your classpath so
            <filename>hbase-site.xml</filename> settings can be found (or
            add <filename>src/test/resources</filename> to pick up the hbase-site.xml
            used by tests).
      </para>
      <para>
          Minimally, a client of HBase needs several libraries in its <varname>CLASSPATH</varname> when connecting to a cluster, including:
          <programlisting>
commons-configuration (commons-configuration-1.6.jar)
commons-lang (commons-lang-2.5.jar)
commons-logging (commons-logging-1.1.1.jar)
hadoop-core (hadoop-core-1.0.0.jar)
hbase (hbase-0.92.0.jar)
log4j (log4j-1.2.16.jar)
slf4j-api (slf4j-api-1.5.8.jar)
slf4j-log4j (slf4j-log4j12-1.5.8.jar)
zookeeper (zookeeper-3.4.2.jar)</programlisting>
      </para>
        <para>
          An example basic <filename>hbase-site.xml</filename> for client only
          might look as follows:
          <programlisting><![CDATA[
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>example1,example2,example3</value>
    <description>The directory shared by region servers.
    </description>
  </property>
</configuration>
]]></programlisting>
        </para>

        <section xml:id="java.client.config">
        <title>Java client configuration</title>
        <para>The configuration used by a Java client is kept
        in an <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HBaseConfiguration">HBaseConfiguration</link> instance.
        The factory method on HBaseConfiguration, <code>HBaseConfiguration.create();</code>,
        on invocation, will read in the content of the first <filename>hbase-site.xml</filename> found on
        the client's <varname>CLASSPATH</varname>, if one is present
        (Invocation will also factor in any <filename>hbase-default.xml</filename> found;
        an hbase-default.xml ships inside the <filename>hbase.X.X.X.jar</filename>).
        It is also possible to specify configuration directly without having to read from a
        <filename>hbase-site.xml</filename>.  For example, to set the ZooKeeper
        ensemble for the cluster programmatically do as follows:
        <programlisting>Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "localhost");  // Here we are running zookeeper locally</programlisting>
        If multiple ZooKeeper instances make up your ZooKeeper ensemble,
        they may be specified in a comma-separated list (just as in the <filename>hbase-site.xml</filename> file).
        This populated <classname>Configuration</classname> instance can then be passed to an
        <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>,
        and so on.
        </para>
       </section>
        </section>

      </section>  <!--  config files -->

      <section xml:id="example_config">
      <title>Example Configurations</title>

      <section>
        <title>Basic Distributed HBase Install</title>

        <para>Here is an example basic configuration for a distributed ten
        node cluster. The nodes are named <varname>example0</varname>,
        <varname>example1</varname>, etc., through node
        <varname>example9</varname> in this example. The HBase Master and the
        HDFS namenode are running on the node <varname>example0</varname>.
        RegionServers run on nodes
        <varname>example1</varname>-<varname>example9</varname>. A 3-node
        ZooKeeper ensemble runs on <varname>example1</varname>,
        <varname>example2</varname>, and <varname>example3</varname> on the
        default ports. ZooKeeper data is persisted to the directory
        <filename>/export/zookeeper</filename>. Below we show what the main
        configuration files -- <filename>hbase-site.xml</filename>,
        <filename>regionservers</filename>, and
        <filename>hbase-env.sh</filename> -- found in the HBase
        <filename>conf</filename> directory might look like.</para>

        <section xml:id="hbase_site">
          <title><filename>hbase-site.xml</filename></title>

          <programlisting>

&lt;?xml version="1.0"?&gt;
&lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
&lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
    &lt;value&gt;example1,example2,example3&lt;/value&gt;
    &lt;description&gt;The directory shared by RegionServers.
    &lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt;
    &lt;value&gt;/export/zookeeper&lt;/value&gt;
    &lt;description&gt;Property from ZooKeeper's config zoo.cfg.
    The directory where the snapshot is stored.
    &lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;hbase.rootdir&lt;/name&gt;
    &lt;value&gt;hdfs://example0:8020/hbase&lt;/value&gt;
    &lt;description&gt;The directory shared by RegionServers.
    &lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
    &lt;description&gt;The mode the cluster will be in. Possible values are
      false: standalone and pseudo-distributed setups with managed Zookeeper
      true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
    &lt;/description&gt;
  &lt;/property&gt;
&lt;/configuration&gt;

    </programlisting>
        </section>

        <section xml:id="regionservers">
          <title><filename>regionservers</filename></title>

          <para>In this file you list the nodes that will run RegionServers.
          In our case, these nodes are <varname>example1</varname>-<varname>example9</varname>.
          </para>

          <programlisting>
    example1
    example2
    example3
    example4
    example5
    example6
    example7
    example8
    example9
    </programlisting>
        </section>

        <section xml:id="hbase_env">
          <title><filename>hbase-env.sh</filename></title>

          <para>Below we use a <command>diff</command> to show the differences
          from default in the <filename>hbase-env.sh</filename> file. Here we
          are setting the HBase heap to be 4G instead of the default
          1G.</para>

          <programlisting>

$ git diff hbase-env.sh
diff --git a/conf/hbase-env.sh b/conf/hbase-env.sh
index e70ebc6..96f8c27 100644
--- a/conf/hbase-env.sh
+++ b/conf/hbase-env.sh
@@ -31,7 +31,7 @@ export JAVA_HOME=/usr/lib//jvm/java-6-sun/
 # export HBASE_CLASSPATH=

 # The maximum amount of heap to use, in MB. Default is 1000.
-# export HBASE_HEAPSIZE=1000
+export HBASE_HEAPSIZE=4096

 # Extra Java runtime options.
 # Below are what we set by default.  May only work with SUN JVM.

    </programlisting>

          <para>Use <command>rsync</command> to copy the content of the
          <filename>conf</filename> directory to all nodes of the
          cluster.</para>
        </section>
      </section>
     </section>       <!-- example config -->


      <section xml:id="important_configurations">
      <title>The Important Configurations</title>
      <para>Below we list what the <emphasis>important</emphasis>
      Configurations.  We've divided this section into
      required configuration and worth-a-look recommended configs.
      </para>


      <section xml:id="required_configuration"><title>Required Configurations</title>
          <para>Review the <xref linkend="os" /> and <xref linkend="hadoop" /> sections.
      </para>
      <section xml:id="big.cluster.config"><title>Big Cluster Configurations</title>
        <para>If a cluster with a lot of regions, it is possible if an eager beaver
            regionserver checks in soon after master start while all the rest in the
            cluster are laggardly, this first server to checkin will be assigned all
            regions.  If lots of regions, this first server could buckle under the
            load.  To prevent the above scenario happening up the
            <varname>hbase.master.wait.on.regionservers.mintostart</varname> from its
            default value of 1.  See
            <link xlink:href="https://issues.apache.org/jira/browse/HBASE-6389">HBASE-6389 Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments</link>
            for more detail.
        </para>
      </section>
      <section xml:id="backup.master.fail.fast"><title>If a backup Master, making primary Master fail fast</title>
          <para>If the primary Master loses its connection with ZooKeeper, it will fall into a loop where it
              keeps trying to reconnect.  Disable this functionality if you are running more than one Master:
              i.e. a backup Master.  Failing to do so, the dying Master may continue to receive RPCs though
              another Master has assumed the role of primary.
              See the configuration <xref linkend="fail.fast.expired.active.master" />.

        </para>
      </section>
      </section>

      <section xml:id="recommended_configurations"><title>Recommended Configurations</title>
          <section xml:id="recommended_configurations.zk">
              <title>ZooKeeper Configuration</title>
          <section xml:id="zookeeper.session.timeout"><title><varname>zookeeper.session.timeout</varname></title>
          <para>The default timeout is three minutes (specified in milliseconds). This means
              that if a server crashes, it will be three minutes before the Master notices
              the crash and starts recovery. You might like to tune the timeout down to
              a minute or even less so the Master notices failures the sooner.
              Before changing this value, be sure you have your JVM garbage collection
              configuration under control otherwise, a long garbage collection that lasts
              beyond the ZooKeeper session timeout will take out
              your RegionServer (You might be fine with this -- you probably want recovery to start
          on the server if a RegionServer has been in GC for a long period of time).</para>

      <para>To change this configuration, edit <filename>hbase-site.xml</filename>,
          copy the changed file around the cluster and restart.</para>

          <para>We set this value high to save our having to field noob questions up on the mailing lists asking
              why a RegionServer went down during a massive import.  The usual cause is that their JVM is untuned and
              they are running into long GC pauses.  Our thinking is that
              while users are  getting familiar with HBase, we'd save them having to know all of its
              intricacies.  Later when they've built some confidence, then they can play
              with configuration such as this.
          </para>
      </section>
          <section xml:id="zookeeper.instances"><title>Number of ZooKeeper Instances</title>
          <para>See <xref linkend="zookeeper"/>.
          </para>
      </section>
      </section>
      <section xml:id="recommended.configurations.hdfs">
          <title>HDFS Configurations</title>
          <section xml:id="dfs.datanode.failed.volumes.tolerated">
              <title>dfs.datanode.failed.volumes.tolerated</title>
              <para>This is the "...number of volumes that are allowed to fail before a datanode stops offering service. By default
                  any volume failure will cause a datanode to shutdown" from the <filename>hdfs-default.xml</filename>
                  description.  If you have > three or four disks, you might want to set this to 1 or if you have many disks,
                  two or more.
              </para>
          </section>
      </section>
          <section xml:id="hbase.regionserver.handler.count"><title><varname>hbase.regionserver.handler.count</varname></title>
          <para>
          This setting defines the number of threads that are kept open to answer
          incoming requests to user tables. The rule of thumb is to keep this
          number low when the payload per request approaches the MB (big puts, scans using
          a large cache) and high when the payload is small (gets, small puts, ICVs, deletes).
          The total size of the queries in progress is limited  by the setting
          "ipc.server.max.callqueue.size".
          </para>
          <para>
          It is safe to set that number to the
          maximum number of incoming clients if their payload is small, the typical example
          being a cluster that serves a website since puts aren't typically buffered
          and most of the operations are gets.
          </para>
          <para>
          The reason why it is dangerous to keep this setting high is that the aggregate
          size of all the puts that are currently happening in a region server may impose
          too much pressure on its memory, or even trigger an OutOfMemoryError. A region server
          running on low memory will trigger its JVM's garbage collector to run more frequently
          up to a point where GC pauses become noticeable (the reason being that all the memory
          used to keep all the requests' payloads cannot be trashed, no matter how hard the
          garbage collector tries). After some time, the overall cluster
          throughput is affected since every request that hits that region server will take longer,
          which exacerbates the problem even more.
          </para>
          <para>You can get a sense of whether you have too little or too many handlers by
            <xref linkend="rpc.logging" />
            on an individual RegionServer then tailing its logs (Queued requests
            consume memory).
            </para>
          </section>
      <section xml:id="big_memory">
        <title>Configuration for large memory machines</title>
        <para>
          HBase ships with a reasonable, conservative configuration that will
          work on nearly all
          machine types that people might want to test with. If you have larger
          machines -- HBase has 8G and larger heap -- you might the following configuration options helpful.
          TODO.
        </para>

      </section>

      <section xml:id="config.compression">
      <title>Compression</title>
      <para>You should consider enabling ColumnFamily compression.  There are several options that are near-frictionless and in most all cases boost
      performance by reducing the size of StoreFiles and thus reducing I/O.
      </para>
      <para>See <xref linkend="compression" /> for more information.</para>
      </section>
      <section xml:id="config.wals"><title>Configuring the size and number of WAL files</title>
      <para>HBase uses <xref linkend="wal" /> to recover the memstore data that has not been flushed to disk in case of an RS failure. These WAL files should be configured to be slightly smaller than HDFS block (by default, HDFS block is 64Mb and WAL file is ~60Mb).</para>
      <para>HBase also has a limit on number of WAL files, designed to ensure there's never too much data that needs to be replayed during recovery. This limit needs to be set according to memstore configuration, so that all the necessary data would fit. It is recommended to allocated enough WAL files to store at least that much data (when all memstores are close to full).
      For example, with 16Gb RS heap, default memstore settings (0.4), and default WAL file size (~60Mb), 16Gb*0.4/60, the starting point for WAL file count is ~109.
      However, as all memstores are not expected to be full all the time, less WAL files can be allocated.</para>
      </section>
      <section xml:id="disable.splitting">
      <title>Managed Splitting</title>
      <para>
      Rather than let HBase auto-split your Regions, manage the splitting manually
      <footnote><para>What follows is taken from the javadoc at the head of
      the <classname>org.apache.hadoop.hbase.util.RegionSplitter</classname> tool
      added to HBase post-0.90.0 release.
      </para>
      </footnote>.
 With growing amounts of data, splits will continually be needed. Since
 you always know exactly what regions you have, long-term debugging and
 profiling is much easier with manual splits. It is hard to trace the logs to
 understand region level problems if it keeps splitting and getting renamed.
 Data offlining bugs + unknown number of split regions == oh crap! If an
 <classname>HLog</classname> or <classname>StoreFile</classname>
 was mistakenly unprocessed by HBase due to a weird bug and
 you notice it a day or so later, you can be assured that the regions
 specified in these files are the same as the current regions and you have
 less headaches trying to restore/replay your data.
 You can finely tune your compaction algorithm. With roughly uniform data
 growth, it's easy to cause split / compaction storms as the regions all
 roughly hit the same data size at the same time. With manual splits, you can
 let staggered, time-based major compactions spread out your network IO load.
      </para>
      <para>
 How do I turn off automatic splitting? Automatic splitting is determined by the configuration value
 <code>hbase.hregion.max.filesize</code>. It is not recommended that you set this
 to <varname>Long.MAX_VALUE</varname> in case you forget about manual splits. A suggested setting
 is 100GB, which would result in > 1hr major compactions if reached.
 </para>
 <para>What's the optimal number of pre-split regions to create?
 Mileage will vary depending upon your application.
 You could start low with 10 pre-split regions / server and watch as data grows
 over time. It's better to err on the side of too little regions and rolling split later.
 A more complicated answer is that this depends upon the largest storefile
 in your region. With a growing data size, this will get larger over time. You
 want the largest region to be just big enough that the <classname>Store</classname> compact
 selection algorithm only compacts it due to a timed major. If you don't, your
 cluster can be prone to compaction storms as the algorithm decides to run
 major compactions on a large series of regions all at once. Note that
 compaction storms are due to the uniform data growth, not the manual split
 decision.
 </para>
<para> If you pre-split your regions too thin, you can increase the major compaction
interval by configuring <varname>HConstants.MAJOR_COMPACTION_PERIOD</varname>. If your data size
grows too large, use the (post-0.90.0 HBase) <classname>org.apache.hadoop.hbase.util.RegionSplitter</classname>
script to perform a network IO safe rolling split
of all regions.
</para>
      </section>
      <section xml:id="managed.compactions"><title>Managed Compactions</title>
      <para>A common administrative technique is to manage major compactions manually, rather than letting
      HBase do it.  By default, <varname>HConstants.MAJOR_COMPACTION_PERIOD</varname> is one day and major compactions
      may kick in when you least desire it - especially on a busy system.  To turn off automatic major compactions set
      the value to <varname>0</varname>.
      </para>
      <para>It is important to stress that major compactions are absolutely necessary for StoreFile cleanup, the only variant is when
      they occur.  They can be administered through the HBase shell, or via
      <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29">HBaseAdmin</link>.
      </para>
      <para>For more information about compactions and the compaction file selection process, see <xref linkend="compaction"/></para>
      </section>

      <section xml:id="spec.ex"><title>Speculative Execution</title>
        <para>Speculative Execution of MapReduce tasks is on by default, and for HBase clusters it is generally advised to turn off
        Speculative Execution at a system-level unless you need it for a specific case, where it can be configured per-job.
        Set the properties <varname>mapred.map.tasks.speculative.execution</varname> and
        <varname>mapred.reduce.tasks.speculative.execution</varname> to false.
        </para>
      </section>
      </section>

      <section xml:id="other_configuration"><title>Other Configurations</title>
         <section xml:id="balancer_config"><title>Balancer</title>
           <para>The balancer is a periodic operation which is run on the master to redistribute regions on the cluster.  It is configured via
           <varname>hbase.balancer.period</varname> and defaults to 300000 (5 minutes). </para>
           <para>See <xref linkend="master.processes.loadbalancer" /> for more information on the LoadBalancer.
           </para>
         </section>
         <section xml:id="disabling.blockcache"><title>Disabling Blockcache</title>
           <para>Do not turn off block cache (You'd do it by setting <varname>hbase.block.cache.size</varname> to zero).
           Currently we do not do well if you do this because the regionserver will spend all its time loading hfile
           indices over and over again.  If your working set it such that block cache does you no good, at least
           size the block cache such that hfile indices will stay up in the cache (you can get a rough idea
           on the size you need by surveying regionserver UIs; you'll see index block size accounted near the
           top of the webpage).</para>
         </section>
    <section xml:id="nagles">
      <title><link xlink:href="http://en.wikipedia.org/wiki/Nagle's_algorithm">Nagle's</link> or the small package problem</title>
      <para>If a big 40ms or so occasional delay is seen in operations against HBase,
      try the Nagles' setting.  For example, see the user mailing list thread,
      <link xlink:href="http://search-hadoop.com/m/pduLg2fydtE/Inconsistent+scan+performance+with+caching+set+&amp;subj=Re+Inconsistent+scan+performance+with+caching+set+to+1">Inconsistent scan performance with caching set to 1</link>
      and the issue cited therein where setting notcpdelay improved scan speeds.  You might also
      see the graphs on the tail of <link xlink:href="https://issues.apache.org/jira/browse/HBASE-7008">HBASE-7008 Set scanner caching to a better default</link>
      where our Lars Hofhansl tries various data sizes w/ Nagle's on and off measuring the effect.</para>
    </section>
    <section xml:id="mttr">
      <title>Better Mean Time to Recover (MTTR)</title>
      <para>This section is about configurations that will make servers come back faster after a fail.
          See the Deveraj Das an Nicolas Liochon blog post
          <link xlink:href="http://hortonworks.com/blog/introduction-to-hbase-mean-time-to-recover-mttr/">Introduction to HBase Mean Time to Recover (MTTR)</link>
          for a brief introduction.</para>
      <para>The issue <link xlink:href="https://issues.apache.org/jira/browse/HBASE-8389">HBASE-8354 forces Namenode into loop with lease recovery requests</link>
          is messy but has a bunch of good discussion toward the end on low timeouts and how to effect faster recovery including citation of fixes
          added to HDFS.  Read the Varun Sharma comments.  The below suggested configurations are Varun's suggestions distilled and tested.  Make sure you are
          running on a late-version HDFS so you have the fixes he refers too and himself adds to HDFS that help HBase MTTR
          (e.g. HDFS-3703, HDFS-3712, and HDFS-4791 -- hadoop 2 for sure has them and late hadoop 1 has some).
          Set the following in the RegionServer.
<![CDATA[<property>
    <name>hbase.lease.recovery.dfs.timeout</name>
    <value>23000</value>
    <description>How much time we allow elapse between calls to recover lease.
    Should be larger than the dfs timeout.</description>
</property>
<property>
    <name>dfs.client.socket-timeout</name>
    <value>10000</value>
    <description>Down the DFS timeout from 60 to 10 seconds.</description>
</property>]]>
And on the namenode/datanode side, set the following to enable 'staleness' introduced in HDFS-3703, HDFS-3912.
<![CDATA[<property>
    <name>dfs.client.socket-timeout</name>
    <value>10000</value>
    <description>Down the DFS timeout from 60 to 10 seconds.</description>
</property>
<property>
    <name>dfs.datanode.socket.write.timeout</name>
    <value>10000</value>
    <description>Down the DFS timeout from 8 * 60 to 10 seconds.</description>
</property>
<property>
    <name>ipc.client.connect.timeout</name>
    <value>3000</value>
    <description>Down from 60 seconds to 3.</description>
</property>
<property>
    <name>ipc.client.connect.max.retries.on.timeouts</name>
    <value>2</value>
    <description>Down from 45 seconds to 3 (2 == 3 retries).</description>
</property>
<property>
    <name>dfs.namenode.avoid.read.stale.datanode</name>
    <value>true</value>
    <description>Enable stale state in hdfs</description>
</property>
<property>
    <name>dfs.namenode.stale.datanode.interval</name>
    <value>20000</value>
    <description>Down from default 30 seconds</description>
</property>
<property>
    <name>dfs.namenode.avoid.write.stale.datanode</name>
    <value>true</value>
    <description>Enable stale state in hdfs</description>
</property>]]>

      </para>
    </section>

      </section>

      </section> <!--  important config -->

  </chapter>
