Google Git
Sign in
apache / hadoop / 5cd5656cd43a176a62c06e0f3d1398da624c0429 / . / branch-0.23.1 / hadoop-common-project / hadoop-common / src / main / docs / src / documentation / content / xdocs / cluster_setup.xml
blob: 1a2de1dd0012c6f14b96dc41bb4b42e2c4158119 [file] [log] [blame]
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">
<document>
<header>
<title>Cluster Setup</title>
</header>
<body>
<section>
<title>Purpose</title>
<p>This document describes how to install, configure and manage non-trivial
Hadoop clusters ranging from a few nodes to extremely large clusters with
thousands of nodes.</p>
<p>
To play with Hadoop, you may first want to install Hadoop on a single machine (see <a href="single_node_setup.html"> Hadoop Quick Start</a>).
</p>
</section>
<section>
<title>Pre-requisites</title>
<ol>
<li>
Make sure all <a href="single_node_setup.html#PreReqs">requisite</a> software
is installed on all nodes in your cluster.
</li>
<li>
<a href="single_node_setup.html#Download">Get</a> the Hadoop software.
</li>
</ol>
</section>
<section>
<title>Installation</title>
<p>Installing a Hadoop cluster typically involves unpacking the software
on all the machines in the cluster.</p>
<p>Typically one machine in the cluster is designated as the
<code>NameNode</code> and another machine the as <code>JobTracker</code>,
exclusively. These are the <em>masters</em>. The rest of the machines in
the cluster act as both <code>DataNode</code> <em>and</em>
<code>TaskTracker</code>. These are the <em>slaves</em>.</p>
<p>The root of the distribution is referred to as
<code>HADOOP_PREFIX</code>. All machines in the cluster usually have the same
<code>HADOOP_PREFIX</code> path.</p>
</section>
<section>
<title>Configuration</title>
<p>The following sections describe how to configure a Hadoop cluster.</p>
<section>
<title>Configuration Files</title>
<p>Hadoop configuration is driven by two types of important
configuration files:</p>
<ol>
<li>
Read-only default configuration -
<a href="ext:core-default">src/core/core-default.xml</a>,
<a href="ext:hdfs-default">src/hdfs/hdfs-default.xml</a>,
<a href="ext:mapred-default">src/mapred/mapred-default.xml</a> and
<a href="ext:mapred-queues">conf/mapred-queues.xml.template</a>.
</li>
<li>
Site-specific configuration -
<a href="#core-site.xml">conf/core-site.xml</a>,
<a href="#hdfs-site.xml">conf/hdfs-site.xml</a>,
<a href="#mapred-site.xml">conf/mapred-site.xml</a> and
<a href="#mapred-queues.xml">conf/mapred-queues.xml</a>.
</li>
</ol>
<p>To learn more about how the Hadoop framework is controlled by these
configuration files, look
<a href="ext:api/org/apache/hadoop/conf/configuration">here</a>.</p>
<p>Additionally, you can control the Hadoop scripts found in the
<code>bin/</code> directory of the distribution, by setting site-specific
values via the <code>conf/hadoop-env.sh</code>.</p>
</section>
<section>
<title>Site Configuration</title>
<p>To configure the Hadoop cluster you will need to configure the
<em>environment</em> in which the Hadoop daemons execute as well as
the <em>configuration parameters</em> for the Hadoop daemons.</p>
<p>The Hadoop daemons are <code>NameNode</code>/<code>DataNode</code>
and <code>JobTracker</code>/<code>TaskTracker</code>.</p>
<section>
<title>Configuring the Environment of the Hadoop Daemons</title>
<p>Administrators should use the <code>conf/hadoop-env.sh</code> script
to do site-specific customization of the Hadoop daemons' process
environment.</p>
<p>At the very least you should specify the
<code>JAVA_HOME</code> so that it is correctly defined on each
remote node.</p>
<p>Administrators can configure individual daemons using the
configuration options <code>HADOOP_*_OPTS</code>. Various options
available are shown below in the table. </p>
<table>
<tr><th>Daemon</th><th>Configure Options</th></tr>
<tr><td>NameNode</td><td>HADOOP_NAMENODE_OPTS</td></tr>
<tr><td>DataNode</td><td>HADOOP_DATANODE_OPTS</td></tr>
<tr><td>SecondaryNamenode</td>
<td>HADOOP_SECONDARYNAMENODE_OPTS</td></tr>
<tr><td>JobTracker</td><td>HADOOP_JOBTRACKER_OPTS</td></tr>
<tr><td>TaskTracker</td><td>HADOOP_TASKTRACKER_OPTS</td></tr>
</table>
<p> For example, To configure Namenode to use parallelGC, the
following statement should be added in <code>hadoop-env.sh</code> :
<br/><code>
export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}"
</code><br/></p>
<p>Other useful configuration parameters that you can customize
include:</p>
<ul>
<li>
<code>HADOOP_LOG_DIR</code> - The directory where the daemons'
log files are stored. They are automatically created if they don't
exist.
</li>
<li>
<code>HADOOP_HEAPSIZE</code> - The maximum amount of heapsize
to use, in MB e.g. <code>1000MB</code>. This is used to
configure the heap size for the hadoop daemon. By default,
the value is <code>1000MB</code>.
</li>
</ul>
</section>
<section>
<title>Configuring the Hadoop Daemons</title>
<p>This section deals with important parameters to be specified in the
following:</p>
<anchor id="core-site.xml"/><p><code>conf/core-site.xml</code>:</p>
<table>
<tr>
<th>Parameter</th>
<th>Value</th>
<th>Notes</th>
</tr>
<tr>
<td>fs.defaultFS</td>
<td>URI of <code>NameNode</code>.</td>
<td><em>hdfs://hostname/</em></td>
</tr>
</table>
<anchor id="hdfs-site.xml"/><p><code>conf/hdfs-site.xml</code>:</p>
<table>
<tr>
<th>Parameter</th>
<th>Value</th>
<th>Notes</th>
</tr>
<tr>
<td>dfs.namenode.name.dir</td>
<td>
Path on the local filesystem where the <code>NameNode</code>
stores the namespace and transactions logs persistently.</td>
<td>
If this is a comma-delimited list of directories then the name
table is replicated in all of the directories, for redundancy.
</td>
</tr>
<tr>
<td>dfs.datanode.data.dir</td>
<td>
Comma separated list of paths on the local filesystem of a
<code>DataNode</code> where it should store its blocks.
</td>
<td>
If this is a comma-delimited list of directories, then data will
be stored in all named directories, typically on different
devices.
</td>
</tr>
</table>
<anchor id="mapred-site.xml"/><p><code>conf/mapred-site.xml</code>:</p>
<table>
<tr>
<th>Parameter</th>
<th>Value</th>
<th>Notes</th>
</tr>
<tr>
<td>mapreduce.jobtracker.address</td>
<td>Host or IP and port of <code>JobTracker</code>.</td>
<td><em>host:port</em> pair.</td>
</tr>
<tr>
<td>mapreduce.jobtracker.system.dir</td>
<td>
Path on the HDFS where where the Map/Reduce framework stores
system files e.g. <code>/hadoop/mapred/system/</code>.
</td>
<td>
This is in the default filesystem (HDFS) and must be accessible
from both the server and client machines.
</td>
</tr>
<tr>
<td>mapreduce.cluster.local.dir</td>
<td>
Comma-separated list of paths on the local filesystem where
temporary Map/Reduce data is written.
</td>
<td>Multiple paths help spread disk i/o.</td>
</tr>
<tr>
<td>mapred.tasktracker.{map|reduce}.tasks.maximum</td>
<td>
The maximum number of Map/Reduce tasks, which are run
simultaneously on a given <code>TaskTracker</code>, individually.
</td>
<td>
Defaults to 2 (2 maps and 2 reduces), but vary it depending on
your hardware.
</td>
</tr>
<tr>
<td>dfs.hosts/dfs.hosts.exclude</td>
<td>List of permitted/excluded DataNodes.</td>
<td>
If necessary, use these files to control the list of allowable
datanodes.
</td>
</tr>
<tr>
<td>mapreduce.jobtracker.hosts.filename/mapreduce.jobtracker.hosts.exclude.filename</td>
<td>List of permitted/excluded TaskTrackers.</td>
<td>
If necessary, use these files to control the list of allowable
TaskTrackers.
</td>
</tr>
<tr>
<td>mapreduce.cluster.acls.enabled</td>
<td>Boolean, specifying whether checks for queue ACLs and job ACLs
are to be done for authorizing users for doing queue operations and
job operations.
</td>
<td>
If <em>true</em>, queue ACLs are checked while submitting
and administering jobs and job ACLs are checked for authorizing
view and modification of jobs. Queue ACLs are specified using the
configuration parameters of the form defined below under
mapred-queues.xml. Job ACLs are described at
mapred-tutorial in "Job Authorization" section.
For enabling this flag(mapreduce.cluster.acls.enabled), this is to be
set to true in mapred-site.xml on JobTracker node and on all
TaskTracker nodes.
</td>
</tr>
</table>
<p>Typically all the above parameters are marked as
<a href="ext:api/org/apache/hadoop/conf/configuration/final_parameters">
final</a> to ensure that they cannot be overriden by user-applications.
</p>
<anchor id="mapred-queues.xml"/><p><code>conf/mapred-queues.xml
</code>:</p>
<p>This file is used to configure the queues in the Map/Reduce
system. Queues are abstract entities in the JobTracker that can be
used to manage collections of jobs. They provide a way for
administrators to organize jobs in specific ways and to enforce
certain policies on such collections, thus providing varying
levels of administrative control and management functions on jobs.
</p>
<p>One can imagine the following sample scenarios:</p>
<ul>
<li> Jobs submitted by a particular group of users can all be
submitted to one queue. </li>
<li> Long running jobs in an organization can be submitted to a
queue. </li>
<li> Short running jobs can be submitted to a queue and the number
of jobs that can run concurrently can be restricted. </li>
</ul>
<p>The usage of queues is closely tied to the scheduler configured
at the JobTracker via <em>mapreduce.jobtracker.taskscheduler</em>.
The degree of support of queues depends on the scheduler used. Some
schedulers support a single queue, while others support more complex
configurations. Schedulers also implement the policies that apply
to jobs in a queue. Some schedulers, such as the Fairshare scheduler,
implement their own mechanisms for collections of jobs and do not rely
on queues provided by the framework. The administrators are
encouraged to refer to the documentation of the scheduler they are
interested in for determining the level of support for queues.</p>
<p>The Map/Reduce framework supports some basic operations on queues
such as job submission to a specific queue, access control for queues,
queue states, viewing configured queues and their properties
and refresh of queue properties. In order to fully implement some of
these operations, the framework takes the help of the configured
scheduler.</p>
<p>The following types of queue configurations are possible:</p>
<ul>
<li> Single queue: The default configuration in Map/Reduce comprises
of a single queue, as supported by the default scheduler. All jobs
are submitted to this default queue which maintains jobs in a priority
based FIFO order.</li>
<li> Multiple single level queues: Multiple queues are defined, and
jobs can be submitted to any of these queues. Different policies
can be applied to these queues by schedulers that support this
configuration to provide a better level of support. For example,
the <a href="ext:capacity-scheduler">capacity scheduler</a>
provides ways of configuring different
capacity and fairness guarantees on these queues.</li>
<li> Hierarchical queues: Hierarchical queues are a configuration in
which queues can contain other queues within them recursively. The
queues that contain other queues are referred to as
container queues. Queues that do not contain other queues are
referred as leaf or job queues. Jobs can only be submitted to leaf
queues. Hierarchical queues can potentially offer a higher level
of control to administrators, as schedulers can now build a
hierarchy of policies where policies applicable to a container
queue can provide context for policies applicable to queues it
contains. It also opens up possibilities for delegating queue
administration where administration of queues in a container queue
can be turned over to a different set of administrators, within
the context provided by the container queue. For example, the
<a href="ext:capacity-scheduler">capacity scheduler</a>
uses hierarchical queues to partition capacity of a cluster
among container queues, and allowing queues they contain to divide
that capacity in more ways.</li>
</ul>
<p>Most of the configuration of the queues can be refreshed/reloaded
without restarting the Map/Reduce sub-system by editing this
configuration file as described in the section on
<a href="commands_manual.html#RefreshQueues">reloading queue
configuration</a>.
Not all configuration properties can be reloaded of course,
as will description of each property below explain.</p>
<p>The format of conf/mapred-queues.xml is different from the other
configuration files, supporting nested configuration
elements to support hierarchical queues. The format is as follows:
</p>
<source>
&lt;queues&gt;
&lt;queue&gt;
&lt;name&gt;$queue-name&lt;/name&gt;
&lt;state&gt;$state&lt;/state&gt;
&lt;queue&gt;
&lt;name&gt;$child-queue1&lt;/name&gt;
&lt;properties&gt;
&lt;property key="$key" value="$value"/&gt;
...
&lt;/properties&gt;
&lt;queue&gt;
&lt;name&gt;$grand-child-queue1&lt;/name&gt;
...
&lt;/queue&gt;
&lt;/queue&gt;
&lt;queue&gt;
&lt;name&gt;$child-queue2&lt;/name&gt;
...
&lt;/queue&gt;
...
...
...
&lt;queue&gt;
&lt;name&gt;$leaf-queue&lt;/name&gt;
&lt;acl-submit-job&gt;$acls&lt;/acl-submit-job&gt;
&lt;acl-administer-jobs&gt;$acls&lt;/acl-administer-jobs&gt;
&lt;properties&gt;
&lt;property key="$key" value="$value"/&gt;
...
&lt;/properties&gt;
&lt;/queue&gt;
&lt;/queue&gt;
&lt;/queues&gt;
</source>
<table>
<tr>
<th>Tag/Attribute</th>
<th>Value</th>
<th>
<a href="commands_manual.html#RefreshQueues">Refresh-able?</a>
</th>
<th>Notes</th>
</tr>
<tr>
<td><anchor id="queues_tag"/>queues</td>
<td>Root element of the configuration file.</td>
<td>Not-applicable</td>
<td>All the queues are nested inside this root element of the
file. There can be only one root queues element in the file.</td>
</tr>
<tr>
<td>aclsEnabled</td>
<td>Boolean attribute to the
<a href="#queues_tag"><em>&lt;queues&gt;</em></a> tag
specifying whether ACLs are supported for controlling job
submission and administration for <em>all</em> the queues
configured.
</td>
<td>Yes</td>
<td>If <em>false</em>, ACLs are ignored for <em>all</em> the
configured queues. <br/><br/>
If <em>true</em>, the user and group details of the user
are checked against the configured ACLs of the corresponding
job-queue while submitting and administering jobs. ACLs can be
specified for each queue using the queue-specific tags
"acl-$acl_name", defined below. ACLs are checked only against
the job-queues, i.e. the leaf-level queues; ACLs configured
for the rest of the queues in the hierarchy are ignored.
</td>
</tr>
<tr>
<td><anchor id="queue_tag"/>queue</td>
<td>A child element of the
<a href="#queues_tag"><em>&lt;queues&gt;</em></a> tag or another
<a href="#queue_tag"><em>&lt;queue&gt;</em></a>. Denotes a queue
in the system.
</td>
<td>Not applicable</td>
<td>Queues can be hierarchical and so this element can contain
children of this same type.</td>
</tr>
<tr>
<td>name</td>
<td>Child element of a
<a href="#queue_tag"><em>&lt;queue&gt;</em></a> specifying the
name of the queue.</td>
<td>No</td>
<td>Name of the queue cannot contain the character <em>":"</em>
which is reserved as the queue-name delimiter when addressing a
queue in a hierarchy.</td>
</tr>
<tr>
<td>state</td>
<td>Child element of a
<a href="#queue_tag"><em>&lt;queue&gt;</em></a> specifying the
state of the queue.
</td>
<td>Yes</td>
<td>Each queue has a corresponding state. A queue in
<em>'running'</em> state can accept new jobs, while a queue in
<em>'stopped'</em> state will stop accepting any new jobs. State
is defined and respected by the framework only for the
leaf-level queues and is ignored for all other queues.
<br/><br/>
The state of the queue can be viewed from the command line using
<code>'bin/mapred queue'</code> command and also on the the Web
UI.<br/><br/>
Administrators can stop and start queues at runtime using the
feature of <a href="commands_manual.html#RefreshQueues">reloading
queue configuration</a>. If a queue is stopped at runtime, it
will complete all the existing running jobs and will stop
accepting any new jobs.
</td>
</tr>
<tr>
<td>acl-submit-job</td>
<td>Child element of a
<a href="#queue_tag"><em>&lt;queue&gt;</em></a> specifying the
list of users and groups that can submit jobs to the specified
queue.</td>
<td>Yes</td>
<td>
Applicable only to leaf-queues.<br/><br/>
The list of users and groups are both comma separated
list of names. The two lists are separated by a blank.
Example: <em>user1,user2 group1,group2</em>.
If you wish to define only a list of groups, provide
a blank at the beginning of the value.
<br/><br/>
</td>
</tr>
<tr>
<td>acl-administer-jobs</td>
<td>Child element of a
<a href="#queue_tag"><em>&lt;queue&gt;</em></a> specifying the
list of users and groups that can view job details, change the
priority of a job or kill a job that has been submitted to the
specified queue.
</td>
<td>Yes</td>
<td>
Applicable only to leaf-queues.<br/><br/>
The list of users and groups are both comma separated
list of names. The two lists are separated by a blank.
Example: <em>user1,user2 group1,group2</em>.
If you wish to define only a list of groups, provide
a blank at the beginning of the value. Note that the
owner of a job can always change the priority or kill
his/her own job, irrespective of the ACLs.
</td>
</tr>
<tr>
<td><anchor id="properties_tag"/>properties</td>
<td>Child element of a
<a href="#queue_tag"><em>&lt;queue&gt;</em></a> specifying the
scheduler specific properties.</td>
<td>Not applicable</td>
<td>The scheduler specific properties are the children of this
element specified as a group of &lt;property&gt; tags described
below. The JobTracker completely ignores these properties. These
can be used as per-queue properties needed by the scheduler
being configured. Please look at the scheduler specific
documentation as to how these properties are used by that
particular scheduler.
</td>
</tr>
<tr>
<td><anchor id="property_tag"/>property</td>
<td>Child element of
<a href="#properties_tag"><em>&lt;properties&gt;</em></a> for a
specific queue.</td>
<td>Not applicable</td>
<td>A single scheduler specific queue-property. Ignored by
the JobTracker and used by the scheduler that is configured.</td>
</tr>
<tr>
<td>key</td>
<td>Attribute of a
<a href="#property_tag"><em>&lt;property&gt;</em></a> for a
specific queue.</td>
<td>Scheduler-specific</td>
<td>The name of a single scheduler specific queue-property.</td>
</tr>
<tr>
<td>value</td>
<td>Attribute of a
<a href="#property_tag"><em>&lt;property&gt;</em></a> for a
specific queue.</td>
<td>Scheduler-specific</td>
<td>The value of a single scheduler specific queue-property.
The value can be anything that is left for the proper
interpretation by the scheduler that is configured.</td>
</tr>
</table>
<p>Once the queues are configured properly and the Map/Reduce
system is up and running, from the command line one can
<a href="commands_manual.html#QueuesList">get the list
of queues</a> and
<a href="commands_manual.html#QueuesInfo">obtain
information specific to each queue</a>. This information is also
available from the web UI. On the web UI, queue information can be
seen by going to queueinfo.jsp, linked to from the queues table-cell
in the cluster-summary table. The queueinfo.jsp prints the hierarchy
of queues as well as the specific information for each queue.
</p>
<p> Users can submit jobs only to a
leaf-level queue by specifying the fully-qualified queue-name for
the property name <em>mapreduce.job.queuename</em> in the job
configuration. The character ':' is the queue-name delimiter and so,
for e.g., if one wants to submit to a configured job-queue 'Queue-C'
which is one of the sub-queues of 'Queue-B' which in-turn is a
sub-queue of 'Queue-A', then the job configuration should contain
property <em>mapreduce.job.queuename</em> set to the <em>
&lt;value&gt;Queue-A:Queue-B:Queue-C&lt;/value&gt;</em></p>
</section>
<section>
<title>Real-World Cluster Configurations</title>
<p>This section lists some non-default configuration parameters which
have been used to run the <em>sort</em> benchmark on very large
clusters.</p>
<ul>
<li>
<p>Some non-default configuration values used to run sort900,
that is 9TB of data sorted on a cluster with 900 nodes:</p>
<table>
<tr>
<th>Configuration File</th>
<th>Parameter</th>
<th>Value</th>
<th>Notes</th>
</tr>
<tr>
<td>conf/hdfs-site.xml</td>
<td>dfs.blocksize</td>
<td>128m</td>
<td>
HDFS blocksize of 128 MB for large file-systems. Sizes can be provided
in size-prefixed values (10k, 128m, 1g, etc.) or simply in bytes (134217728 for 128 MB, etc.).
</td>
</tr>
<tr>
<td>conf/hdfs-site.xml</td>
<td>dfs.namenode.handler.count</td>
<td>40</td>
<td>
More NameNode server threads to handle RPCs from large
number of DataNodes.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.reduce.shuffle.parallelcopies</td>
<td>20</td>
<td>
Higher number of parallel copies run by reduces to fetch
outputs from very large number of maps.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.map.java.opts</td>
<td>-Xmx512M</td>
<td>
Larger heap-size for child jvms of maps.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.reduce.java.opts</td>
<td>-Xmx512M</td>
<td>
Larger heap-size for child jvms of reduces.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.reduce.shuffle.input.buffer.percent</td>
<td>0.80</td>
<td>
Larger amount of memory allocated for merging map output
in memory during the shuffle. Expressed as a fraction of
the total heap.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.reduce.input.buffer.percent</td>
<td>0.80</td>
<td>
Larger amount of memory allocated for retaining map output
in memory during the reduce. Expressed as a fraction of
the total heap.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.task.io.sort.factor</td>
<td>100</td>
<td>More streams merged at once while sorting files.</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.task.io.sort.mb</td>
<td>200</td>
<td>Higher memory-limit while sorting data.</td>
</tr>
<tr>
<td>conf/core-site.xml</td>
<td>io.file.buffer.size</td>
<td>131072</td>
<td>Size of read/write buffer used in SequenceFiles.</td>
</tr>
</table>
</li>
<li>
<p>Updates to some configuration values to run sort1400 and
sort2000, that is 14TB of data sorted on 1400 nodes and 20TB of
data sorted on 2000 nodes:</p>
<table>
<tr>
<th>Configuration File</th>
<th>Parameter</th>
<th>Value</th>
<th>Notes</th>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.jobtracker.handler.count</td>
<td>60</td>
<td>
More JobTracker server threads to handle RPCs from large
number of TaskTrackers.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.reduce.shuffle.parallelcopies</td>
<td>50</td>
<td></td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.tasktracker.http.threads</td>
<td>50</td>
<td>
More worker threads for the TaskTracker's http server. The
http server is used by reduces to fetch intermediate
map-outputs.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.map.java.opts</td>
<td>-Xmx512M</td>
<td>
Larger heap-size for child jvms of maps.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.reduce.java.opts</td>
<td>-Xmx1024M</td>
<td>Larger heap-size for child jvms of reduces.</td>
</tr>
</table>
</li>
</ul>
</section>
<section>
<title>Configuring Memory Parameters for MapReduce Jobs</title>
<p>
As MapReduce jobs could use varying amounts of memory, Hadoop
provides various configuration options to users and administrators
for managing memory effectively. Some of these options are job
specific and can be used by users. While setting up a cluster,
administrators can configure appropriate default values for these
options so that users jobs run out of the box. Other options are
cluster specific and can be used by administrators to enforce
limits and prevent misconfigured or memory intensive jobs from
causing undesired side effects on the cluster.
</p>
<p>
The values configured should
take into account the hardware resources of the cluster, such as the
amount of physical and virtual memory available for tasks,
the number of slots configured on the slaves and the requirements
for other processes running on the slaves. If right values are not
set, it is likely that jobs start failing with memory related
errors or in the worst case, even affect other tasks or
the slaves themselves.
</p>
<section>
<title>Monitoring Task Memory Usage</title>
<p>
Before describing the memory options, it is
useful to look at a feature provided by Hadoop to monitor
memory usage of MapReduce tasks it runs. The basic objective
of this feature is to prevent MapReduce tasks from consuming
memory beyond a limit that would result in their affecting
other processes running on the slave, including other tasks
and daemons like the DataNode or TaskTracker.
</p>
<p>
<em>Note:</em> For the time being, this feature is available
only for the Linux platform.
</p>
<p>
Hadoop allows monitoring to be done both for virtual
and physical memory usage of tasks. This monitoring
can be done independently of each other, and therefore the
options can be configured independently of each other. It
has been found in some environments, particularly related
to streaming, that virtual memory recorded for tasks is high
because of libraries loaded by the programs used to run
the tasks. However, this memory is largely unused and does
not affect the slaves's memory itself. In such cases,
monitoring based on physical memory can provide a more
accurate picture of memory usage.
</p>
<p>
This feature considers that there is a limit on
the amount of virtual or physical memory on the slaves
that can be used by
the running MapReduce tasks. The rest of the memory is
assumed to be required for the system and other processes.
Since some jobs may require higher amount of memory for their
tasks than others, Hadoop allows jobs to specify how much
memory they expect to use at a maximum. Then by using
resource aware scheduling and monitoring, Hadoop tries to
ensure that at any time, only enough tasks are running on
the slaves as can meet the dual constraints of an individual
job's memory requirements and the total amount of memory
available for all MapReduce tasks.
</p>
<p>
The TaskTracker monitors tasks in regular intervals. Each time,
it operates in two steps:
</p>
<ul>
<li>
In the first step, it
checks that a job's task and any child processes it
launches are not cumulatively using more virtual or physical
memory than specified. If both virtual and physical memory
monitoring is enabled, then virtual memory usage is checked
first, followed by physical memory usage.
Any task that is found to
use more memory is killed along with any child processes it
might have launched, and the task status is marked
<em>failed</em>. Repeated failures such as this will terminate
the job.
</li>
<li>
In the next step, it checks that the cumulative virtual and
physical memory
used by all running tasks and their child processes
does not exceed the total virtual and physical memory limit,
respectively. Again, virtual memory limit is checked first,
followed by physical memory limit. In this case, it kills
enough number of tasks, along with any child processes they
might have launched, until the cumulative memory usage
is brought under limit. In the case of virtual memory limit
being exceeded, the tasks chosen for killing are
the ones that have made the least progress. In the case of
physical memory limit being exceeded, the tasks chosen
for killing are the ones that have used the maximum amount
of physical memory. Also, the status
of these tasks is marked as <em>killed</em>, and hence repeated
occurrence of this will not result in a job failure.
</li>
</ul>
<p>
In either case, the task's diagnostic message will indicate the
reason why the task was terminated.
</p>
<p>
Resource aware scheduling can ensure that tasks are scheduled
on a slave only if their memory requirement can be satisfied
by the slave. The Capacity Scheduler, for example,
takes virtual memory requirements into account while
scheduling tasks, as described in the section on
<a href="ext:capacity-scheduler/MemoryBasedTaskScheduling">
memory based scheduling</a>.
</p>
<p>
Memory monitoring is enabled when certain configuration
variables are defined with non-zero values, as described below.
</p>
</section>
<section>
<title>Job Specific Options</title>
<p>
Memory related options that can be configured individually per
job are described in detail in the section on
<a href="ext:mapred-tutorial/ConfiguringMemoryRequirements">
Configuring Memory Requirements For A Job</a> in the MapReduce
tutorial. While setting up
the cluster, the Hadoop defaults for these options can be reviewed
and changed to better suit the job profiles expected to be run on
the clusters, as also the hardware configuration.
</p>
<p>
As with any other configuration option in Hadoop, if the
administrators desire to prevent users from overriding these
options in jobs they submit, these values can be marked as
<em>final</em> in the cluster configuration.
</p>
</section>
<section>
<title>Cluster Specific Options</title>
<p>
This section describes the memory related options that are
used by the JobTracker and TaskTrackers, and cannot be changed
by jobs. The values set for these options should be the same
for all the slave nodes in a cluster.
</p>
<ul>
<li>
<code>mapreduce.cluster.{map|reduce}memory.mb</code>: These
options define the default amount of virtual memory that should be
allocated for MapReduce tasks running in the cluster. They
typically match the default values set for the options
<code>mapreduce.{map|reduce}.memory.mb</code>. They help in the
calculation of the total amount of virtual memory available for
MapReduce tasks on a slave, using the following equation:<br/>
<em>Total virtual memory for all MapReduce tasks =
(mapreduce.cluster.mapmemory.mb *
mapreduce.tasktracker.map.tasks.maximum) +
(mapreduce.cluster.reducememory.mb *
mapreduce.tasktracker.reduce.tasks.maximum)</em><br/>
Typically, reduce tasks require more memory than map tasks.
Hence a higher value is recommended for
<em>mapreduce.cluster.reducememory.mb</em>. The value is
specified in MB. To set a value of 2GB for reduce tasks, set
<em>mapreduce.cluster.reducememory.mb</em> to 2048.
</li>
<li>
<code>mapreduce.jobtracker.max{map|reduce}memory.mb</code>:
These options define the maximum amount of virtual memory that
can be requested by jobs using the parameters
<code>mapreduce.{map|reduce}.memory.mb</code>. The system
will reject any job that is submitted requesting for more
memory than these limits. Typically, the values for these
options should be set to satisfy the following constraint:<br/>
<em>mapreduce.jobtracker.maxmapmemory.mb =
mapreduce.cluster.mapmemory.mb *
mapreduce.tasktracker.map.tasks.maximum<br/>
mapreduce.jobtracker.maxreducememory.mb =
mapreduce.cluster.reducememory.mb *
mapreduce.tasktracker.reduce.tasks.maximum</em><br/>
The value is specified in MB. If
<code>mapreduce.cluster.reducememory.mb</code> is set to 2GB and
there are 2 reduce slots configured in the slaves, the value
for <code>mapreduce.jobtracker.maxreducememory.mb</code> should
be set to 4096.
</li>
<li>
<code>mapreduce.tasktracker.reserved.physicalmemory.mb</code>:
This option defines the amount of physical memory that is
marked for system and daemon processes. Using this, the amount
of physical memory available for MapReduce tasks is calculated
using the following equation:<br/>
<em>Total physical memory for all MapReduce tasks =
Total physical memory available on the system -
mapreduce.tasktracker.reserved.physicalmemory.mb</em><br/>
The value is specified in MB. To set this value to 2GB,
specify the value as 2048.
</li>
<li>
<code>mapreduce.tasktracker.taskmemorymanager.monitoringinterval</code>:
This option defines the time the TaskTracker waits between
two cycles of memory monitoring. The value is specified in
milliseconds.
</li>
</ul>
<p>
<em>Note:</em> The virtual memory monitoring function is only
enabled if
the variables <code>mapreduce.cluster.{map|reduce}memory.mb</code>
and <code>mapreduce.jobtracker.max{map|reduce}memory.mb</code>
are set to values greater than zero. Likewise, the physical
memory monitoring function is only enabled if the variable
<code>mapreduce.tasktracker.reserved.physicalmemory.mb</code>
is set to a value greater than zero.
</p>
</section>
</section>
<section>
<title>Task Controllers</title>
<p>Task controllers are classes in the Hadoop Map/Reduce
framework that define how user's map and reduce tasks
are launched and controlled. They can
be used in clusters that require some customization in
the process of launching or controlling the user tasks.
For example, in some
clusters, there may be a requirement to run tasks as
the user who submitted the job, instead of as the task
tracker user, which is how tasks are launched by default.
This section describes how to configure and use
task controllers.</p>
<p>The following task controllers are the available in
Hadoop.
</p>
<table>
<tr><th>Name</th><th>Class Name</th><th>Description</th></tr>
<tr>
<td>DefaultTaskController</td>
<td>org.apache.hadoop.mapred.DefaultTaskController</td>
<td> The default task controller which Hadoop uses to manage task
execution. The tasks run as the task tracker user.</td>
</tr>
<tr>
<td>LinuxTaskController</td>
<td>org.apache.hadoop.mapred.LinuxTaskController</td>
<td>This task controller, which is supported only on Linux,
runs the tasks as the user who submitted the job. It requires
these user accounts to be created on the cluster nodes
where the tasks are launched. It
uses a setuid executable that is included in the Hadoop
distribution. The task tracker uses this executable to
launch and kill tasks. The setuid executable switches to
the user who has submitted the job and launches or kills
the tasks. For maximum security, this task controller
sets up restricted permissions and user/group ownership of
local files and directories used by the tasks such as the
job jar files, intermediate files, task log files and distributed
cache files. Particularly note that, because of this, except the
job owner and tasktracker, no other user can access any of the
local files/directories including those localized as part of the
distributed cache.
</td>
</tr>
</table>
<section>
<title>Configuring Task Controllers</title>
<p>The task controller to be used can be configured by setting the
value of the following key in mapred-site.xml</p>
<table>
<tr>
<th>Property</th><th>Value</th><th>Notes</th>
</tr>
<tr>
<td>mapreduce.tasktracker.taskcontroller</td>
<td>Fully qualified class name of the task controller class</td>
<td>Currently there are two implementations of task controller
in the Hadoop system, DefaultTaskController and LinuxTaskController.
Refer to the class names mentioned above to determine the value
to set for the class of choice.
</td>
</tr>
</table>
</section>
<section>
<title>Using the LinuxTaskController</title>
<p>This section of the document describes the steps required to
use the LinuxTaskController.</p>
<p>In order to use the LinuxTaskController, a setuid executable
should be built and deployed on the compute nodes. The
executable is named task-controller. To build the executable,
execute
<em>ant task-controller -Dhadoop.conf.dir=/path/to/conf/dir.
</em>
The path passed in <em>-Dhadoop.conf.dir</em> should be the path
on the cluster nodes where a configuration file for the setuid
executable would be located. The executable would be built to
<em>build.dir/dist.dir/bin</em> and should be installed to
<em>$HADOOP_PREFIX/bin</em>.
</p>
<p>
The executable must have specific permissions as follows. The
executable should have <em>6050 or --Sr-s---</em> permissions
user-owned by root(super-user) and group-owned by a special group
of which the TaskTracker's user is the group member and no job
submitter is. If any job submitter belongs to this special group,
security will be compromised. This special group name should be
specified for the configuration property
<em>"mapreduce.tasktracker.group"</em> in both mapred-site.xml and
<a href="#task-controller.cfg">task-controller.cfg</a>.
For example, let's say that the TaskTracker is run as user
<em>mapred</em> who is part of the groups <em>users</em> and
<em>specialGroup</em> any of them being the primary group.
Let also be that <em>users</em> has both <em>mapred</em> and
another user (job submitter) <em>X</em> as its members, and X does
not belong to <em>specialGroup</em>. Going by the above
description, the setuid/setgid executable should be set
<em>6050 or --Sr-s---</em> with user-owner as <em>mapred</em> and
group-owner as <em>specialGroup</em> which has
<em>mapred</em> as its member(and not <em>users</em> which has
<em>X</em> also as its member besides <em>mapred</em>).
</p>
<p>
The LinuxTaskController requires that paths including and leading up
to the directories specified in
<em>mapreduce.cluster.local.dir</em> and <em>hadoop.log.dir</em> to
be set 755 permissions.
</p>
<section>
<title>task-controller.cfg</title>
<p>The executable requires a configuration file called
<em>taskcontroller.cfg</em> to be
present in the configuration directory passed to the ant target
mentioned above. If the binary was not built with a specific
conf directory, the path defaults to
<em>/path-to-binary/../conf</em>. The configuration file must be
owned by the user running TaskTracker (user <em>mapred</em> in the
above example), group-owned by anyone and should have the
permissions <em>0400 or r--------</em>.
</p>
<p>The executable requires following configuration items to be
present in the <em>taskcontroller.cfg</em> file. The items should
be mentioned as simple <em>key=value</em> pairs.
</p>
<table><tr><th>Name</th><th>Description</th></tr>
<tr>
<td>mapreduce.cluster.local.dir</td>
<td>Path to mapreduce.cluster.local.directories. Should be same as the value
which was provided to key in mapred-site.xml. This is required to
validate paths passed to the setuid executable in order to prevent
arbitrary paths being passed to it.</td>
</tr>
<tr>
<td>hadoop.log.dir</td>
<td>Path to hadoop log directory. Should be same as the value which
the TaskTracker is started with. This is required to set proper
permissions on the log files so that they can be written to by the user's
tasks and read by the TaskTracker for serving on the web UI.</td>
</tr>
<tr>
<td>mapreduce.tasktracker.group</td>
<td>Group to which the TaskTracker belongs. The group owner of the
taskcontroller binary should be this group. Should be same as
the value with which the TaskTracker is configured. This
configuration is required for validating the secure access of the
task-controller binary.</td>
</tr>
</table>
</section>
</section>
</section>
<section>
<title>Monitoring Health of TaskTracker Nodes</title>
<p>Hadoop Map/Reduce provides a mechanism by which administrators
can configure the TaskTracker to run an administrator supplied
script periodically to determine if a node is healthy or not.
Administrators can determine if the node is in a healthy state
by performing any checks of their choice in the script. If the
script detects the node to be in an unhealthy state, it must print
a line to standard output beginning with the string <em>ERROR</em>.
The TaskTracker spawns the script periodically and checks its
output. If the script's output contains the string <em>ERROR</em>,
as described above, the node's status is reported as 'unhealthy'
and the node is black-listed on the JobTracker. No further tasks
will be assigned to this node. However, the
TaskTracker continues to run the script, so that if the node
becomes healthy again, it will be removed from the blacklisted
nodes on the JobTracker automatically. The node's health
along with the output of the script, if it is unhealthy, is
available to the administrator in the JobTracker's web interface.
The time since the node was healthy is also displayed on the
web interface.
</p>
<section>
<title>Configuring the Node Health Check Script</title>
<p>The following parameters can be used to control the node health
monitoring script in <em>mapred-site.xml</em>.</p>
<table>
<tr><th>Name</th><th>Description</th></tr>
<tr><td><code>mapreduce.tasktracker.healthchecker.script.path</code></td>
<td>Absolute path to the script which is periodically run by the
TaskTracker to determine if the node is
healthy or not. The file should be executable by the TaskTracker.
If the value of this key is empty or the file does
not exist or is not executable, node health monitoring
is not started.</td>
</tr>
<tr>
<td><code>mapreduce.tasktracker.healthchecker.interval</code></td>
<td>Frequency at which the node health script is run,
in milliseconds</td>
</tr>
<tr>
<td><code>mapreduce.tasktracker.healthchecker.script.timeout</code></td>
<td>Time after which the node health script will be killed by
the TaskTracker if unresponsive.
The node is marked unhealthy. if node health script times out.</td>
</tr>
<tr>
<td><code>mapreduce.tasktracker.healthchecker.script.args</code></td>
<td>Extra arguments that can be passed to the node health script
when launched.
These should be comma separated list of arguments. </td>
</tr>
</table>
</section>
</section>
</section>
<section>
<title>Slaves</title>
<p>Typically you choose one machine in the cluster to act as the
<code>NameNode</code> and one machine as to act as the
<code>JobTracker</code>, exclusively. The rest of the machines act as
both a <code>DataNode</code> and <code>TaskTracker</code> and are
referred to as <em>slaves</em>.</p>
<p>List all slave hostnames or IP addresses in your
<code>conf/slaves</code> file, one per line.</p>
</section>
<section>
<title>Logging</title>
<p>Hadoop uses the <a href="http://logging.apache.org/log4j/">Apache
log4j</a> via the <a href="http://commons.apache.org/logging/">Apache
Commons Logging</a> framework for logging. Edit the
<code>conf/log4j.properties</code> file to customize the Hadoop
daemons' logging configuration (log-formats and so on).</p>
<section>
<title>History Logging</title>
<p> The job history files are stored in central location
<code> mapreduce.jobtracker.jobhistory.location </code> which can be on DFS also,
whose default value is <code>${HADOOP_LOG_DIR}/history</code>.
The history web UI is accessible from job tracker web UI.</p>
<p> The history files are also logged to user specified directory
<code>mapreduce.job.userhistorylocation</code>
which defaults to job output directory. The files are stored in
"_logs/history/" in the specified directory. Hence, by default
they will be in "mapreduce.output.fileoutputformat.outputdir/_logs/history/". User can stop
logging by giving the value <code>none</code> for
<code>mapreduce.job.userhistorylocation</code> </p>
<p> User can view the history logs summary in specified directory
using the following command <br/>
<code>$ bin/hadoop job -history output-dir</code><br/>
This command will print job details, failed and killed tip
details. <br/>
More details about the job such as successful tasks and
task attempts made for each task can be viewed using the
following command <br/>
<code>$ bin/hadoop job -history all output-dir</code><br/></p>
</section>
</section>
<p>Once all the necessary configuration is complete, distribute the files
to the <code>HADOOP_CONF_DIR</code> directory on all the machines,
typically <code>${HADOOP_PREFIX}/conf</code>.</p>
</section>
<section>
<title>Cluster Restartability</title>
<section>
<title>Map/Reduce</title>
<p>The job tracker restart can recover running jobs if
<code>mapreduce.jobtracker.restart.recover</code> is set true and
<a href="#Logging">JobHistory logging</a> is enabled. Also
<code>mapreduce.jobtracker.jobhistory.block.size</code> value should be
set to an optimal value to dump job history to disk as soon as
possible, the typical value is 3145728(3MB).</p>
</section>
</section>
<section>
<title>Hadoop Rack Awareness</title>
<p>The HDFS and the Map/Reduce components are rack-aware.</p>
<p>The <code>NameNode</code> and the <code>JobTracker</code> obtains the
<code>rack id</code> of the slaves in the cluster by invoking an API
<a href="ext:api/org/apache/hadoop/net/dnstoswitchmapping/resolve
">resolve</a> in an administrator configured
module. The API resolves the slave's DNS name (also IP address) to a
rack id. What module to use can be configured using the configuration
item <code>net.topology.node.switch.mapping.impl</code>. The default
implementation of the same runs a script/command configured using
<code>net.topology.script.file.name</code>. If topology.script.file.name is
not set, the rack id <code>/default-rack</code> is returned for any
passed IP address. The additional configuration in the Map/Reduce
part is <code>mapred.cache.task.levels</code> which determines the number
of levels (in the network topology) of caches. So, for example, if it is
the default value of 2, two levels of caches will be constructed -
one for hosts (host -> task mapping) and another for racks
(rack -> task mapping).
</p>
</section>
<section>
<title>Hadoop Startup</title>
<p>To start a Hadoop cluster you will need to start both the HDFS and
Map/Reduce cluster.</p>
<p>
Format a new distributed filesystem:<br/>
<code>$ bin/hadoop namenode -format</code>
</p>
<p>
Start the HDFS with the following command, run on the designated
<code>NameNode</code>:<br/>
<code>$ bin/start-dfs.sh</code>
</p>
<p>The <code>bin/start-dfs.sh</code> script also consults the
<code>${HADOOP_CONF_DIR}/slaves</code> file on the <code>NameNode</code>
and starts the <code>DataNode</code> daemon on all the listed slaves.</p>
<p>
Start Map-Reduce with the following command, run on the designated
<code>JobTracker</code>:<br/>
<code>$ bin/start-mapred.sh</code>
</p>
<p>The <code>bin/start-mapred.sh</code> script also consults the
<code>${HADOOP_CONF_DIR}/slaves</code> file on the <code>JobTracker</code>
and starts the <code>TaskTracker</code> daemon on all the listed slaves.
</p>
</section>
<section>
<title>Hadoop Shutdown</title>
<p>
Stop HDFS with the following command, run on the designated
<code>NameNode</code>:<br/>
<code>$ bin/stop-dfs.sh</code>
</p>
<p>The <code>bin/stop-dfs.sh</code> script also consults the
<code>${HADOOP_CONF_DIR}/slaves</code> file on the <code>NameNode</code>
and stops the <code>DataNode</code> daemon on all the listed slaves.</p>
<p>
Stop Map/Reduce with the following command, run on the designated
the designated <code>JobTracker</code>:<br/>
<code>$ bin/stop-mapred.sh</code><br/>
</p>
<p>The <code>bin/stop-mapred.sh</code> script also consults the
<code>${HADOOP_CONF_DIR}/slaves</code> file on the <code>JobTracker</code>
and stops the <code>TaskTracker</code> daemon on all the listed slaves.</p>
</section>
</body>
</document>