<?xml version="1.0"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">

<document>
  
  <header>
    <title>Capacity Scheduler</title>
  </header>
  
  <body>
  
    <section>
      <title>Purpose</title>
      
      <p>This document describes the Capacity Scheduler, a pluggable 
      MapReduce scheduler for Hadoop which provides a way to share 
      large clusters.</p>
    </section>
    
    <section>
      <title>Features</title>
      
      <p>The Capacity Scheduler supports the following features:</p> 
      <ul>
        <li>
          Multiple queues, possibly hierarchical/recursive, where a job is
          submitted to a queue.
        </li>
        <li>
          Queues are allocated a fraction of the capacity of the grid in the 
          sense that a certain capacity of resources will be at their 
          disposal. All jobs submitted to a queue will have access to the 
          capacity allocated to the queue.
        </li>
        <li>
          Free resources can be allocated to any queue beyond it's capacity. 
          When there is demand for these resources from queues running below 
          capacity at a future point in time, as tasks scheduled on these 
          resources complete, they will be assigned to jobs on queues 
          running below the capacity.
        </li>
        <li>
          Queues optionally support job priorities (disabled by default).
        </li>
        <li>
          Within a queue, jobs with higher priority will have access to the 
          queue's resources before jobs with lower priority. However, once a 
          job is running, it will not be preempted for a higher priority job,
          though new tasks from the higher priority job will be 
          preferentially scheduled.
        </li>
        <li>
          In order to prevent one or more users from monopolizing its 
          resources, each queue enforces a limit on the percentage of 
          resources allocated to a user at any given time, if there is 
          competition for them.  
        </li>
        <li>
          Queues can use idle resources of other queues. In order to prevent
          monopolizing of resources by particular queues, each queue can be
          set a cap on the maximum number of resources it can expand to in
          the presence of idle resources in other queues of the cluster.
        </li>
        <li>
          Support for memory-intensive jobs, wherein a job can optionally 
          specify higher memory-requirements than the default, and the tasks 
          of the job will only be run on TaskTrackers that have enough memory 
          to spare.
        </li>
        <li>
          Support for refreshing/reloading some of the queue-properties
          without restarting the JobTracker, taking advantage of the
          <a href="ext:cluster-setup/RefreshingQueueConfiguration">
          queue-refresh</a> feature in the framework.
        </li>
      </ul>
    </section>
    
    <section>
      <title>Picking a Task to Run</title>
      
      <p>Note that many of these steps can be, and will be, enhanced over time
      to provide better algorithms.</p>
      
      <p>Whenever a TaskTracker is free, the Capacity Scheduler picks 
      a queue which has most free space (whose ratio of # of running slots to 
      capacity is the lowest).</p>
      
      <p>Once a queue is selected, the Scheduler picks a job in the queue. Jobs
      are sorted based on when they're submitted and their priorities (if the 
      queue supports priorities). Jobs are considered in order, and a job is 
      selected if its user is within the user-quota for the queue, i.e., the 
      user is not already using queue resources above his/her limit. The 
      Scheduler also makes sure that there is enough free memory in the 
      TaskTracker to tun the job's task, in case the job has special memory
      requirements.</p>
      
      <p>Once a job is selected, the Scheduler picks a task to run. This logic 
      to pick a task remains unchanged from earlier versions.</p> 

      <section>
        <title>Scheduling Tasks Considering Memory Requirements</title>
      
        <p>
        The Capacity Scheduler supports scheduling of tasks on a
        TaskTracker based on a job's virtual memory requirements and 
        the availability
        of enough virtual memory on the TaskTracker node. By doing so, it
        simplifies the virtual memory monitoring function on the 
        TaskTracker node, described in the section on 
        <a href="ext:cluster-setup/ConfiguringMemoryParameters">
        Monitoring Task Memory Usage</a> in the Cluster Setup guide. 
        Refer to that section for more details on how memory for 
        MapReduce tasks is handled.
        </p>
        
        <p>
        Virtual memory based task scheduling uses the same parameters as
        the memory monitoring function of the TaskTracker, and is enabled
        along with virtual memory monitoring. When enabled, the scheduler
        ensures that a task is scheduled on a TaskTracker only when the
        virtual memory required by the map or reduce task can be assured 
        by the TaskTracker. That is, the task is scheduled only if the 
        following constraint is satisfied:<br/>
        <code>
        Job's mapreduce.{map|reduce}.memory.mb of the job &lt;=
          total virtual memory for all map or reduce tasks on the TaskTracker -
            total virtual memory required for all running map or reduce tasks 
            on the TaskTracker
        </code><br/>
        </p>
        
        <p>
        When a task at the front of the scheduler's queue cannot be scheduled
        on a TaskTracker due to insufficient memory, the scheduler creates
        a virtual <em>reservation</em> for this task. This can continue
        for all pending tasks on a job, subject to other capacity constraints.
        Once all tasks are either scheduled or have reservations, the 
        scheduler will proceed to schedule other jobs's tasks that are not 
        necessarily at the front of the queue, but meet memory constraints 
        of the TaskTracker.
        By following this reservation procedure of reserving just enough
        TaskTrackers, the scheduler balances between not starving jobs with 
        high memory requirements and under-utilizing cluster resources.
        </p>

        <p>
        Tasks of jobs that require more virtual memory than the
        per slot <code>mapreduce.cluster.{map|reduce}memory.mb</code>
        value, are treated as occupying more than one slot, and account
        for a corresponding increased capacity usage for their queue.
        The number of slots they occupy is determined as:<br/>
        <code>
        Number of slots for a task = mapreduce.{map|reduce}.memory.mb / 
        mapreduce.cluster.{map|reduce}memory.mb
        </code><br/>
        However, special tasks run by the framework like setup 
        and cleanup tasks do not count for more than 1 slot, 
        irrespective of their job's memory requirements.
        </p>

      </section>
      
    </section>
    
    <section>
      <title>Installation</title>
      
        <p>The Capacity Scheduler is available as a JAR file in the Hadoop
        tarball under the <em>contrib/capacity-scheduler</em> directory. The name of 
        the JAR file would be on the lines of hadoop-*-capacity-scheduler.jar.</p>
        <p>You can also build the Scheduler from source by executing
        <em>ant package</em>, in which case it would be available under
        <em>build/contrib/capacity-scheduler</em>.</p>
        <p>To run the Capacity Scheduler in your Hadoop installation, you need 
        to put it on the <em>CLASSPATH</em>. The easiest way is to copy the 
        <code>hadoop-*-capacity-scheduler.jar</code> from 
        to <code>HADOOP_HOME/lib</code>. Alternatively, you can modify 
        <em>HADOOP_CLASSPATH</em> to include this jar, in 
        <code>conf/hadoop-env.sh</code>.</p>
    </section>

    <section>
      <title>Configuration</title>

      <section>
        <title>Using the Capacity Scheduler</title>
        <p>
          To make the Hadoop framework use the Capacity Scheduler, set up
          the following property in the site configuration:</p>
          <table>
            <tr>
              <th>Name</th>
              <th>Value</th>
            </tr>
            <tr>
              <td>mapreduce.jobtracker.taskscheduler</td>
              <td>org.apache.hadoop.mapred.CapacityTaskScheduler</td>
            </tr>
          </table>
      </section>

      <section>
        <title>Setting Up Queues</title>
        <p>
          You can define multiple, possibly hierarchical queues to which users
          can submit jobs with the Capacity Scheduler. To define queues,
          various properties should be set in two configuration files -
          <a href="ext:cluster-setup/mapred-queues.xml">mapred-queues.xml</a>
          and
          <a href="ext:capacity-scheduler-conf">conf/capacity-scheduler.xml</a>
          .</p>
          <p><em>conf/capacity-scheduler.xml</em> can be used to configure (1)
          job-initialization-poller related properties and (2) the
          default values for various properties in the queues</p>
          <p><em>conf/mapred-queues.xml</em> contains the actual queue
          configuration including (1) framework specific properties like ACLs
          for controlling which users or groups have access to the queues and
          state of the queues and (2) the scheduler specific properties for
          each queue. If any of these scheduler specific properties are
          missing and not configured for a queue, then the properties in
          <em>conf/capacity-scheduler.xml</em> are used to set default values.
          More details about the properties that can be configured, and their
          semantics is mentioned below. Also, a default template for 
          mapred-queues.xml tailored for using with
          Capacity-scheduler can be found
          <a href="ext:mapred-queues-capacity-scheduler">here</a>.</p>
      </section>
  
      <section>
        <title>Configuring Properties for Queues</title>

        <p>The Capacity Scheduler can be configured with several properties
        for each queue that control the behavior of the Scheduler. As
        described above, this scheduler specific configuration has to be in
        the <em>conf/mapred-queues.xml</em> along with the rest of the
        framework specific configuration. By
        default, the configuration is set up for one queue, named 
        <em>default</em>.</p>
        <p>To specify a property for a specific queue that is defined in the
        mapred-queues.xml, you should set the corresponding property in a
        &lt;property&gt; tag explained
        <a href="ext:cluster-setup/property_tag">here</a>.
        </p>

        <p>The properties defined for queues and their descriptions are
        listed in the table below:</p>

        <table>
          <tr>
          	<th>Name</th>
            <th>
            	<a href="ext:commands-manual/RefreshQueues">
            	Refresh-able?</a>
           	</th>
            <th>Applicable to?</th>
            <th>Description</th>
          </tr>
          <tr>
          	<td>capacity</td>
          	<td>Yes</td>
          	<td>Container queues as well as leaf queues</td>
          	<td>For a root-level container queue, this is the percentage of the
          	number of slots in the cluster that will be available for all its
          	immediate children together. For a root-level leaf-queue, this is
          	the percentage of the number of slots in the cluster that will be
          	available for all its jobs.	For a non-root level container queue,
          	this is the percentage of the number of slots in its parent queue
          	that will be available for all its	children together. For a
          	non-root-level leaf queue, this	is the percentage of the number of
          	slots in its parent queue that will be available for jobs in this
          	queue. The sum of capacities for all children of a container queue
          	should be less than or equal 100. The sum of capacities of all the
          	root-level queues should be less than or equal to 100.
            </td>
          </tr>
          <tr>
            <td>maximum-capacity</td>
            <td>Yes</td>
          	<td>Container queues as well as leaf queues</td>
            <td>
	          A limit in percentage beyond which a non-root-level queue cannot use
	          the capacity of its parent queue; for a root-level queue, this is
	          the limit in percentage beyond which it cannot use the
	          cluster-capacity. This property provides a means to limit how much
	          excess capacity a queue can use.  It can be used to prevent queues
	          with long running jobs from occupying more than a certain percentage
	          of the parent-queue or the cluster, which, in the absence of
	          pre-emption, can lead to capacity guarantees of other queues getting
	          affected.
	
	          The maximum-capacity of a queue can only be greater than or equal to
	          its capacity. By default, there is no limit for a queue. For a
	          non-root-level queue this means it can occupy till the
	          maximum-capacity of its parent, for a root-level queue, it means that
	          it can occupy the whole cluster. A value of 100 implies that a queue
	          can use the complete capacity of its parent, or the complete
	          cluster-capacity in case of root-level-queues.
            </td>
          </tr>
          <tr>
          	<td>supports-priority</td>
          	<td>No</td>
          	<td>Leaf queues only</td>
          	<td>If true, priorities of jobs will be taken into account in scheduling 
          	decisions.
          	</td>
          </tr>
          <tr>
          	<td>minimum-user-limit-percent</td>
          	<td>Yes</td>
          	<td>Leaf queues only</td>
          	<td>Each queue enforces a limit on the percentage of resources 
          	allocated to a user at any given time, if there is competition 
          	for them. This user limit can vary between a minimum and maximum 
          	value. The former depends on the number of users who have submitted
          	jobs, and the latter is set to this property value. For example, 
          	suppose the value of this property is 25. If two users have 
          	submitted jobs to a queue, no single user can use more than 50% 
          	of the queue resources. If a third user submits a job, no single 
          	user can use more than 33% of the queue resources. With 4 or more 
          	users, no user can use more than 25% of the queue's resources. A 
          	value of 100 implies no user limits are imposed.
	        </td>
          </tr>
          <tr>
            <td>maximum-initialized-jobs-per-user</td>
            <td>Yes</td>
          	<td>Leaf queues only</td>
            <td>
              Maximum number of jobs which are allowed to be pre-initialized for
              a particular user in the queue. Once a job is scheduled, i.e.
              it starts running, then that job is not considered
              while scheduler computes the maximum job a user is allowed to
              initialize.
            </td>
          </tr>  
        </table>
        <p>See  <a href="ext:mapred-queues-capacity-scheduler">
        this configuration file</a> for a default configuration of queues in
        capacity-scheduler.</p>
      </section>
      
   <section>
        <title>Job Initialization Parameters</title>
        <p>Capacity scheduler lazily initializes the jobs before they are
        scheduled, for reducing the memory footprint on jobtracker. 
        Following are the parameters, by which you can control the laziness
        of the job initialization. The following parameters can be 
        configured in capacity-scheduler.xml:
        </p>
        
        <table>
          <tr><th>Name</th><th>Description</th></tr>
          <tr>
            <td>
              mapred.capacity-scheduler.init-poll-interval
            </td>
            <td>
              Amount of time in miliseconds which is used to poll the scheduler
              job queue to look for jobs to be initialized.
            </td>
          </tr>
          <tr>
            <td>
              mapred.capacity-scheduler.init-worker-threads
            </td>
            <td>
              Number of worker threads which would be used by Initialization
              poller to initialize jobs in a set of queue. If number mentioned 
              in property is equal to number of job queues then a thread is 
              assigned jobs from one queue. If the number configured is lesser than
              number of queues, then a thread can get jobs from more than one queue
              which it initializes in a round robin fashion. If the number configured
              is greater than number of queues, then number of threads spawned
              would be equal to number of job queues.
            </td>
          </tr>
        </table>
      </section>   
      <section>
        <title>Reviewing the Configuration of the Capacity Scheduler</title>
        <p>
          Once the installation and configuration is completed, you can review
          it after starting the MapReduce cluster from the admin UI.
        </p>
        <ul>
          <li>Start the MapReduce cluster as usual.</li>
          <li>Open the JobTracker web UI.</li>
          <li>The queues you have configured should be listed under the <em>Scheduling
              Information</em> section of the page.</li>
          <li>The properties for the queues should be visible in the <em>Scheduling
              Information</em> column against each queue.</li>
        </ul>
      </section>
      
   </section>
  </body>
  
</document>