| <?xml version="1.0"?> |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| <!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd"> |
| |
| <document> |
| |
| <header> |
| <title>Gridmix</title> |
| </header> |
| |
| <body> |
| |
| <section> |
| <title>Overview</title> |
| |
| <p>Gridmix is a benchmark for live clusters. It submits a mix of synthetic |
| jobs, modeling a profile mined from production loads.</p> |
| |
| <p>There exist three versions of the Gridmix tool. This document discusses |
| the third (checked into contrib), distinct from the two checked into the |
| benchmarks subdirectory. While the first two versions of the tool included |
| stripped-down versions of common jobs, both were principally saturation |
| tools for stressing the framework at scale. In support of a broader range of |
| deployments and finer-tuned job mixes, this version of the tool will attempt |
| to model the resource profiles of production jobs to identify bottlenecks, |
| guide development, and serve as a replacement for the existing gridmix |
| benchmarks.</p> |
| |
| </section> |
| |
| <section id="usage"> |
| |
| <title>Usage</title> |
| |
| <p>To run Gridmix, one requires a job trace describing the job mix for a |
| given cluster. Such traces are typically genenerated by Rumen (see related |
| documentation). Gridmix also requires input data from which the synthetic |
| jobs will draw bytes. The input data need not be in any particular format, |
| as the synthetic jobs are currently binary readers. If one is running on a |
| new cluster, an optional step generating input data may precede the run.</p> |
| |
| <p>Basic command line usage:</p> |
| <source> |
| |
| bin/mapred org.apache.hadoop.mapred.gridmix.Gridmix [-generate <MiB>] <iopath> <trace> |
| </source> |
| |
| <p>The <code>-generate</code> parameter accepts standard units, e.g. |
| <code>100g</code> will generate 100 * 2<sup>30</sup> bytes. The |
| <iopath> parameter is the destination directory for generated and/or |
| the directory from which input data will be read. The <trace> |
| parameter is a path to a job trace. The following configuration parameters |
| are also accepted in the standard idiom, before other Gridmix |
| parameters.</p> |
| |
| <section> |
| <title>Configuration parameters</title> |
| <p></p> |
| <table> |
| <tr><th> Parameter </th><th> Description </th><th> Notes </th></tr> |
| <tr><td><code>gridmix.output.directory</code></td> |
| <td>The directory into which output will be written. If specified, the |
| <code>iopath</code> will be relative to this parameter.</td> |
| <td>The submitting user must have read/write access to this |
| directory. The user should also be mindful of any quota issues that |
| may arise during a run.</td></tr> |
| <tr><td><code>gridmix.client.submit.threads</code></td> |
| <td>The number of threads submitting jobs to the cluster. This also |
| controls how many splits will be loaded into memory at a given time, |
| pending the submit time in the trace.</td> |
| <td>Splits are pregenerated to hit submission deadlines, so |
| particularly dense traces may want more submitting threads. However, |
| storing splits in memory is reasonably expensive, so one should raise |
| this cautiously.</td></tr> |
| <tr><td><code>gridmix.client.pending.queue.depth</code></td> |
| <td>The depth of the queue of job descriptions awaiting split |
| generation.</td> |
| <td>The jobs read from the trace occupy a queue of this depth before |
| being processed by the submission threads. It is unusual to configure |
| this.</td></tr> |
| <tr><td><code>gridmix.min.key.length</code></td> |
| <td>The key size for jobs submitted to the cluster.</td> |
| <td>While this is clearly a job-specific, even task-specific property, |
| no data on key length is currently available. Since the intermediate |
| data are random, memcomparable data, not even the sort is likely |
| affected. It exists as a tunable as no default value is appropriate, |
| but future versions will likely replace it with trace data.</td></tr> |
| </table> |
| |
| </section> |
| </section> |
| |
| <section id="assumptions"> |
| |
| <title>Simplifying Assumptions</title> |
| |
| <p>Gridmix will be developed in stages, incorporating feedback and patches |
| from the community. Currently, its intent is to evaluate Map/Reduce and HDFS |
| performance and not the layers on top of them (i.e. the extensive lib and |
| subproject space). Given these two limitations, the following |
| characteristics of job load are not currently captured in job traces and |
| cannot be accurately reproduced in Gridmix.</p> |
| |
| <table> |
| <tr><th>Property</th><th>Notes</th></tr> |
| <tr><td>CPU usage</td><td>We have no data for per-task CPU usage, so we |
| cannot attempt even an approximation. Gridmix tasks are never CPU bound |
| independent of I/O, though this surely happens in practice.</td></tr> |
| <tr><td>Filesystem properties</td><td>No attempt is made to match block |
| sizes, namespace hierarchies, or any property of input, intermediate, or |
| output data other than the bytes/records consumed and emitted from a given |
| task. This implies that some of the most heavily used parts of the system- |
| the compression libraries, text processing, streaming, etc.- cannot be |
| meaningfully tested with the current implementation.</td></tr> |
| <tr><td>I/O rates</td><td>The rate at which records are consumed/emitted is |
| assumed to be limited only by the speed of the reader/writer and constant |
| throughout the task.</td></tr> |
| <tr><td>Memory profile</td><td>No data on tasks' memory usage over time is |
| available, though the max heap size is retained.</td></tr> |
| <tr><td>Skew</td><td>The records consumed and emitted to/from a given task |
| are assumed to follow observed averages, i.e. records will be more regular |
| than may be seen in the wild. Each map also generates a proportional |
| percentage of data for each reduce, so a job with unbalanced input will be |
| flattened.</td></tr> |
| <tr><td>Job failure</td><td>User code is assumed to be correct.</td></tr> |
| <tr><td>Job independence</td><td>The output or outcome of one job does not |
| affect when or whether a subsequent job will run.</td></tr> |
| </table> |
| |
| </section> |
| |
| <section> |
| |
| <title>Appendix</title> |
| |
| <p>Issues tracking the implementations of <a |
| href="https://issues.apache.org/jira/browse/HADOOP-2369">gridmix1</a>, <a |
| href="https://issues.apache.org/jira/browse/HADOOP-3770">gridmix2</a>, and |
| <a href="https://issues.apache.org/jira/browse/MAPREDUCE-776">gridmix3</a>. |
| Other issues tracking the development of Gridmix can be found by searching |
| the Map/Reduce <a |
| href="https://issues.apache.org/jira/browse/MAPREDUCE">JIRA</a></p> |
| |
| </section> |
| |
| </body> |
| |
| </document> |