| ### "Gridmix" Benchmark ### |
| |
| Contents: |
| |
| 0 Overview |
| 1 Getting Started |
| 1.0 Build |
| 1.1 Configure |
| 1.2 Generate test data |
| 2 Running |
| 2.0 General |
| 2.1 Non-Hod cluster |
| 2.2 Hod |
| 2.2.0 Static cluster |
| 2.2.1 Hod cluster |
| |
| |
| * 0 Overview |
| |
| The scripts in this package model a cluster workload. The workload is |
| simulated by generating random data and submitting map/reduce jobs that |
| mimic observed data-access patterns in user jobs. The full benchmark |
| generates approximately 2.5TB of (often compressed) input data operated on |
| by the following simulated jobs: |
| |
| 1) Three stage map/reduce job |
| Input: 500GB compressed (2TB uncompressed) SequenceFile |
| (k,v) = (5 words, 100 words) |
| hadoop-env: FIXCOMPSEQ |
| Compute1: keep 10% map, 40% reduce |
| Compute2: keep 100% map, 77% reduce |
| Input from Compute1 |
| Compute3: keep 116% map, 91% reduce |
| Input from Compute2 |
| Motivation: Many user workloads are implemented as pipelined map/reduce |
| jobs, including Pig workloads |
| |
| 2) Large sort of variable key/value size |
| Input: 500GB compressed (2TB uncompressed) SequenceFile |
| (k,v) = (5-10 words, 100-10000 words) |
| hadoop-env: VARCOMPSEQ |
| Compute: keep 100% map, 100% reduce |
| Motivation: Processing large, compressed datsets is common. |
| |
| 3) Reference select |
| Input: 500GB compressed (2TB uncompressed) SequenceFile |
| (k,v) = (5-10 words, 100-10000 words) |
| hadoop-env: VARCOMPSEQ |
| Compute: keep 0.2% map, 5% reduce |
| 1 Reducer |
| Motivation: Sampling from a large, reference dataset is common. |
| |
| 4) Indirect Read |
| Input: 500GB compressed (2TB uncompressed) Text |
| (k,v) = (5 words, 20 words) |
| hadoop-env: FIXCOMPTEXT |
| Compute: keep 50% map, 100% reduce Each map reads 1 input file, |
| adding additional input files from the output of the |
| previous iteration for 10 iterations |
| Motivation: User jobs in the wild will often take input data without |
| consulting the framework. This simulates an iterative job |
| whose input data is all "indirect," i.e. given to the |
| framework sans locality metadata. |
| |
| 5) API text sort (java, pipes, streaming) |
| Input: 500GB uncompressed Text |
| (k,v) = (1-10 words, 0-200 words) |
| hadoop-env: VARINFLTEXT |
| Compute: keep 100% map, 100% reduce |
| Motivation: This benchmark should exercise each of the APIs to |
| map/reduce |
| |
| Each of these jobs may be run individually or- using the scripts provided- |
| as a simulation of user activity sized to run in approximately 4 hours on a |
| 480-500 node cluster using Hadoop 0.15.0. The benchmark runs a mix of small, |
| medium, and large jobs simultaneously, submitting each at fixed intervals. |
| |
| Notes(1-4): Since input data are compressed, this means that each mapper |
| outputs a lot more bytes than it reads in, typically causing map output |
| spills. |
| |
| |
| |
| * 1 Getting Started |
| |
| 1.0 Build |
| |
| 1) Compile the examples, including the C++ sources: |
| > ant -Dcompile.c++=yes examples |
| 2) Copy the pipe sort example to a location in the default filesystem |
| (usually HDFS, default /gridmix/programs) |
| > $HADOOP_HOME/hadoop dfs -mkdir $GRID_MIX_PROG |
| > $HADOOP_HOME/hadoop dfs -put build/c++-examples/$PLATFORM_STR/bin/pipes-sort $GRID_MIX_PROG |
| |
| 1.1 Configure |
| |
| One must modify hadoop-env to supply the following information: |
| |
| HADOOP_HOME The hadoop install location |
| GRID_MIX_HOME The location of these scripts |
| APP_JAR The location of the hadoop example |
| GRID_MIX_DATA The location of the datsets for these benchmarks |
| GRID_MIX_PROG The location of the pipe-sort example |
| |
| Reasonable defaults are provided for all but HADOOP_HOME. The datasets used |
| by each of the respective benchmarks are recorded in the Input::hadoop-env |
| comment in section 0 and their location may be changed in hadoop-env. Note |
| that each job expects particular input data and the parameters given to it |
| must be changed in each script if a different InputFormat, keytype, or |
| valuetype is desired. |
| |
| Note that NUM_OF_REDUCERS_FOR_*_JOB properties should be sized to the |
| cluster on which the benchmarks will be run. The default assumes a large |
| (450-500 node) cluster. |
| |
| 1.2 Generate test data |
| |
| Test data is generated using the generateData.sh script. While one may |
| modify the structure and size of the data generated here, note that many of |
| the scripts- particularly for medium and small sized jobs- rely not only on |
| specific InputFormats and key/value types, but also on a particular |
| structure to the input data. Changing these values will likely be necessary |
| to run on small and medium-sized clusters, but any modifications must be |
| informed by an explicit familiarity with the underlying scripts. |
| |
| It is sufficient to run the script without modification, though it may |
| require up to 4TB of free space in the default filesystem. Changing the size |
| of the input data (COMPRESSED_DATA_BYTES, UNCOMPRESSED_DATA_BYTES, |
| INDIRECT_DATA_BYTES) is safe. A 4x compression ratio for generated, block |
| compressed data is typical. |
| |
| * 2 Running |
| |
| 2.0 General |
| |
| The submissionScripts directory contains the high-level scripts submitting |
| sized jobs for the gridmix benchmark. Each submits $NUM_OF_*_JOBS_PER_CLASS |
| instances as specified in the gridmix-env script, where an instance is an |
| invocation of a script as in $JOBTYPE/$JOBTYPE.$CLASS (e.g. |
| javasort/text-sort.large). Each instance may submit one or more map/reduce |
| jobs. |
| |
| There is a backoff script, submissionScripts/sleep_if_too_busy that can be |
| modified to define throttling criteria. By default, it simply counts running |
| java processes. |
| |
| 2.1 Non-Hod cluster |
| |
| The submissionScripts/allToSameCluster script will invoke each of the other |
| submission scripts for the gridmix benchmark. Depending on how your cluster |
| manages job submission, these scripts may require modification. The details |
| are very context-dependent. |
| |
| 2.2 Hod |
| |
| Note that there are options in hadoop-env that control jobs sumitted thruogh |
| Hod. One may specify the location of a config (HOD_CONFIG), the number of |
| nodes to allocate for classes of jobs, and any additional options one wants |
| to apply. The default includes an example for supplying a Hadoop tarball for |
| testing platform changes (see Hod documentation). |
| |
| 2.2.0 Static Cluster |
| |
| > hod --hod.script=submissionScripts/allToSameCluster -m 500 |
| |
| 2.2.1 Hod-allocated cluster |
| |
| > ./submissionScripts/allThroughHod |