_posts/2012-05-22-apache_mrunit_0_9_0.html

---
layout: post
status: PUBLISHED
published: true
title: Apache MRUnit 0.9.0-incubating has been released!
id: 5c036e07-ed39-47a6-adba-18d4f9b9b17b
date: '2012-05-22 01:01:44 -0400'
categories: mrunit
tags:
- mapreduce
- hadoop
- mrunit
permalink: mrunit/entry/apache_mrunit_0_9_0
---
<p>
We (the Apache MRUnit team) have just released Apache MRUnit 0.9.0-incubating (<a href="http://www.apache.org/dyn/closer.cgi/incubator/mrunit/">tarball</a>, <a href="https://repository.apache.org/index.html#nexus-search;gav~org.apache.mrunit~~~~">nexus</a>, <a href="http://incubator.apache.org/mrunit/documentation/javadocs/0.9.0-incubating/index.html">javadoc</a>). Apache MRUnit is an Apache Incubator project. MRUnit is a Java library that helps developers unit test Apache Hadoop MapReduce jobs. Unit testing is a technique for improving project quality and reducing overall costs by writing a small amount of code that can automatically verify the software you write performs as intended. This is considered a best practice in software development since it helps identify defects early, before they're deployed to a production system.</p>
<p>
The MRUnit project is quite active, 0.9.0 is our fourth release since entering the incubator and we have added 4 new committers beyond the projects initial charter! We are very interested in having new contributors and committers join the project! Please join our <a href="http://incubator.apache.org/mrunit/community/mailing_lists.html">mailing list</a> to find out how you can help!</p>
<p>
The MRUnit build process has changed to produce mrunit-0.9.0-hadoop1.jar and mrunit-0.9.0-hadoop2.jar instead of mrunit-0.9.0-hadoop020.jar, mrunit-0.9.0-hadoop100.jar and mrunit-0.9.0-hadoop023.jar. The hadoop1 classifier is for all Apahce Hadoop versions based off the 0.20.X line including 1.0.X. The hadoop2 classifier is for all Apache Hadoop versions based off the 0.23.X line including the unreleased 2.0.X.</p>
<p>
This <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311292&version=12316360">release</a> contains 6 bug fixes, 15 improvements, and 2 new features. I will highlight a few below:</p>
<ul>
<li>Support custom counter checking in <a href="https://issues.apache.org/jira/browse/MRUNIT-68">MRUNIT-68</a></li>
<li>runTest() should optionally ignore output order in <a href="https://issues.apache.org/jira/browse/MRUNIT-91">MRUNIT-91</a></li>
<li>Driver.runTest throws RuntimeException should it throw AssertionError in <a href="https://issues.apache.org/jira/browse/MRUNIT-54">MRUNIT-54</a></li>
<li>o.a.h.mrunit.mapreduce.MapReduceDriver should support a combiner in <a href="https://issues.apache.org/jira/browse/MRUNIT-67">MRUNIT-67</a></li>
<li>Better support for other serializations besides Writable:  <a href="https://issues.apache.org/jira/browse/MRUNIT-70">MRUNIT-70</a>,  <a href="https://issues.apache.org/jira/browse/MRUNIT-86">MRUNIT-86</a>,  <a href="https://issues.apache.org/jira/browse/MRUNIT-99">MRUNIT-99</a>,  <a href="https://issues.apache.org/jira/browse/MRUNIT-77">MRUNIT-77</a></li>
<li>Better error messages from validate, null checking and forgetting to set mappers and reducers: <a href="https://issues.apache.org/jira/browse/MRUNIT-74">MRUNIT-74</a>, <a href="https://issues.apache.org/jira/browse/MRUNIT-66">MRUNIT-66</a>, <a href="https://issues.apache.org/jira/browse/MRUNIT-65">MRUNIT-65</a></li>
<li>add static convenience methods to PipelineMapReduceDriver class in <a href="https://issues.apache.org/jira/browse/MRUNIT-89">MRUNIT-89</a></li>
<li>Test and Deprecate Driver.{*OutputFromString,*InputFromString} Methods in <a href="https://issues.apache.org/jira/browse/MRUNIT-48">MRUNIT-48</a></li>
</ul>
<p><h3><u>Support custom counter checking</u></h3></p>
<p>It has always been possible to check the counter values like so:</p>
<pre>
assertEquals(2, mapDriver.getCounters().findCounter(CustomMapper.CustomCounter.NAME).getValue()); 
</pre>
<p>but this is quite tedious. As such Jarek Jarcec Cecho (our second newest committer) added this feature directly to the drivers:</p>
<pre>
.withCounter(CustomMapper.CustomCounter.Name, 2);
</pre>
<p><h3><u>runTest() should optionally ignore output order</u></h3></p>
<p>Previous to this change MRUnit required Mapper/Reducer classes to output key value pairs in the order specified on the test. Well defined output order is common, but strictly not universal. Dave Beech (our newest committer) contributed a patch so you optionally turn this ordered requirement off by using:</p>
<pre>
.runTest(false)
</pre>
<p>instead of</p>
<pre>
.runTest()
</pre>
<p><h3><u>Driver.runTest throws RuntimeException should it throw AssertionError</u></h3></p>
<p>Previous versions of MRUnit threw a RuntimeException when a test failed. This worked well, but it meant that testing frameworks saw the the test as having erred, not failed. We have changed this to AssertionError so that testing frameworks see the tests as failed. The distinction is small but important.</p>
<p><h3><u>o.a.h.mrunit.mapreduce.MapReduceDriver should support a combiner</u></h3></p>
<p>Previously the MRUnit only supported a combiner in the mapred MapReduceDriver class but now the mapreduce MapReduceDriver also supports a combiner by:</p>
<pre>MapReduceDriver.newMapReduceDriver(mapper, reducer, combiner)</pre>
<p>or</p>
<pre>.withCombiner(combiner) or .setCombiner(combiner)</pre>
<p><h3><u>Better support for other serializations besides Writable</u></h3></p>
<p>Previous versions of MRUnit did not support JavaSerialization, Avro or other Serialization frameworks well. We improved alternative serialization support by not forcing K2 in MapReduceDriver to be Comparable and supporting serializations that cannot clone into a object or that do not have default constructors.</p>
<p><h3><u>Better error messages from validate, null checking and forgetting to set mappers and reducers</u></h3></p>
<p>We have improved checking of parameters passed to MRUnit and the error messages when the parameters are invalid including throwing NullPointerException immediately when receiving a null value and throwing a IllegalStateExcpetion when no mapper or reducer class is provided instead of a NullPointerException.</p>
<p><h3><u>add static convenience methods to PipelineMapReduceDriver class</u></h3></p>
<p>add static convenience constructors similar to those in the other driver classes:</p>
<pre>PipelineMapReduceDriver.newPipelineMapReduceDriver()</pre>
<p>or</p>
<pre>PipelineMapReduceDriver.newPipelineMapReduceDriver(list of Pair<Mapper, Reducer>)</pre>
<p><h3><u>Test and Deprecate Driver.{*OutputFromString,*InputFromString} Methods</u></h3></p>
<p>The OutputFromString and InputFromString methods are now deprecated because they required Text inputs or outputs with no way to enforce that the inputs or outputs from a mapper or reducer were actually Text. These methods also provided little convenience as a user can just pass the string they intended to new Text(string)</p>