MRUnit 0.9.0-incubating release.
git-svn-id: https://svn.apache.org/repos/asf/incubator/mrunit/tags/release-0.9.0-incubating@1327662 13f79535-47bb-0310-9956-ffa450edef68
diff --git a/BIN-NOTICE.txt b/BIN-NOTICE.txt
new file mode 100644
index 0000000..8c7f807
--- /dev/null
+++ b/BIN-NOTICE.txt
@@ -0,0 +1,14 @@
+Apache MRUnit
+Copyright 2011-2012 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+This product includes Mockito (Copyright (c) 2007 Mockito contributors)
+(http://code.google.com/p/mockito/) distributed under the MIT license.
+
+This product includes Hamcrest (Copyright (c) 2000-2006, www.hamcrest.org)
+(http://code.google.com/p/hamcrest/) distributed under the BSD 2-Clause License.
+
+This product includes JUnit (Copyright (c) 2006 JUnit.org)
+(http://junit.sourceforge.net/) distributed under the Common Public License.
diff --git a/BUILD.txt b/BUILD.txt
index a2733be..859488d 100644
--- a/BUILD.txt
+++ b/BUILD.txt
@@ -12,17 +12,13 @@
Building MRUnit is dependent upon the version of Hadoop Map Reduce you intend to use.
-Hadoop Map Reduce 1.0.0:
+Hadoop Map Reduce 1.X or 0.20.X:
$ mvn package
-This produces mrunit-${VERSION}-hadoop100.tar.gz and mrunit-${VERSION}-hadoop100.jar in the target directory.
+This produces mrunit-${VERSION}-hadoop1.jar in the target directory.
-Hadoop Map Reduce 0.23.1:
-$ mvn package -Dhadoop.version=0.23
-This produces mrunit-${VERSION}-hadoop023.tar.gz and mrunit-${VERSION}-hadoop023.jar in the target directory.
-
-Hadoop Map Reduce 0.20.2:
-$ mvn package -Dhadoop.version=0.20
-This produces mrunit-${VERSION}-hadoop020.tar.gz and mrunit-${VERSION}-hadoop020.jar in the target directory.
+Hadoop Map Reduce 2.X or 0.23.X:
+$ mvn package -Dhadoop.version=2
+This produces mrunit-${VERSION}-hadoop2.jar in the target directory.
If you wish you skip the unit tests (not advised):
diff --git a/CHANGES.txt b/CHANGES.txt
index 877a81a..5595137 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -5,6 +5,8 @@
* [MRUNIT-70] - copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
* [MRUNIT-77] - cannot use serializations that do not clone into object, or serializations without no arg constructors with reduce driver
* [MRUNIT-86] - Configuration not passed to individual mappers/reducers in mapred MapReducerDriver, PipelineMapReduceDriver
+ * [MRUNIT-97] - InputSplit causes NullPointerException in mapreduce api
+ * [MRUNIT-99] - dont force K2 in MapReduceDriver's to be Comparable since Hadoop dont enforce this on the map output key and value classes
** Dependency upgrade
* [MRUNIT-90] - upgrade junit to 4.10 and change ExpectedSuppliedException to implement TestRule instead
@@ -24,11 +26,17 @@
* [MRUNIT-89] - add static convenience methods to PipelineMapReduceDriver class
* [MRUNIT-91] - runTest() should optionally ignore output order
* [MRUNIT-92] - cleanup multiple release profile and javadoc handling in pom
+ * [MRUNIT-96] - Cleanup release: remove unnecesssary artifacts from nexus and distribute single binary distribution with 2 build artifacts, 1 for hadoop 1+ and 1 for hadoop 2+
** New Feature
* [MRUNIT-67] - o.a.h.mrunit.mapreduce.MapReduceDriver should support a combiner
* [MRUNIT-68] - Support custom counter checking
+** Task
+ * [MRUNIT-61] - Build separate source and binary tar balls
+ * [MRUNIT-94] - upgrade parent pom to version 10 and remove info in pom that is inherited from parent
+ * [MRUNIT-103] - JUnit license not in NOTICE.txt in a binary tarball
+
Release Notes - MRUnit - Version 0.8.1
** Bug
diff --git a/NOTICE-src.txt b/NOTICE-src.txt
deleted file mode 100644
index 3e09c89..0000000
--- a/NOTICE-src.txt
+++ /dev/null
@@ -1,5 +0,0 @@
-Apache MRUnit
-Copyright 2011-2012 The Apache Software Foundation
-
-This product includes software developed at
-The Apache Software Foundation (http://www.apache.org/).
diff --git a/NOTICE.txt b/NOTICE.txt
index 48344e0..3e09c89 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -3,9 +3,3 @@
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
-
-This product includes Mockito (Copyright (c) 2007 Mockito contributors)
-(http://code.google.com/p/mockito/) distributed under the MIT license.
-
-This product includes Hamcrest (Copyright (c) 2000-2006, www.hamcrest.org)
-(http://code.google.com/p/hamcrest/) distributed under the BSD 2-Clause License.
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index 49a318b..0ac909b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -105,10 +105,36 @@
<email>brock@apache.org</email>
<name>Brock Noland</name>
</developer>
+ <developer>
+ <id>jdonofrio.apache.org</id>
+ <email>jdonofrio@apache.org</email>
+ <name>Jim Donofrio</name>
+ </developer>
+ <developer>
+ <id>jarcec.apache.org</id>
+ <email>jarcec@apache.org</email>
+ <name>Jarek Jarcec Cecho</name>
+ </developer>
</developers>
<build>
<plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-clean-plugin</artifactId>
+ <version>2.4.1</version>
+ <configuration>
+ <excludeDefaultDirectories>true</excludeDefaultDirectories>
+ <filesets>
+ <fileset>
+ <directory>${project.build.directory}</directory>
+ <excludes>
+ <exclude>*hadoop?.jar</exclude>
+ </excludes>
+ </fileset>
+ </filesets>
+ </configuration>
+ </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
@@ -130,56 +156,12 @@
</plugin>
<plugin>
- <artifactId>maven-jar-plugin</artifactId>
- <configuration>
- <classifier>${envClassifier}</classifier>
- </configuration>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-assembly-plugin</artifactId>
- <version>2.3</version>
- <configuration>
- <tarLongFileMode>gnu</tarLongFileMode>
- </configuration>
- <executions>
- <execution>
- <id>src</id>
- <phase>package</phase>
- <goals>
- <goal>single</goal>
- </goals>
- <configuration>
- <descriptors>
- <descriptor>src/main/assembly/src-assembly.xml</descriptor>
- </descriptors>
- </configuration>
- </execution>
- <execution>
- <id>bin</id>
- <phase>package</phase>
- <goals>
- <goal>single</goal>
- </goals>
- <configuration>
- <descriptors>
- <descriptor>src/main/assembly/bin-assembly.xml</descriptor>
- </descriptors>
- <finalName>${project.build.finalName}-${envClassifier}</finalName>
- <appendAssemblyId>false</appendAssemblyId>
- </configuration>
- </execution>
- </executions>
- </plugin>
-
- <plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<version>0.8</version>
<executions>
<execution>
- <phase>verify</phase>
+ <phase>prepare-package</phase>
<goals>
<goal>check</goal>
</goals>
@@ -187,7 +169,7 @@
<excludes>
<exclude>BUILD.txt</exclude>
<exclude>CHANGES.txt</exclude>
- <exclude>NOTICE-src.txt</exclude>
+ <exclude>BIN-NOTICE.txt</exclude>
</excludes>
</configuration>
</execution>
@@ -196,8 +178,11 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-site-plugin</artifactId>
- <version>3.0-beta-3</version>
+ <artifactId>maven-jar-plugin</artifactId>
+ <version>2.4</version>
+ <configuration>
+ <classifier>${envClassifier}</classifier>
+ </configuration>
</plugin>
<plugin>
@@ -229,11 +214,129 @@
</executions>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <version>2.7</version>
+ </plugin>
</plugins>
</build>
- <dependencies>
+ <profiles>
+ <profile>
+ <id>hadoop1</id>
+ <activation>
+ <property>
+ <name>!hadoop.version</name>
+ </property>
+ </activation>
+ <properties>
+ <envClassifier>hadoop1</envClassifier>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-core</artifactId>
+ <version>1.0.2</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <profile>
+ <id>hadoop2</id>
+ <activation>
+ <property>
+ <name>hadoop.version</name>
+ <value>2</value>
+ </property>
+ </activation>
+
+ <properties>
+ <envClassifier>hadoop2</envClassifier>
+ <specificHadoopVersion>0.23.1</specificHadoopVersion>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${specificHadoopVersion}</version>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <version>${specificHadoopVersion}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <profile>
+ <id>sign</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-gpg-plugin</artifactId>
+ <version>1.4</version>
+ <executions>
+ <execution>
+ <phase>verify</phase>
+ <goals>
+ <goal>sign</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+
+ <profile>
+ <id>dist-src</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <version>2.3</version>
+ <configuration>
+ <tarLongFileMode>gnu</tarLongFileMode>
+ <descriptors>
+ <descriptor>src/main/assembly/src-assembly.xml</descriptor>
+ </descriptors>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+
+ <profile>
+ <id>dist-bin</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <version>2.3</version>
+ <configuration>
+ <tarLongFileMode>gnu</tarLongFileMode>
+ <descriptors>
+ <descriptor>src/main/assembly/bin-assembly.xml</descriptor>
+ </descriptors>
+ <appendAssemblyId>false</appendAssemblyId>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
+ <dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
@@ -250,7 +353,6 @@
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.15</version>
-
<exclusions>
<exclusion>
<groupId>com.sun.jdmk</groupId>
@@ -276,144 +378,5 @@
<artifactId>mockito-all</artifactId>
<version>1.8.5</version>
</dependency>
-
</dependencies>
-
- <reporting>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.4</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-report-plugin</artifactId>
- <version>2.8.1</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-jxr-plugin</artifactId>
- <version>2.2</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <version>2.8.1</version>
- <configuration>
- <excludePackageNames>*.internal</excludePackageNames>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
-
- <profiles>
- <profile>
- <!-- Used only when cutting a full release to sign the artifacts -->
- <id>deploy</id>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-gpg-plugin</artifactId>
- <version>1.1</version>
- <executions>
- <execution>
- <goals>
- <goal>sign</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- <properties>
- <update-release-info>true</update-release-info>
- </properties>
- </profile>
- <profile>
- <id>hadoop-0.20</id>
- <activation>
- <property>
- <name>hadoop.version</name>
- <value>0.20</value>
- </property>
- </activation>
- <properties>
- <envClassifier>hadoop020</envClassifier>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-core</artifactId>
- <version>0.20.2</version>
- <scope>provided</scope>
- </dependency>
- </dependencies>
- </profile>
- <profile>
- <id>hadoop-1.0</id>
- <activation>
- <property>
- <name>!hadoop.version</name>
- </property>
- </activation>
- <properties>
- <envClassifier>hadoop100</envClassifier>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-core</artifactId>
- <version>1.0.0</version>
- <scope>provided</scope>
- </dependency>
- </dependencies>
- </profile>
- <profile>
- <id>hadoop-0.23</id>
- <activation>
- <property>
- <name>hadoop.version</name>
- <value>0.23</value>
- </property>
- </activation>
- <properties>
- <envClassifier>hadoop023</envClassifier>
- <specificHadoopVersion>0.23.1</specificHadoopVersion>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${specificHadoopVersion}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-common</artifactId>
- <version>${specificHadoopVersion}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
- <version>${specificHadoopVersion}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.codehaus.jackson</groupId>
- <artifactId>jackson-core-asl</artifactId>
- <version>1.8.2</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.codehaus.jackson</groupId>
- <artifactId>jackson-mapper-asl</artifactId>
- <version>1.8.2</version>
- <scope>provided</scope>
- </dependency>
- </dependencies>
- </profile>
- </profiles>
</project>
diff --git a/src/main/assembly/bin-assembly.xml b/src/main/assembly/bin-assembly.xml
index df7002f..79ae96f 100644
--- a/src/main/assembly/bin-assembly.xml
+++ b/src/main/assembly/bin-assembly.xml
@@ -23,8 +23,6 @@
<id>bin</id>
- <baseDirectory>${project.build.finalName}-${envClassifier}</baseDirectory>
-
<formats>
<format>tar.gz</format>
</formats>
@@ -35,8 +33,9 @@
<files>
<file>
- <source>NOTICE.txt</source>
+ <source>BIN-NOTICE.txt</source>
<outputDirectory>/</outputDirectory>
+ <destName>NOTICE.txt</destName>
</file>
</files>
@@ -45,7 +44,7 @@
<directory>${project.build.directory}</directory>
<outputDirectory>lib</outputDirectory>
<includes>
- <include>${project.build.finalName}-${envClassifier}.jar</include>
+ <include>*hadoop*.jar</include>
</includes>
</fileSet>
</fileSets>
@@ -53,11 +52,6 @@
<dependencySets>
<dependencySet>
<outputDirectory>lib</outputDirectory>
- <!-- assembly plugin throws exception on target/classes folder is not
- a file if dont exclude itself -->
- <excludes>
- <exclude>org.apache.mrunit:mrunit</exclude>
- </excludes>
</dependencySet>
</dependencySets>
</assembly>
diff --git a/src/main/assembly/src-assembly.xml b/src/main/assembly/src-assembly.xml
index 8edc51a..19e32e0 100644
--- a/src/main/assembly/src-assembly.xml
+++ b/src/main/assembly/src-assembly.xml
@@ -35,9 +35,8 @@
<files>
<file>
- <source>NOTICE-src.txt</source>
+ <source>NOTICE.txt</source>
<outputDirectory>/</outputDirectory>
- <destName>NOTICE.txt</destName>
</file>
</files>
diff --git a/src/main/java/org/apache/hadoop/mrunit/MapReduceDriver.java b/src/main/java/org/apache/hadoop/mrunit/MapReduceDriver.java
index 6a6e5ed..fa4dda1 100644
--- a/src/main/java/org/apache/hadoop/mrunit/MapReduceDriver.java
+++ b/src/main/java/org/apache/hadoop/mrunit/MapReduceDriver.java
@@ -47,7 +47,7 @@
* and before the Reducer.
*/
@SuppressWarnings({ "deprecation", "rawtypes" })
-public class MapReduceDriver<K1, V1, K2 extends Comparable, V2, K3, V3> extends
+public class MapReduceDriver<K1, V1, K2, V2, K3, V3> extends
MapReduceDriverBase<K1, V1, K2, V2, K3, V3> {
public static final Log LOG = LogFactory.getLog(MapReduceDriver.class);
@@ -393,7 +393,7 @@
*
* @return new MapReduceDriver
*/
- public static <K1, V1, K2 extends Comparable, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver() {
+ public static <K1, V1, K2, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver() {
return new MapReduceDriver<K1, V1, K2, V2, K3, V3>();
}
@@ -407,7 +407,7 @@
* passed to MapReduceDriver constructor
* @return new MapReduceDriver
*/
- public static <K1, V1, K2 extends Comparable, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver(
+ public static <K1, V1, K2, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver(
final Mapper<K1, V1, K2, V2> mapper, final Reducer<K2, V2, K3, V3> reducer) {
return new MapReduceDriver<K1, V1, K2, V2, K3, V3>(mapper, reducer);
}
@@ -424,7 +424,7 @@
* passed to MapReduceDriver constructor
* @return new MapReduceDriver
*/
- public static <K1, V1, K2 extends Comparable, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver(
+ public static <K1, V1, K2, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver(
final Mapper<K1, V1, K2, V2> mapper,
final Reducer<K2, V2, K3, V3> reducer,
final Reducer<K2, V2, K2, V2> combiner) {
diff --git a/src/main/java/org/apache/hadoop/mrunit/MapReduceDriverBase.java b/src/main/java/org/apache/hadoop/mrunit/MapReduceDriverBase.java
index a85b443..04029df 100644
--- a/src/main/java/org/apache/hadoop/mrunit/MapReduceDriverBase.java
+++ b/src/main/java/org/apache/hadoop/mrunit/MapReduceDriverBase.java
@@ -45,8 +45,8 @@
* pair, representing a single unit test.
*/
@SuppressWarnings("rawtypes")
-public abstract class MapReduceDriverBase<K1, V1, K2 extends Comparable, V2, K3, V3>
- extends TestDriver<K1, V1, K3, V3> {
+public abstract class MapReduceDriverBase<K1, V1, K2, V2, K3, V3> extends
+ TestDriver<K1, V1, K3, V3> {
public static final Log LOG = LogFactory.getLog(MapReduceDriverBase.class);
@@ -60,19 +60,6 @@
public MapReduceDriverBase() {
inputList = new ArrayList<Pair<K1, V1>>();
-
- // create a simple comparator for key grouping and ordering
- final Comparator<K2> simpleComparator = new Comparator<K2>() {
- @SuppressWarnings("unchecked")
- @Override
- public int compare(final K2 o1, final K2 o2) {
- return o1.compareTo(o2);
- }
- };
-
- // assign simple key grouping and ordering comparator
- keyGroupComparator = simpleComparator;
- keyValueOrderComparator = null;
}
/**
@@ -170,6 +157,13 @@
*/
public List<Pair<K2, List<V2>>> shuffle(final List<Pair<K2, V2>> mapOutputs) {
// step 1 - use the key group comparator to organise map outputs
+ final Comparator<K2> keyGroupComparator;
+ if (this.keyGroupComparator == null) {
+ keyGroupComparator = new JobConf(getConfiguration())
+ .getOutputValueGroupingComparator();
+ } else {
+ keyGroupComparator = this.keyGroupComparator;
+ }
final TreeMap<K2, List<Pair<K2, V2>>> groupedByKey = new TreeMap<K2, List<Pair<K2, V2>>>(
keyGroupComparator);
diff --git a/src/main/java/org/apache/hadoop/mrunit/mapreduce/MapDriver.java b/src/main/java/org/apache/hadoop/mrunit/mapreduce/MapDriver.java
index 782fd36..01fb9d6 100644
--- a/src/main/java/org/apache/hadoop/mrunit/mapreduce/MapDriver.java
+++ b/src/main/java/org/apache/hadoop/mrunit/mapreduce/MapDriver.java
@@ -28,9 +28,11 @@
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.Counters;
+import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mrunit.MapDriverBase;
import org.apache.hadoop.mrunit.counters.CounterWrapper;
+import org.apache.hadoop.mrunit.mapreduce.mock.MockInputSplit;
import org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper;
import org.apache.hadoop.mrunit.types.Pair;
@@ -209,9 +211,11 @@
final List<Pair<K1, V1>> inputs = new ArrayList<Pair<K1, V1>>();
inputs.add(new Pair<K1, V1>(inputKey, inputVal));
+ final InputSplit inputSplit = new MockInputSplit();
+
try {
final MockMapContextWrapper<K1, V1, K2, V2> wrapper = new MockMapContextWrapper<K1, V1, K2, V2>(
- inputs, getCounters(), getConfiguration());
+ inputs, getCounters(), getConfiguration(), inputSplit);
final Mapper<K1, V1, K2, V2>.Context context = wrapper.getMockContext();
myMapper.run(context);
diff --git a/src/main/java/org/apache/hadoop/mrunit/mapreduce/MapReduceDriver.java b/src/main/java/org/apache/hadoop/mrunit/mapreduce/MapReduceDriver.java
index 5c0f4ea..4c3080e 100644
--- a/src/main/java/org/apache/hadoop/mrunit/mapreduce/MapReduceDriver.java
+++ b/src/main/java/org/apache/hadoop/mrunit/mapreduce/MapReduceDriver.java
@@ -45,7 +45,7 @@
* pair, representing a single unit test.
*/
@SuppressWarnings("rawtypes")
-public class MapReduceDriver<K1, V1, K2 extends Comparable, V2, K3, V3> extends
+public class MapReduceDriver<K1, V1, K2, V2, K3, V3> extends
MapReduceDriverBase<K1, V1, K2, V2, K3, V3> {
public static final Log LOG = LogFactory.getLog(MapReduceDriver.class);
@@ -391,7 +391,7 @@
*
* @return new MapReduceDriver
*/
- public static <K1, V1, K2 extends Comparable, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver() {
+ public static <K1, V1, K2, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver() {
return new MapReduceDriver<K1, V1, K2, V2, K3, V3>();
}
@@ -405,7 +405,7 @@
* passed to MapReduceDriver constructor
* @return new MapReduceDriver
*/
- public static <K1, V1, K2 extends Comparable, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver(
+ public static <K1, V1, K2, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver(
final Mapper<K1, V1, K2, V2> mapper, final Reducer<K2, V2, K3, V3> reducer) {
return new MapReduceDriver<K1, V1, K2, V2, K3, V3>(mapper, reducer);
}
@@ -422,7 +422,7 @@
* passed to MapReduceDriver constructor
* @return new MapReduceDriver
*/
- public static <K1, V1, K2 extends Comparable, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver(
+ public static <K1, V1, K2, V2, K3, V3> MapReduceDriver<K1, V1, K2, V2, K3, V3> newMapReduceDriver(
final Mapper<K1, V1, K2, V2> mapper,
final Reducer<K2, V2, K3, V3> reducer,
final Reducer<K2, V2, K2, V2> combiner) {
diff --git a/src/main/java/org/apache/hadoop/mrunit/mapreduce/mock/MockMapContextWrapper.java b/src/main/java/org/apache/hadoop/mrunit/mapreduce/mock/MockMapContextWrapper.java
index 3f46d16..0fd02fa 100644
--- a/src/main/java/org/apache/hadoop/mrunit/mapreduce/mock/MockMapContextWrapper.java
+++ b/src/main/java/org/apache/hadoop/mrunit/mapreduce/mock/MockMapContextWrapper.java
@@ -28,6 +28,7 @@
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.Counters;
+import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mrunit.types.Pair;
import org.mockito.invocation.InvocationOnMock;
@@ -50,12 +51,14 @@
protected final List<Pair<KEYIN, VALUEIN>> inputs;
protected Pair<KEYIN, VALUEIN> currentKeyValue;
protected final Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context context;
-
+ protected InputSplit inputSplit;
+
public MockMapContextWrapper(final List<Pair<KEYIN, VALUEIN>> inputs,
- final Counters counters, final Configuration conf) throws IOException,
- InterruptedException {
+ final Counters counters, final Configuration conf, final InputSplit inputSplit)
+ throws IOException, InterruptedException {
super(counters, conf);
this.inputs = inputs;
+ this.inputSplit = inputSplit;
context = create();
}
@@ -94,6 +97,12 @@
return currentKeyValue.getSecond();
}
});
+ when(context.getInputSplit()).thenAnswer(new Answer<InputSplit>() {
+ @Override
+ public InputSplit answer(InvocationOnMock invocation) throws Throwable {
+ return inputSplit;
+ }
+ });
return context;
}
diff --git a/src/site/site.xml b/src/site/site.xml
deleted file mode 100644
index 341dffc..0000000
--- a/src/site/site.xml
+++ /dev/null
@@ -1,70 +0,0 @@
-<?xml version="1.0" encoding="ISO-8859-1"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<project name="MRUnit">
- <bannerLeft>
- <name>MRUnit</name>
- <href>http://incubator.apache.org/mrunit/</href>
- </bannerLeft>
- <bannerRight>
- <src>http://incubator.apache.org/images/apache-incubator-logo.png</src>
- <href>http://incubator.apache.org/</href>
- </bannerRight>
-
- <publishDate position="right"/>
- <version position="right"/>
-
- <body>
- <links position="left">
- <item name="Wiki" href="https://cwiki.apache.org/confluence/display/MRUNIT" />
- <item name="JIRA" href="https://issues.apache.org/jira/browse/MRUNIT" />
- <item name="SVN" href="https://svn.apache.org/repos/asf/incubator/mrunit/" />
- </links>
-
- <breadcrumbs position="left">
- <item name="Apache" href="http://www.apache.org/" />
- <item name="Apache Incubator" href="http://incubator.apache.org/" />
- <item name="MRUnit" href="http://incubator.apache.org/mrunit/"/>
- </breadcrumbs>
-
- <menu name="MRUnit">
- <item name="About" href="index.html"/>
- <item name="Download" href="http://www.apache.org/dyn/closer.cgi/incubator/mrunit/"/>
- <item name="Road Map" href="https://issues.apache.org/jira/browse/MRUNIT#selectedTab=com.atlassian.jira.plugin.system.project%3Aroadmap-panel" />
- <item name="Wiki" href="https://cwiki.apache.org/confluence/display/MRUNIT" />
- </menu>
-
- <menu name="Project Information">
- <item name="Summary" href="project-info.html" />
- <item name="Mailing Lists" href="mail-lists.html" />
- <item name="Issue Tracking" href="issue-tracking.html" />
- <item name="Source" href="source-repository.html" />
- <item name="Team" href="team-list.html" />
- <item name="License" href="license.html" />
- <item name="Dependencies" href="dependencies.html" />
- </menu>
-
- <menu name="Reports">
- <item name="Summary" href="project-reports.html" />
- <item name="Unit Test Report" href="surefire-report.html" />
- <item name="Javadoc" href="apidocs" />
- <item name="Source Xref" href="xref/index.html" />
- <item name="Test Source Xref" href="xref-test/index.html" />
- </menu>
-
- </body>
-</project>
diff --git a/src/test/java/org/apache/hadoop/mrunit/mapreduce/TestMapDriver.java b/src/test/java/org/apache/hadoop/mrunit/mapreduce/TestMapDriver.java
index 4169ee0..66cdc5e 100644
--- a/src/test/java/org/apache/hadoop/mrunit/mapreduce/TestMapDriver.java
+++ b/src/test/java/org/apache/hadoop/mrunit/mapreduce/TestMapDriver.java
@@ -26,9 +26,11 @@
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
+import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.lib.map.InverseMapper;
import org.apache.hadoop.mrunit.ExpectedSuppliedException;
import org.apache.hadoop.mrunit.types.Pair;
@@ -244,6 +246,24 @@
}
@Test
+ public void testInputSplitDetails() {
+ final MapDriver<NullWritable, NullWritable, Text, LongWritable> driver =
+ MapDriver.newMapDriver(new InputSplitDetailMapper());
+ driver.withInput(NullWritable.get(), NullWritable.get())
+ .withOutput(new Text("somefile"), new LongWritable(0L)).runTest();
+ }
+
+ public static class InputSplitDetailMapper
+ extends Mapper<NullWritable, NullWritable, Text, LongWritable> {
+ protected void map(NullWritable key, NullWritable value, Context context)
+ throws IOException, InterruptedException {
+ FileSplit split = (FileSplit)context.getInputSplit();
+ context.write(new Text(split.getPath().toString()),
+ new LongWritable(split.getLength()));
+ }
+ }
+
+ @Test
public void testNoMapper() {
driver = MapDriver.newMapDriver();
thrown.expectMessage(IllegalStateException.class,
diff --git a/src/test/sh/run-artifact-unit-tests.sh b/src/test/sh/run-artifact-unit-tests.sh
deleted file mode 100755
index bc497fa..0000000
--- a/src/test/sh/run-artifact-unit-tests.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/bash
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-# run all the tests, using the mrunit jar in the cwd
-set -e
-MRUNIT_JAR=$(ls mrunit-*.jar)
-TEST_CLASSPATH=$(hadoop classpath | tr ':' '\n' | grep -vi 'mrunit' | tr '\n' ':')
-TEST_CLASSPATH=src/test/java/:$MRUNIT_JAR:$TEST_CLASSPATH
-find src/test/java/ -name '*.java' | xargs javac -classpath $TEST_CLASSPATH
-TEST_CLASSES=$(cd src/test/java/ && find . -name '*.java' | xargs grep -Fl '@Test' | perl -pe 's@^./@@g;s@.java@@g;s@/@.@g')
-java -classpath $TEST_CLASSPATH org.junit.runner.JUnitCore $TEST_CLASSES
-echo "MRUnit Tests Pass!"
