| //// |
| /** |
| * |
| * Licensed to the Apache Software Foundation (ASF) under one |
| * or more contributor license agreements. See the NOTICE file |
| * distributed with this work for additional information |
| * regarding copyright ownership. The ASF licenses this file |
| * to you under the Apache License, Version 2.0 (the |
| * "License"); you may not use this file except in compliance |
| * with the License. You may obtain a copy of the License at |
| * |
| * http://www.apache.org/licenses/LICENSE-2.0 |
| * |
| * Unless required by applicable law or agreed to in writing, software |
| * distributed under the License is distributed on an "AS IS" BASIS, |
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| * See the License for the specific language governing permissions and |
| * limitations under the License. |
| */ |
| //// |
| |
| [[unit.tests]] |
| = Unit Testing HBase Applications |
| :doctype: book |
| :numbered: |
| :toc: left |
| :icons: font |
| :experimental: |
| |
| This chapter discusses unit testing your HBase application using JUnit, Mockito, MRUnit, and HBaseTestingUtility. |
| Much of the information comes from link:http://blog.cloudera.com/blog/2013/09/how-to-test-hbase-applications-using-popular-tools/[a community blog post about testing HBase applications]. |
| For information on unit tests for HBase itself, see <<hbase.tests,hbase.tests>>. |
| |
| == JUnit |
| |
| HBase uses link:http://junit.org[JUnit] 4 for unit tests |
| |
| This example will add unit tests to the following example class: |
| |
| [source,java] |
| ---- |
| |
| public class MyHBaseDAO { |
| |
| public static void insertRecord(Table.getTable(table), HBaseTestObj obj) |
| throws Exception { |
| Put put = createPut(obj); |
| table.put(put); |
| } |
| |
| private static Put createPut(HBaseTestObj obj) { |
| Put put = new Put(Bytes.toBytes(obj.getRowKey())); |
| put.add(Bytes.toBytes("CF"), Bytes.toBytes("CQ-1"), |
| Bytes.toBytes(obj.getData1())); |
| put.add(Bytes.toBytes("CF"), Bytes.toBytes("CQ-2"), |
| Bytes.toBytes(obj.getData2())); |
| return put; |
| } |
| } |
| ---- |
| |
| The first step is to add JUnit dependencies to your Maven POM file: |
| |
| [source,xml] |
| ---- |
| |
| <dependency> |
| <groupId>junit</groupId> |
| <artifactId>junit</artifactId> |
| <version>4.11</version> |
| <scope>test</scope> |
| </dependency> |
| ---- |
| |
| Next, add some unit tests to your code. |
| Tests are annotated with `@Test`. |
| Here, the unit tests are in bold. |
| |
| [source,java] |
| ---- |
| |
| public class TestMyHbaseDAOData { |
| @Test |
| public void testCreatePut() throws Exception { |
| HBaseTestObj obj = new HBaseTestObj(); |
| obj.setRowKey("ROWKEY-1"); |
| obj.setData1("DATA-1"); |
| obj.setData2("DATA-2"); |
| Put put = MyHBaseDAO.createPut(obj); |
| assertEquals(obj.getRowKey(), Bytes.toString(put.getRow())); |
| assertEquals(obj.getData1(), Bytes.toString(put.get(Bytes.toBytes("CF"), Bytes.toBytes("CQ-1")).get(0).getValue())); |
| assertEquals(obj.getData2(), Bytes.toString(put.get(Bytes.toBytes("CF"), Bytes.toBytes("CQ-2")).get(0).getValue())); |
| } |
| } |
| ---- |
| |
| These tests ensure that your `createPut` method creates, populates, and returns a `Put` object with expected values. |
| Of course, JUnit can do much more than this. |
| For an introduction to JUnit, see link:https://github.com/junit-team/junit/wiki/Getting-started. |
| |
| == Mockito |
| |
| Mockito is a mocking framework. |
| It goes further than JUnit by allowing you to test the interactions between objects without having to replicate the entire environment. |
| You can read more about Mockito at its project site, link:https://code.google.com/p/mockito/. |
| |
| You can use Mockito to do unit testing on smaller units. |
| For instance, you can mock a `org.apache.hadoop.hbase.Server` instance or a `org.apache.hadoop.hbase.master.MasterServices` interface reference rather than a full-blown `org.apache.hadoop.hbase.master.HMaster`. |
| |
| This example builds upon the example code in <<unit.tests,unit.tests>>, to test the `insertRecord` method. |
| |
| First, add a dependency for Mockito to your Maven POM file. |
| |
| [source,xml] |
| ---- |
| |
| <dependency> |
| <groupId>org.mockito</groupId> |
| <artifactId>mockito-all</artifactId> |
| <version>1.9.5</version> |
| <scope>test</scope> |
| </dependency> |
| ---- |
| |
| Next, add a `@RunWith` annotation to your test class, to direct it to use Mockito. |
| |
| [source,java] |
| ---- |
| |
| @RunWith(MockitoJUnitRunner.class) |
| public class TestMyHBaseDAO{ |
| @Mock |
| Configuration config = HBaseConfiguration.create(); |
| @Mock |
| Connection connection = ConnectionFactory.createConnection(config); |
| @Mock |
| private Table table; |
| @Captor |
| private ArgumentCaptor putCaptor; |
| |
| @Test |
| public void testInsertRecord() throws Exception { |
| //return mock table when getTable is called |
| when(connection.getTable(TableName.valueOf("tablename")).thenReturn(table); |
| //create test object and make a call to the DAO that needs testing |
| HBaseTestObj obj = new HBaseTestObj(); |
| obj.setRowKey("ROWKEY-1"); |
| obj.setData1("DATA-1"); |
| obj.setData2("DATA-2"); |
| MyHBaseDAO.insertRecord(table, obj); |
| verify(table).put(putCaptor.capture()); |
| Put put = putCaptor.getValue(); |
| |
| assertEquals(Bytes.toString(put.getRow()), obj.getRowKey()); |
| assert(put.has(Bytes.toBytes("CF"), Bytes.toBytes("CQ-1"))); |
| assert(put.has(Bytes.toBytes("CF"), Bytes.toBytes("CQ-2"))); |
| assertEquals(Bytes.toString(put.get(Bytes.toBytes("CF"),Bytes.toBytes("CQ-1")).get(0).getValue()), "DATA-1"); |
| assertEquals(Bytes.toString(put.get(Bytes.toBytes("CF"),Bytes.toBytes("CQ-2")).get(0).getValue()), "DATA-2"); |
| } |
| } |
| ---- |
| |
| This code populates `HBaseTestObj` with ``ROWKEY-1'', ``DATA-1'', ``DATA-2'' as values. |
| It then inserts the record into the mocked table. |
| The Put that the DAO would have inserted is captured, and values are tested to verify that they are what you expected them to be. |
| |
| The key here is to manage Connection and Table instance creation outside the DAO. |
| This allows you to mock them cleanly and test Puts as shown above. |
| Similarly, you can now expand into other operations such as Get, Scan, or Delete. |
| |
| == MRUnit |
| |
| link:http://mrunit.apache.org/[Apache MRUnit] is a library that allows you to unit-test MapReduce jobs. |
| You can use it to test HBase jobs in the same way as other MapReduce jobs. |
| |
| Given a MapReduce job that writes to an HBase table called `MyTest`, which has one column family called `CF`, the reducer of such a job could look like the following: |
| |
| [source,java] |
| ---- |
| |
| public class MyReducer extends TableReducer<Text, Text, ImmutableBytesWritable> { |
| public static final byte[] CF = "CF".getBytes(); |
| public static final byte[] QUALIFIER = "CQ-1".getBytes(); |
| public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException { |
| //bunch of processing to extract data to be inserted, in our case, lets say we are simply |
| //appending all the records we receive from the mapper for this particular |
| //key and insert one record into HBase |
| StringBuffer data = new StringBuffer(); |
| Put put = new Put(Bytes.toBytes(key.toString())); |
| for (Text val : values) { |
| data = data.append(val); |
| } |
| put.add(CF, QUALIFIER, Bytes.toBytes(data.toString())); |
| //write to HBase |
| context.write(new ImmutableBytesWritable(Bytes.toBytes(key.toString())), put); |
| } |
| } |
| ---- |
| |
| To test this code, the first step is to add a dependency to MRUnit to your Maven POM file. |
| |
| [source,xml] |
| ---- |
| |
| <dependency> |
| <groupId>org.apache.mrunit</groupId> |
| <artifactId>mrunit</artifactId> |
| <version>1.0.0 </version> |
| <scope>test</scope> |
| </dependency> |
| ---- |
| |
| Next, use the ReducerDriver provided by MRUnit, in your Reducer job. |
| |
| [source,java] |
| ---- |
| |
| public class MyReducerTest { |
| ReduceDriver<Text, Text, ImmutableBytesWritable, Writable> reduceDriver; |
| byte[] CF = "CF".getBytes(); |
| byte[] QUALIFIER = "CQ-1".getBytes(); |
| |
| @Before |
| public void setUp() { |
| MyReducer reducer = new MyReducer(); |
| reduceDriver = ReduceDriver.newReduceDriver(reducer); |
| } |
| |
| @Test |
| public void testHBaseInsert() throws IOException { |
| String strKey = "RowKey-1", strValue = "DATA", strValue1 = "DATA1", |
| strValue2 = "DATA2"; |
| List<Text> list = new ArrayList<Text>(); |
| list.add(new Text(strValue)); |
| list.add(new Text(strValue1)); |
| list.add(new Text(strValue2)); |
| //since in our case all that the reducer is doing is appending the records that the mapper |
| //sends it, we should get the following back |
| String expectedOutput = strValue + strValue1 + strValue2; |
| //Setup Input, mimic what mapper would have passed |
| //to the reducer and run test |
| reduceDriver.withInput(new Text(strKey), list); |
| //run the reducer and get its output |
| List<Pair<ImmutableBytesWritable, Writable>> result = reduceDriver.run(); |
| |
| //extract key from result and verify |
| assertEquals(Bytes.toString(result.get(0).getFirst().get()), strKey); |
| |
| //extract value for CF/QUALIFIER and verify |
| Put a = (Put)result.get(0).getSecond(); |
| String c = Bytes.toString(a.get(CF, QUALIFIER).get(0).getValue()); |
| assertEquals(expectedOutput,c ); |
| } |
| |
| } |
| ---- |
| |
| Your MRUnit test verifies that the output is as expected, the Put that is inserted into HBase has the correct value, and the ColumnFamily and ColumnQualifier have the correct values. |
| |
| MRUnit includes a MapperDriver to test mapping jobs, and you can use MRUnit to test other operations, including reading from HBase, processing data, or writing to HDFS, |
| |
| == Integration Testing with a HBase Mini-Cluster |
| |
| HBase ships with HBaseTestingUtility, which makes it easy to write integration tests using a [firstterm]_mini-cluster_. |
| The first step is to add some dependencies to your Maven POM file. |
| Check the versions to be sure they are appropriate. |
| |
| [source,xml] |
| ---- |
| |
| <dependency> |
| <groupId>org.apache.hadoop</groupId> |
| <artifactId>hadoop-common</artifactId> |
| <version>2.0.0</version> |
| <type>test-jar</type> |
| <scope>test</scope> |
| </dependency> |
| |
| <dependency> |
| <groupId>org.apache.hbase</groupId> |
| <artifactId>hbase</artifactId> |
| <version>0.98.3</version> |
| <type>test-jar</type> |
| <scope>test</scope> |
| </dependency> |
| |
| <dependency> |
| <groupId>org.apache.hadoop</groupId> |
| <artifactId>hadoop-hdfs</artifactId> |
| <version>2.0.0</version> |
| <type>test-jar</type> |
| <scope>test</scope> |
| </dependency> |
| |
| <dependency> |
| <groupId>org.apache.hadoop</groupId> |
| <artifactId>hadoop-hdfs</artifactId> |
| <version>2.0.0</version> |
| <scope>test</scope> |
| </dependency> |
| ---- |
| |
| This code represents an integration test for the MyDAO insert shown in <<unit.tests,unit.tests>>. |
| |
| [source,java] |
| ---- |
| |
| public class MyHBaseIntegrationTest { |
| private static HBaseTestingUtility utility; |
| byte[] CF = "CF".getBytes(); |
| byte[] QUALIFIER = "CQ-1".getBytes(); |
| |
| @Before |
| public void setup() throws Exception { |
| utility = new HBaseTestingUtility(); |
| utility.startMiniCluster(); |
| } |
| |
| @Test |
| public void testInsert() throws Exception { |
| HTableInterface table = utility.createTable(Bytes.toBytes("MyTest"), |
| Bytes.toBytes("CF")); |
| HBaseTestObj obj = new HBaseTestObj(); |
| obj.setRowKey("ROWKEY-1"); |
| obj.setData1("DATA-1"); |
| obj.setData2("DATA-2"); |
| MyHBaseDAO.insertRecord(table, obj); |
| Get get1 = new Get(Bytes.toBytes(obj.getRowKey())); |
| get1.addColumn(CF, CQ1); |
| Result result1 = table.get(get1); |
| assertEquals(Bytes.toString(result1.getRow()), obj.getRowKey()); |
| assertEquals(Bytes.toString(result1.value()), obj.getData1()); |
| Get get2 = new Get(Bytes.toBytes(obj.getRowKey())); |
| get2.addColumn(CF, CQ2); |
| Result result2 = table.get(get2); |
| assertEquals(Bytes.toString(result2.getRow()), obj.getRowKey()); |
| assertEquals(Bytes.toString(result2.value()), obj.getData2()); |
| } |
| } |
| ---- |
| |
| This code creates an HBase mini-cluster and starts it. |
| Next, it creates a table called `MyTest` with one column family, `CF`. |
| A record is inserted, a Get is performed from the same table, and the insertion is verified. |
| |
| NOTE: Starting the mini-cluster takes about 20-30 seconds, but that should be appropriate for integration testing. |
| |
| To use an HBase mini-cluster on Microsoft Windows, you need to use a Cygwin environment. |
| |
| See the paper at link:http://blog.sematext.com/2010/08/30/hbase-case-study-using-hbasetestingutility-for-local-testing-development/[HBase Case-Study: Using HBaseTestingUtility for Local Testing and |
| Development] (2010) for more information about HBaseTestingUtility. |