bigtop-tests/smoke-tests/README

# This is the new smoke testing module for bigtop

## Goals

This gradle based test framework aims at making it extremely simple to
customize and test Apache BigTop's hadoop distribution.

 - Easily add new tests to bigtop without adding new directories / xml files.
 - Debug tests easily and make logging extremely transparent.
 - Geneate easy to read test reports.
 - Run bigtop tests without needing a jar file.
 - Run bigtop tests easily, ideally without gradle/maven

So overall, this framework makes testing with apache bigtop easy as well as fun!

## Running

You need to export several environment variables.   For example,
```
export HADOOP_CONF_DIR=/etc/hadoop/conf/
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
export HIVE_HOME=/usr/lib/hive/
export PIG_HOME=/usr/lib/pig/
export HIVE_CONF_DIR=/etc/hive/conf/
export JAVA_HOME="/usr/lib/jvm/java-openjdk/"
export SPARK_HOME="/usr/lib/spark"
export OOZIE_TAR_HOME="/usr/lib/oozie/doc"
```

Fallback mode for shark-shell submission is 'yarn-client'. For a good reason,
we aren't using YARN to provision cluster services. Hence, in order to execute
the test against a standalone Spark cluster, like we deploy would require two
more environment variables:
`
export SPARK_MASTER_IP=bigtop1.docker
export SPARK_MASTER_PORT=7077
`

Then, simply invoke the tests you want to run, as described below.

Each directory is a gradle "subproject" mapping to an ecosystem component.
Invoking test task of hbase subproject will execute hbase tests. To test hive as well,
you would add the invocation of hive:test, and so on. To run the tests from the top-level
Bigtop folder, follow the example below:
```
    ./gradlew bigtop-tests:smoke-tests:hbase:test bigtop-tests:smoke-tests:hive:test -Psmoke.tests --info
```
You also can run smoke tests from bigtop-tests/smoke-tests directory,
or from any subproject of it. E.g.
```
    cd bigtop-tests/smoke-tests
    ../../gradlew hive:test -Psmoke.tests
```
or
```
    cd bigtop-tests/smoke-tests/hive
    ../../../gradlew test -Psmoke.tests
```
Instead of using top-level Gradle wrapper one can also use system-wide
Gradle, if installed and available.

Setting up project property smoke.tests is important, cause this is
required to to enable test tasks for smoke projects.
The --info option is a nice feature to have here:  You can easily debug your tests this way,
just make sure you have log4j.properties in your conf/ directory locally.

## Adding more tests

To add a new test you will want to add a groovy file to an existing ecosystem test or create a new folder
(if you are adding a new component to bigtop that has no tests).

This can be done easily by copying an existing test subproject, and simply modifying it
for your own needs.  The settings.gradle file will automatically see the new folder you have added,
and run your build.gradle.

## How it works

Subprojects in gradle are configured at the top level by the root build.gradle file.  Each
subproject is injected with bits of functionality from the top level (i.e. generic functions
for confirming environment variables are set, calling of the test task, etc).  Each subproject
then is responsible for defining its source directories and resources, and does little more.

Right now, this library leverages tests both from test-artifacts, as well as locally.
Hopefully over time all tests will be merged here, because it is easier to maintain
this script based framework.

## Typical recipes

There are a lot of ways to run the bigtop tests, and not all are applicable to all clusters.
So, below, we've accumulated some typical modifications which you can easily implement
when running tests through this gradle interface (rather than as jars).

1) Add new tests to hive , other than the "basic" hive test.
2) Modify the Sqoop test to write more data via the embedded database.
3) Add your own custom Groovy test in a directory which calls a shell script of your own desgn.
And so on.