| # This is the new smoke testing module for bigtop |
| |
| ## Goals |
| |
| This gradle based test framework aims at making it extremely simple to |
| customize and test Apache BigTop's hadoop distribution. |
| |
| - Easily add new tests to bigtop without adding new directories / xml files. |
| - Debug tests easily and make logging extremely transparent. |
| - Geneate easy to read test reports. |
| - Run bigtop tests without needing a jar file. |
| - Run bigtop tests easily, ideally without gradle/maven |
| |
| So overall, this framework makes testing with apache bigtop easy as well as fun! |
| |
| ## Running |
| |
| You need to export several environment variables. For example, |
| ``` |
| export HADOOP_CONF_DIR=/etc/hadoop/conf/ |
| export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/ |
| export HIVE_HOME=/usr/lib/hive/ |
| export PIG_HOME=/usr/lib/pig/ |
| export FLUME_HOME=/usr/lib/flume/ |
| export HIVE_CONF_DIR=/etc/hive/conf/ |
| export JAVA_HOME="/usr/lib/jvm/java-openjdk/" |
| export MAHOUT_HOME="/usr/lib/mahout" |
| export SPARK_HOME="/usr/lib/spark" |
| export OOZIE_TAR_HOME="/usr/lib/oozie/doc" |
| ``` |
| |
| Fallback mode for shark-shell submission is 'yarn-client'. For a good reason, |
| we aren't using YARN to provision cluster services. Hence, in order to execute |
| the test against a standalone Spark cluster, like we deploy would require two |
| more environment variables: |
| ` |
| export SPARK_MASTER_IP=bigtop1.docker |
| export SPARK_MASTER_PORT=7077 |
| ` |
| |
| Then, simply invoke the tests you want to run, as described below. |
| |
| Each directory is a gradle "subproject" mapping to an ecosystem component. |
| Invoking test task of flume subproject will execute flume tests. To test hive as well, |
| you would add the invocation of hive:test, and so on. To run the tests from the top-level |
| Bigtop folder, follow the example below: |
| ``` |
| ./gradlew bigtop-tests:smoke-tests:flume:test bigtop-tests:smoke-tests:hive:test -Psmoke.tests --info |
| ``` |
| You also can run smoke tests from bigtop-tests/smoke-tests directory, |
| or from any subproject of it. E.g. |
| ``` |
| cd bigtop-tests/smoke-tests |
| ../../gradlew hive:test -Psmoke.tests |
| ``` |
| or |
| ``` |
| cd bigtop-tests/smoke-tests/hive |
| ../../../gradlew test -Psmoke.tests |
| ``` |
| Instead of using top-level Gradle wrapper one can also use system-wide |
| Gradle, if installed and available. |
| |
| Setting up project property smoke.tests is important, cause this is |
| required to to enable test tasks for smoke projects. |
| The --info option is a nice feature to have here: You can easily debug your tests this way, |
| just make sure you have log4j.properties in your conf/ directory locally. |
| |
| ## Adding more tests |
| |
| To add a new test you will want to add a groovy file to an existing ecosystem test or create a new folder |
| (if you are adding a new component to bigtop that has no tests). |
| |
| This can be done easily by copying an existing test subproject, and simply modifying it |
| for your own needs. The settings.gradle file will automatically see the new folder you have added, |
| and run your build.gradle. |
| |
| ## How it works |
| |
| Subprojects in gradle are configured at the top level by the root build.gradle file. Each |
| subproject is injected with bits of functionality from the top level (i.e. generic functions |
| for confirming environment variables are set, calling of the test task, etc). Each subproject |
| then is responsible for defining its source directories and resources, and does little more. |
| |
| Right now, this library leverages tests both from test-artifacts, as well as locally. |
| Hopefully over time all tests will be merged here, because it is easier to maintain |
| this script based framework. |
| |
| ## Typical recipes |
| |
| There are a lot of ways to run the bigtop tests, and not all are applicable to all clusters. |
| So, below, we've accumulated some typical modifications which you can easily implement |
| when running tests through this gradle interface (rather than as jars). |
| |
| 1) Modify the TestMahoutExamples.groovy file to exclude clustering tests, and re run. |
| 2) Add new tests to hive , other than the "basic" hive test. |
| 3) Modify the Sqoop test to write more data via the embedded database. |
| 4) Add your own custom Groovy test in a directory which calls a shell script of your own desgn. |
| And so on. |