tree: f05ad6f3d59e9c63af62886c13b986a1c0df75cd [path history] [tgz]
  1. README.md
  2. build.sh
  3. run-cluster.sh
  4. ubuntu12.04-jdk7-apache2.7/
  5. ubuntu12.04-jdk7-hdp2.1.15/
  6. ubuntu12.04-jdk7-hdp2.2.7/
  7. ubuntu12.04-jdk7-hdp2.2.8/
  8. ubuntu12.04-jdk7-hdp2.3.2/
  9. ubuntu12.04-jdk7-hdp2.4/
  10. ubuntu12.04-jdk7-mesos0.24/
  11. ubuntu12.04-jdk7-mesos0.25/
  12. ubuntu12.04-jdk7-mesos0.26/
  13. ubuntu12.04-jdk7-mesos0.27/
  14. ubuntu12.04-jdk7-mesos0.28/
  15. ubuntu12.04-jdk7/
dev/docker/README.md

Apache REEF™ Docker Tests

Apache REEF™ is an approach to simplify and unify the lower layers of big data systems on modern resource managers. This project maintains docker images to simulate the target underlying systems as similar as possible.

Docker-based Test Cluster

EnvironmentOSHadoop VersionDescriptionalias
YARNUbuntu 12.04HDP 2.1.15 (Hadoop 2.4.0)HDInsight 3.1hdi3.1
YARNUbuntu 12.04HDP 2.2.7 (Hadoop 2.6.0)HDInsight 3.2hdi3.2
YARNUbuntu 12.04HDP 2.2.8 (Hadoop 2.6.0)hdp2.2
YARNUbuntu 12.04HDP 2.3.2 (Hadoop 2.7.1)hdp2.3
YARNUbuntu 12.04HDP 2.4 (Hadoop 2.7.1)hdp2.4
YARNUbuntu 12.04Apache Hadoop 2.7.3apache2.7
MESOSUbuntu 12.04Apache Mesos 0.24~0.28mesos0.24 ~ 0.28

Please note that all images use Oracle JDK 7u80.

Requirements

First, download and build Apache REEF. You can find a guide for this here. Second, set $REEF_HOME to the directory of Apache REEF for the script to know its location. Tests will use $REEF_HOME/lang/java/reef-tests/target/reef-tests-*-test-jar-with-dependencies.jar.

Build docker images (Optional)

This step is optional. We provide pre-built docker images in DockerHub. Please go to next step if you do not want to build images. To build all images manually, run build.sh.

$ ./build.sh

Run a docker-based cluster

Run run-cluster.sh reefrt/{alias}. It will create a small cluster with 4 nodes and run a shell on the master node. For all cluster, Apache Hadoop (HDFS/YARN) is installed. The hostnames are hnn-001-01 (master) and hnn-001-01 ~ hnn-001-03 (slaves).

$ ./run-cluster.sh reefrt/hdi3.2
$ root@hnn-001-01:~#

Test Apache REEF on a docker-based YARN cluster

Among reefrt/hdi3.1, reefrt/hdi3.2, reefrt/hdp2.2 ~ 2.4, and reefrt/apache2.7, choose one and run it as described in section Run a docker-based cluster. If you want to test on Hadoop 2.7.3, choose reefrt/apache2.7. To simulate HDInsight 3.2, choose reefrt/hdi3.2. After running a cluster, use the following commands.

$ root@hnn-001-01:~# cd /reef
$ root@hnn-001-01:~# ./bin/runyarntests.sh
...
OK (# tests)
$ root@hnn-001-01:~# exit

If you see ‘OK’, it means all tests are passed. exit command will terminate your cluster automatically.

Test Apache REEF on a docker-based Mesos cluster

From reefrt/mesos0.24 to reefrt/mesos0.28, choose one and run it as described in section Run a docker-based cluster. If you want to test on Mesos 0.28, choose reefrt/mesos0.28. After running a cluster, use the following commands.

$ root@hnn-001-01:~# cd /reef
$ root@hnn-001-01:~# ./bin/runmesostests.sh hnn-001-01:5050
...
OK (# tests)
$ root@hnn-001-01:~# exit

If you see ‘OK’, it means all tests are passed. exit command will terminate your cluster automatically.

Reference