samza-test/src/main/config/join/README

INTEGRATION TEST

* What does the test do? *
This is a correctness test that attempts to do partitioned messaging and use state. It is meant to be run while killing samsa and kafka machines to test fault-tolerance.
It runs in iterations and each iteration has a correctness criteria that is checked before launching the next iteration. Here are the jobs and their function

emitter.samza:
  This job takes input from the "epoch" topic. Epochs are number 0, 1, 2,...
  For each new epoch each emitter task does something like the following:
     for i = 0...count:
       send("emitted", i, partition)
  where partition is the task partition id.
  
joiner.samza:
  This job takes in the emitted values from emitter and joins them together by key.
  When it has received an emitted value from each partition it outputs the key to the topic "completed".
  To track which partitions have emitted their value it keeps a store with | seperated numbers. 
  The first entry is the epoch and the remaining entries are partitions that have emitted the key.
  
checker.samza:
  This job has a single partition and stores all the completed keys. When all the keys are completed it sends an incremented epoch to the epoch topic, kicking off a new round.
  
watcher.samza:
  This job watches the epoch topic. If the epoch doesn't advance within some SLA this job sends an alert email.
  
The state maintained by some of these jobs is slightly complex because of the need to make everything idempotent. So, for example, instead of keeping the partition count
in the joiner job we keep the set of partitions so that double counting can not occur.

To run, simply start all four jobs at once.

* How to setup test ? *

NOTE: You will need to have Paramiko installed in order to run these tests.

This test is meant to be used with hello-samza's bin/grid script. If you use hello-samza, and bin/grid bootstrap, then DEPLOY_DIR can be set to samza-hello-samza/deploy/samza,
and you can run the integration tests with ./bin/setup-int-test.sh path/to/samza-hello-samza/deploy.

The steps to setup the integration tests is similar to the samza-hello-samza setup, using the grid script.
Once you deploy Zookeeper, YARN and Kafka, you have to generate the tar ball with the tests.
Before generating the tar ball, update the "yarn.package.path" in $SAMZA_SRC/src/config/join/common.properties file to the path where the published tar will be made available.

--> Release the tar
cd $SAMZA_SRC
./gradlew releaseTestJobs
cp ~/samza-test/build/distributions/samza-test_*.tgz $DEPLOY_DIR
tar -xvf $DEPLOY_DIR/samza-test_*.tgz -C $DEPLOY_DIR/samza

--> Create the topics and start the samza jobs
./bin/setup-int-test.sh $DEPLOY_DIR

Now, you should be able to view all 4 jobs running in the Yarn UI.

FAILURE TESTING:

* What does the test do? *
This is used to test the resilience of the system. It periodically brings down a random container or kafka broker in the system and waits to see if it recovers correctly.

* How to setup test? *
Ensure that the 4 jobs are running via the YARN UI.
In order to trigger the failure testing, run the python script: $SAMZA_SRC/samza-test/src/main/python/samza_failure_testing.py

Usage: samza_failure_testing.py [options]

Options:
  -h, --help            show this help message and exit
  --node-list=nodes.txt
                        A list of nodes in the YARN cluster
  --kill-time=s         The time in seconds to sleep between
  --kafka-dir=dir       The directory in which to find kafka
  --kafka-host=localhost
                        Host on which Kafka is installed
  --yarn-dir=dir        The directory in which to find yarn
  --kill-kafka          Should we kill Kafka?
  --kill-container      Should we kill Application Container?
  --yarn-host=localhost
                        Host that will respond to Yarn REST queries