This module contains tests that verify end-to-end behaviour of Flink. We categorize end-to-end tests as either pre-commit tests or nightly tests. The former should be run on every commit, that is every Travis run, while the second category should be run by a nightly job or when manually verifying a release or making sure that the tests pass.
Tests in the pre-commit category should be more lightweight while tests in the nightly category can be quite heavyweight because we don't run them for every commit.
You can run all pre-commit tests by executing
$ FLINK_DIR=<flink dir> flink-end-to-end-tests/run-pre-commit-tests.sh
and all nightly tests via
$ FLINK_DIR=<flink dir> flink-end-to-end-tests/run-nightly-tests.sh
where is a Flink distribution directory.
You can also run tests individually via
$ FLINK_DIR=<flink dir> flink-end-to-end-tests/run-single-test.sh your_test.sh arg1 arg2
Have a look at test_batch_wordcount.sh
for a very basic test and test_streaming_kafka010.sh
for a more involved example. Whenever possible, try to put new functionality in common.sh
so that it can be reused by other tests.
In order to add a new test case you need add it to either test-scripts/run-nightly-tests.sh
and / or test-scripts/run-pre-commit-tests.sh
. Templates on how to add tests can be found in those respective files.
Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually.
A test is considered to have passed if it:
Note: There is a whitelist for exceptions and errors that do not lead to failure, which can be found in the check_logs_for_errors
and check_logs_for_exceptions
in test-scripts/common.sh
.
Please note that a previously supported pattern where you could assign a value the global variable PASS
to have your tests fail is not supported anymore.
The test runner performs a cleanup after each test case, which includes:
In some cases your test is required to do to some additional cleanup, for example shutting down external systems like Kafka or Elasticsearch. In this case it is a common pattern to trap a test_cleanup
function to EXIT
like this:
function test_cleanup { # do your custom cleanup here } trap test_cleanup EXIT