You‘ve played with the existing tests and you are ready to create a new test. This section walks you through the process. If you are converting an existing test, then see the conversion guide instead. The details of each step are covered in other files, we’ll link them from here.
The first quesetion is: should your new test go into an existing category, or should you create a new one?
You should use an existing category if:
You should create a new category if:
When your test can reuse an existing cluser definition, then the question is about time. It takes significan time (minutes) to start a Docker cluster. We clearly don‘t want to pay that cost for a test that runs for seconds, if we could just add the test to another category. On the other hand, if you’ve gone crazy and added a huge suite of tests that take 20 minutes to run, then there is a huge win to be had by running the tests in parallel, even if they reuse an existing cluster configuration. Use your best judgment.
The existing categories are listed in the org.apache.druid.testsEx.categories package. The classes there represent JUnit categories. See Test Category for details.
If you create a new category, but want to reuse the configuration of an existing category, add the @Cluster annotation as described in the above link. Note: be sure to link to a “base” category, not to a category that, itself, has a @Cluster annotation.
If you use the @Cluster annotation, you must also add a mapping in the cluster.sh file. See the top of the file for an example.
If you create a new category, you must define a new cluster. There are two parts:
Create a new folder: custer/<category>, then create a docker-compose.yaml file in that folder. Define your cluster by borrowing heavily from existing files. See compose for details.
The only trick is if you want to include a new external dependency. The preferred approach is to use an “official” image. If you must, you can create a custom image in the it-image module. (We‘ve not yet done that, so if you need a custom image, let us know and we’ll figure it out.)
Tests need a variety of configuration information. This is, at present, more complex than we might like. You will at least need:
You may also need:
The cluster and properties are defined in a config file. Create a folder src/test/resources/cluster/<category>. Then add a file called docker.yaml. Crib the contents from the same category from which you borrowed the Docker Compose definitions. Strip out properties and metastore statements you don't need. Add those you do need. See Test Configuration for the gory details of this file.
You may also want to customize Guice, environment variable bindings, etc. This is done in the test setup method in your test.
There are many things that can go wrong. It is best to start simple.
Start by ensuring your cluster works.
it.sh up <category>.target/<category>/logs/<service>.log) files.Next, create your test file as described above and in Tests.
docker.yaml file as decribed above.If so, then this means that your test connected to your custer and verified the health of all the services declared in your docker.yaml file.
If something goes wrong, you'll know it is in the basics. Check your cluster status. Double-check the docker.yaml structure. Check ports. Etc.
Every test is a Druid client. Determine which service API you need. Find an existing test client. The DruidClusterAdminClient is the “modern” way to interact with the cluster, but thus far has a limited set of methods. There are older clients as well, but they tend to be quirky. Feel free to extend DruidClusterAdminClient, or use the older one: whatever works.
Inject the client into your test. See existing tests for how this is done.
Revise your “starter” test to do some trivial operation using the client. Retest to ensure things work.
From here, you can start writing tests. Explore the existing mechanisms (including those in the original druid-integration-tests module which may not yet have been ported to the new framework yet.) For example, there are ways to store specs as files and parameterize them in tests. There is a syntax for running queries and specifying expected results.
You may have to create a new tool to help with your test. If you do, try to use the new mechanisms, such as ResolvedClusterConfig rather than using the old, cumbersome ones. Post questions in Slack so we can help.
Your test may need a “non-default” extension. See Special Environment Variables for how to specify test-specific extensions. (Hint: don't copy/paste the full load list!)
Extensions have two aspects in ITs. They act like extensions in the Druid servers running in Docker. So, the extension must be avaialble in the Docker image. All standard Druid extensions which are available in the Druid distribution, are also available in the image. The may not be enabled, however. Hence the need to define the custom load list.
Your test may use code from the extension. To the tests, however, the extension is just another jar: it must be listed in the pom.xml file. There is no such thing as a “Druid extensions” to the tests themselves.
If you test an extension that is not part of the Druid distributeion, then it has to get into the image. Reach out on the slack mailing list so we can discuss solutions (such as mounting a directory that contains the extension).
The old IT framework was very liberal in its use of retries. Retires were used to handle:
The new framework takes a stricter view. The framework itself will ensure service are ready (using the Druid API for that purpose.) If a server reports itself ready, but still fails on one of your API calls, then we‘ve got a bug to fix. Don’t use retries to work around this issue because users won't know to do this.
In the new framwork, tests should not be flaky. Flaky tests are a drag on development; they waste time. If your test is flaky, please fix it. Don't count on the amount of times things take: a busy build system will run much slower than your dedicated laptop. And so on.
Ideally, Druid would provide a way to positively confirm that an action has occurred. Perhaps this might be a test-only API. Otherwise, a retry is fine, but should be coded into your test. (Or, better, implemented in a client.) Do this only if we document that, for that API, users should poll. Otherwise, again, users of the API under test won‘t know to retry, and so the test shouldn’t do so either.
This leaves random failures. The right place to handle those is in the client, since they are independent of the usage of the API.
The result of the above is that you should not need (or use) the ITRetryUtil mechanism. No reason for your test to retry 240 times if something is really wrong or your test is flaky.
This is an area under development. If you see a reason to retry, lets discuss it and put it in the proper place.
Run your tests in the IDE. Try them using it.sh test <category>. If that passes add the test to Travis. The details on how to do so are still being worked out. Likely, you will just copy/paste an existing test “stanza” to define your new test. Your test will run in parallel with all other IT categories, which is why we offered the advice above: the test has to have a good reason to fire up yet another build task.
Tests should run on the Middle Manager by default. Tests can optionally run on the Indexer. To run on Indexer:
export USE_INDEXER=indexer. (Use middleManager otherwise. If the variable is not set, middleManager is the default.)Then, there are two ways to handle indexer-specific configuration: the crude-but-effective way and the subtle way.
The crude way, which involves much copy/paste and results in two files which must be maintained in sync:
cluster/<category>/docker-compose.yaml file should be for the Middle manager. Create a separate file called cluster/<category>/docker-compose-indexer.yaml to define the Indexer-based cluster.The fancy way is to use the docker-compose.yaml generation template described elsewhere. In that case, the script will automatically generate either the Middle Manager, or the Indexer, depending on the environment variable mentioned above.
The client will choose Middle Manager or Indexer automatially if you set the USE_INDEXER environment variable in your IDE. (When run via the build process, the environment variable is already set.)
src/test/resources/cluster/<category>/docker.yaml file should contain a conditional entry to select define either the Middle Manager or Indexer. Example:middlemanager: if: middleManager instances: - port: 8091 indexer: if: indexer instances: - port: 8091
Now, the test will run on Indexer if the above environment variable is set, Middle Manager otherwise.
You may have a test that can run only on Middle Manager or Indexer. The crude-but-effective way to handle this is:
@Test
public void myMMOnlyTest()
{
if (ClusterConfig.isIndexer()) {
return; // Runs only on MM
}
// The MM-only test code here
}
It would be possible to define an annotation, managed by the DruidTestRunner, if this becomes something we need to do often.