| Testing |
| ======= |
| |
| Examples below assume cwd at the gradlew script in the top directory of |
| the project's checkout. |
| |
| |
| Generic test/ checkup commands |
| ------------------------------ |
| |
| Run all unit tests: |
| |
| gradlew test |
| |
| Run all verification tasks, including tests: |
| |
| gradlew check |
| |
| Run all verification tasks, excluding tests (-x is gradle's generic task |
| exclusion mechanism): |
| |
| gradlew check -x test |
| |
| Run verification for a selected module only: |
| |
| gradlew :lucene:core:check # By full gradle project path |
| gradlew -p lucene/core check # By folder designation + task |
| |
| |
| Randomization |
| ------------- |
| |
| To run tests with the given starting seed pass 'tests.seed' |
| property: |
| |
| gradlew :lucene:misc:test -Ptests.seed=DEADBEEF |
| |
| There are a lot of other test randomization properties |
| available. To list them, their defaults and current values |
| run the testOpts task against a project that has tests. |
| For example: |
| |
| gradlew -p lucene/core testOpts |
| |
| |
| Filtering |
| --------- |
| |
| Run tests of lucene-core module: |
| |
| gradlew -p lucene/core test |
| |
| Run a single test case (from a single module). Uses gradle's built-in filtering |
| (https://docs.gradle.org/current/userguide/java_testing.html#test_filtering): |
| |
| gradlew -p lucene/core test --tests TestDemo |
| |
| Run all tests in a package: |
| |
| gradlew -p lucene/core test --tests "org.apache.lucene.document.*" |
| |
| Run all test classes/ methods that match this pattern: |
| |
| gradlew -p lucene/core test --tests "*testFeatureMissing*" |
| |
| |
| Test groups |
| ----------- |
| |
| Tests can be filtered by an annotation they're marked with. |
| Some test group annotations include: @AwaitsFix, @Nightly, @Slow |
| |
| This uses filtering infrastructure on the *runner* (randomizedtesting), |
| not gradle's built-in mechanisms (but it can be combined with "--tests"). |
| For example, run all lucene-core tests annotated as @Slow: |
| |
| gradlew -p lucene/core test -Ptests.filter=@Slow |
| |
| Test group filters can be combined into Boolean expressions: |
| |
| gradlew -p lucene/core test "default and not(@awaitsfix or @slow)" |
| |
| |
| Reiteration ("beasting") |
| ------------------------ |
| |
| Multiply each test case N times (this works by repeating the same test |
| within the same JVM; it also works in IDEs): |
| |
| gradlew -p lucene/core test --tests TestDemo -Ptests.iters=5 |
| |
| Tests tasks will be (by default) re-executed on each invocation because |
| we pick a random global tests.seed and because we want them to (force |
| the test task to run). If you want to make the 'tests' task obey up-to-date |
| gradle rules, fix the seed and turn off up to date trigger: |
| |
| gradlew -p lucene/core test -Ptests.seed=deadbeef -Ptests.neverUpToDate=false |
| |
| These properties can be set in your local gradle.properties. To force re-execution |
| of tests, even for the same master seed, apply cleanTest task: |
| |
| gradlew -p lucene/core cleanTest test -Ptests.seed=deadbeef |
| |
| The 'tests.iters' option should be sufficient for individual test cases |
| and is *much* faster than trying to duplicate re-runs of the entire |
| test suites. When it is absolutely needed to re-run an entire suite (because |
| of randomization in the static initialization, for example), you can do it |
| by running the 'beast' task with 'tests.dups' option: |
| |
| gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat |
| |
| Note the filter (--tests) used to narrow down test reiterations to a particular |
| class. You can use any filter, including no filter at all, but it rarely makes |
| sense (will take ages). By default the test tasks generated by the 'beast' mode |
| use a random starting seed for randomization. If you pass an explicit seed, this |
| won't be the case (all tasks will use exactly the same starting seed): |
| |
| gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat -Dtests.seed=deadbeef |
| |
| |
| Verbose mode and debugging |
| -------------------------- |
| |
| The "tests.verbose" mode switch enables standard streams from tests |
| to be dumped directly to the console. Run your verbose tests explicitly |
| specifying the project and test task or a fully qualified task path. Example: |
| |
| gradlew -p lucene/core test -Ptests.verbose=true --tests "TestDemo" |
| |
| |
| Profiling slow tests |
| -------------------- |
| |
| The "tests.profile" mode switch turns on a sampling profiler during test execution, |
| and prints a simple summary at the end. |
| |
| For example, top-10 histogram of methods (cpu samples): |
| |
| gradlew -p lucene/core test -Ptests.profile=true |
| |
| Alternatively, you can profile heap allocations instead: |
| |
| gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.mode=heap |
| |
| By default, results are computed (deduplicated) on just the method name, folding |
| together all events from the same method. To drill down further, you can increase the |
| stack size from the default of 1, to get a histogram of stacktraces instead: |
| |
| gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.stacksize=8 |
| |
| For big methods, it can also be helpful to include line numbers for more granularity: |
| |
| gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.linenumbers=true |
| |
| Using these additional options will make the results more sparse, so it may be useful |
| to increase the top-N count: |
| |
| gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.count=100 |
| |
| |
| Generating Coverage Reports |
| ------------------------------ |
| |
| Running the "coverage" task (or setting the property "tests.coverage" to true) |
| will run the tests with instrumentation to record code coverage. |
| |
| Example: |
| |
| gradlew -p lucene/core coverage |
| open lucene/core/build/reports/jacoco/test/html/index.html |
| |
| If you want to use test filtering to just check a particular test, specify |
| the "test" task explicitly before "coverage": |
| |
| gradlew -p lucene/core test --tests TestDemo coverage |
| open lucene/core/build/reports/jacoco/test/html/index.html |
| |
| |
| External data sets |
| ------------------ |
| |
| Some tests may require external (and large) data sets. To see relevant tasks |
| that download and extract these data files automatically, run the following: |
| |
| gradlew tasks --group "Data set download" |