help/tests.txt

Testing
=======

Examples below assume cwd at the gradlew script in the top directory of
the project's checkout.


Generic test/ checkup commands
------------------------------

Run all unit tests:

gradlew test

Run all(*) verification tasks, including tests:

gradlew check

(*) This step may omit certain validation tasks (errorprone, for example)
    that are slow or require external resources. A pull request runs all 
    those locally omitted tasks on the CI (jenkins, gh actions).
    For this reason, it's a good idea to run patches via the CI.

Run all verification tasks, excluding tests (-x is gradle's generic task
exclusion mechanism):

gradlew check -x test

Run verification for a selected module only:

gradlew :lucene:core:check     # By full gradle project path
gradlew -p lucene/core check   # By folder designation + task


Randomization
-------------

To run tests with the given starting seed pass 'tests.seed'
property:

gradlew :lucene:misc:test -Ptests.seed=DEADBEEF

There are a lot of other test randomization properties
available. To list them, their defaults and current values
run the testOpts task against a project that has tests.
For example:

gradlew -p lucene/core testOpts


Filtering
---------

Run tests of lucene-core module:

gradlew -p lucene/core test

Run a single test case (from a single module). Uses gradle's built-in filtering
(https://docs.gradle.org/current/userguide/java_testing.html#test_filtering):

gradlew -p lucene/core test --tests TestDemo

Run all tests in a package:

gradlew -p lucene/core test --tests "org.apache.lucene.document.*"

Run all test classes/ methods that match this pattern:

gradlew -p lucene/core test --tests "*testFeatureMissing*"


Test groups
-----------

Tests can be filtered by an annotation they're marked with.
Some test group annotations include: @AwaitsFix, @Nightly

This uses filtering infrastructure on the *runner* (randomizedtesting), 
not gradle's built-in mechanisms (but it can be combined with "--tests").
For example, run all lucene-core tests annotated as @Nightly:

gradlew -p lucene/core test -Ptests.filter=@Nightly

Test group filters can be combined into Boolean expressions:

gradlew -p lucene/core test "default and not(@awaitsfix or @nightly)"


Reiteration ("beasting")
------------------------

Multiply each test case N times (this works by repeating the same test
within the same JVM; it also works in IDEs):

gradlew -p lucene/core test --tests TestDemo -Ptests.iters=5

Tests tasks will be (by default) re-executed on each invocation because
we pick a random global tests.seed and because we want them to (force
the test task to run). If you want to make the 'tests' task obey up-to-date
gradle rules, fix the seed and turn off up to date trigger:

gradlew -p lucene/core test -Ptests.seed=deadbeef -Ptests.neverUpToDate=false

These properties can be set in your local gradle.properties. To force re-execution 
of tests, even for the same master seed, apply cleanTest task:

gradlew -p lucene/core cleanTest test -Ptests.seed=deadbeef

The 'tests.iters' option should be sufficient for individual test cases
and is *much* faster than trying to duplicate re-runs of the entire
test suites. When it is absolutely needed to re-run an entire suite (because
of randomization in the static initialization, for example), you can do it
by running the 'beast' task with 'tests.dups' option:

gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat

Note the filter (--tests) used to narrow down test reiterations to a particular
class. You can use any filter, including no filter at all, but it rarely makes
sense (will take ages). By default the test tasks generated by the 'beast' mode
use a random starting seed for randomization. If you pass an explicit seed, this
won't be the case (all tasks will use exactly the same starting seed):

gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat -Dtests.seed=deadbeef


Verbose mode and debugging
--------------------------

The "tests.verbose" mode switch enables standard streams from tests
to be dumped directly to the console. Run your verbose tests explicitly
specifying the project and test task or a fully qualified task path. Example:

gradlew -p lucene/core test -Ptests.verbose=true --tests "TestDemo"


Larger heap size
--------------------------

By default tests run with a 512 MB max heap.  But some tests (monster/nightly)
need more heap.  Use "-Dtests.heapsize" for this:

  gradlew -p lucene/core test --tests "Test2BFST" -Dtests.heapsize=32g


Run GUI tests headlessly with Xvfb (Linux only)
-----------------------------------------------

GUI test for Luke application launches a window, this might mess up your
monitor depending on the display manager you are using. In that case,
you can install Xvfb (X Virtual Frame Buffer) so that the test runs on the
virtual display and does not open a real window.

# redhat-type OS
$ sudo yum install Xvfb

# ubuntu/debian-type OS
$ sudo apt install xvfb

# arch
$ sudo pacman -S xorg-server-xvfb


Profiling slow tests
--------------------

The "tests.profile" mode switch turns on a sampling profiler during test execution,
and prints a simple summary at the end.

For example, top-10 histogram of methods (cpu samples):

gradlew -p lucene/core test -Ptests.profile=true

Alternatively, you can profile heap allocations instead:

gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.mode=heap

By default, results are computed (deduplicated) on just the method name, folding
together all events from the same method. To drill down further, you can increase the
stack size from the default of 1, to get a histogram of stacktraces instead:

gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.stacksize=8

For big methods, it can also be helpful to include line numbers for more granularity:

gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.linenumbers=true

Using these additional options will make the results more sparse, so it may be useful
to increase the top-N count:

gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.count=100


Generating Coverage Reports
------------------------------

Running the "coverage" task (or setting the property "tests.coverage" to true)
will run the tests with instrumentation to record code coverage.

Example:

gradlew -p lucene/core coverage
open lucene/core/build/reports/jacoco/test/html/index.html

If you want to use test filtering to just check a particular test, specify
the "test" task explicitly before "coverage":

gradlew -p lucene/core test --tests TestDemo coverage
open lucene/core/build/reports/jacoco/test/html/index.html


External data sets
------------------

Some tests may require external (and large) data sets. To see relevant tasks
that download and extract these data files automatically, run the following:

gradlew tasks --group "Data set download"