help/tests.md

# Testing

Examples below assume cwd at the gradlew script in the top directory of the project's checkout.

## Generic test/ checkup commands

Run all unit tests:

```bash
gradlew test
```

Run all(\*) verification tasks, including tests:

```bash
gradlew check
```

(\*) This step may omit certain validation tasks (errorprone, for example) that are slow or require external
resources. A pull request runs all those locally omitted tasks on the CI (jenkins, gh actions). For this reason, it's
a good idea to run patches via the CI.

Run all verification tasks, excluding tests (`-x` is gradle's generic task exclusion mechanism):

```bash
gradlew check -x test
```

Run verification for a selected module only:

```bash
gradlew :lucene:core:check     # By full gradle project path
gradlew -p lucene/core check   # By folder designation + task
```

## Randomization

To run tests with the given starting seed pass `tests.seed` property:

```bash
gradlew :lucene:misc:test -Ptests.seed=DEADBEEF
```

There are a lot of other test randomization properties available. To list them, their defaults and current values run
the `buildOptions` task against a project that has tests. For example:

```bash
gradlew -p lucene/core buildOptions
```

or list all options with:

```bash
gradlew allOptions
```

## Filtering

Run tests of lucene-core module:

```bash
gradlew -p lucene/core test
```

Run a single test case (from a single module). Uses gradle's built-in filtering
(<https://docs.gradle.org/current/userguide/java_testing.html#test_filtering>):

```bash
gradlew -p lucene/core test --tests TestDemo
```

Run all tests in a package:

```bash
gradlew -p lucene/core test --tests "org.apache.lucene.document.*"
```

Run all test classes/ methods that match this pattern:

```bash
gradlew -p lucene/core test --tests "*testFeatureMissing*"
```

## Test groups

Tests can be filtered by an annotation they're marked with. Some test group annotations include: `@AwaitsFix`,
`@Nightly`

This uses filtering infrastructure on the *runner* (randomizedtesting), not gradle's built-in mechanisms (but it can be
combined with `--tests`). For example, run all lucene-core tests annotated as `@Nightly`:

```bash
gradlew -p lucene/core test -Ptests.filter=@Nightly
```

Test group filters can be combined into Boolean expressions:

```bash
gradlew -p lucene/core test "default and not(@awaitsfix or @nightly)"
```

## Reiteration ("beasting")

Multiply each test case N times (this works by repeating the same test within the same JVM; it also works in IDEs):

```bash
gradlew -p lucene/core test --tests TestDemo -Ptests.iters=5
```

Tests tasks will be (by default) re-executed on each invocation because we pick a random global `tests.seed` and
because we want them to (force the test task to run). If you want to make the `tests` task obey up-to-date gradle
rules, fix the seed and force rerunning:

```bash
gradlew -p lucene/core test -Ptests.seed=deadbeef -Ptests.rerun=true
```

These properties can be set in your local `gradle.properties`. To force re-execution of tests, even for the same master
seed, apply `cleanTest` task:

```bash
gradlew -p lucene/core cleanTest test -Ptests.seed=deadbeef
```

The `tests.iters` option should be sufficient for individual test cases and is *much* faster than trying to duplicate
re-runs of the entire test suites. When it is absolutely needed to re-run an entire suite (because of randomization in
the static initialization, for example), you can do it by running the `beast` task with `tests.dups` option:

```bash
gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat
```

Note the filter (`--tests`) used to narrow down test reiterations to a particular class. You can use any filter,
including no filter at all, but it rarely makes sense (will take ages). By default the test tasks generated by the
`beast` mode use a random starting seed for randomization. If you pass an explicit seed, this won't be the case (all
tasks will use exactly the same starting seed):

```bash
gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat -Dtests.seed=deadbeef
```

## Verbose mode and debugging

The `tests.verbose` mode switch enables standard streams from tests to be dumped directly to the console. Run your
verbose tests explicitly specifying the project and test task or a fully qualified task path. Example:

```bash
gradlew -p lucene/core test -Ptests.verbose=true --tests "TestDemo"
```

## Larger heap size

By default tests run with a 512 MB max heap. But some tests (monster/nightly) need more heap. Use `-Ptests.heapsize`
for this:

```bash
gradlew -p lucene/core test --tests "Test2BFST" -Ptests.heapsize=32g
```

## Run GUI tests headlessly with Xvfb (Linux only)

GUI test for Luke application launches a window, this might mess up your monitor depending on the display manager you
are using. In that case, you can install Xvfb (X Virtual Frame Buffer) so that the test runs on the virtual display
and does not open a real window.

```bash
# redhat-type OS
sudo yum install Xvfb

# ubuntu/debian-type OS
sudo apt install xvfb

# arch
sudo pacman -S xorg-server-xvfb
```

## Profiling slow tests

The `tests.profile` mode switch turns on a sampling profiler during test execution, and prints a simple summary at the
end.

For example, top-10 histogram of methods (cpu samples):

```bash
gradlew -p lucene/core test -Ptests.profile=true
```

Alternatively, you can profile heap allocations instead:

```bash
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.mode=heap
```

By default, results are computed (deduplicated) on just the method name, folding together all events from the same
method. To drill down further, you can increase the stack size from the default of 1, to get a histogram of
stacktraces instead:

```bash
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.stacksize=8
```

For big methods, it can also be helpful to include line numbers for more granularity:

```bash
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.linenumbers=true
```

Using these additional options will make the results more sparse, so it may be useful to increase the top-N count:

```bash
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.count=100
```

## Generating coverage reports

Running the `coverage` task (or setting the property `tests.coverage` to true) will run the tests with instrumentation
to record code coverage.

Example:

```bash
gradlew -p lucene/core coverage
open lucene/core/build/reports/jacoco/test/html/index.html
```

If you want to use test filtering to just check a particular test, specify the `test` task explicitly before
`coverage`:

```bash
gradlew -p lucene/core test --tests TestDemo coverage
open lucene/core/build/reports/jacoco/test/html/index.html
```

## External data sets

Some tests may require external (and large) data sets. To see relevant tasks that download and extract these data
files automatically, run the following:

```bash
gradlew tasks --group "Data set download"
```