test/README.pytest

Apache httpd pytest suite
=========================
Using pytest (<https://docs.pytest.org/en/6.2.x/>) and a Python >= 3.8
for a more flexible testing of Apache httpd.

Install
-------
The Python dependencies (pytest, pyOpenSSL, etc.) are managed with uv
(https://docs.astral.sh/uv/), which reads test/pyproject.toml + test/uv.lock and
creates test/.venv, shared with pytest_suite. Install uv, then let
runtests.sh build the venv on first run (it invokes `uv sync` for you);
or create it yourself, from test/:
> uv sync

And for 'h2load':
> apt install nghttp2-client


Usage
-----
In your httpd source checkout, do:

> make install
> pytest

and all tests defined run on the installed executable and modules.
> pytest test/modules/core
runs all the core tests. You can run tests also by name selection:
> pytest -k test_core
runs all test cases starting with 'test_core'. Similar:
> pytest -k test_core_001_04
runs the test cases starting with that.

Test output gets more verbose, by adding one or several '-v'. This
also raises the error log level of the tested modules.
> pytest -vvv -k test_h2_004_01
run the specific test with mod_http2 at log level TRACE2.

There is an option to archive the results across different
modules and httpd versions. The archiving will preserve
error_log and access_log (cumulative for all tests in the module),
and save a separate config file for each test case.
> pytest test/modules/core --archive=/path/to/archive

If you don't provide any specific module, pytest will execute all of them
and add a new folder per module to the existing structure.
> pytest --archive=/path/to/archive

Always use --archive=<path> (with =, not a space) to avoid pytest rootdir issues.

The archive folder is named after the httpd version and the git commit hash
so results from different code versions are kept separate.
Running the same module again with the same commit updates its folder in place.

What gets archived:
- error_log and access_log for the module run
- per-test configs: test.conf snapshot after each test
- module config: modules.conf
- shared configs and infrastructure in _shared/ to avoid duplication

Example archive structure with 2 httpd versions:
    ├── /path/to/archive
    │   ├── 2.4.65-a3f8c1d
    │   │   ├── _shared
    │   │   │   ├── conf
    │   │   │   │   ├── httpd.conf
    │   │   │   │   ├── mime.types
    │   │   │   │   └── stop.conf
    │   │   │   ├── ca
    │   │   ├── core
    │   │   │   ├── conf
    │   │   │   │   ├── modules.conf
    │   │   │   │   ├── test_core_001_01.conf
    │   │   │   └── logs
    │   │   │       ├── error_log
    │   │   │       └── access_log
    │   │   ├── http1
    │   │   └── http2
    │   └── 2.4.66-b7d2e4f
    │        ├── _shared
    │        └── core

By default, test cases will configure httpd with mpm_event. You
can change that with the invocation:
> MPM=worker pytest test/modules/http2

Some tests rely on additional tools installed in your environment
and will 'skip' if those are not present. In a non-verbose run,
these will appear as 's' in the output. If you run pytest more
verbose, the skipped test cases will mention a reason for why
they are disabled.

For example, most tests in test/modules/md require an installation
of 'pebble', an ACME test server, and look for it in $PATH.


Strict optional dependencies
----------------------------
A skip is convenient locally and dangerous in CI: a run that was meant
to cover SSL, HTTP/2 and mod_md stays green while quietly covering none
of them, because the machine was misconfigured. Strict mode turns that
silence into a failure.

> pytest --strict-optional
> PYHTTPD_STRICT_OPTIONAL=1 pytest

Both are off by default, so nothing changes unless you ask for it. The
env var is the one to set in CI; the flag is the one to use by hand.

Not every skip becomes a failure. Skips come in two kinds:

  - "does not apply here" -- wrong MPM, httpd too old, STRESS_TEST not
    set, $SSL selects a different TLS module. These always skip, in
    strict mode too, and a test gated on both kinds skips rather than
    fails.

  - "something we expected to be installed is missing" -- curl, nghttp,
    h2load, the 'websockets' and 'multipart' python packages, mod_ssl,
    mod_tls, mod_md, a2md, the ACME test server, an OCSP responder.
    These fail under strict mode.

The OCSP responder is the one judgement call in that split. pebble
serves no OCSP, boulder does, so with ACME=pebble the stapling tests
have nothing to talk to. That is a property of the environment rather
than of this machine's packages, and a pipeline that is meant to cover
OCSP stapling and runs pebble has lost that coverage silently -- which
is the thing strict mode exists to catch. So it fails, and the message
names ACME=pebble as what was detected. Run ACME=boulder to cover
stapling, or leave strict mode off.

A strict failure is raised during test setup, so pytest reports it as
ERROR rather than FAILED; the exit code is non-zero either way. The
message is prefixed '[strict-optional]' so it greps cleanly out of a CI
log, and names the dependency, what was expected, what was found
instead, and how to switch strict mode back off.

curl is the one dependency gated for the suite as a whole rather than
test by test: apache_restart() proves the server came up by making a
curl request, so without curl nothing under test/modules/ can run. That
gate lives in the root conftest.py.


Workings
--------
All tests start httpd on their own and try hard to shut it down
afterwards. You can abort tests with ^C and they clean up.

httpd is started/stopped repeatedly in testing as configurations
for individual test cases are changed. This is a main difference to
the Perl test framework which starts the server with all possible
combinations configured that are needed by tests.

In test/gen/apache a server root is created with config, logs and htdocs
directories. test/gen/apache/logs/error_log will be the log.
Configs start in test/gen/apache/conf/httpd.conf. modules.conf is
dynamically created for the list of modules that a test suite needs.

Test cases write their specific setup in test.conf and reload/restart
the httpd process. This makes for a small configuration in a test case.


Development
-----------

Adding a test in an existing file is done by adding a method. Its name
must start with 'test_' and the common practice is to have the name
of the test suite there as well. All http2 tests start with 'test_h2_'.

Following this can be any characters. If you make test cases of a
certain feature with a common prefix, it is easier to invoke just
them using the '-k' selector on the command line.

You can also add just a new file to a test suite, if you do a new
range of test cases that do not fit anywhere else. A very simple
one is found in test/modules/http2/test_001_httpd_alive.py.

There is a python class defined with 2 methods. One is the test
method itself and the other one ' is

    @pytest.fixture(autouse=True, scope='class')
    def _class_scope(self, env):
        code

is marked as a pytest 'fixture'. This is some pytest magic.
'autouse=True' means that this fixture is run, even though
no test case uses it directly. scope='class' means that it
is run once for all test cases in this class.

As you see, this fixture gets a parameter named 'env' and
that is the name of another pytest fixture, defined in the
file 'conftest.py' in the same directory.

    @pytest.fixture(scope="package")
    def env(pytestconfig) -> H2TestEnv:
        code

This one runs one time per 'package', meaning for all test
cases in this directory. And it gets the 'pytestconfig'
as parameter which is a standard pytest fixture.

So, when you run 'pytest -k test_h2_004', pytest will look
at _all_ test cases defined and collect those with that
prefix. For each directory with test cases found, it will
process the 'conftest.py', boot-strapping the 'env' fixture,
and the process the files with active test cases.

As a result, if you invoke just a single test case, only
the fixtures needed for that test case are created. This
gives good turn around times when debugging a test case.

If you want to add a new test suite, create a new directory.
Add the files '__init__.py', 'conftest.py' and a first
'test_suitename_something.py'. test/modules/core is the
simplest example. 'test/modules/http2' shows how you load
other modules. 'test/modules/md' checks and starts external
processes (an ACME test server).

If your test needs something that may not be installed, say
so with the decorator from 'pyhttpd.depends'. It takes the
condition itself, so it works on a method, a class or a
module-level 'pytestmark':

    from pyhttpd.depends import needs_dependency

    @needs_dependency("a2md", MDTestEnv.has_a2md(),
                      reason="no a2md available")
    class TestStore:

For the tools the suite gates on most often there are ready
made versions of that decorator, which probe the tool once
and cache the answer:

    @H2TestEnv.needs_nghttp()             # and (assets=True)
    @H2TestEnv.needs_h2load("1.41.0")
    @HttpdTestEnv.needs_multipart()

For a check that can only be made once the test is running,
'env.require()' does the same job from inside the body:

    env.require(env.curl_is_at_least('8.0.0'),
                'need at least curl v8.0.0 for this',
                dep='curl',
                detected=f"curl {env.curl_version_str()}")

Both skip normally and fail under strict mode; see "Strict
optional dependencies" above. When the reason is instead
"this test does not apply here", keep using a plain
'pytest.mark.skipif' or 'pytest.skip()' -- those never
become failures.


Infrastructure
--------------
The test cases rely on the classes provided in `test/pyhttpd/` (where
`test/runtests.sh` runs the suite) for common code in managing
a httpd test instance and do provide some base setups:
- a small set of virtual hosts with some files
- access to paths and definitions (env fixture)
- start/stop httpd and inspect the error log
- run clients like curl and nghttp
- create certificates for your hosts and make curl use
  the root certs (so no --insecure calls by curl).


Tests of changes missing from 2.4.x
-----------------------------------
This directory is the same on trunk and on 2.4.x: a test is written on trunk,
then `test/` is copied to 2.4.x as it is, and `check-test-sync.sh` fails if
the two branches differ. A test covering a trunk change that 2.4.x does not
have yet carries a mark from `pyhttpd/backport.py` (or, in `pytest_suite/`,
`apache_pytest.not_backported`) naming the trunk revision:

    from pyhttpd.backport import not_backported, skip_not_backported

    @not_backported("r1939033", "nonce-count validation")
    def test_digest_001(self, env): ...

`not_backported` makes the test an expected failure on a server older than
2.5.0, `skip_not_backported` skips it where failing would hang or crash the
server or leave errors in the error log. Neither does anything on trunk.
Only a failure of the test function itself is expected: an error in the
setup or teardown of a fixture (a server that does not start, the error log
check) stays an error. Mark only the tests, or the parameters (with
`pytest.param(..., marks=not_backported(...))`), that fail without the
change; on 2.4.x a mark on a test that passes there would turn a regression
into an expected failure.
Both marks are applied when the tests are collected, by `pyhttpd/gating.py`,
against the httpd that `pyhttpd/build.py` found (`--apxs`, `--httpd`, or the
build's `config.ini`); the `need_*` markers of `pytest_suite/` are evaluated
there too, and a missing module or PHP/SSL support is a missing dependency:
skipped, or failed under `--strict-optional`.

When the change is backported, add the 2.4.x release it first appears in:

    @not_backported("r1939033", "nonce-count validation", backported_in="2.4.71")

on trunk, then copy `test/` to 2.4.x again. Older 2.4.x servers keep the
expectation. `platform="win32"` limits it to one `sys.platform`.