README.md - httpd-tests - Git at Google

 # Testing Apache httpd with the Perl Test Harness

 ## Prerequisites

 These Perl modules must installed to be able to use the test harness:

 - `ExtUtils::MakeMaker`
 - `Test`
 - `Apache::Test`

 The
 [`Bundle::ApacheTest`](https://metacpan.org/pod/Bundle%3A%3AApacheTest)
 CPAN module can be installed to gather many dependencies of `Apache::Test`:

 ```
 perl -MCPAN -e 'install Bundle::ApacheTest'
 ```

 > **Note:** `Crypt::SSLeay` requires OpenSSL to be installed (only required
 > for `t/TEST -ssl`): https://www.openssl-library.org/
 > More accurate results may be obtained by using the same openssl command
 > line and libraries as consumed by APR-util and `mod_ssl`, due to X.509
 > formatting behavior differences.

 For an extensive documentation see
 https://perl.apache.org/docs/general/testing/testing.html

 To run the tests for all Apache web server modules, some additional
 CPAN modules will be required. If the tests don't work, make sure
 that you have up to date versions of each of these perl modules:

 ```
 cpan App::cpanminus
 cpanm Bundle::ApacheTest
 cpanm HTTP::DAV DateTime Time::HiRes \
   Test::Harness IO::Socket::SSL \
   IO::Socket::IP IO::Select LWP::Protocol::https AnyEvent \
   AnyEvent::WebSocket::Client LWP::Protocol::AnyEvent::http FCGI
 ```

 If the Bundle::ApacheTest fails due to native compilation issues with Crypt::SSLeay, you
 can install the remaining packages:

 ```
 cpanm --quiet --showdeps  Bundle::ApacheTest|grep -v Crypt::SSLeay|xargs cpanm
 ```

 ## Quick Start

 If you don't care how it works and just want to run the tests:

 1. Clone this repository:
    ```bash
    git clone --recurse-submodule https://github.com/apache/httpd-tests
    ```
 2. You need an installation of Apache httpd (2.4.x or trunk).
 3. Any DSOs you wish to use should be configured in that Apache's
    `httpd.conf` (the test harness will pick this configuration up).
 4. Setup:
    ```bash
    perl Makefile.PL -apxs /path/to/apache/bin/apxs
    ```
 5. Run the tests:
    ```bash
    t/TEST
    ```
 5. Evaluate test output.

 ## Getting a Little Deeper

 The test harness will run every `.t` file under the `t/` directory. To run
 only the tests for PHP:

 ```
 t/TEST -httpd /path/to/apache/bin/httpd t/php
 ```

 That will start the test server, run the `.t` tests under `t/php` and shut
 down the test server. You can also control each of these steps individually.

 Start the test server:
 ```
 t/TEST -httpd /path/to/apache/bin/httpd -start
 ```

 Run the PHP tests in the test environment:
 ```
 t/TEST t/php
 ```

 Stop the test server:
 ```
 t/TEST -stop
 ```

 Run the server under gdb (using `-X`):
 ```
 t/TEST -d gdb
 ```

 > **Note:** At this point, you have a working test environment. You can look
 > in `t/conf` for the test server configuration files. These are generated by
 > the test harness. Once you have a working test environment, you do not need
 > to specify `httpd` on the `t/TEST` command line. For instance, to start the
 > server up again:
 > ```
 > t/TEST -start
 > ```

 ## Running Regression Tests

 For a full regression test, you should have all modules loaded. Build the
 server with:

 ```
 configure --enable-modules=reallyall --enable-load-all-modules ...
 ```

 among other things. Edit the generated `httpd.conf` and comment all mpm
 modules that you do not want. Run `t/TEST -clean` again.

 You will see some:
 ```
 skipped: cannot find module 'XXX'
 ```
 as not all modules are in every Apache release (but the tests run for all).

 All in all, some >4k tests will run and the result needs to be: **PASS**

 ## Trouble Shooting

 If you have a "PASS" at the end of `t/TEST`, congratulations! If not, this
 section gives some advice in order to find the cause.

 0. If your test startup hangs forever in "waiting for server to warm up", but
    the test server is reachable under port 8529, you might be the victim of
    IPv4/6 confusion. The default servername configured is `localhost` and
    some operating systems define `127.0.0.1` *as well as* `::1` in
    `/etc/hosts`. If the test server listens only on `0.0.0.0` it might not
    answer requests to `::1` and that causes the test startup to hang.
    **Solution:** comment the `::1` line in `/etc/hosts` and see if that
    improves matters.

 1. Run `t/TEST -clean` every time you change something in your Apache
    configuration. The test suite picks up certain things from your installed
    `httpd.conf` (such as `LoadModule` statements) and will not see your
    changes unless you clean it.

 2. Failures in `proxy.t` may originate from the fact that the test script
    cannot open the specified port. This happens on some machines if you abort
    a test run and the socket is not properly shut down. Check if the error
    goes away after a reboot. (`proxy.t` tests are slow, so chances you
    interrupt tests at that point are good.)

 3. Failures in `access.t` may result from reverse lookups not working or
    giving other answers than expected. In the case 0 above, if the test
    client connects via `127.0.0.1`, a "Grant for localhost" might resolve to
    `::1` and therefore will not match the access rules of the tests.
    **Solution:** check that your servername is `localhost` (which is the
    default) and that it *always* resolves to `127.0.0.1`.

 4. If some ssl test cases fail, especially when `t/ssl/proxy.t` fails, the
    reason can be mismatches between your installed SSL library and the one
    httpd uses. The `openssl` binary found in your `$PATH` is used to create
    binary setup files by `t/TEST`. If another version of openssl then tries
    to read these from your Apache server process, it might fail. Try:
    ```
    t/TEST -clean
    PATH=<bin dir of correct openssl>:$PATH t/TEST
    ```
    If a lot of ssl tests fail, check in the error log for the presence of a
    certificate validation error. If you find it, check the expiration date of
    the TLS/SSL certificates used by the tests; they might be expired. Running
    `t/TEST -clean` should delete the old ssl certificates, so they'll be
    regenerated during the next run.

 5. If you see failures in the `modules/h2.t` test cases, please notify the
    dev mailing list with a description of your setup. These tests are quite
    young, currently only valid in 2.4.x and later and interact with quite
    some other modules as well as OpenSSL versions installed. Some tests
    require `mod_ssl` so make sure to load it in the httpd conf.

 6. Segmentation faults and core dumps occurring while executing the test suite
    might indicate a real problem but always run again the tests after a clean
    `make install` to avoid inconsistencies from old objects.

 7. If you see error messages like "Parse errors: Bad plan. You planned X tests
    but ran Y." it usually means that you are missing a perl module or the
    tested httpd module depends on another one not loaded in the httpd config.

 8. If you see SSL certificate errors, remove `t/conf/ssl/ca` prior to
    `t/TEST -clean`.

 9. perl 5.28 in MacOS homebrew seems to hang the test suite. Invoking
    `/usr/bin/perl Makefile.PL -apxs ...` will cause an older perl to be used.

 ## Smoking Tests

 Sometimes it's possible that the test is passing properly for the first time,
 but when you run it again, the test might fail. It's important to run
 repetition smoke testing. For example to repeat the tests 5 times:

 ```
 t/SMOKE -times=5
 ```

 It's also possible that a test will pass when it's run after a particular
 test, but if moved to run after a different state it may fail. For this reason
 by default the tests run in random order.

 Since it's important to be able to reproduce the problem with random testing,
 whenever `-order=random` is used, the used seed is printed to STDERR. It can
 then be fed into future tests via the `APACHE_TEST_SEED` environment variable.

 By adding the option `-order=repeat`, the tests will be run in alphabetical
 order.

 Combining these two important smoke testing techniques, one can run tests with:

 ```
 t/SMOKE -times=N -order=(repeat|random)
 ```

 For example, to run the mod_rewrite tests 5 times:

 ```
 t/SMOKE -times=5 -verbose t/modules/rewrite.t
 ```

 The tests can be repeated N times, and run in the following modes:

 - randomize all tests
 - repeat the whole test suite N times

 For configuration options and default settings run:

 ```
 t/SMOKE -help
 ```

 For more information refer to the `Apache::TestSmoke` manpage.

 ## Test Environment Configuration

 The test server is configured with conf files like any normal Apache server.
 The tricky part is those conf files are generated by the harness just prior
 to starting the server. `t/conf/httpd.conf` is generated by
 `t/conf/httpd.conf.in`. If that does not exist, the harness will generate a
 working configuration and will include `LoadModule` (and `AddModule` for
 Apache 1.3) directives from the `httpd.conf` associated with the httpd binary
 you are using for testing.

 If `t/conf/extra.conf.in` exists, `t/conf/extra.conf` will be generated from
 that, and an `Include` directive for that file will be put in the generated
 `t/conf/httpd.conf`.

 `t/conf/apache_test_config.pm` is generated from the test configuration. It
 contains all the information about the configuration of your test server. You
 can access this information in test scripts by:

 ```perl
 my $env = Apache::TestConfig->thaw;
 ```

 `Apache::TestConfig` accesses `apache_test_config.pm` and returns a hash
 reference with all the information. Look through `apache_test_config.pm`;
 it's a lot of stuff. Once these conf files are generated, you have a working
 test environment, and they must be 'cleaned' if you wish to make changes to
 them:

 ```
 t/TEST -clean
 ```

 (Now you will have to specify your httpd binary when starting back up again.)

 ## More Information

 For more information on using the test harness and writing tests, see the
 README in `Apache-Test` and the examples in `Apache-Test/t`.

 The test harness was originally written by Doug MacEachern and is discussed
 on the [httpd dev mailing list](mailto:dev@httpd.apache.org).

 It is also included in modperl-2.0 source along with tests for modperl-2.0.
	# Testing Apache httpd with the Perl Test Harness

	## Prerequisites

	These Perl modules must installed to be able to use the test harness:

	- `ExtUtils::MakeMaker`
	- `Test`
	- `Apache::Test`

	The
	[`Bundle::ApacheTest`](https://metacpan.org/pod/Bundle%3A%3AApacheTest)
	CPAN module can be installed to gather many dependencies of `Apache::Test`:

	```
	perl -MCPAN -e 'install Bundle::ApacheTest'
	```

	> Note: `Crypt::SSLeay` requires OpenSSL to be installed (only required
	> for `t/TEST -ssl`): https://www.openssl-library.org/
	> More accurate results may be obtained by using the same openssl command
	> line and libraries as consumed by APR-util and `mod_ssl`, due to X.509
	> formatting behavior differences.

	For an extensive documentation see
	https://perl.apache.org/docs/general/testing/testing.html

	To run the tests for all Apache web server modules, some additional
	CPAN modules will be required. If the tests don't work, make sure
	that you have up to date versions of each of these perl modules:

	```
	cpan App::cpanminus
	cpanm Bundle::ApacheTest
	cpanm HTTP::DAV DateTime Time::HiRes \
	Test::Harness IO::Socket::SSL \
	IO::Socket::IP IO::Select LWP::Protocol::https AnyEvent \
	AnyEvent::WebSocket::Client LWP::Protocol::AnyEvent::http FCGI
	```

	If the Bundle::ApacheTest fails due to native compilation issues with Crypt::SSLeay, you
	can install the remaining packages:

	```
	cpanm --quiet --showdeps Bundle::ApacheTest\|grep -v Crypt::SSLeay\|xargs cpanm
	```

	## Quick Start

	If you don't care how it works and just want to run the tests:

	1. Clone this repository:
	```bash
	git clone --recurse-submodule https://github.com/apache/httpd-tests
	```
	2. You need an installation of Apache httpd (2.4.x or trunk).
	3. Any DSOs you wish to use should be configured in that Apache's
	`httpd.conf` (the test harness will pick this configuration up).
	4. Setup:
	```bash
	perl Makefile.PL -apxs /path/to/apache/bin/apxs
	```
	5. Run the tests:
	```bash
	t/TEST
	```
	5. Evaluate test output.

	## Getting a Little Deeper

	The test harness will run every `.t` file under the `t/` directory. To run
	only the tests for PHP:

	```
	t/TEST -httpd /path/to/apache/bin/httpd t/php
	```

	That will start the test server, run the `.t` tests under `t/php` and shut
	down the test server. You can also control each of these steps individually.

	Start the test server:
	```
	t/TEST -httpd /path/to/apache/bin/httpd -start
	```

	Run the PHP tests in the test environment:
	```
	t/TEST t/php
	```

	Stop the test server:
	```
	t/TEST -stop
	```

	Run the server under gdb (using `-X`):
	```
	t/TEST -d gdb
	```

	> Note: At this point, you have a working test environment. You can look
	> in `t/conf` for the test server configuration files. These are generated by
	> the test harness. Once you have a working test environment, you do not need
	> to specify `httpd` on the `t/TEST` command line. For instance, to start the
	> server up again:
	> ```
	> t/TEST -start
	> ```

	## Running Regression Tests

	For a full regression test, you should have all modules loaded. Build the
	server with:

	```
	configure --enable-modules=reallyall --enable-load-all-modules ...
	```

	among other things. Edit the generated `httpd.conf` and comment all mpm
	modules that you do not want. Run `t/TEST -clean` again.

	You will see some:
	```
	skipped: cannot find module 'XXX'
	```
	as not all modules are in every Apache release (but the tests run for all).

	All in all, some >4k tests will run and the result needs to be: PASS

	## Trouble Shooting

	If you have a "PASS" at the end of `t/TEST`, congratulations! If not, this
	section gives some advice in order to find the cause.

	0. If your test startup hangs forever in "waiting for server to warm up", but
	the test server is reachable under port 8529, you might be the victim of
	IPv4/6 confusion. The default servername configured is `localhost` and
	some operating systems define `127.0.0.1` as well as `::1` in
	`/etc/hosts`. If the test server listens only on `0.0.0.0` it might not
	answer requests to `::1` and that causes the test startup to hang.
	Solution: comment the `::1` line in `/etc/hosts` and see if that
	improves matters.

	1. Run `t/TEST -clean` every time you change something in your Apache
	configuration. The test suite picks up certain things from your installed
	`httpd.conf` (such as `LoadModule` statements) and will not see your
	changes unless you clean it.

	2. Failures in `proxy.t` may originate from the fact that the test script
	cannot open the specified port. This happens on some machines if you abort
	a test run and the socket is not properly shut down. Check if the error
	goes away after a reboot. (`proxy.t` tests are slow, so chances you
	interrupt tests at that point are good.)

	3. Failures in `access.t` may result from reverse lookups not working or
	giving other answers than expected. In the case 0 above, if the test
	client connects via `127.0.0.1`, a "Grant for localhost" might resolve to
	`::1` and therefore will not match the access rules of the tests.
	Solution: check that your servername is `localhost` (which is the
	default) and that it always resolves to `127.0.0.1`.

	4. If some ssl test cases fail, especially when `t/ssl/proxy.t` fails, the
	reason can be mismatches between your installed SSL library and the one
	httpd uses. The `openssl` binary found in your `$PATH` is used to create
	binary setup files by `t/TEST`. If another version of openssl then tries
	to read these from your Apache server process, it might fail. Try:
	```
	t/TEST -clean
	PATH=<bin dir of correct openssl>:$PATH t/TEST
	```
	If a lot of ssl tests fail, check in the error log for the presence of a
	certificate validation error. If you find it, check the expiration date of
	the TLS/SSL certificates used by the tests; they might be expired. Running
	`t/TEST -clean` should delete the old ssl certificates, so they'll be
	regenerated during the next run.

	5. If you see failures in the `modules/h2.t` test cases, please notify the
	dev mailing list with a description of your setup. These tests are quite
	young, currently only valid in 2.4.x and later and interact with quite
	some other modules as well as OpenSSL versions installed. Some tests
	require `mod_ssl` so make sure to load it in the httpd conf.

	6. Segmentation faults and core dumps occurring while executing the test suite
	might indicate a real problem but always run again the tests after a clean
	`make install` to avoid inconsistencies from old objects.

	7. If you see error messages like "Parse errors: Bad plan. You planned X tests
	but ran Y." it usually means that you are missing a perl module or the
	tested httpd module depends on another one not loaded in the httpd config.

	8. If you see SSL certificate errors, remove `t/conf/ssl/ca` prior to
	`t/TEST -clean`.

	9. perl 5.28 in MacOS homebrew seems to hang the test suite. Invoking
	`/usr/bin/perl Makefile.PL -apxs ...` will cause an older perl to be used.

	## Smoking Tests

	Sometimes it's possible that the test is passing properly for the first time,
	but when you run it again, the test might fail. It's important to run
	repetition smoke testing. For example to repeat the tests 5 times:

	```
	t/SMOKE -times=5
	```

	It's also possible that a test will pass when it's run after a particular
	test, but if moved to run after a different state it may fail. For this reason
	by default the tests run in random order.

	Since it's important to be able to reproduce the problem with random testing,
	whenever `-order=random` is used, the used seed is printed to STDERR. It can
	then be fed into future tests via the `APACHE_TEST_SEED` environment variable.

	By adding the option `-order=repeat`, the tests will be run in alphabetical
	order.

	Combining these two important smoke testing techniques, one can run tests with:

	```
	t/SMOKE -times=N -order=(repeat\|random)
	```

	For example, to run the mod_rewrite tests 5 times:

	```
	t/SMOKE -times=5 -verbose t/modules/rewrite.t
	```

	The tests can be repeated N times, and run in the following modes:

	- randomize all tests
	- repeat the whole test suite N times

	For configuration options and default settings run:

	```
	t/SMOKE -help
	```

	For more information refer to the `Apache::TestSmoke` manpage.

	## Test Environment Configuration

	The test server is configured with conf files like any normal Apache server.
	The tricky part is those conf files are generated by the harness just prior
	to starting the server. `t/conf/httpd.conf` is generated by
	`t/conf/httpd.conf.in`. If that does not exist, the harness will generate a
	working configuration and will include `LoadModule` (and `AddModule` for
	Apache 1.3) directives from the `httpd.conf` associated with the httpd binary
	you are using for testing.

	If `t/conf/extra.conf.in` exists, `t/conf/extra.conf` will be generated from
	that, and an `Include` directive for that file will be put in the generated
	`t/conf/httpd.conf`.

	`t/conf/apache_test_config.pm` is generated from the test configuration. It
	contains all the information about the configuration of your test server. You
	can access this information in test scripts by:

	```perl
	my $env = Apache::TestConfig->thaw;
	```

	`Apache::TestConfig` accesses `apache_test_config.pm` and returns a hash
	reference with all the information. Look through `apache_test_config.pm`;
	it's a lot of stuff. Once these conf files are generated, you have a working
	test environment, and they must be 'cleaned' if you wish to make changes to
	them:

	```
	t/TEST -clean
	```

	(Now you will have to specify your httpd binary when starting back up again.)

	## More Information

	For more information on using the test harness and writing tests, see the
	README in `Apache-Test` and the examples in `Apache-Test/t`.

	The test harness was originally written by Doug MacEachern and is discussed
	on the [httpd dev mailing list](mailto:dev@httpd.apache.org).

	It is also included in modperl-2.0 source along with tests for modperl-2.0.