test/README.ci

Variables
---------

The CI scripts use the following environment variables:

* APR_VERSION - if set, APR of this version is built and installed in
  $HOME/root/apr-$APR_VERSION - a value of "trunk" means trunk is
  used, "*.x" means a branch, otherwise a tagged version is implied.

* APR_CONFIG - arguments to pass when running APR's configure script
  if APR_VERSION is set

* APU_VERSION - if set, APR-util of this version is built and
  installed in $HOME/root/apr-util-$APU_VERSION - a value of "*.x"
  means a branch, otherwise a tagged version is implied.  (Since there
  is no "trunk" for apr-util, that value cannot be used here.)

* APU_CONFIG - arguments to pass when running APR-util's configure
  script if APU_VERSION is set

* CONFIG - arguments to pass to httpd's configure script.

* BUILDCONFIG - arguments to pass when running httpd's ./buildconf script

* MFLAGS - arguments to pass when running "make", both for httpd and
  for any dependency built from source.  The Linux workflow defaults
  this to "-j<number of CPUs>"; a matrix entry can override it, e.g.
  setting it empty to force a serial build.

* SKIP_TESTING - if set, no testing is done at all

* NO_TEST_FRAMEWORK - if set, the Perl-based test framework and
                      libcheck-based unit tests are skipped

* TEST_LOGNO - if set, checks for APLOGNO() macro errors (duplicates
               or empty arguments) and exits; no build or testing is done.

* TEST_UBSAN - set for job using UBSan ("Undefined Behaviour Sanitizer")

* TEST_ASAN - set for job using ASan ("Address Sanitizer")

* TEST_MALLOC - set for job using enhanced malloc debugging
  (sets MALLOC_PERTURB_, MALLOC_CHECK_ and LIBC_FATAL_STDERR_).

* TEST_INSTALL - set for job testing "make install"

* TEST_VPATH - set for job testing srcdir!=builddir (VPATH build)

* TEST_LDAP - set for job with slapd, running LDAP tests

* TEST_SSL - set for job with SSL/TLS testing variants (encrypted
  private keys, session cache backends via redis/memcache/shmcb, etc.)

* TEST_OPENSSL3 - if set to a version string (e.g. "3.4.7"), that
  version of OpenSSL is built from source and used for mod_ssl.
  Requires non-system APR/APR-util (APR_VERSION must also be set).

* TEST_OPENSSL3_BRANCH - if set together with TEST_OPENSSL3, the
  OpenSSL source is cloned from the named git branch instead of
  downloading a release tarball.

* OPENSSL_CONFIG - extra arguments passed to OpenSSL's ./Configure
  when building from source (e.g. "no-engine").

* TEST_PYTEST - if set, runs all available pytest-based test suites
  via "make check-all-pytest" (pytest_suite/ and test/modules/*/
  suites, in one pytest session).  Replaces the old per-suite TEST_CORE
  / TEST_PROXY / TEST_H2 / TEST_MD flags.  A suite whose dependencies
  are missing skips itself: modules/md without `pebble` on PATH,
  modules/http2 when mod_http2 was not built.

* PYTEST_ARGS - with TEST_PYTEST, test paths (relative to test/) and
  options passed through to pytest, to narrow the run: "modules/http2",
  "modules" (skip pytest_suite/), "pytest_suite/tests" (skip modules/),
  "-k expr", "-m expr".

* TEST_NGHTTP2 - if set to a version string (e.g. "1.68.0"), that
  release of nghttp2 is built from its GitHub tag (library only) and
  used for mod_http2/mod_proxy_http2 via --with-nghttp2.  Used by the
  HTTP/2 workflow to test against nghttp2 versions other than the one
  packaged for the runner image.

* TEST_MD - set together with TEST_PYTEST; installs pebble first so
  that modules/md runs instead of skipping.  See the
  "Python pytest test suites" job in .github/workflows/linux.yml.

* TEST_APR_TARBALL - if set, APR/APR-util source is fetched as a
  release tarball from archive.apache.org rather than cloned from git.
  (This is necessary for old releases where running ./buildconf from
  a tag may now be difficult.)

* TESTS - a list of Perl framework tests to run

* TEST_ARGS - arguments to pass to ./t/TEST in the Perl test framework

* LITMUS - if set, runs the litmus WebDAV test suite against mod_dav.

* PHP_FPM - path to the php-fpm binary.  If set, PHP tests are
  included in the pytest test suites.  If not set by the matrix
  entry, the workflow sets it to the newest /usr/sbin/php-fpm* found
  in the runner image (e.g. "/usr/sbin/php-fpm8.3"), failing the job
  if there is none.

Caching
-------

Two directories are cached.

~/root (key "cache-libs-<serial>-<image>-<arch>-$ROOTID") holds the
APR, APR-util and OpenSSL installs.  gha-resolve-deps.sh computes
$ROOTID before the cache is restored, hashing only what determines
those installs: the versions, their configure arguments and $CC.  The
job's own configuration is deliberately not included, so jobs building
the same dependencies share one entry however differently they
configure or test httpd - the five pool-debug jobs, for instance.  A job
which builds no dependencies gets no $ROOTID and does not use the cache
at all.

For anything built from a branch rather than a tag - APR_VERSION=trunk
or 1.7.x, and TEST_OPENSSL3_BRANCH - the version alone does not identify
what gets built, so the branch is resolved to a commit and that goes
into $ROOTID too.  The commit is exported as
$APR_COMMIT/$APU_COMMIT/$OPENSSL_COMMIT so that the build uses exactly
what the key names rather than resolving the branch a second time,
minutes later, by which point it may have moved.

It is restored before the dependencies are built and saved afterwards,
only if they built successfully - a failed job would otherwise write a
half-built tree to a key which can never be updated.

~/perl5 (key "cache-cpan-<serial>-$PERLID-<arch>-<hash>") holds the
CPAN modules.  Nothing in it varies per job: the module list is fixed
and the modules are always built with CC=gcc, so the content depends
only on the perl ABI and on the module list itself, hashed from
travis_before_linux.sh.  One cache is therefore shared by every job.

$PERLID is a hash of the "perl -V" output, computed by
gha-resolve-deps.sh, which also logs that output so a change to it can
be seen.  The image name is deliberately not used: the runner image is
rebuilt regularly and its name does not change when perl does, so the
key outlived a perl upgrade which the modules did not.  The key has no
restore-keys fallback, so the modules restored always match the perl
running the job.

The cached CPAN modules still become gradually stale as versions in
CPAN move on from whatever was installed for the first build.  Bump
the serial in the key ("cache-cpan-1-" -> "cache-cpan-2-") to discard
them and rebuild against current CPAN.

APR and APR-util must stay in a single cache entry.  APR-util's
buildconf needs APR's *source* tree, which lives in $HOME/build and is
not cached, so APR-util can only be built in a job which also builds
APR.  $ROOTID covers both, and the entry is saved only once both have
built, so a restored entry never holds one without the other.

Artifacts
---------

A job which fails uploads its config.log, and the error_log from any
test run, as artifacts named "config.log-$JOBID" and "error_log-$JOBID".

$JOBID is the matrix index of the job followed by its name, with every
character upload-artifact rejects - "/" among them - replaced, so job
zero uploads "error_log-0-Shared-MPMs-event-pool-debug-SSL-TLS-variants".

The index is what makes the name unique.  It cannot collide however the
matrix changes, including for an entry which sets no name at all,
whereas hashing a hand-picked subset of the job configuration - as this
used to do - collides silently as soon as two entries differ only in a
field which the hash omits.  Two entries already differ only in "os",
which was never hashed: they stayed apart only because $ImageOS, which
was, happens to differ on an ARM runner.  The job name is appended
purely so that an artifact can be matched to the job it came from
without counting matrix entries; it is not needed for uniqueness, and
an entry which sets none is still named unambiguously.

The name reaches the shell as $MATRIX_NAME through the step's env
rather than being interpolated into the script, so that no job name can
be written which breaks the shell.

The value computed for a job can be read from its log without waiting
for a failure: it is listed in the environment which the runner prints
under the "Run" group of every step after the one which sets it.

Travis to Github Actions Migration TODO
---------------------------------------

* test across different Ubuntu versions again
 - and test against OpenSSL 1.x since we're now ONLY building against 3.x
* container testing for non-Ubuntu hosts
* introduce some job ordering rather than having a flat/concurrent
  set, if the default "./configure && make && test" works *then* start
  jobs doing 200 different variations on ./configure --enable-XXX 
  i.e. stop burning CPU time for a typo which breaks every job

TODO list
---------

* MacOS build
* clang-on-Linux build
* Use containers for non-Ubuntu-based Linux testing

Known Failures
--------------

- A few mod_http2 Python tests sometimes fail spuriously, e.g.
  FAILED modules/http2/test_106_shutdown.py::TestShutdown::test_h2_106_02 - AssertionError: failed on 2. request:  * Added test1.tests.httpd.apache.org:5001:127.0.0.1 to DNS cache

- mod_md tests can deadlock pebble v2.10.1 which hangs the test suite,
  hopefully fixed for a future release. See e.g.
  https://github.com/letsencrypt/pebble/pull/555

- ERROR at teardown of TestConf.test_md_300_022 - some race condition?

Testing from a Feature Branch
-----------------------------

An SVN branch off trunk should be mirrored to github, and will be
tested in the same way that trunk is in CI, so this workflow is
available for those familiar with using Subversion and the standard
ASF/httpd repository layout.

Test runs are listed at: https://github.com/apache/httpd/actions

A workflow to enable testing would be as follows, substituting
$USERNAME for your github username:

  $ git clone https://github.com/apache/httpd
  $ cd httpd
  $ git remote add $USERNAME git@github.com:$USERNAME/httpd.git
  $ git checkout -b my-feature origin/trunk
  ... do some work ...
  $ git commit ...
  $ git push -u $USERNAME my-feature:my-feature

To create a Pull Request, go to the URL produced in the "git push"
command output when pushing to your fork, which is something like:
https://github.com/apache/httpd/compare/trunk...$USERNAME:trunk

Once a PR has been created, the tests will run and test results
displayed directly in the PR.

To merge from github back to SVN trunk, create a patch from e.g.:

  $ git diff origin/trunk..my-feature

and then apply it in SVN.  To rebase a feature once trunk has
diverged, from a feature branch run:

  $ git pull
  $ git rebase -i origin/trunk

and follow the standard rebase steps.

An alternative way to apply a PR into an SVN checkout is to use the
apply_trunk_pr.sh script:
https://svn.apache.org/repos/asf/httpd/dev-tools/github/apply_trunk_pr.sh

If you use keywords in the SVN commit message, the PR will be closed
when the commit is mirrored to Github, e.g. a commit message like:

   mod_foo: Fix the baz feature.

   PR: 12345
   Submitted by: Jane Doe <jane example.com>
   GitHub: closes #1

will close PR #1 when it's merged.

2.4.x
-----

The 2.4.x workflow runs the pytest jobs of trunk's, and TEST_PYTEST and
"make check-all-pytest" work the same on both branches, so a test
backported from trunk needs no change in how it is run.  A job that
configures an MPM which the branch does not have yet (MotorZ, for now)
succeeds without doing anything.  travis_before_linux.sh and
travis_run_linux.sh are the same on both branches and do what the
workflow leaves to them:

* CLEAR_CACHE - if set, the cached $HOME/root and $HOME/perl5 are
  removed before the build.

* MFLAGS - the 2.4.x workflow does not set it; the build script then
  uses "-j2".

* TEST_MOD_TLS - with RUSTLS_VERSION, builds that tag of rustls-ffi
  into $HOME/root/rustls and configures httpd with --with-tls and
  --with-rustls.

When the workflow does not set $ROOTID, its cache key does not say what
was built, so the scripts check for themselves.  APR and APR-util are
rebuilt unless $HOME/root/.key-apr (.key-apr-util) names the same
version, commit, configuration and $CC; OpenSSL is rebuilt unless
$HOME/root/openssl-is-$TEST_OPENSSL3 exists and, for a branch, names its
current commit; the CPAN modules are rebuilt when the output of
"perl -V" differs from the one saved in $HOME/perl5/.perlver.