docs/how_to/dev/pytest_target_parametrization.rst

..  Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

..    http://www.apache.org/licenses/LICENSE-2.0

..  Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

Python Target Parametrization
=============================

Summary
-------

For any supported runtime, TVM should produce numerically
correct results.  Therefore, when writing unit tests that validate
the numeric output, these unit tests should be run on all supported
runtimes.  Since this is a very common use case, TVM has helper
functions to parametrize unit tests such that they will run on all
targets that are enabled and have a compatible device.

A single python function in the test suite can expand to several
parameterized unit tests, each of which tests a single target device.
In order for a test to be run, all of the following must be true.

- The test exists in a file or directory that has been passed to
  `pytest`.

- The pytest marks applied to the function, either explicitly or
  through target parametrization, must be compatible with the
  expression passed to pytest's `-m` argument.

- For parametrized tests using the `target` fixture, the target must
  appear in the environment variable `TVM_TEST_TARGETS`.

- For parametrized tests using the `target` fixture, the build
  configuration in `config.cmake` must enable the corresponding
  runtime.

Unit-Test File Contents
-----------------------

.. _pytest-marks: https://docs.pytest.org/en/stable/how-to/mark.html

The recommended method to run a test on multiple targets is by
parametrizing the test.  This can be done explicitly for a fixed list
of targets by decorating with
``@tvm.testing.parametrize_targets('target_1', 'target_2', ...)``, and
accepting ``target`` or ``dev`` as function arguments.  The function
will be run once for each target listed, and the success/failure of
each target is reported separately.  If a target cannot be run because
it is disabled in the `config.cmake`, or because no appropriate
hardware is present, then that target will be reported as skipped.

.. code-block:: python

    # Explicit listing of targets to use.
    @tvm.testing.parametrize_targets('llvm', 'cuda')
    def test_function(target, dev):
        # Test code goes here

For tests that should run correctly on all targets, the decorator can
be omitted.  Any test that accepts a ``target`` or ``dev`` argument
will automatically be parametrized over all targets specified in
``TVM_TEST_TARGETS``.  The parametrization provides the same
pass/fail/skipped report for each target, while allowing the test
suite to be easily extended to cover additional targets.

.. code-block:: python

    # Implicitly parametrized to run on all targets
    # in environment variable TVM_TEST_TARGETS
    def test_function(target, dev):
        # Test code goes here

The ``@tvm.testing.parametrize_targets`` can also be used as a bare
decorator to explicitly draw attention to the parametrization, but has
no additional effect.

.. code-block:: python

    # Explicitly parametrized to run on all targets
    # in environment variable TVM_TEST_TARGETS
    @tvm.testing.parametrize_targets
    def test_function(target, dev):
        # Test code goes here


Specific targets can be excluded or marked as expected to fail using
the ``@tvm.testing.exclude_targets`` or
``@tvm.testing.known_failing_targets`` decorators.  For more
information on their intended use cases, please see their docstrings.

In some cases it may be necessary to parametrize across multiple
parameters.  For instance, there may be target-specific
implementations that should be tested, where some targets have more
than one implementation.  These can be done by explicitly
parametrizing over tuples of arguments, such as shown below.  In these
cases, only the explicitly listed targets will run, but they will
still have the appropriate ``@tvm.testing.requires_RUNTIME`` mark
applied to them.

.. code-block:: python

   @pytest.mark.parametrize('target,impl', [
        ('llvm', cpu_implementation),
        ('cuda', gpu_implementation_small_batch),
        ('cuda', gpu_implementation_large_batch),
    ])
    def test_function(target, dev, impl):
        # Test code goes here


The parametrization functionality is implemented
on top of pytest marks.  Each test function can
be decorated with `pytest marks <pytest-marks>`_
to include metadata.  The most frequently applied
marks are as follows.

- ``@pytest.mark.gpu`` - Tags a function as using GPU
  capabilities. This has no effect on its own, but can be paired with
  command-line arguments ``-m gpu`` or ``-m 'not gpu'`` to restrict
  which tests pytest will execute.  This should not be called on its
  own, but is part of other marks used in unit-tests.

- ``@tvm.testing.uses_gpu`` - Applies ``@pytest.mark.gpu``.  This
  should be used to mark unit tests that may use the GPU, if one is
  present.  This decorator is only needed for tests that explicitly
  loop over ``tvm.testing.enabled_targets()``, but that is no longer
  the preferred style of writing unit tests (see below).  When using
  ``tvm.testing.parametrize_targets()``, this decorator is implicit
  for GPU targets, and does not need to be explicitly applied.

- ``@tvm.testing.requires_gpu`` - Applies ``@tvm.testing.uses_gpu``,
  and additionally marks that the test should be skipped
  (``@pytest.mark.skipif``) entirely if no GPU is present.

- ``@tvm.testing.requires_RUNTIME`` - Several decorators
  (e.g. ``@tvm.testing.requires_cuda``), each of which skips a test if
  the specified runtime cannot be used. A runtime cannot be used if it
  is disabled in the ``config.cmake``, or if a compatible device is
  not present. For runtimes that use the GPU, this includes
  ``@tvm.testing.requires_gpu``.

When using parametrized targets, each test run is decorated with the
``@tvm.testing.requires_RUNTIME`` that corresponds to the target
being used.  As a result, if a target is disabled in ``config.cmake``
or does not have appropriate hardware to run, it will be explicitly
listed as skipped.

There also exists a ``tvm.testing.enabled_targets()`` that returns
all targets that are enabled and runnable on the current machine,
based on the environment variable ``TVM_TEST_TARGETS``, the build
configuration, and the physical hardware present.  Most current tests
explicitly loop over the targets returned from ``enabled_targets()``,
but it should not be used for new tests.  The pytest output for this
style silently skips runtimes that are disabled in ``config.cmake``,
or do not have a device on which they can run.  In addition, the test
halts on the first target to fail, which is ambiguous as to whether
the error occurs on a particular target, or on every target.

.. code-block:: python

    # Old style, do not use.
    def test_function():
        for target,dev in tvm.testing.enabled_targets():
            # Test code goes here



Running locally
---------------

To run the python unit-tests locally, use the command ``pytest`` in
the ``${TVM_HOME}`` directory.

- Environment variables
    - ``TVM_TEST_TARGETS`` should be a semicolon-separated list of
      targets to run. If unset, will default to the targets defined in
      ``tvm.testing.DEFAULT_TEST_TARGETS``.

      Note: If ``TVM_TEST_TARGETS`` does not contain any targets that
      are both enabled, and have an accessible device of that type,
      then the tests will fall back to running on the ``llvm`` target
      only.

    - ``TVM_LIBRARY_PATH`` should be a path to the ``libtvm.so``
      library. This can be used, for example, to run tests using a
      debug build. If unset, will search for ``libtvm.so`` relative to
      the TVM source directory.

- Command-line arguments

    - Passing a path to a folder or file will run only the unit tests
      in that folder or file. This can be useful, for example, to
      avoid running tests located in ``tests/python/contrib`` on a
      system without a specific backend installed.

    - The ``-m`` argument only runs unit tests that are tagged with a
      specific pytest marker. The most frequent usage is to use ``m
      gpu`` to run only tests that are marked with
      ``@pytest.mark.gpu`` and use a GPU to run. It can also be used
      to run only tests that do not use a GPU, by passing ``m 'not
      gpu'``.

      Note: This filtering takes place after the selection of targets
      based on the ``TVM_TEST_TARGETS`` environment variable.  Even if
      ``-m gpu`` is specified, if ``TVM_TEST_TARGETS`` does not
      contain GPU targets, no GPU tests will be run.

Running in local docker container
---------------------------------

.. _tlcpack: https://hub.docker.com/u/tlcpack

The ``docker/bash.sh`` script can be used to run unit tests inside the
same docker image as is used by the CI.  The first argument should
specify which docker image to run (e.g. ``docker/bash.sh ci_gpu``).
Allowed image names are defined in ``ci/jenkins/data.py`` in the TVM source directory,
and map to images at `tlcpack`_.

If no additional arguments are given, the docker image will be loaded
with an interactive bash session.  If a script is passed as an
optional argument (e.g. ``docker/bash.sh ci_gpu tests/scripts/task_python_unittest.sh``), then that script will be
executed inside the docker image.

Note: The docker images contain all system dependencies, but do not
include the ``build/config.cmake`` configuration file for those
systems.  The TVM source directory is used as the home directory of
the docker image, and so this will default to using the same
config/build directories as the local config.  One solution is to
maintain separate ``build_local`` and ``build_docker`` directories,
and make a symlink from ``build`` to the appropriate folder when
entering/exiting docker.

Running in CI
-------------

Everything in the CI starts from the task definitions present in the
Jenkinsfile.  This includes defining which docker image gets used,
what the compile-time configuration is, and which tests are included
in which stages.

- Docker images

  Each task of the Jenkinsfile (e.g. 'BUILD: CPU') makes calls to
  ``docker/bash.sh``.  The argument following the call to
  docker/bash.sh defines the docker image in CI, just as it does
  locally.

- Compile-time configuration

  The docker image does not have the ``config.cmake`` file built into
  it, so this is the first step in each of the ``BUILD`` tasks.  This
  is done using the ``tests/scripts/task_config_build_*.sh`` scripts.
  Which script is used depends on the build being tested, and is
  specified in the Jenkinsfile.

  Each ``BUILD`` task concludes by packing a library for use in later
  tests.

- Which tests run

  The ``Unit Test`` and ``Integration Test`` stages of the Jenkinsfile
  determine how ``pytest`` is called.  Each task starts by unpacking a
  compiled library that was previous compiled in the ``BUILD`` stage,
  then runs a test script
  (e.g. ``tests/scripts/task_python_unittest.sh``).  These scripts set
  the files/folders and command-line options that are passed to
  ``pytest``.

  Several of these scripts include the ``-m gpu`` option, which
  restricts the tests to only run tests that include the
  ``@pytest.mark.gpu`` mark.