blob: a1dfd9b10b2d2a09908dd8e95949bd60e684c112 [file] [log] [blame]
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
Rev. Date Changes
2 2015/09/07 Addition of tu_restart(); clarification of exported
preprocessor symbols.
1 2015/08/28
Automated testing can be done in either of two ways:
1. Testing a set of packages on embedded hardware or a simulated
2. Testing an individual package in a simulated environment.
Automated testing of package sets consists of two components:
1. "PC software" which administers tests and manages results.
2. Test images which run on the units under test (UUTs).
The PC software will need to perform the following tasks:
1. Connect to UUT (e.g., via JTAG and OpenOCD) (if embedded).
2. Upload test image (if embedded).
3. Run test image.
4. Read test results from UUT.
5. Generate test report.
If any of these tasks fail, a fatal error would need to be logged to the test
The details of the PC software are TBD, and are not discussed further here.
Test images are build using the stack tool. The user will need to create a
separate stack tool target for each platform under test. In addition, there
will be a single stack tool project which contains code to call the appropriate
test code. Each test image typically consists of the following:
1. The test cases from all relevant packages.
2. Some "glue code" from a special test project.
A package may optionally contain a set of test cases. Test cases are not
normally compiled and linked when a package is built; they are only included
when the "test" identity is specified. All of a
package's test code goes in its 'src/test' directory. For example, the ffs
package's test code is located in the following directory:
This directory contains the source and header files that implement the ffs test
The test code has access to all the header files in the following directories:
* src
* src/arch/<target-arch>
* include
* src/test
* src/test/arch/<target-arch>
* include directories of all package dependencies
Package test code typically depends on the testutil package, described later in
this document. If a package's test code uses testutil, then the package itself
needs to have testutil in its dependency list.
Some test cases or test initialization code may be platform-specific. In such
cases, the platform-specific function definitions are placed in arch
subdirectories within the package test directory.
When test code is built (i.e., when the "test" identity is specified), the
stack tool defines the "TEST" macro. This macro is defined during complilation
of all C source files in all projects and packages..
The testutil package is a test framework that provides facilities for
specifying test cases and recording test results.
Tests are structures according to the following hierarchy:
/ \
[suite] [suite]
/ \ / \
[case] [case] [case] [case]
I.e., a test consists of test suites, and a test suite consits of test cases.
The test code uses testutil to define test suites and test cases.
The following example demonstrates how to create a simple test suite.
int sum;
sum = 5 + 10;
TEST_ASSERT(sum == 15, "actual value: %d", sum);
TEST_ASSERT(3 * 0 == 0);
TEST_ASSERT(4 * 0 == 0);
TEST_ASSERT(712 * 0 == 0);
The test suite would then be executed via a call to test_suite_arithmetic().
The testutil package optionally writes test results to a file system residing
on the UUT. The contents of the file system can then be retrieved from the UUT
by the PC software. When run in an embedded environment, the results are
written to an ffs file system. When run in a simulated environment, the
results are written to the native file system.
Results directory structure:
└── test-results
├── <test-suite-1>
| ├── <test-case-1>
| │   ├── <result-1>.txt
| │   ├── [<result-2>.txt]
| │   ├── [...]
| │   └── [other-data-files]
| ├── <test-case-2>
| │   ├── <result-1>.txt
| │   ├── [<result-2>.txt]
| │   ├── [...]
| │   └── [other-data-files]
| └── [...]
├── <test-suite-2>
| └── [...]
└── [...]
The name of the top-level directory is configurable. This document uses
'test-results' in its examples.
Each test case directory contains a set of result files, and optionally
contains data files containing information collected during the test. Result
files are named according to one of the following templates:
* pass.txt
* fail-xxxx.txt
Where "xxxx" is a numeric index automatically generated by testutil. If a test
passes, its results directory is populated with a single 'pass' file and no
'fail' files. If a test fails, its results directory is populated with one or
more 'fail' files and no 'pass' files. A result file (pass or fail) optionally
contains a text string.
In addition, a results directory can contains data files containing information
collected during the test. For example, a test case may record timing
information or statistics. Test case code specifies the name and contents of a
data file when it writes one. Data files can be written during a test case
regardless of the test outcome. There are no restrictions on the size or
contents of data files, but their filenames should be distinct from the test
result naming scheme described above. There is no limit to the number of data
files that a particular test case can write.
Returning to the earlier arithmetic test suite example, suppose the test suite
completed with the following results:
* test_addition: 1 assertion success, 0 assertion failures
* test_times_0: 1 assertion success, 2 assertion failures
Such a test run would produce the following directory tree:
└── test-results
└── test_suite_arithmetic
├── test_addition
│   └── pass.txt
└── test_times_0
├── fail-0000.txt
└── fail-0001.txt
Each 'fail' file would contain a string describing the failure.
struct tu_config {
/** If nonzero, test results are printed to stdout. */
int tc_print_results;
* Name of the base results directory (e.g., "test-results"). If null,
* results are not written to disk.
const char *tc_results_path;
* Array of flash areas where an ffs file system should be located. Unused
* if run in a simulated environment. This array must be terminated with a
* zero-length area.
const struct ffs_area_desc *tc_area_descs;
extern struct tu_config tu_config;
The global tu_config struct contains all the testutil package's settings.
The struct must be populated before tu_init() is called.
int tu_init(void)
Initializes the test framework according to the contents of the tu_config
struct. If run in an embedded environment and the
tu_config.tc_results_path and tu_config.tc_area_descs fields are non-null,
this function ensures an ffs file system is present within the specified
flash areas. If testutil is using an ffs file system, and no valid ffs
file system is detected in the specified region of flash, the region is
formatted with a new file system. This function must be called before any
tests are run.
If, immediately prior to a call to tu_init(), the UUT was restarted via a
call to tu_restart(), this function skips all the test cases that have
already been executed. See tu_restart() for details.
Returns 0 on success; nonzero on failure.
TEST_ASSERT(expression, fail_msg, ...)
Asserts that the specified condition is true. If the expression is true,
nothing gets reported. The expression argument is mandatory; the rest are
optional. The fail_msg argument is a printf format string which specifies
how the remaining arguments are parsed.
If the expression is false, testutil reports the following message:
|<file>:<line-number>| failed assertion: <expression>
If no fail_msg is specified, only the first line gets reported.
The test case proceeds as normal after this macro is invoked.
TEST_ASSERT(num_blocks == 1024,
"expected: 1024 blocks; got: %d", num_blocks);
On failure, the above assertion would result in a report like the
|ffs_test.c:426| failed assertion: num_blocks == 1024
expected: 1024 blocks; got 12
TEST_ASSERT_FATAL(expression, fail_msg, ...)
These are identical to their TEST_ASSERT counterparts, except that a
failure causes the current test case to be aborted.
TEST_PASS(msg, ...)
Reports a success result for the current test. This function is not
normally needed, as all successful tests automatically write an empty pass
result at completion. This function is only needed when the success result
report should contain text. The msg argument is a printf format string
which specifies how the remaining arguments are parsed. The result file
produced by this function contains the following text:
|<file>:<line-number>| manual pass
After this function is called, the remainder of the test case is not
executed. If the current test has already failed when this macro is
invoked, the macro has no effect.
testutil_write_data(const char *filename, const uint8_t *data, size_t data_len)
Writes a data file to the current test results directory. The file's
contents are fully specified by the data and data_len arguments. If a file
with the specified name already exists, it is overwritten. After this
function is called, the test case proceeds as normal.
Data does not get written to stdout; it only gets written to disk. This
function has no effect if testutil is configured to not write results to
TEST_CASE(test_case_name) { /* test case code */ }
Defines a test case function with the following type:
int test_case_name(void)
The return value is 0 if the test case passed; nonzero if it failed.
Generally, the return code is not used.
TEST_SUITE(test_suite_name) { /* test suite code */ }
Defines a test suite function with the following type:
int test_suite_name(void)
The return value is 0 if the test suite passed; nonzero if it failed.
Generally, the return code is not used.
void tu_restart(void)
This function is used when a system reset is necessary to proceed with
testing. For example, the OS is designed to run forever once started, so a
test which creates several OS tasks and then starts the OS has no means of
completing. This function, when called from such a test, gracefully ends
the current test case and proceeds to the next test case.
The particulars of this function depend on whether it is called from a
simulated environment. In a simulated environment, this function uses a
longjmp() call to break out of the current test case.
In a non-simulated environment, this function performs the following
1. Record the current test status to the file system.
2. Perform a software reset of the UUT.
3. Read test status and delete it from the file system.
4. Skip all previously completed test cases.
If this function is called before any results have been reported for the
current test case, a "pass" result gets reported.
Note: after this function performs a software reset, the UUT may not be
fully initialized, depending on the test environment. For example, if the
UUT has been set up to run the test image from RAM via JTAG, variables in
the .data section will not get properly initialized by the software reset.
Proper initialization in this case would require that data be re-read from
the image file via an additional run of the JTAG script.
The effect of this macro depends on whether the TEST macro is defined:
If defined: expands to assert().
If not defined: expands to nothing.
This macro is not intended for use within test cases. Instead, it should
be used in normal non-test code to perform computationally-expensive
checks. In non-test builds, the checks do not get performed.
Testing individual packages is performed via the "target test" stack tool
command. When testing is done in this manner, the stack tool automatically
runs the test code on the host machine. There is no need for an extra stack
project for producing a test image.
Testing a package in this manner requires a special target that is specific to
the package. The target specifies the package to test via the "pkg"
configuration variable. The target must not specify a project, or the test
command will fail.
Below is an example target called testffs that tests the ffs package:
compiler: sim
compiler_def: debug
pkg: libs/ffs
bsp: hw/bsp/native
arch: sim
name: testffs
The target specifies the "native" bsp. This is necessary so that the ffs test
code can link with the appropriate implementations of the hal functions that it
depends on.
The test is run by specifying the following stack arguments:
target test testffs
When the "target test" command is used, the PKG_TEST macro is defined during
compilation of the package under test. This is in addition to the TEST macro,
which is defined during compilation of all packages.
* For individual package tests, the requirement for a separate target for each
package may become unwieldy. Perhaps the stack tool should be modified to
allow arbitrary key-value pairs to be specified on the command line. That
way, the target's "pkg" variable could be unset, and the user would specify
the package to test when he executes the test command. This would allow the
same target to be used for all packages, e.g.,
target test testtarget pkg=libs/ffs
target test testtarget pkg=libs/os
* Most test frameworks define a lot of extra assertions (e.g., assert_equal(),
assert_not_null(), assert_false(), etc). These don't seem particularly
useful to me, so I did not add them. Typing TEST_ASSERT(x == 5) is no more
difficult than TEST_ASSERT_EQUAL(x, 5). The benefit of the more specific
assertions is that they can automatically produce more helpful failure
messages. For example, if TEST_ASSERT_EQUAL(x, 5) fails, the failure message
can contain the actual value of x in most cases, whereas a plain
TEST_ASSERT() won't automatically log that information. This didn't seem
useful enough to me to justify growing the interface, but I am interested in
hearing what others think.
* During testing on embedded devices, the PC software will need some
notification when testing is complete, so that it knows when to retrieve the
test results. The testutil package should expose some state that the PC
software can poll for this purpose.