blob: b908eec799a0479cd002a8ef8da3434e2e0efdbb [file] [log] [blame]
Write a Test Suite for a Package
This document guides the reader through creating a test suite for a
Mynewt package.
Writing a test suite involves using the
:doc:`test/testutil <../../os/modules/testutil>` package. The
testutil library provides the functionality needed to define test suites
and test cases.
.. contents::
:depth: 2
Choose Your Package Under Test
Choose the package you want to write a test suite for. In this tutorial,
we will use the ``time/datetime`` in the apache-mynewt-core repo.
Throughout this tutorial, we will be inside the apache-mynewt-core repo
directory, unlike most tutorials which operate from the top-level
project directory.
Create A Test Package
Typically, a library has only one test package. The convention is name
the test package by appending ``/test`` to the host library name. For
example, the test package for ``encoding/json`` is
``encoding/json/test``. The directory structure of the json package is
shown below:
.. code-block:: console
├── include
│   └── json
│   └── json.h
├── pkg.yml
├── src
│   ├── json_decode.c
│   └── json_encode.c
└── test
├── pkg.yml
└── src
├── test_json.c
├── test_json.h
├── test_json_utils.c
└── testcases
├── json_simple_decode.c
└── json_simple_encode.c
The top-level ``test`` directory contains the json test package. To
create a test package for the datetime package, we need to create a
similar package called ``time/datetime/test``.
.. code-block:: console
$ newt pkg new time/datetime/test -t unittest
Download package template for package type pkg.
Package successfuly installed into /home/me/mynewt-core/time/datetime/test.
We now have a test package inside ``time/datetime``:
.. code-block:: console
├── include
│   └── datetime
│   └── datetime.h
├── pkg.yml
├── src
│   └── datetime.c
└── test
├── pkg.yml
├── src
│   └── main.c
└── syscfg.yml
There is one modification we need to make to the new package before we
can start writing unit test code. A test package needs access to the
code it will be testing, so we need to add a dependency on
``@apache-mynewt-core/time/datetime`` to our ``pkg.yml`` file:
.. code-block:: console
:emphasize-lines: 10 "time/datetime/test"
pkg.type: unittest
pkg.description: "Description of your package" "You <>"
pkg.homepage: ""
- '@apache-mynewt-core/test/testutil'
- '@apache-mynewt-core/time/datetime'
- '@apache-mynewt-core/sys/console/stub'
While we have the ``pkg.yml`` file open, let's take a look at what newt
filled in automatically:
- ``pkg.type: unittest`` designates this as a test package. A *test
package* is special in that it can be built and executed using the
``newt test`` command.
- A test package always depends on
``@apache-mynewt-core/test/testutil``. The testutil library provides
the tools necessary for verifying package behavior,
- The ``SELFTEST`` suffix indicates that a setting should only be
applied when the ``newt test`` command is used.
Regarding the conditional dependency on ``sys/console/stub``, the
datetime package requires some form of console to function. In a regular
application, the console dependency would be supplied by a higher order
package. Because ``newt test`` runs the test package without an
application present, the test package needs to supply all unresolved
dependencies itself when run in self-test mode.
Create Your Test Suite Code
We will be adding a *test suite* to the ``main.c`` file. The test suite
will be empty for now. We also need to invoke the test suite from
Our ``main.c`` file now looks like this:
.. code-block:: c
#include "sysinit/sysinit.h"
#include "testutil/testutil.h"
TEST_SUITE(test_datetime_suite) {
/* Empty for now; add test cases later. */
main(int argc, char **argv)
/* Initialize all packages. */
/* Indicate whether all test cases passed. */
return tu_any_failed;
Try It Out
We now have a working test suite with no tests. Let's make sure we get a
passing result when we run ``newt test``:
.. code-block:: c
$ newt test time/datetime
<build output>
Executing test: /home/me/mynewt-core/bin/targets/unittest/time_datetime_test/app/time/datetime/test/time_datetime_test.elf
Passed tests: [time/datetime/test]
All tests passed
Create a Test
To create a test within your test suite, there are two things to do.
1. Implement the test case function using the ``testutil`` macros.
2. Call the test case function from within the test suite.
For this tutorial we will create a test case to verify the
``datetime_parse()`` function. The ``datetime_parse()`` function is
declared as follows:
.. code-block:: c
* Parses an RFC 3339 datetime string. Some examples of valid datetime strings
* are:
* 2016-03-02T22:44:00 UTC time (implicit)
* 2016-03-02T22:44:00Z UTC time (explicit)
* 2016-03-02T22:44:00-08:00 PST timezone
* 2016-03-02T22:44:00.1 fractional seconds
* 2016-03-02T22:44:00.101+05:30 fractional seconds with timezone
* On success, the two output parameters are filled in (tv and tz).
* @return 0 on success;
* nonzero on parse error.
datetime_parse(const char *input, struct os_timeval *tv, struct os_timezone *tz)
Our test case should make sure this function rejects invalid input, and
that it parses valid input correctly. The updated ``main.c`` file looks
like this:
.. code-block:: c
#include "sysinit/sysinit.h"
#include "testutil/testutil.h"
#include "os/os_time.h"
#include "datetime/datetime.h"
struct os_timezone tz;
struct os_timeval tv;
int rc;
/*** Valid input. */
/* No timezone; UTC implied. */
rc = datetime_parse("2017-06-28T22:37:59", &tv, &tz);
TEST_ASSERT(tv.tv_sec == 1498689479);
TEST_ASSERT(tv.tv_usec == 0);
TEST_ASSERT(tz.tz_minuteswest == 0);
TEST_ASSERT(tz.tz_dsttime == 0);
/* PDT timezone. */
rc = datetime_parse("2013-12-05T02:43:07-07:00", &tv, &tz);
TEST_ASSERT(tv.tv_sec == 1386236587);
TEST_ASSERT(tv.tv_usec == 0);
TEST_ASSERT(tz.tz_minuteswest == 420);
TEST_ASSERT(tz.tz_dsttime == 0);
/*** Invalid input. */
/* Nonsense. */
rc = datetime_parse("abc", &tv, &tz);
TEST_ASSERT(rc != 0);
/* Date-only. */
rc = datetime_parse("2017-01-02", &tv, &tz);
TEST_ASSERT(rc != 0);
/* Zero month. */
rc = datetime_parse("2017-00-28T22:37:59", &tv, &tz);
TEST_ASSERT(rc != 0);
/* 13 month. */
rc = datetime_parse("2017-13-28T22:37:59", &tv, &tz);
TEST_ASSERT(rc != 0);
main(int argc, char **argv)
/* Initialize all packages. */
/* Indicate whether all test cases passed. */
return tu_any_failed;
Take a few minutes to review the above code. Then keep reading for some
The ``test/testutil`` package provides two tools for verifying the
correctness of a package:
Both of these macros check if the supplied condition is true. They
differ in how they behave when the condition is not true. On failure,
``TEST_ASSERT`` reports the error and proceeds with the remainder of the
test case. ``TEST_ASSERT_FATAL``, on the other hand, aborts the test
case on failure.
The general rule is to only use ``TEST_ASSERT_FATAL`` when subsequent
assertions depend on the condition being checked. For example, when
``datetime_parse()`` is expected to succeed, the return code is checked
with ``TEST_ASSERT_FATAL``. If ``datetime_parse()`` unexpectedly failed,
the contents of the ``tv`` and ``tz`` objects would be indeterminate, so
it is desirable to abort the test instead of checking them and reporting
spurious failures.
Scaling Up
The above example is small and self contained, so it is reasonable to
put everything in a single C file. A typical package will need a lot
more test code, and it helps to follow some conventions to maintain
organization. Let's take a look at a more realistic example. Here is the
directory structure of the ``fs/nffs/test`` package:
.. code-block:: console
├── pkg.yml
└── src
├── nffs_test.c
├── nffs_test.h
├── nffs_test_debug.c
├── nffs_test_priv.h
├── nffs_test_system_01.c
├── nffs_test_utils.c
├── nffs_test_utils.h
└── testcases
├── append_test.c
├── cache_large_file_test.c
├── corrupt_block_test.c
├── corrupt_scratch_test.c
├── gc_on_oom_test.c
├── gc_test.c
├── incomplete_block_test.c
├── large_system_test.c
├── large_unlink_test.c
├── large_write_test.c
├── long_filename_test.c
├── lost_found_test.c
├── many_children_test.c
├── mkdir_test.c
├── open_test.c
├── overwrite_many_test.c
├── overwrite_one_test.c
├── overwrite_three_test.c
├── overwrite_two_test.c
├── read_test.c
├── readdir_test.c
├── rename_test.c
├── split_file_test.c
├── truncate_test.c
├── unlink_test.c
└── wear_level_test.c
The ``fs/nffs/test`` package follows these conventions:
1. A maximum of one test case per C file.
2. Each test case file goes in the ``testcases`` subdirectory.
3. Test suites and utility functions go directly in the ``src``
Test packages contributed to the Mynewt project should follow these
Now you can begin the work of validating your packages.