| Write a Test Suite for a Package |
| ================================ |
| |
| This document guides the reader through creating a test suite for a |
| Mynewt package. |
| |
| Introduction |
| ------------ |
| |
| Writing a test suite involves using the |
| :doc:`test/testutil <../../os/modules/testutil>` package. The |
| testutil library provides the functionality needed to define test suites |
| and test cases. |
| |
| .. contents:: |
| :local: |
| :depth: 2 |
| |
| Choose Your Package Under Test |
| ------------------------------ |
| |
| Choose the package you want to write a test suite for. In this tutorial, |
| we will use the ``time/datetime`` in the apache-mynewt-core repo. |
| Throughout this tutorial, we will be inside the apache-mynewt-core repo |
| directory, unlike most tutorials which operate from the top-level |
| project directory. |
| |
| Create A Test Package |
| --------------------- |
| |
| Typically, a library has only one test package. The convention is name |
| the test package by appending ``/test`` to the host library name. For |
| example, the test package for ``encoding/json`` is |
| ``encoding/json/test``. The directory structure of the json package is |
| shown below: |
| |
| .. code-block:: console |
| |
| encoding/json |
| ├── include |
| │ └── json |
| │ └── json.h |
| ├── pkg.yml |
| ├── src |
| │ ├── json_decode.c |
| │ └── json_encode.c |
| └── test |
| ├── pkg.yml |
| └── src |
| ├── test_json.c |
| ├── test_json.h |
| ├── test_json_utils.c |
| └── testcases |
| ├── json_simple_decode.c |
| └── json_simple_encode.c |
| |
| The top-level ``test`` directory contains the json test package. To |
| create a test package for the datetime package, we need to create a |
| similar package called ``time/datetime/test``. |
| |
| .. code-block:: console |
| |
| $ newt pkg new time/datetime/test -t unittest |
| Download package template for package type pkg. |
| Package successfuly installed into /home/me/mynewt-core/time/datetime/test. |
| |
| We now have a test package inside ``time/datetime``: |
| |
| .. code-block:: console |
| |
| time/datetime |
| ├── include |
| │ └── datetime |
| │ └── datetime.h |
| ├── pkg.yml |
| ├── src |
| │ └── datetime.c |
| └── test |
| ├── README.md |
| ├── pkg.yml |
| ├── src |
| │ └── main.c |
| └── syscfg.yml |
| |
| There is one modification we need to make to the new package before we |
| can start writing unit test code. A test package needs access to the |
| code it will be testing, so we need to add a dependency on |
| ``@apache-mynewt-core/time/datetime`` to our ``pkg.yml`` file: |
| |
| .. code-block:: console |
| :emphasize-lines: 10 |
| |
| pkg.name: "time/datetime/test" |
| pkg.type: unittest |
| pkg.description: "Description of your package" |
| pkg.author: "You <you@you.org>" |
| pkg.homepage: "http://your-url.org/" |
| pkg.keywords: |
| |
| pkg.deps: |
| - '@apache-mynewt-core/test/testutil' |
| - '@apache-mynewt-core/time/datetime' |
| |
| pkg.deps.SELFTEST: |
| - '@apache-mynewt-core/sys/console/stub' |
| |
| While we have the ``pkg.yml`` file open, let's take a look at what newt |
| filled in automatically: |
| |
| - ``pkg.type: unittest`` designates this as a test package. A *test |
| package* is special in that it can be built and executed using the |
| ``newt test`` command. |
| - A test package always depends on |
| ``@apache-mynewt-core/test/testutil``. The testutil library provides |
| the tools necessary for verifying package behavior, |
| - The ``SELFTEST`` suffix indicates that a setting should only be |
| applied when the ``newt test`` command is used. |
| |
| Regarding the conditional dependency on ``sys/console/stub``, the |
| datetime package requires some form of console to function. In a regular |
| application, the console dependency would be supplied by a higher order |
| package. Because ``newt test`` runs the test package without an |
| application present, the test package needs to supply all unresolved |
| dependencies itself when run in self-test mode. |
| |
| Create Your Test Suite Code |
| --------------------------- |
| |
| We will be adding a *test suite* to the ``main.c`` file. The test suite |
| will be empty for now. We also need to invoke the test suite from |
| ``main()``. |
| |
| Our ``main.c`` file now looks like this: |
| |
| .. code-block:: c |
| |
| #include "sysinit/sysinit.h" |
| #include "testutil/testutil.h" |
| |
| TEST_SUITE(test_datetime_suite) { |
| /* Empty for now; add test cases later. */ |
| } |
| |
| #if MYNEWT_VAL(SELFTEST) |
| int |
| main(int argc, char **argv) |
| { |
| /* Initialize all packages. */ |
| sysinit(); |
| |
| test_datetime_suite(); |
| |
| /* Indicate whether all test cases passed. */ |
| return tu_any_failed; |
| } |
| #endif |
| |
| Try It Out |
| ---------- |
| |
| We now have a working test suite with no tests. Let's make sure we get a |
| passing result when we run ``newt test``: |
| |
| .. code-block:: c |
| |
| $ newt test time/datetime |
| <build output> |
| Executing test: /home/me/mynewt-core/bin/targets/unittest/time_datetime_test/app/time/datetime/test/time_datetime_test.elf |
| Passed tests: [time/datetime/test] |
| All tests passed |
| |
| Create a Test |
| ------------- |
| |
| To create a test within your test suite, there are two things to do. |
| |
| 1. Implement the test case function using the ``testutil`` macros. |
| 2. Call the test case function from within the test suite. |
| |
| For this tutorial we will create a test case to verify the |
| ``datetime_parse()`` function. The ``datetime_parse()`` function is |
| declared as follows: |
| |
| .. code-block:: c |
| |
| /** |
| * Parses an RFC 3339 datetime string. Some examples of valid datetime strings |
| * are: |
| * 2016-03-02T22:44:00 UTC time (implicit) |
| * 2016-03-02T22:44:00Z UTC time (explicit) |
| * 2016-03-02T22:44:00-08:00 PST timezone |
| * 2016-03-02T22:44:00.1 fractional seconds |
| * 2016-03-02T22:44:00.101+05:30 fractional seconds with timezone |
| * |
| * On success, the two output parameters are filled in (tv and tz). |
| * |
| * @return 0 on success; |
| * nonzero on parse error. |
| */ |
| int |
| datetime_parse(const char *input, struct os_timeval *tv, struct os_timezone *tz) |
| |
| Our test case should make sure this function rejects invalid input, and |
| that it parses valid input correctly. The updated ``main.c`` file looks |
| like this: |
| |
| .. code-block:: c |
| |
| #include "sysinit/sysinit.h" |
| #include "testutil/testutil.h" |
| #include "os/os_time.h" |
| #include "datetime/datetime.h" |
| |
| TEST_SUITE(test_datetime_suite) |
| { |
| test_datetime_parse_simple(); |
| } |
| |
| TEST_CASE(test_datetime_parse_simple) |
| { |
| struct os_timezone tz; |
| struct os_timeval tv; |
| int rc; |
| |
| /*** Valid input. */ |
| |
| /* No timezone; UTC implied. */ |
| rc = datetime_parse("2017-06-28T22:37:59", &tv, &tz); |
| TEST_ASSERT_FATAL(rc == 0); |
| TEST_ASSERT(tv.tv_sec == 1498689479); |
| TEST_ASSERT(tv.tv_usec == 0); |
| TEST_ASSERT(tz.tz_minuteswest == 0); |
| TEST_ASSERT(tz.tz_dsttime == 0); |
| |
| /* PDT timezone. */ |
| rc = datetime_parse("2013-12-05T02:43:07-07:00", &tv, &tz); |
| TEST_ASSERT_FATAL(rc == 0); |
| TEST_ASSERT(tv.tv_sec == 1386236587); |
| TEST_ASSERT(tv.tv_usec == 0); |
| TEST_ASSERT(tz.tz_minuteswest == 420); |
| TEST_ASSERT(tz.tz_dsttime == 0); |
| |
| /*** Invalid input. */ |
| |
| /* Nonsense. */ |
| rc = datetime_parse("abc", &tv, &tz); |
| TEST_ASSERT(rc != 0); |
| |
| /* Date-only. */ |
| rc = datetime_parse("2017-01-02", &tv, &tz); |
| TEST_ASSERT(rc != 0); |
| |
| /* Zero month. */ |
| rc = datetime_parse("2017-00-28T22:37:59", &tv, &tz); |
| TEST_ASSERT(rc != 0); |
| |
| /* 13 month. */ |
| rc = datetime_parse("2017-13-28T22:37:59", &tv, &tz); |
| TEST_ASSERT(rc != 0); |
| } |
| |
| #if MYNEWT_VAL(SELFTEST) |
| int |
| main(int argc, char **argv) |
| { |
| /* Initialize all packages. */ |
| sysinit(); |
| |
| test_datetime_suite(); |
| |
| /* Indicate whether all test cases passed. */ |
| return tu_any_failed; |
| } |
| #endif |
| |
| Take a few minutes to review the above code. Then keep reading for some |
| specifics. |
| |
| Asserting |
| ~~~~~~~~~ |
| |
| The ``test/testutil`` package provides two tools for verifying the |
| correctness of a package: |
| |
| - ``TEST_ASSERT`` |
| - ``TEST_ASSERT_FATAL`` |
| |
| Both of these macros check if the supplied condition is true. They |
| differ in how they behave when the condition is not true. On failure, |
| ``TEST_ASSERT`` reports the error and proceeds with the remainder of the |
| test case. ``TEST_ASSERT_FATAL``, on the other hand, aborts the test |
| case on failure. |
| |
| The general rule is to only use ``TEST_ASSERT_FATAL`` when subsequent |
| assertions depend on the condition being checked. For example, when |
| ``datetime_parse()`` is expected to succeed, the return code is checked |
| with ``TEST_ASSERT_FATAL``. If ``datetime_parse()`` unexpectedly failed, |
| the contents of the ``tv`` and ``tz`` objects would be indeterminate, so |
| it is desirable to abort the test instead of checking them and reporting |
| spurious failures. |
| |
| Scaling Up |
| ~~~~~~~~~~ |
| |
| The above example is small and self contained, so it is reasonable to |
| put everything in a single C file. A typical package will need a lot |
| more test code, and it helps to follow some conventions to maintain |
| organization. Let's take a look at a more realistic example. Here is the |
| directory structure of the ``fs/nffs/test`` package: |
| |
| .. code-block:: console |
| |
| fs/nffs/test |
| ├── pkg.yml |
| └── src |
| ├── nffs_test.c |
| ├── nffs_test.h |
| ├── nffs_test_debug.c |
| ├── nffs_test_priv.h |
| ├── nffs_test_system_01.c |
| ├── nffs_test_utils.c |
| ├── nffs_test_utils.h |
| └── testcases |
| ├── append_test.c |
| ├── cache_large_file_test.c |
| ├── corrupt_block_test.c |
| ├── corrupt_scratch_test.c |
| ├── gc_on_oom_test.c |
| ├── gc_test.c |
| ├── incomplete_block_test.c |
| ├── large_system_test.c |
| ├── large_unlink_test.c |
| ├── large_write_test.c |
| ├── long_filename_test.c |
| ├── lost_found_test.c |
| ├── many_children_test.c |
| ├── mkdir_test.c |
| ├── open_test.c |
| ├── overwrite_many_test.c |
| ├── overwrite_one_test.c |
| ├── overwrite_three_test.c |
| ├── overwrite_two_test.c |
| ├── read_test.c |
| ├── readdir_test.c |
| ├── rename_test.c |
| ├── split_file_test.c |
| ├── truncate_test.c |
| ├── unlink_test.c |
| └── wear_level_test.c |
| |
| The ``fs/nffs/test`` package follows these conventions: |
| |
| 1. A maximum of one test case per C file. |
| 2. Each test case file goes in the ``testcases`` subdirectory. |
| 3. Test suites and utility functions go directly in the ``src`` |
| directory. |
| |
| Test packages contributed to the Mynewt project should follow these |
| conventions. |
| |
| Congratulations |
| --------------- |
| |
| Now you can begin the work of validating your packages. |
