| Writing APR tests |
| |
| All APR tests should be executable in 2 ways, as an individual program, or |
| as a part of the full test suite. The full test suite is controlled with |
| the testall program. At the beginning of the testall.c file, there is an |
| array of functions called tests. The testall program loops through this |
| array calling each function. Each function returns a CuSuite variable, which |
| is then added to the SuiteList. Once all Suites have been added, the SuiteList |
| is executed, and the output is printed to the screen. All functions in the |
| array should follow the same basic format: |
| |
| The Full Suite |
| -------------- |
| |
| /* The driver function. This must return a CuSuite variable, which will |
| * then be used to actually run the tests. Essentially, all Suites are a |
| * collection of tests. The driver will take each Suite, and put it in a |
| * SuiteList, which is a collection of Suites. |
| */ |
| CuSuite *testtime(void) |
| { |
| /* The actual suite, this must be created for each test program. Please |
| * give it a useful name, that will inform the user of the feature being |
| * tested. |
| */ |
| CuSuite *suite = CuSuiteNew("Test Time"); |
| |
| /* Each function must be added to the suite. Each function represents |
| * a single test. It is possible to test multiple features in a single |
| * function, although no tests currently do that. |
| */ |
| SUITE_ADD_TEST(suite, test_now); |
| SUITE_ADD_TEST(suite, test_gmtstr); |
| SUITE_ADD_TEST(suite, test_localstr); |
| SUITE_ADD_TEST(suite, test_exp_get_gmt); |
| SUITE_ADD_TEST(suite, test_exp_get_lt); |
| SUITE_ADD_TEST(suite, test_imp_gmt); |
| SUITE_ADD_TEST(suite, test_rfcstr); |
| SUITE_ADD_TEST(suite, test_ctime); |
| SUITE_ADD_TEST(suite, test_strftime); |
| SUITE_ADD_TEST(suite, test_strftimesmall); |
| SUITE_ADD_TEST(suite, test_exp_tz); |
| SUITE_ADD_TEST(suite, test_strftimeoffset); |
| |
| /* You must return the suite so that the driver knows which suites to |
| * run. |
| */ |
| return suite; |
| } |
| |
| Building the full driver |
| ------------------------ |
| |
| All you need to do to build the full driver is run: |
| |
| make testall |
| |
| To run it, run: |
| |
| ./testall |
| |
| Running individual tests |
| --------------------------------- |
| |
| It is not possible to build individual tests, however it is possible to |
| run individual tests. When running the test suite, specify the name of the |
| tests that you want to run on the command line. For example: |
| |
| ./testall teststr testrand |
| |
| Will run the Strings and Random generator tests. |
| |
| Reading the test suite output |
| ----------------------------- |
| |
| Once you run the test suite, you will get output like: |
| |
| All APR Tests: |
| Test Strings: .... |
| Test Time: ............ |
| |
| 16 tests run: 16 passed, 0 failed, 0 not implemented. |
| |
| Known test failures are documented in ../STATUS. |
| |
| There are a couple of things to look at with this. First, if you look at the |
| first function in this document, you should notice that the string passed to |
| the CuSuiteNew function is in the output. That is why the string should |
| explain the feature you are testing. |
| |
| Second, this test passed completely. This is obvious in two ways. First, and |
| most obvious, the summary line tells you that 16 tests were run and 16 tests |
| passed. However, the results can also be found in the lines above. Every |
| '.' in the output represents a passed test. |
| |
| If a test fails, the output will look like: |
| |
| All APR Tests: |
| Test Strings: .... |
| Test Time: ..F......... |
| |
| 16 tests run: 15 passed, 1 failed, 0 not implemented. |
| |
| This is not very useful, because you don't know which test failed. However, |
| once you know that a test failed, you can run the suite again, with the |
| -v option. If you do this, you will get something like: |
| |
| All APR Tests: |
| Test Strings: .... |
| Test Time: ..F......... |
| |
| 16 tests run: 15 passed, 1 failed, 0 not implemented. |
| Failed tests: |
| 1) test_localstr: assert failed |
| |
| In this case, we know the test_localstr function failed, and there is an |
| Assert in this that failed (I modified the test to fail for this document). |
| Now, you can look at what that test does, and why it would have failed. |
| |
| There is one other possible output for the test suite (run with -v): |
| |
| All APR Tests: |
| Test Strings: .... |
| Test Time: ..N......... |
| |
| 16 tests run: 15 passed, 0 failed, 1 not implemented. |
| |
| Not Implemented tests: |
| |
| Not Implemented tests: |
| 1) test_localstr: apr_time_exp_lt not implemented on this platform |
| |
| The 'N' means that a function has returned APR_ENOTIMPL. This should be |
| treated as an error, and the function should be implemented as soon as |
| possible. |
| |
| Adding New test Suites to the full driver |
| ------------------------------------------- |
| |
| To add a new Suite to the full driver, you must make a couple of modifications. |
| |
| 1) Edit test_apr.h, and add the prototype for the function. |
| 2) Edit testall.c, and add the function and name to the tests array. |
| 3) Edit Makefile.in, and add the .lo file to the testall target. |
| |
| Once those four things are done, your tests will automatically be added |
| to the suite. |
| |
| Writing tests |
| ------------- |
| |
| There are a couple of rules for writing good tests for the test suite. |
| |
| 1) All tests can determine for themselves if it passed or not. This means |
| that there is no reason for the person running the test suite to interpret |
| the results of the tests. |
| 2) Never use printf to add to the output of the test suite. The suite |
| library should be able to print all of the information required to debug |
| a problem. |
| 3) Functions should be tested with both positive and negative tests. This |
| means that you should test things that should both succeed and fail. |
| 4) Just checking the return code does _NOT_ make a useful test. You must |
| check to determine that the test actually did what you expected it to do. |
| |
| An example test |
| --------------- |
| |
| Finally, we will look at a quick test: |
| |
| /* All tests are passed a CuTest variable. This is how the suite determines |
| * if the test succeeded or failed. |
| */ |
| static void test_localstr(CuTest *tc) |
| { |
| apr_status_t rv; |
| apr_time_exp_t xt; |
| time_t os_now; |
| |
| rv = apr_time_exp_lt(&xt, now); |
| os_now = now / APR_USEC_PER_SEC; |
| |
| /* If the function can return APR_ENOTIMPL, then you should check for it. |
| * This allows platform implementors to know if they have to implement |
| * the function. |
| */ |
| if (rv == APR_ENOTIMPL) { |
| CuNotImpl(tc, "apr_time_exp_lt"); |
| } |
| |
| /* It often helps to ensure that the return code was APR_SUCESS. If it |
| * wasn't, then we know the test failed. |
| */ |
| CuAssertTrue(tc, rv == APR_SUCCESS); |
| |
| /* Now that we know APR thinks it worked properly, we need to check the |
| * output to ensure that we got what we expected. |
| */ |
| CuAssertStrEquals(tc, "2002-08-14 12:05:36.186711 -25200 [257 Sat] DST", |
| print_time(p, &xt)); |
| } |
| |
| Notice, the same test can fail for any of a number of reasons. The first |
| test to fail ends the test. |
| |
| CuTest |
| ------ |
| |
| CuTest is an open source test suite written by Asim Jalis. It has been |
| released under the zlib/libpng license. That license can be found in the |
| CuTest.c and CuTest.h files. |
| |
| The version of CuTest that is included in the APR test suite has been modified |
| from the original distribution in the following ways: |
| |
| 1) The original distribution does not have a -v flag, the details are always |
| printed. |
| 2) The NotImplemented result does not exist. |
| 3) SuiteLists do not exist. In the original distribution, you can add suites |
| to suites, but it just adds the tests in the first suite to the list of tests |
| in the original suite. The output wasn't as detailed as I wanted, so I created |
| SuiteLists. |
| |
| The first two modifications have been sent to the original author of CuTest, |
| but they have not been integrated into the base distribution. The SuiteList |
| changes will be sent to the original author soon. |
| |
| The modified version of CuTest is not currently in any CVS or Subversion |
| server. In time, it will be hosted at rkbloom.net. |
| |
| There are currently no docs for how to write tests, but the teststr and |
| testtime programs should give an idea of how it is done. In time, a document |
| should be written to define how tests are written. |
| |