| Writing APR tests |
| |
| All APR tests should be executable in 2 ways, as an individual program, or |
| as a part of the full test suite. The full test suite is controlled with |
| the testall program. At the beginning of the testall.c file, there is an |
| array of functions called tests. The testall program loops through this |
| array calling each function. Each function returns a CuSuite variable, which |
| is then added to the SuiteList. Once all Suites have been added, the SuiteList |
| is executed, and the output is printed to the screen. All functions in the |
| array should follow the same basic format: |
| |
| The Full Suite |
| -------------- |
| |
| /* The driver function. This must return a CuSuite variable, which will |
| * then be used to actually run the tests. Essentially, all Suites are a |
| * collection of tests. The driver will take each Suite, and put it in a |
| * SuiteList, which is a collection of Suites. |
| */ |
| CuSuite *testtime(void) |
| { |
| /* The actual suite, this must be created for each test program. Please |
| * give it a useful name, that will inform the user of the feature being |
| * tested. |
| */ |
| CuSuite *suite = CuSuiteNew("Test Time"); |
| |
| /* Each function must be added to the suite. Each function represents |
| * a single test. It is possible to test multiple features in a single |
| * function, although no tests currently do that. |
| */ |
| SUITE_ADD_TEST(suite, test_now); |
| SUITE_ADD_TEST(suite, test_gmtstr); |
| SUITE_ADD_TEST(suite, test_localstr); |
| SUITE_ADD_TEST(suite, test_exp_get_gmt); |
| SUITE_ADD_TEST(suite, test_exp_get_lt); |
| SUITE_ADD_TEST(suite, test_imp_gmt); |
| SUITE_ADD_TEST(suite, test_rfcstr); |
| SUITE_ADD_TEST(suite, test_ctime); |
| SUITE_ADD_TEST(suite, test_strftime); |
| SUITE_ADD_TEST(suite, test_strftimesmall); |
| SUITE_ADD_TEST(suite, test_exp_tz); |
| SUITE_ADD_TEST(suite, test_strftimeoffset); |
| |
| /* You must return the suite so that the driver knows which suites to |
| * run. |
| */ |
| return suite; |
| } |
| |
| Building the full driver |
| ------------------------ |
| |
| All you need to do to build the full driver is run: |
| |
| make |
| |
| To run it, run: |
| |
| ./testall |
| |
| Running individual tests |
| ------------------------ |
| |
| It is not possible to build individual tests, however it is possible to |
| run individual tests. When running the test suite, specify the name of the |
| tests that you want to run on the command line. For example: |
| |
| ./testall teststr testrand |
| |
| Will run the Strings and Random generator tests. |
| |
| Reading the test suite output |
| ----------------------------- |
| |
| Once you run the test suite, you will get output like: |
| |
| All APR Tests: |
| Test Strings: .... |
| Test Time: ............ |
| |
| 16 tests run: 16 passed, 0 failed, 0 not implemented. |
| |
| Known test failures are documented in ../STATUS. |
| |
| There are a couple of things to look at with this. First, if you look at the |
| first function in this document, you should notice that the string passed to |
| the CuSuiteNew function is in the output. That is why the string should |
| explain the feature you are testing. |
| |
| Second, this test passed completely. This is obvious in two ways. First, and |
| most obvious, the summary line tells you that 16 tests were run and 16 tests |
| passed. However, the results can also be found in the lines above. Every |
| '.' in the output represents a passed test. |
| |
| If a test fails, the output will look like: |
| |
| All APR Tests: |
| Test Strings: .... |
| Test Time: ..F......... |
| |
| 16 tests run: 15 passed, 1 failed, 0 not implemented. |
| |
| This is not very useful, because you don't know which test failed. However, |
| once you know that a test failed, you can run the suite again, with the |
| -v option. If you do this, you will get something like: |
| |
| All APR Tests: |
| Test Strings: .... |
| Test Time: ..F......... |
| |
| 16 tests run: 15 passed, 1 failed, 0 not implemented. |
| Failed tests: |
| 1) test_localstr: assert failed |
| |
| In this case, we know the test_localstr function failed, and there is an |
| Assert in this that failed (I modified the test to fail for this document). |
| Now, you can look at what that test does, and why it would have failed. |
| |
| There is one other possible output for the test suite (run with -v): |
| |
| All APR Tests: |
| Test Strings: .... |
| Test Time: ..N......... |
| |
| 16 tests run: 15 passed, 0 failed, 1 not implemented. |
| |
| Not Implemented tests: |
| |
| Not Implemented tests: |
| 1) test_localstr: apr_time_exp_lt not implemented on this platform |
| |
| The 'N' means that a function has returned APR_ENOTIMPL. This should be |
| treated as an error, and the function should be implemented as soon as |
| possible. |
| |
| Adding New test Suites to the full driver |
| ------------------------------------------- |
| |
| To add a new Suite to the full driver, you must make a couple of modifications. |
| |
| 1) Edit test_apr.h, and add the prototype for the function. |
| 2) Edit testall.c, and add the function and name to the tests array. |
| 3) Edit Makefile.in, and add the .lo file to the testall target. |
| |
| Once those four things are done, your tests will automatically be added |
| to the suite. |
| |
| Writting an ABTS unit test |
| -------------------------- |
| |
| The aim of this quick and dirty Howto is to give a short introduction |
| to APR (Apache Portable Runtime) unit tests, and how to write |
| one. During my Google's Summer of Code 2005 project, I discovered a |
| small bug in the APR-Util's date parsing routines, and I needed to |
| write a unit test for the fixed code. I decided to write this |
| documentation because I did not find any. Thanks to Garrett Rooney for |
| his help on writing the unit test ! |
| |
| The APR and APR-Util libraries provide a platform independent API for |
| software developers. They contain a lot of modules, including network |
| programming, threads, string and memory management, etc. All these |
| functions need to be heavily tested so that developers can be sure the |
| library is reliable. |
| |
| The ABTS give APR developers the ability to build a complete test |
| suite for the bunch of tests they wrote, which can then be ran under |
| various platforms. In this Howto, I will try teach you how to write an |
| ABTS unit test. |
| |
| As you may probably know, a unit test is a simple routine which tests |
| a very specific feature of the tested software or library. To build a |
| unit test, you need three different things : |
| |
| * the to-be-tested function, |
| * the input data that will be given to the function, |
| * the expected output data. |
| |
| The principle of a unit test is very simple : for each entry in your |
| set of input data, we pass it to our function, fetch what the function |
| returned and compare it to the corresponding expected output data. Of |
| course, the more edge cases you can test, the better your input data |
| set is. |
| |
| The ABTS aims to quicken the write of unit test, and make them |
| available to the whole test suite by providing a set of preprocessor |
| macros. Adding a unit test to a test suite can be easily done by the |
| following piece of code : |
| |
| abts_suite *testdaterfc(abts_suite *suite) |
| { |
| suite = ADD_SUITE(suite); |
| abts_run_test(suite, test_date_rfc, NULL); |
| |
| return suite; |
| } |
| |
| Where test_date_rfc is the name of the function performing the |
| test. Writing such a function is, in the light of the explanation I |
| just gave, pretty much easy too. As I said, we need to check every |
| entry of our input data set. That gives us a loop. For each loop |
| iteration, we call our to-be-tested function, grab its result and |
| compare the returned value with the expected one. |
| |
| Test functions must have the following prototype : |
| |
| static void my_test_function(abts_case *tc, void *data); |
| |
| The comparison step is performed by the ABTS, thus giving the |
| whole test suite the correct behavior if your unit test fails. Here |
| comes a list of the available test methods : |
| |
| ABTS_INT_EQUAL(tc, a, b) |
| ABTS_INT_NEQUAL(tc, a, b) |
| ABTS_STR_EQUAL(tc, a, b) |
| ABTS_STR_NEQUAL(tc, a, b, c) |
| ABTS_PTR_NOTNULL(tc, b) |
| ABTS_PTR_EQUAL(tc, a, b) |
| ABTS_TRUE(tc, b) |
| ABTS_FAIL(tc, b) |
| ABTS_NOT_IMPL(tc, b) |
| ABTS_ASSERT(tc, a, b) |
| |
| The first argument, tc is a reference to the unit test currently |
| processed by the test suite (passed to your test function). The other |
| parameters are the data to be tested. For example, the following line |
| will never make your unit test fail : |
| |
| ABTS_INT_EQUAL(tc, 1, 1); |
| |
| See, it's easy ! Let's take a look at the complete example : |
| testdaterfc. We want to test our date string parser. For this, we will |
| use some chosen date strings (from mail headers for example) written |
| in various formats but that should all be handled by our function, and |
| their equivalents in correct RFC822 format. |
| |
| The function we want to test returns an apr_time_t}, which will be |
| directly given as input to the apr_rfc822_date() function, thus |
| producing the corresponding RFC822 date string. All we need to do |
| after this is to call the correct test method from the ABTS macros ! |
| |
| You can take a look at the apr-util/test/testdaterfc.c file for the |
| complete source code of this unit test. |
| |
| Although this Howto is very small and mostly dedicated to the |
| testdaterfc unit test, I hope you'll find it useful. Good luck ! |
| |
| Writing tests for CuTest (no longer used) |
| ----------------------------------------- |
| |
| There are a couple of rules for writing good tests for the test suite. |
| |
| 1) All tests can determine for themselves if it passed or not. This means |
| that there is no reason for the person running the test suite to interpret |
| the results of the tests. |
| 2) Never use printf to add to the output of the test suite. The suite |
| library should be able to print all of the information required to debug |
| a problem. |
| 3) Functions should be tested with both positive and negative tests. This |
| means that you should test things that should both succeed and fail. |
| 4) Just checking the return code does _NOT_ make a useful test. You must |
| check to determine that the test actually did what you expected it to do. |
| |
| An example test |
| --------------- |
| |
| Finally, we will look at a quick test: |
| |
| /* All tests are passed a CuTest variable. This is how the suite determines |
| * if the test succeeded or failed. |
| */ |
| static void test_localstr(CuTest *tc) |
| { |
| apr_status_t rv; |
| apr_time_exp_t xt; |
| time_t os_now; |
| |
| rv = apr_time_exp_lt(&xt, now); |
| os_now = now / APR_USEC_PER_SEC; |
| |
| /* If the function can return APR_ENOTIMPL, then you should check for it. |
| * This allows platform implementors to know if they have to implement |
| * the function. |
| */ |
| if (rv == APR_ENOTIMPL) { |
| CuNotImpl(tc, "apr_time_exp_lt"); |
| } |
| |
| /* It often helps to ensure that the return code was APR_SUCESS. If it |
| * wasn't, then we know the test failed. |
| */ |
| CuAssertTrue(tc, rv == APR_SUCCESS); |
| |
| /* Now that we know APR thinks it worked properly, we need to check the |
| * output to ensure that we got what we expected. |
| */ |
| CuAssertStrEquals(tc, "2002-08-14 12:05:36.186711 -25200 [257 Sat] DST", |
| print_time(p, &xt)); |
| } |
| |
| Notice, the same test can fail for any of a number of reasons. The first |
| test to fail ends the test. |
| |
| CuTest |
| ------ |
| |
| CuTest is an open source test suite written by Asim Jalis. It has been |
| released under the zlib/libpng license. That license can be found in the |
| CuTest.c and CuTest.h files. |
| |
| The version of CuTest that is included in the APR test suite has been modified |
| from the original distribution in the following ways: |
| |
| 1) The original distribution does not have a -v flag, the details are always |
| printed. |
| 2) The NotImplemented result does not exist. |
| 3) SuiteLists do not exist. In the original distribution, you can add suites |
| to suites, but it just adds the tests in the first suite to the list of tests |
| in the original suite. The output wasn't as detailed as I wanted, so I created |
| SuiteLists. |
| |
| The first two modifications have been sent to the original author of CuTest, |
| but they have not been integrated into the base distribution. The SuiteList |
| changes will be sent to the original author soon. |
| |
| The modified version of CuTest is not currently in any CVS or Subversion |
| server. In time, it will be hosted at rkbloom.net. |
| |
| There are currently no docs for how to write tests, but the teststr and |
| testtime programs should give an idea of how it is done. In time, a document |
| should be written to define how tests are written. |
| |