| |
| ================================ |
| A Subversion Testing Framework |
| ================================ |
| |
| |
| The three goals of Subversion's automated test-suite: |
| |
| 1. It must be easy to run. |
| 2. It must be easy to understand the results. |
| 3. It must be easy to add new tests. |
| |
| |
| |
| Definition of an SVN "test program" |
| ----------------------------------- |
| |
| A Subversion test program is any executable that contains a number of |
| sub-tests it can run. It has a standard interface: |
| |
| 1. If run with a numeric argument N, the program runs sub-test N. |
| |
| 2. If run with the argument `--list', it will list the names of all sub-tests. |
| |
| 3. If run with no arguments, the program runs *all* sub-tests. |
| |
| 4. The program returns either 0 (success) or 1 (if any sub-test failed). |
| |
| 5. Upon finishing a test, the program reports the results in a format |
| which is both machine-readable (for the benefit of automatic |
| regression tracking scripts), and human-readable (for the sake of |
| painstaking grovelling by hand in the dead of night): |
| |
| (PASS | FAIL): (argv[0]) (argv[1]): (description) |
| |
| For example, |
| |
| [sussman@newton:~] ./frobtest 2 |
| PASS: frobtest 2: frobnicating fragile data |
| [sussman@newton:~] |
| |
| Note that no particular programming language is required to write a |
| set of tests; they just needs to export this user interface. |
| |
| |
| |
| How to write new C tests |
| ------------------------ |
| |
| The C test framework tests library APIs, both internal and external. |
| |
| All test programs use a standard `main' function. You write .c files |
| that contain only test functions --- you should not define your own |
| `main' function. |
| |
| Instead, your code should define an externally visible array |
| `test_funcs', like this: |
| |
| /* The test table. */ |
| struct svn_test_descriptor_t test_funcs[] = |
| { |
| SVN_TEST_NULL, |
| SVN_TEST_PASS(test_a), |
| SVN_TEST_PASS(test_b), |
| SVN_TEST_PASS(test_c), |
| SVN_TEST_NULL |
| }; |
| |
| In this example, `test_a', `test_b', and `test_c' are the names of |
| test functions. The first and last elements of the array must be |
| SVN_TEST_NULL. The first SVN_TEST_NULL is there to leave room for |
| Buddha. The standard `main' function searches for the final |
| SVN_TEST_NULL to determine the size of the array. |
| |
| Instead of SVN_TEST_PASS, you can use SVN_TEST_XFAIL to declare that a |
| test is expected to fail. The status of such tests is then no longer |
| marked as PASS or FAIL, but rather as XFAIL (eXpected FAILure) or |
| XPASS (uneXpected PASS). |
| |
| The purpose of XFAIL tests is to confirm that a known bug still |
| exists. When you see such a test uneXpectedly PASS, you've probably |
| fixed the bug it tests for, even if that wasn't your intention. :-) |
| XFAIL is not to be used as a way of testing a deliberately invalid |
| operation that is expected to fail when Subversion is working |
| correctly, nor as a place-holder for a test that is not yet written. |
| |
| Each test function conforms to the svn_test_driver_t prototype: |
| |
| svn_error_t *f (const char **MSG, |
| svn_boolean_t MSG_ONLY |
| apr_pool_t *POOL); |
| |
| When called, a test function should first set *MSG to a brief (as in, |
| half-line) description of the test. Then, if MSG_ONLY is TRUE, the |
| test should immediately return SVN_NO_ERROR. Else it should perform a |
| test. If the test passes, the function should return SVN_NO_ERROR; |
| otherwise, it should return an error object, built using the functions |
| in svn_error.h. |
| |
| Once you've got a .c file with a bunch of tests and a `test_funcs' |
| array, you should link it against the `libsvn_tests_main.la' libtool |
| library, in this directory, `subversion/tests'. That library provides |
| a `main' function which will check the command-line arguments, pick |
| the appropriate tests to run from your `test_funcs' array, and print |
| the results in the standard way. |
| |
| |
| How to write new Python tests |
| ----------------------------- |
| |
| The python test framework exercises the command-line client as a |
| "black box". |
| |
| To write python tests, please look at the README file inside the |
| cmdline/ subdirectory. |
| |
| |
| When to write new tests |
| ----------------------- |
| |
| In the world of CVS development, people have noticed that the same |
| bugs tend to recur over and over. Thus the CVS community has adopted |
| a hard-and-fast rule that whenever somebody fixes a bug, a *new* test |
| is added to the suite to specifically check for it. It's a common |
| case that in the process of fixing a bug, several old bugs are |
| accidentally resurrected... and then quickly revealed by the test |
| suite. |
| |
| This same rule applies to Subversion development: ** If you fix a |
| bug, write a test for it. ** |
| |
| |
| When to file a related issue |
| ---------------------------- |
| |
| By definition, if you write a new test which is set to XFail, then it |
| assumed that the test is for a known bug. In these cases it is |
| recommended that you associate an issue in the issue tracker with the |
| XFailing test. This ensures that the issue tracker is the authoritative |
| list of known bugs -- see http://subversion.tigris.org/issue-tracker.html. |
| You may need to create a new issue if one doesn't already exist. |
| |
| For C tests simply add a comment noting any associated issue: |
| |
| /* This is for issue #3234. */ |
| static svn_error_t * |
| test_copy_crash(const svn_test_opts_t *opts, |
| apr_pool_t *pool) |
| { |
| apr_array_header_t *sources; |
| svn_opt_revision_t rev; |
| . |
| . |
| |
| For Python tests use the @Issue() decorator (a summary comment of the |
| issue never hurts either): |
| |
| #--------------------------------------------------------------------- |
| # Test for issue #3657 'dav update report handler in skelta mode can |
| # cause spurious conflicts'. |
| @Issue(3657) |
| @XFail() |
| def dav_skelta_mode_causes_spurious_conflicts(sbox): |
| "dav skelta mode can cause spurious conflicts" |
| . |
| . |
| |
| Of course it isn't *always* necessary to create an associated issue. |
| If a the fix for an new XFailing test is imminent, you are probably |
| better off simply fixing the bug and moving on. Use common sense, but |
| when in doubt associate a new issue. |
| |
| |
| What not to test |
| ---------------- |
| |
| Regression tests are for testing interface promises. This might |
| include semi-private interfaces (such as the non-public .h files |
| inside module subdirs), but does not include implementation details |
| behind the interfaces. For example, this is a good way to test |
| svn_fs_txn_name: |
| |
| /* Test that svn_fs_txn_name fulfills its promise. */ |
| char *txn_name = NULL; |
| SVN_ERR = svn_fs_txn_name (&txn_name, txn, pool); |
| if (txn_name == NULL) |
| return fail(); |
| |
| But this is not: |
| |
| /* Test that the txn got id "0", since it's the first txn. */ |
| char *txn_name = NULL; |
| SVN_ERR = svn_fs_txn_name (&txn_name, txn, pool); |
| if (txn_name && (strcmp (txn_name, "0") != 0)) |
| return fail(); |
| |
| During development, it may sometimes be very convenient to |
| *temporarily* test implementation details via the regular test suite. |
| It's okay to do that, but please remove the test when you're done and |
| make sure it's clearly marked in the meantime. Since implementation |
| details are not interface promises, they might legitimately change -- |
| and when they change, that test will break. At which point whoever |
| encountered the problem will look into the test suite and find the |
| temporary test you forgot to remove. As long as it's marked like |
| this... |
| |
| /* Temporary test for debugging only: Test that the txn got id |
| * "0", since it's the first txn. |
| * NOTE: If the test suite is failing because of this test, then |
| * just remove the test. It was written to help me debug an |
| * implementation detail that might have changed by now, so its |
| * failure does not necessarily mean there's anything wrong with |
| * Subversion. */ |
| char *txn_name = NULL; |
| SVN_ERR = svn_fs_txn_name (&txn_name, txn, pool); |
| if (txn_name && (strcmp (txn_name, "0") != 0)) |
| return fail(); |
| |
| ...then they won't have wasted much time. |
| |
| |
| What's here |
| ----------- |
| |
| * svn_test_main.c |
| [shared library "libsvn_tests_main"] |
| A standardized main() function to drive tests. Link this into |
| your automated test-programs. |
| |
| * svn_test_editor.c |
| [shared library "libsvn_tests_editor"] |
| An editor for testing drivers of svn_delta_edit_fns_t. This |
| editor's functions simply print information to stdout. |
| |
| * cmdline/ |
| A collection of python scripts to test the command-line client. |
| |
| |
| `make check` |
| ------------ |
| |
| The file `build.conf' (at the top level of the tree) defines a |
| [test-scripts] section. These are a list of scripts that will be run |
| whenever someone types `make check`. |
| |
| Each script is expected to output sub-test information as described in |
| the first section of this document; the `make check` rule scans for |
| FAIL codes, and logs all the sub-test output into a top-level file |
| called `tests.log'. |
| |
| If you write a new C executable that contains subtests, be sure to add |
| a build "target" under the TESTING TARGETS section of build.conf. |
| |
| If you write a new python-script, be sure to add to the [test-scripts] |
| section. |
| |
| |
| Testing Over DAV |
| ---------------- |
| |
| Please see subversion/tests/cmdline/README for how to run the |
| command-line client test suite against a remote repository. |
| |
| Conclusion |
| ---------- |
| |
| Our test suite... |
| |
| |
| 1. ...must be easy to run. |
| |
| * run `make check` |
| |
| 2. ...must be easy to understand the results. |
| |
| * test programs output standardized messages |
| * all messages are logged |
| * `make check` only displays errors (not successes!) |
| |
| 3. ...must be easy to add new tests. |
| |
| * add your own sub-test to an existing test program, or |
| * add a new test program using template C or python code. |