notes/test-writing

                                                        -*- text -*-

                     ============================
                     Writing tests for Subversion
                     ============================



 * Test structure
 * Test results
** C test coding
** Python test coding
 * On-disk state
 * 'svn status' status
 * Differences between on-disk and status trees
 * 'svn ci' output
 * Gotcha's


Test structure
==============

Tests start with a clean repository and working copy.  For the
purpose of testing, we use a versioned tree going by the name
'greektree'.  See subversion/tests/greek-tree.txt for more.

This tree is then modified to become in a state in which we want
to test our program.  This can involve changing the working copy
as well as the repository.  Several commands (add, rm, update,
commit) can be required to bring the repository/working copy in
the desired state.

If the working copy and repository are in the required
pre-condition, the command-to-be-tested is executed.  After
execution, the output (stdout, stderr), on-disk state and
'svn status' are checked to verify the command worked as expected.

If you need commands to construct the working copy+repository state,
checks as described above apply to each of the intermediate commands
just as they do to the final command.  That way, failure of the final
command can be narrowed down to just that command, because the
working copy/repository combination was knowingly in the correct
state.


Test results
============

Tests can generate 2 results:

  - Success, signalled by normal function termination
  - Failure, signalled by raising an exception
     In case of python tests: an exception of type SVNFailure
     In case of C tests:      return an svn_error_t * != SVN_NO_ERROR

Sometimes it's necessary to code tests which are supposed to fail,
if Subversion should behave a certain way, but does not yet.  Tests
like these are marked XFail (eXpected-to-FAIL).  If the program is
changed to support the tested behaviour, but the test is not adjusted,
it will XPASS (uneXpectedly-PASS).

Next to normal and XFAIL status tests, there's also conditional
execution of tests, by marking them Skip().  A condition can be
given for which the skip should take effect, executing the test
otherwise.



** C test coding
================

(Could someone fill in this section please?!)



** Python test coding
=====================

The python tests abstract from ordering problems by storing status
information in trees.  Comparing expected and actual status means
comparing trees - there are routines to do the comparison for you.

Every command you issue should use the
svntest.actions.run_and_verify_* API. If there's no such function
for the operation you want to execute, you can use
svntest.main.run_svn.  Note that this is an escape route only:
the results of this command are not checked meaning you should
include any checks in your test yourself.


On-disk state
=============

On-disk state objects can be generated with the
svntest.tree.build_tree_from_wc() function which describe the actual
state on disk.  If you need an object which describes the unchanged
(virginal) state, you can use svntest.actions.get_virginal_state().

Testing for on-disk states is required in several instances, among
which:
 - Checking for specific file contents (after a merge for example)
 - Checking for properties and their values


'svn status' status
===================

Normally any change is at least validated (pre- and post-processing)
by running run_and_verify_status, or passing an expected_status to
one of the other run_and_verify_* methods.

A clean expected_status can be obtained by calling
svntest.actions.get_virginal_state(<wc_dir>, <revision>).


Differences between on-disk and status trees
============================================

Both on-disk and status information is recorded in equal structures,
but there are some differences in the elements that are assigned to
files in each case:

    Fieldname                    On-disk         status

    Contents                        X               -
    Properties                      X               -
    Status                          -               X

###Note: maybe others?

'svn ci' output
===============

Most methods in the run_and_verify_* API take an expected_output
parameter.  This parameter describes which actions the command line
client should report to be taking on each target.  So far there are:

 - 'Adding'
 - 'Deleting'
 - 'Replacing'
 - 'Sending'


Gotcha's
========

 * Minimize the use of 'run_command' and 'run_svn'

   The output of these commands is not checked by the test suite
   itself, so if you really need to use them, be sure to check
   any relevant output yourself.

   If you have any choice at all not to use them, please don't.

 * Tests which check for failure as expected behaviour should PASS

   The XFAIL test status is *only* meant for tests which check for
   not-yet-but-expected-to-be supported program behaviour.

 * File accesses can't use hardcoded '/' characters

   Because the tests need to run on platforms with different path
   separators too (MS Windows), you need to use the os.path.join()
   function to concatenate path strings.

 * Paths within status structures *do* use '/' characters

   Paths within expected_status or expected_disk structures use '/'
   characters as path separators.

 * Don't forget to check output for correct output

   You need to check not only whether a command generated output, but
   also if that output meets your expectations:

     - If the program is supposed to generate an error, check
       if it generates the error you expect it to.
     - If the program does not generate an error, check that
       it gives you the confirmation you expect it to.

 * Don't forget to check pre- and post-command conditions

   You need to verify that the status and on-disk structures are
   actually what you think they are before invoking the command
   you're testing.  Likewise, you need to verify that the command
   resulted in expected output, status and on-disk structure.

 * Don't forget to check!

   Yes, just check anything you can check.  If you don't, your test
   may be passing for all the wrong reasons.