| (-*- text -*-) |
| |
| Subversion Commandline Client: Test Suite |
| ========================================== |
| |
| The cmdline client test suite doesn't use the C-level testing |
| framework, but is structured similarly. Instead of testing library |
| APIs, it drives the client just like a user would, examining the |
| output and the on-disk results (i.e., the working copy) carefully as |
| it goes. In other words, this is "black box" testing of the |
| command-line client. It has no access to code internals; it never |
| looks inside the .svn/ directory; it only performs actions that a |
| human user would do. |
| |
| These tests require Python 2.7 or later. |
| |
| [ For more general information on Subversion's testing system, |
| please read the README in subversion/tests/. ] |
| |
| |
| How To Run The Tests |
| ==================== |
| |
| To run a test script over ra_local, invoke it from THIS DIRECTORY. |
| |
| $ cd subversion/tests/cmdline/ |
| |
| Invoke the script with no arguments to run all the tests in that |
| script: |
| |
| $ ./basic_tests.py |
| |
| Invoke with one or more numeric arguments to run those particular tests: |
| |
| $ ./basic_tests.py 7 13 17 |
| |
| Invoke with one or more function names to run those particular tests: |
| |
| $ ./basic_tests.py basic_mkdir_wc_with_parents basic_switch basic_import |
| |
| And invoke with the "--list" option to list information about some or |
| all tests available in that script: |
| |
| $ ./basic_tests.py --list 2 3 4 |
| $ ./basic_tests.py --list |
| |
| Note: if you are building Subversion in a directory other than the source |
| directory (q.v. INSTALL), you will have to invoke the tests from within |
| the build directory: |
| |
| $ cd obj/subversion/tests/cmdline |
| $ ../../../../svn/subversion/tests/cmdline/basic_tests.py |
| |
| |
| Running over mod_dav_svn |
| ------------------------ |
| |
| Running a script over mod_dav_svn is basically the same, but you have to |
| set up httpd 2.0 first (on the same machine, since the tests create |
| repositories on the fly), and pass a URL argument to the test scripts. |
| |
| Assuming you have httpd 2.0 installed in /usr/local/apache2, just add |
| two Location directives to /usr/local/apache2/conf/httpd.conf, with |
| paths adjusted appropriately: |
| |
| <Location /svn-test-work/repositories> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/repositories |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| Require valid-user |
| </Location> |
| |
| <Location /svn-test-work/local_tmp/repos> |
| DAV svn |
| SVNPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp/repos |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| Require valid-user |
| </Location> |
| |
| <Location /authz-test-work/anon> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| # This may seem unnecessary but granting access to everyone here is necessary |
| # to exercise a bug with httpd 2.3.x+. The "Require all granted" syntax is |
| # new to 2.3.x+ which we can detect with the mod_authz_core.c module |
| # signature. Use the "Allow from all" syntax with older versions for symmetry. |
| <IfModule mod_authz_core.c> |
| Require all granted |
| </IfModule> |
| <IfModule !mod_authz_core.c> |
| Allow from all |
| </IfMOdule> |
| </Location> |
| <Location /authz-test-work/mixed> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| Require valid-user |
| Satisfy Any |
| </Location> |
| <Location /authz-test-work/mixed-noauthwhenanon> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| Require valid-user |
| AuthzSVNNoAuthWhenAnonymousAllowed On |
| </Location> |
| <Location /authz-test-work/authn> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| Require valid-user |
| </Location> |
| <Location /authz-test-work/authn-anonoff> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| Require valid-user |
| AuthzSVNAnonymous Off |
| </Location> |
| <Location /authz-test-work/authn-lcuser> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| Require valid-user |
| AuthzForceUsernameCase Lower |
| </Location> |
| <Location /authz-test-work/authn-lcuser> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| Require valid-user |
| AuthzForceUsernameCase Lower |
| </Location> |
| <Location /authz-test-work/authn-group> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| AuthGroupFile /usr/local/apache2/conf/groups |
| Require group random |
| AuthzSVNAuthoritative Off |
| </Location> |
| <IfModule mod_authz_core.c> |
| <Location /authz-test-work/sallrany> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| AuthzSendForbiddenOnFailure On |
| Satisfy All |
| <RequireAny> |
| Require valid-user |
| Require expr req('ALLOW') == '1' |
| </RequireAny> |
| </Location> |
| <Location /authz-test-work/sallrall> |
| DAV svn |
| SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp |
| AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz |
| SVNListParentPath On |
| AuthType Basic |
| AuthName "Subversion Repository" |
| AuthUserFile /usr/local/apache2/conf/users |
| AuthzSendForbiddenOnFailure On |
| Satisfy All |
| <RequireAll> |
| Require valid-user |
| Require expr req('ALLOW') == '1' |
| </RequireAll> |
| </Location> |
| </IfModule> |
| |
| |
| RedirectMatch permanent ^/svn-test-work/repositories/REDIRECT-PERM-(.*)$ /svn-test-work/repositories/$1 |
| RedirectMatch ^/svn-test-work/repositories/REDIRECT-TEMP-(.*)$ /svn-test-work/repositories/$1 |
| |
| Httpd should be running on port 80. You may also need to ensure that |
| it's running as you, so it has read/write access to the repositories |
| that are probably living in your Subversion working copy. To do this, |
| set the User and Group directives in httpd.conf, something like this: |
| |
| User yourusernamehere |
| Group users |
| |
| You need to run the tests over mod_dav_svn with authentication enabled, so |
| just drop the following 2-line snippet into the |
| /usr/local/apache2/conf/users file [1]: |
| |
| ---------------------------- |
| jrandom:xCGl35kV9oWCY |
| jconstant:xCGl35kV9oWCY |
| JRANDOM:xCGl35kV9oWCY |
| JCONSTANT:xCGl35kV9oWCY |
| ---------------------------- |
| |
| and these lines into the |
| /usr/local/apache/conf/groups file: |
| ---------------------------- |
| random: jrandom |
| constant: jconstant |
| ---------------------------- |
| |
| Now, (re)start Apache and run the tests over mod_dav_svn. |
| |
| You can run a test script over DAV: |
| |
| $ ./basic_tests.py --url http://localhost |
| $ ./basic_tests.py --url http://localhost 3 |
| |
| or |
| |
| $ ./basic_tests.py --url=http://localhost |
| $ ./basic_tests.py --url=http://localhost 3 |
| |
| If you run httpd on a port other than 80, you can specify the port in |
| the URL: "http://localhost:15835" for example. |
| |
| To run all tests over DAV, pass BASE_URL when running 'make check' |
| from the top of the build dir: |
| |
| $ make check BASE_URL=http://localhost |
| |
| BASE_URL=URL can also be used when running individual tests: |
| |
| $ ./basic_tests.py BASE_URL=http://localhost |
| $ ./basic_tests.py BASE_URL=http://localhost 3 |
| |
| |
| Note [1]: It would be quite too much to expect those password entries |
| to work on Windows... Apache httpd on Windows doesn't |
| understand crypted passwords, but it does understand |
| MD5-hashed passwords. The correct password entries for |
| Windows are: |
| |
| ---------------------------- |
| jrandom:$apr1$3p1.....$FQW6RceW5QhJ2blWDQgKn0 |
| jconstant:$apr1$jp1.....$Usrqji1c9H6AbOxOGAzzb0 |
| JRANDOM:$apr1$3p1.....$FQW6RceW5QhJ2blWDQgKn0 |
| JCONSTANT:$apr1$jp1.....$Usrqji1c9H6AbOxOGAzzb0 |
| ---------------------------- |
| |
| |
| As a shorthand to all of the above, ./davautocheck.sh will generate |
| an Apache configuration listening on a random port on localhost and |
| run some tests. Without arguments, or when invoking 'make davautocheck' |
| on the top-level Makefile, it will run all tests. With arguments, |
| it will run just one suite or just one test: |
| |
| $ ./davautocheck.sh |
| $ ./davautocheck.sh basic |
| $ ./davautocheck.sh basic 15 |
| |
| With '--no-tests' argument, it will start httpd but not run any tests. This is |
| useful for manual testing --- create repositories in |
| ./svn-test-work/repositories/<$repo> and they will be accessible at |
| <URL>/svn-test-work/repositories/<$repo>. You can also run individual tests by passing the --url option to them, as described above. |
| |
| davautocheck.sh also respects some environment variables; see the comments at |
| the top of the script for details. |
| |
| Running over ra_svn |
| ------------------- |
| |
| It's also easy to run the tests against a local svnserve: |
| |
| $ subversion/svnserve/svnserve -d -r `pwd`/subversion/tests/cmdline |
| $ make check BASE_URL=svn://localhost |
| |
| or, to run individual tests, |
| |
| $ ./basic_tests.py --url=svn://localhost 3 |
| |
| To enable Cyrus SASL on the server side you should either set the |
| ENABLE_SASL variable when calling make: |
| |
| $ make check BASE_URL=svn://localhost ENABLE_SASL=true |
| |
| or if you're running an individual test, |
| |
| $ ./basic_tests.py --url=svn://localhost --enable-sasl 3 |
| |
| Note that to do this you'll have to have a svn.conf file in your |
| SASL lib dir (i.e. something like /usr/lib/sasl2/svn.conf), it |
| should contain something like: |
| |
| pwcheck_method: auxprop |
| mech_list: CRAM-MD5 |
| |
| And then you'll have to add the users jrandom and jconstant to your |
| SASL password db, |
| |
| $ saslpasswd2 -c -u svntest jrandom |
| $ saslpasswd2 -c -u svntest jconstant |
| |
| As usual, both users should use the password 'rayjandom'. |
| |
| To enable DUMP_LOAD_CROSS_CHECK to work a third user is required, |
| |
| $ saslpasswd2 -c -u svntest __dumpster__ |
| |
| with password '__loadster__'. |
| |
| The user running the tests will need read access to the sasl database |
| and on some systems this can be arranged by adding the user to the sasl |
| group. |
| |
| There are 'make svnserveautocheck' and ./svnserveautocheck.sh commands, |
| analogous to davautocheck.sh documented above. |
| |
| |
| Running tests in a RAM disk |
| -------------------------- |
| |
| Test execution can be dramatically sped up by keeping Subversion test |
| data on a RAM disk. On a Linux system, you can mount a RAM disk on the |
| fly with the command: |
| |
| mount -t tmpfs tmpfs /path/to/src/subversion/tests/cmdline/svn-test-work \ |
| -o uid=$USER,mode=770,size=32m |
| |
| Or, for a more permanent solution, add lines like the following in your |
| /etc/fstab file: |
| |
| tmpfs /path/to/src/svn/subversion/tests/cmdline/svn-test-work tmpfs defaults,user,noauto,exec,size=32m |
| |
| The minimum required size for testing ramdisk is approximately 700MB. |
| However, flagging your test targets for cleanup dramatically reduces |
| the space requirements (as shown in the example configuration above), |
| and thus your memory usage. Cleanup means more I/O, but since test |
| data is in-memory, there will be no performance degradation. Example: |
| |
| make check CLEANUP=true |
| |
| See http://svn.haxx.se/dev/archive-2003-02/0068.shtml for the original |
| authoritative discussion on use of RAM disks. |
| |
| |
| Directory Contents |
| ================== |
| |
| *.py The tests themselves. |
| |
| svntest/ Python package, provides test suite framework |
| |
| /main.py: Global vars, utility routines; exports |
| run_tests(), the main test routine. |
| |
| /tree.py: Infrastructure for SVNTreeNode class. |
| - tree constructors, tree comparison routines. |
| - routines to parse subcommand output into |
| specific kinds of trees. |
| - routines to parse a working copy and |
| entries files into specific kinds of trees. |
| |
| /wc.py: Functions for interacting with a working |
| copy, and converting to/from trees. |
| |
| /actions.py: Main API for driving subversion client and |
| using trees to verify results. |
| |
| /verify.py: Verifies output from Subversion. |
| |
| /testcase.py: Control of test case execution - contains |
| decorators for expected failures and conditionally |
| executed tests. |
| |
| /sandbox.py: Tools for manipulating a test's working area |
| ("a sandbox"), those are handy for most simple |
| actions a test might want to perform on a wc. |
| |
| /objects.py: Objects that keep track of state during a test. |
| (not directly used by the test scripts.) |
| |
| /mergetrees.py: Routines that create merge scenarios. |
| |
| /factory.py: Automatically generate a (near-)complete new |
| cmdline test from a series of shell commands. |
| |
| /error.py: Error codes as constants, for convenience. |
| (auto-generated by tools/dev/gen-py-error.py) |
| |
| |
| What the Python Tests are Doing |
| =============================== |
| |
| I. Theory |
| |
| A. Types of Verification |
| |
| The point of this test system is that it's *automated*: that is, |
| each test can algorithmically verify the results and indicate "PASS" |
| or "FAIL". |
| |
| We've identified two broad classes of verification: |
| |
| 1. Verifying svn subcommand output. |
| |
| Most important subcommands (co, up, ci, im, st) print results to |
| stdout as a list of paths. Even though the paths may be printed |
| out in an unpredictable order, we still want to make sure this |
| list is exactly the *set* of lines we expect to get. |
| |
| 2. Verifying the working copy itself. |
| |
| Every time a subcommand could potentially change something on |
| disk, we need to inspect the working copy. Specifically, this |
| means we need to make sure the working copy has exactly the |
| tree-structure we expect, and each file has exactly the contents |
| and properties we expect. |
| |
| |
| II. Practice: Trees |
| |
| Sam TH <sam@uchicago.edu> proposed and began work on a solution |
| whereby all important, inspectable information is parsed into a |
| general, in-memory tree representation. By comparing actual |
| vs. expected tree structures, we get automated verification. |
| |
| A. Tree node structure |
| |
| Each "tree node" in a tree has these fields: |
| |
| - name : the name of the node |
| - children: list of child nodes (if the node is a dir) |
| - contents: textual contents (if the node is a file) |
| - properties: a hash to hold subversion props |
| - atts: a hash of meta-information about tree nodes themselves |
| |
| |
| B. Parsing subcommand output into a tree |
| |
| Special parsers examine lines printed by subcommands, and |
| convert them into a tree of tree-nodes. The 'contents' and |
| 'properties' fields are empty; but prepending on the subcommand, |
| specific attributes in the 'atts' field are set in tree-nodes: |
| |
| - svn co/up: a 'status' attribute is set to a two-character |
| value from the set (A, D, G, U, C, _, ' ') or |
| a 'verb' attribute is set to ('Restored') |
| |
| - svn status: a 'status' attribute (as above), plus 'wc_rev' |
| and 'repos_rev' attributes to hold the wc |
| and repos revision numbers. |
| |
| - svn ci/im: a 'verb' attribute is set to one of |
| (Adding, Sending, Deleting) |
| |
| |
| C. Parsing a working copy into a tree |
| |
| We also have a routines that walks a regular working copy and |
| returns a tree representing disk contents and props. In this |
| case the 'atts' hash in each node is empty, but the 'contents' |
| and 'props' fields are filled in. |
| |
| |
| |
| How to Write New Tests |
| ====================== |
| |
| If you'd like to write a new python test, first decide which file it |
| might fit into; test scripts each contain collections of tests grouped |
| by rough categories. (Is it testing a new subcommand? New |
| enhancement? Tricky use-case? Regression test?) |
| |
| Next, read the long documentation comment at the top of |
| svntest/tree.py. It will explain the general API that most tests use. |
| |
| Finally, try copying-and-pasting a simple test and then edit from |
| there. Don't forget to add your test to the 'test_list' variable at |
| the bottom of the file. To avoid renumbering of existing tests, you |
| should add new tests to the end of the list. |
| |
| |
| Testing Compatibility With Previous Release |
| =========================================== |
| |
| You can run the Python test suite against older installed versions of |
| the Subversion servers. This mail fragment introduces the ability: |
| |
| Message-ID: <1ea387f60804091828q48c9d18ah7bf8d89ef7d39461@mail.gmail.com> |
| Date: Wed, 9 Apr 2008 18:28:40 -0700 |
| From: "David Glasser" <glasser@davidglasser.net> |
| To: "Subversion Developers" <dev@subversion.tigris.org> |
| Subject: backwards-compatibility testing! |
| |
| I've updated the expectations on trunk so that you can cleanly run the |
| test suite against 1.4.x svnserve or DAV. You do this by adding |
| SERVER_MINOR_VERSION=4 with make check, or --server-minor-version 4 to |
| run_tests.py or a specific Python test. |
| |
| [...] |
| |
| We expect that post-1.5, this support will expand in the obvious ways |
| (allowing "--server-minor-version 5" and SERVER_MINOR_VERSION=5). |