| -*- text -*- |
| |
| TREE CONFLICT TESTING STRATEGY |
| |
| |
| This document describes how we are testing the code that detects, |
| reports and resolves tree conflicts. We'd like to make the testing, |
| and the tree conflicts feature itself, more transparent and open to |
| contributions. |
| |
| For tree conflicts, there already exist cmdline tests for the update, |
| switch, merge, commit, status, info and revert commands. We've added |
| tree_conflicts_tests.py, not to replace the other tests, but rather to |
| complement them by offering a generic way to create lots of |
| tree-conflict detection test scenarios. The generic framework is not |
| yet finished, but we think it will be useful as we extend the |
| tree-conflict feature beyond its original use cases. |
| |
| ================ |
| The Declarations |
| ================ |
| |
| The new tree-conflict testing framework offers a compact, declarative |
| format for test definitions. Elementary actions are combined into |
| scenarios, scenarios are bundled into sets, and the sets are fed into |
| a generic tree-conflict-maker. A scenario can be committed to the |
| test repository and then applied to the working copy by an update, |
| switch or merge operation. In another test, the same scenario can |
| modify the working copy prior to an update or switch operation. |
| |
| An advantage of this abstraction is that it allows us to create |
| additional tests easily through code reuse. It also helps us to see |
| beyond our 6 original use cases. |
| |
| A disadvantage is that test failures are rather opaque, but that could |
| probably be fixed with some Python wizardry. |
| |
| The changes that can cause tree conflicts are composed from a set of |
| elementary actions, each named according to its function. For |
| instance, fD signifies running 'svn delete' on a file. |
| |
| The first character of an action name specifies the type of the item |
| acted upon. The names of the items are fixed. |
| |
| f_ Item is the file 'F' |
| d_ Item is the directory 'D' |
| |
| The second character of an action name can specify an svn operation. |
| |
| _P - Change a Property. |
| (Note: Presently just sets a particular property value, even |
| if that property already had that value.) |
| |
| _A - Create the item by Adding an unversioned item |
| (and set a property on it as well). |
| (Suggestion: Don't set a property here; let the user do so |
| explicitly with '_P' when desired. We'd have to ensure that |
| a subsequent '_P' later in the same test would still cause |
| a change; presently it would not.) |
| _C - Create the item by Copying from an existing one. |
| |
| _M - Move the item to a different name. |
| _D - Delete the item. |
| |
| Alternately, the second character can specify a non-svn filesystem |
| operation. |
| |
| _t Append text to the file. |
| |
| _a Create the item on disk. |
| |
| _d Delete the item from disk. |
| |
| To help detect bugs in the test scenarios, each action, except for _a, |
| first asserts that the item (F or D) exists on disk. |
| |
| Some actions operate on 2 items. _C copies F1 to F (or D1 to D) and |
| _M moves F to F2 (or D to D2). The items F, F1, D and D1 are created, |
| added and committed by a generic test-setup function. F2 and D2 do not |
| exist at the start of a test. |
| |
| The arguments for copy and move are not symmetrical because we are |
| interested only in the destination of a copy and the source of a move. |
| The source of a copy is uninteresting, and the destination of a move |
| is the same as that of a copy. |
| |
| The elementary actions are combined to form "scenarios". A scenario |
| is a literal Python tuple containing two lists of actions: the first |
| list creates the starting point for the change to be tested, and the |
| second being the actual change to be tested. For example, this |
| scenario represents a simple deletion of a file: |
| |
| ( ['fa','fA'], ['fD'] ) |
| |
| ================= |
| Behind the Scenes |
| ================= |
| |
| How are the scenarios actually used? |
| |
| The generic tree-conflict-maker is ensure_tree_conflict(). This |
| function applies two sets of scenarios. The "incoming" set is |
| applied to the repository, and the "localmod" set is applied to the |
| working copy. Each possible combination of incoming and localmod |
| scenarios is tested as an independent subtest. |
| |
| The incoming scenarios are prepared as follows. |
| |
| 1. Run the usual Subversion test setup, sbox.build(), which creates a |
| test repository containing the "greek tree" (as revision 1) and checks |
| out a working copy of it. |
| |
| 2. For each incoming scenario, create the scenario path via 'svn |
| mkdir'. |
| |
| 3. For each incoming scenario, execute its initialisation actions which |
| will typically create the file or directory that will be acted on later. |
| |
| 4. Commit as revision 2. |
| |
| 5. For each incoming scenario, execute its actions on the F or D in |
| its scenario path. |
| |
| 6. Commit as revision 3. |
| |
| Now the repository is loaded with all of the incoming scenarios. To |
| run the actual subtests, each incoming scenario must be applied to |
| each localmod scenario. |
| |
| 1. Check out a fresh working copy at revision 2. |
| |
| 2. Execute the localmod scenario's actions on the F or D in its |
| scenario path. |
| |
| 3. For each incoming scenario, run the given svn command (e.g. update) |
| on the incoming scenario's path, then run 'svn status' on the same |
| path. If the path is tree-conflicted, we're happy. |
| |
| The working copy is deleted and the steps are repeated for next |
| localmod scenario. If any failure occurs, the whole test is marked as |
| a failure in the test output. |
| |
| Each test scenario is executed in a unique path created from the actions |
| in the action list, concatenated with "_" between them. For the above |
| example, that path would be simply "fD". The use of a unique path could |
| allow running many of them in parallel. Currently, we run the scenarios |
| one-by-one, each in a fresh working copy. |
| |
| ============== |
| Current Status |
| ============== |
| |
| The following features are sketched out in the scenario data, but not |
| tested: |
| |
| Obstructions |
| |
| Replacement (file->file, dir->dir) |
| |
| 'svn switch' |
| |