               A Streamlined HTTP Protocol for Subversion

GOAL
====

Write a new HTTP protocol for svn -- one which has no pretense of
adhering to some pre-existing standard, and which is designed for
speed and comprehensibility.  This ain't your Daddy's WebDAV.


PURPOSE / HISTORY
=================

Subversion standardized on Apache and the WebDAV/DeltaV protocol as a
back in the earliest days of development, based on some very strong
value propositions:

  A. Able to go through corporate firewalls
  B. Zillions of authn/authz options via Apache
  C. Standardized encryption (SSL)
  D. Excellent logging
  E. Built-in repository browsing
  F. Caching within intermediate proxies
  G. Interoperability with other WebDAV clients

Unfortunately, DeltaV is an insanely complex and inefficient protocol,
and doesn't fit Subversion's model well at all.  The result is that
Subversion speaks a "limited portion" of DeltaV, and pays a huge
performance price for this complexity.


REQUIREMENTS
============

Write a new HTTP protocol for svn ("HTTP v2").  Map RA requests
directly to HTTP requests.  Subversion over HTTP should:

 * be much faster (eliminate extra turnarounds)
 * be almost as easy to extend as Subversionserve
 * be comprehensible to devs and users without knowledge of DeltaV concepts
 * be designed for optimum cacheability by web proxies
 * make use of pipelined and parallel requests when possible


OUR PLANS, IN A NUTSHELL
========================

 * Phase 1:  Remove all DeltaV mechanics & formalities

   - Get rid of all the PROPFIND 'discovery' turnarounds.
   - Stop doing CHECKOUT requests before each PUT
   - Publish a public URI syntax for browsing historical objects

 * Phase 2:  Speed up commits

   - Make PUT requests pipelined, the way ra_svn does.

 * Phase 3:  (maybe) get rid of XML in request/response bodies

   - If there's a worthwhile speed gain, use serialized Thrift objects.


PHASE 1 IN DETAIL
=================

 * Deprecated DeltaV resources and resource types:

   In compliance with the DeltaV spec, Subversion clients prior using
   that standard protocol have to "discover" and manipulate the
   following DeltaV objects:

     - Version Controlled Resource (VCC):   !svn/vcc
     - Baseline resource:                   !svn/bln
     - Working baseline resource:           !svn/wbl
     - Baseline collection resource:        !svn/bc/REV/
     - Activity collection:                 !svn/act/ACTIVITY-UUID/
     - Versioned resource:                  !svn/ver/REV/path
     - Working resource:                    !svn/wrk/ACTIVITY-UUID/path

   All of these objects will be deprecated and no longer used.
   mod_dav_svn will still support older clients, of course, but new
   clients will be able to automatically construct all of the URIs
   they need.

 * New resources and resource types:

   The following are some new resource and resource type concepts
   we're introducing in HTTP protocol v2:

     - me resource (!svn/me)

       Represents the "repository itself".  This is the URI that
       custom REPORTS are sent against.  (This eliminates our need for
       the VCC resource.)

     - revision resource (!svn/rev/REV)
  
       Represents a Subversion revision at the metadata level, and
       maps conceptually to a "revision" in the FS layer.  Standard
       PROPFIND and PROPATCH requests can be used against a revision
       resource, with the understanding that the name/value pairs
       being accessed are unversioned revision props, rather than file
       or directory props.  (This eliminates our need for baseline or
       working baseline resources.)

     - revision root resource (!svn/rvr/REV/[PATH])

       Represents the directory tree snapshot associated with a
       Subversion revision, and maps conceptually to a revision-type
       svn_fs_root_t/path pair in the FS layer.  GET, PROPFIND, and
       certain REPORT requests can be issued against these resources.

     - transaction resource (!svn/txn/TXN-NAME)

       Represents a Subversion commit transaction, and maps
       conceptually to an svn_fs_txn_t in the FS layer.  PROPFIND and
       PROPATCH requests can be used against a transaction resource,
       with the understanding that the name/value pairs being accessed
       are unversioned transaction props, rather than file or directory
       props.

     - transaction root resource (!svn/txr/TXN-NAME/[PATH])

       Represents the directory tree snapshot associated with a
       Subversion commit transaction, and maps conceptually to a
       transaction-type svn_fs_root_t/path pair in the FS layer.
       Various read- and write-type requests can be issued against
       these resources (MKCOL, PUT, PROPFIND, PROPPATCH, GET, etc.).

     - alternate transaction resource (!svn/vtxn/VTXN-NAME)
     - alternate transaction root resource (!svn/vtxr/VTXN-NAME/[PATH])

       Alternative names for the transaction based on a virtual, or
       visible, name supplied by the client when the transaction
       was created.  The client supplied name is optional, if not
       supplied these resource names are not valid.

 * Opening an RA session:

   ra_serf will send an OPTIONS request when creating a new
   ra_session.  mod_dav_svn will send back what it already sends now,
   but will also return new information as custom headers in the
   OPTIONS response:

     SVN-Youngest-Rev:  REV
     SVN-Me-Resource:  /REPOS-ROOT/!svn/me

   Additionally, this response will contain some new URL stub values:

     SVN-Rev-Stub:  /REPOS-ROOT/!svn/rev
     SVN-Rev-Root-Stub:  /REPOS-ROOT/!svn/rvr
     SVN-Txn-Stub:  /REPOS-ROOT/!svn/txn
     SVN-Txn-Root-Stub:  /REPOS-ROOT/!svn/txr
     SVN-VTxn-Stub:  /REPOS-ROOT/!svn/vtxn
     SVN-VTxn-Root-Stub:  /REPOS-ROOT/!svn/vtxr

   The presence of these new stubs (which can be appended to by the
   client to create full-fledged resource URLs) tells ra_serf that
   this is a new server, and that the new streamlined HTTP protocol
   can be used.  ra_serf then caches them in the ra_session object.
   If these new OPTIONS responses are not returned, ra_serf falls back
   to 'classic' DeltaV protocol.

   NOTE:  Recall that the !svn, while shown in this document as its
   default value, can be changed via server configuration.  Clients
   MUST NOT assume that the "special URI" is "!svn".  We're just
   trying to keep this document as readable as possible.

 * Changes to Read Requests

   Most of the read requests performed by ra_serf fall into one of a
   few categories, issuing GETs, PROPFINDs, or REPORTs against public
   URIs, baseline collection URIs, or the default VCC (with some
   exceptions).  Now, these will all be changed to use the new
   resources, all of which have URLs that can be constructed from the
   information returned by the new OPTIONS response information.

   The implementations of the higher level Subversion update, switch,
   status -u, and diff operations in a given RA module share almost
   all their code.  They really are just that similar to one another.
   Like the changes to the other read requests, this family of
   requests will now operate against our new resources.  But
   additionally, we'll be able to stop using the "wcprops" abstraction
   layer, which is today used to cache version resource URLs in
   Subversion's working copy layer (since the client can construct
   URLs as easily as it can fetch them from a cache).

 * Simple Write Requests

   The 'lock' and 'unlock' operations won't change, because they
   already operate on public HEAD URIs today.  But revprop changes
   will now happen as PROPPATCH's against a resource URI (which can be
   constructed by the client).

 * Commits

   Commits will change significantly.  The current methodology looks like:

     OPTIONS to start ra_session
     PROPFINDs to discover various opaque URIs
     MKACTIVITY to create a transaction
     try:
       for each changed object:
         CHECKOUT object to get working resource
         PUT/PROPPATCH/DELETE/COPY working resource
         MKCOL to create new directories
       MERGE to commit the transaction
     finally:
       DELETE the activity

   The new sequence is simpler, and looks like:

     OPTIONS to start ra_session
     POST against "me resource", to create a transaction
     try:
       for each changed object:
         PUT/PROPPATCH/DELETE/COPY/MKCOL against transaction resources
       MERGE to commit the transaction
     except:
       DELETE the transaction resource

   Specific new changes:

     - The activity-UUID-to-Subversion-txn-name abstraction is gone.
       We now expose the Subversion txn names explicitly through the
       protocol.

     - The new POST request replaces the MKACTIVITY request.

       - no more need to "discover" the activity URI;  !svn/act/ is gone.

       - client no longer needs to create an activity UUID itself.

       - instead, POST returns the name of the transaction it created,
         as TXN-NAME, which can then be appended to the transaction
         stub and transaction root stub as necessary.

       - if the client does choose to supply a UUID with the POST
         request then the POST returns that UUID as VTXN-NAME, instead of
         returning TXN-NAME, and the client then uses that with the
         alternate transaction stub and transaction root stub in subsequent
         requests.

     - Once the commit transaction is created, the client is free to
       send write requests against transaction resources it constructs
       itself.  This eliminates the CHECKOUT requests, and also
       removes our need to use versioned resources (!svn/ver) or
       working resources (!svn/wrk).

     - When modifying transaction resources, clients should send
       'X-SVN-Version-Name:' headers (whose value carries the base
       revision) to facilitate server-side out-of-dateness checks.


STATUS
======

 * Teach mod_dav_svn to answer the OPTIONS request with these
   additional pieces of information:

      repository root URI    [DONE]
      repository UUID        [DONE]
      youngest revision      [DONE]
      me resource URI        [DONE]
      revision stub          [DONE]
      revision root stub     [DONE]
      transaction stub       [DONE]
      transaction root stub  [DONE]

 * Teach mod_dav_svn to recognize and correctly interpret URLs which
   make use of the new URI stubs:

      me resource URI       -> !svn/me   [DONE]
      revision stub         -> !svn/rev  [STARTED]
      revision root stub    -> !svn/rvr  [DONE]
      transaction stub      -> !svn/txn  [STARTED]
      transaction root stub -> !svn/txr  [STARTED]

 * Teach mod_dav_svn to handle POST against the "me resource",
   returning a transaction URI stub and transaction prop URI stub for
   further use in the commit.  [DONE]

 * Teach mod_dav_svn to notice and use X-SVN-Version-Name headers in
   write requests aimed at transaction root resources for
   out-of-dateness checks.  This maps conceptually to the
   'base_revision' svn_delta_editor_t concept.  (Note that some write
   requests -- such as MKCOL, COPY and MOVE -- map to editor
   functionality that doesn't carry a base_revision concept.)

      PROPPATCH  [DONE]
      DELETE     [DONE]
      PUT        [DONE]

 * Teach mod_dav_svn to handle HTTPv2 requests in its mirroring
   logic.  [STARTED (by Dave Brown)]

 * Teach ra_serf operations to not do the multi-PROPFIND dance any
   more, but to fetch the information they seek from mod_dav_svn using
   the new stub URIs:

      get-file              -> GET (against pegrev URI)            [DONE]
      get-dir               -> PROPFIND (against pegrev URI)       [DONE]
      rev-prop              -> PROPFIND (against revision URI)     [DONE]
      rev-proplist          -> PROPFIND (against revision URI)     [DONE]
      check-path            -> PROPFIND (against pegrev URI)       [DONE]
      stat                  -> PROPFIND (against pegrev URI)       [DONE]
      get-lock              -> PROPFIND (against public HEAD URI)  [DONE]

 * Teach ra_serf REPORT-type requests to use the URI stubs where
   applicable, too:

      log                   -> REPORT (against pegrev URI)      [DONE]
      get-dated-rev         -> REPORT (against "me resource")   [DONE]
      get-deleted-rev       -> REPORT (against pegrev URI)      [DONE]
      get-locations         -> REPORT (against pegrev URI)      [DONE]
      get-location-segments -> REPORT (against pegrev URI)      [DONE]
      get-file-revs         -> REPORT (against pegrev URI)      [DONE]
      get-locks             -> REPORT (against public HEAD URI) [DONE]
      get-mergeinfo         -> REPORT (against pegrev URI)      [DONE]
      replay                -> REPORT (against "me resource")   [DONE]
      replay-range          -> REPORTs (against "me resource")  [DONE]

 * Teach ra_serf simple write requests to use new URI stubs:

      change-rev-prop       -> PROPPATCH (against revision URI)  [DONE]
      lock                  -> LOCK (against public HEAD URI)    [DONE]
      unlock                -> UNLOCK (against public HEAD URI)  [DONE]

 * Teach ra_serf to do update-style REPORTs a little differently:

    - REPORT against the new "me resource" instead of VCC URI  [DONE]
    - use new URIs to avoid unnecessary PROPFIND discovery     [DONE]
    - eliminate now-unnecessary wcprops cache                  [DEFERRED]

 * Rework ra_serf commit editor implementation to use new direct
   methods as described in the design doc.

    - use POST to 'me' resource to get txn name           [DONE]
    - set revprops using PROPPATCH on the txn resource    [DONE]
    - abort edit with DELETE against the txn resource     [DONE]
    - send write requests against txn and txn root URLs   [DONE]
    - send X-SVN-Version-Name for out-of-dateness checks  [DONE]
    - enable pipelined PUTs                               [OUT-OF-SCOPE]

 * Optional:  Do some of this stuff for ra_neon, too:

    - get and cache UUID and repos_root from OPTIONS         [DONE]
    - get me resource, and pegrev stub from OPTIONS          [DONE]
    - use me resource instead of the VCC                     [STARTED]
    - use pegrev stubs instead of get_baseline_info() walks  [STARTED]
    - use rev stubs for revprop stuff                        [DONE]
    - use POST to 'me' resource to get txn name              [DONE]
    - set revprops using PROPPATCH on the txn resource       [DONE]
    - abort edit with DELETE against the txn resource        [DONE]
    - send write requests against txn and txn root URLs      [DONE]
    - send X-SVN-Version-Name for out-of-dateness checks     [DONE]
    - enable pipelined PUTs                                  [OUT-OF-SCOPE]