notes/http-protocol-v2.txt

               A Streamlined HTTP Protocol for Subversion

GOAL
====

Write a new HTTP protocol for svn -- one which is entirely proprietary
and designed for speed and comprehensibility.


PURPOSE / HISTORY
=================

Subversion standardized on Apache and the WebDAV/DeltaV protocol as a
back in the earliest days of development, based on some very strong
value propositions:

  A. Able to go through corporate firewalls
  B. Zillions of authn/authz options via Apache
  C. Standardized encryption (SSL)
  D. Excellent logging
  E. Built-in repository browsing
  F. Caching within intermediate proxies
  G. Interoperability with other WebDAV clients

Unfortunately, DeltaV is an insanely complex and inefficient protocol,
and doesn't fit Subversion's model well at all.  The result is that
Subversion speaks a "limited portion" of DeltaV, and pays a huge
performance price for this complexity.


REQUIREMENTS
============

Write a new HTTP protocol for svn ("HTTP v2").  Map RA requests
directly to HTTP requests.

  * svn over HTTP should be much faster (eliminate extra turnarounds)
 
  * svn over HTTP should be almost as easy to extend as svnserve.

  * svn over HTTP should be comprehensible to devs and users both
    (require no knowledge of DeltaV concepts).

  * svn over HTTP should be designed for optimum cacheability by web
    proxies.

  * svn over HTTP should make use of pipelined and parallel requests
    when possible.



Our Plans, in a Nutshell
========================

* Phase 1:  Remove all DeltaV mechanics & formalities

  - get rid of all the PROPFIND 'discovery' turnarounds.
  - stop doing CHECKOUT requests before each PUT
  - publish a public URI syntax for browsing historical objects

* Phase 2:  Speed up commits

  - make PUT requests pipelined, the way ra_svn does.

* Phase 3:  (maybe) get rid of XML in request/response bodies

  - if there's a worthwhile speed gain, use serialized Thrift objects.



Phase 1 in Detail
=================

At the moment, ra_serf has to 'discover' and manipulate the following
DeltaV objects:

   - Version Controlled Resource (VCC) :  !svn/vcc
   - Baseline resource:                   !svn/bln
   - Working baseline resource:           !svn/wbl
   - Baseline collection resource:        !svn/bc/REV/
   - Activity collection:                 !svn/act/activityUUID/
   - Versioned resource:                  !svn/ver/REV/path
   - Working resource:                    !svn/wrk/activityUUID/path

All of these objects will be deprecated and no longer used.
mod_dav_svn will still support older clients, of course, but new
clients will be able to automatically construct all of the URIs they
need.


 * Opening an RA session:

   ra_serf will send an OPTIONS request when creating a new
   ra_session.  mod_dav_svn will send back what it already sends now,
   but will also return new information:

         youngest revision:  number
         "me resource" URI:  !svn/me
             revision stub:  !svn/rev
        revision root stub:  !svn/bc   [TODO: make this !svn/rvr]
          transaction stub:  !svn/txn
     transaction root stub:  !svn/txr

   The presence of these new stubs tells ra_serf that this is a new
   server, and that the new streamlined HTTP protocol can be used.
   ra_serf then caches them in the ra_session object.  If these new
   OPTIONS responses are not returned, ra_serf falls back to 'classic'
   DeltaV protocol.


 * What the new stubs are used for:

   - me resource:  represents the "repository itself".  This is the URI
     that custom REPORTS are sent against.

     Note:  this eliminates our need for the VCC resource.

   - revision stub: represents an opaque string to append to, whenever
     the client wants to access a revision's revprops (either reading
     or writing).  Specifically, /REV is appended, e.g.:

          PROPFIND !svn/rev/2398

     This maps conceptually to a "revision" in the FS.

     Standard PROPFIND and PROPATCH requests can be used against the
     constructed URI, with the understanding that the name/value pairs
     being accessed are unversioned revision props, rather than file
     or directory props.

     Note:  this eliminates our need for baseline (bln) or working
     baseline (wbl) resources.

   - revision root stub: an opaque string to append to, whenever the
     client wants to refer to a (pegrev, path) in the repository.
     Specifically, /REV/[PATH] are appended, e.g.:

          GET !svn/bc/2398/trunk/foo.c

     This maps conceptually to a "revision root" FS object.

     Note:  that this syntax is already the one mod_dav_svn understands;
     what's changing here is that we no longer need to do a bunch of
     PROPFINDs to discover it -- we get the stub right up front when
     the session is opened.

   - transaction stub: represents an opaque string to append to
     whenever the client wants to access an uncommitted transaction's
     properties.  Specifically, /TXN_NAME is appended, e.g.:

          PROPFIND !svn/txn/e4b

     This maps conceptually to an svn_fs_txn_t in the FS.

   - transaction root stub: an opaque string to append to, whenever the
     client wants to refer to a (txn-name, path) in the repository.
     Specifically, /TXN_NAME/[PATH] are appended, e.g.:

          GET !svn/txr/e4b/trunk/foo.c

     This maps conceptually to a "txn root" FS object.


 * Simple read requests

   These RA functions each send single request/response, either GET or
   PROPFIND.

   The only changes here is that we no longer need to "discover"
   pegrev or revision URIs with extra turnarounds; instead we construct
   them directly.

    get-latest-rev    -> already present in ra_session (via OPTIONS)

    get-file          -> GET (against a pegrev URI)

    get-dir           -> PROPFIND depth 1 (against a pegrev URI)

    rev-prop          -> PROPFIND (against a revision URI)

    rev-proplist      -> PROPFIND (against a revision URI, but recursive)

    check-path        -> PROPFIND (against a pegrev URI)

    stat              -> PROPFIND (against a pegrev URI)

    get-lock          -> PROPFIND (against a public HEAD URI)


 * Complex read requests

   These RA functions are each accomplished in a single REPORT
   request/response.

   These REPORTs are not changing, except that they'll be sent against
   the "me resource" URI (!svn/me) rather than a VCC URI.  Again, we're
   eliminating all "discovery" turnarounds which used to preceed these
   requests.

   log                      -> REPORT (against a pegrev URI)

   get-dated-rev            -> REPORT (against "me resource")

   get-locations            -> REPORT (against a pegrev URI)

   get-locations-segments   -> REPORT (against a pegrev URI)

   get-file-revs            -> REPORT (against a pegrev URI)

   get-locks                -> REPORT (against a public HEAD URI)

   get-mergeinfo            -> REPORT (against a pegrev URI)

   replay                   -> REPORT (against "me resource")

   replay-range             -> pipelined REPORT requests (against "me
                               resource") on each revision in the range


* The "update" family of requests

   update
   switch
   status
   diff

   For these RA functions, the existing ra_serf strategy stays the same:

    1. Client sends custom REPORT describing state of working copy;
       it does *not* request text-deltas in response (the way ra_neon does).

    2. Server responds with a 'skeletal' editor-drive.

    3. Client pipelines bunches of GET and PROPFIND requests.


   The only changes we plan to make:

    - the REPORT happens against the new '"me resource"', rather than a
      discovered VCC URI.

    - no need to cache the !svn/ver "wcprops" in the working copy
      anymore, since our commit process has changed (see below).

    - no need to do any PROPFIND discovery of pegrev objects to fetch;
      client can construct them at will using the 'pegrev stub' it
      received when the ra_session began.


* Simple write requests

   change-rev-prop          -> PROPPATCH (against a revision URI)

   lock                     -> LOCK (against a public HEAD URI)

   unlock                   -> UNLOCK (against a public HEAD URI)


* Commit process

  This will change significantly.  The current methodology looks like:

      OPTIONS to start ra_session
      PROPFINDs to discover various opaque URIs
      MKACTIVITY to create a transaction
      for each changed object:
         CHECKOUT object to get working resource
         {PUT, PROPPATCH, DELETE, COPY} working resource
         MKCOL to create new directories
      MERGE to commit the transaction

  The new sequence looks like:

      OPTIONS to start ra_session
      POST against "me resource", to create a transaction
      for each changed object:
         {PUT, PROPPATCH, DELETE, COPY, MKCOL} against transaction resources
      MERGE to commit the transaction

  Specific new changes:

    - The activity-UUID-to-Subversion-txn-name abstraction is gone.
      We now expose the Subversion txn names explicitly through the
      protocol.

    - The new POST request replaces the MKACTIVITY request.

       - no more need to "discover" the activity URI;  !svn/act/ is gone.

       - client no longer creates an activity UUID itself.

       - instead, POST returns the name of the transaction it created,
         which can then be appended to the transaction stub and
         transaction root stub as necessary.

    - Once the commit transaction is created, the client is free to
      send write requests against transaction resources it constructs itself.

      NOTE:  this eliminates the CHECKOUT requests, and also removes
      our need to use versioned resources (!svn/ver) or working
      resources (!svn/wrk).

    - When modifying transaction resources, clients should send
      'If-match:' headers to facilitate server-side out-of-dateness
      checks.  (TODO:  value of header is probably an etag?)