                Implementing Sparse Directory Support in SVN

   #########################################################################
   ###                                                                   ###
   ###  Note: This feature used to be called "incomplete directories";   ###
   ###  It is now called "sparse directories", because "incomplete"      ###
   ###  made it sound like something was wrong with your directories.    ###
   ###                                                                   ###
   #########################################################################

Contents
========

   1. Design
   2. User Interface
   3. Examples
   4. Implementation Strategy
   5. Compatability Matters
   6. Current Status

1. Design
=========

   This design document started out as a post by Eric Gillespie:

      http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=117053
      From: Eric Gillespie <epg@pretzelnet.org>
      To: dev@subversion.tigris.org
      Subject: [PROPOSAL] Incomplete working copies (issue #695)
      Date: Thu, 22 Jun 2006 22:35:06 -0700
      Message-ID: <25668.1151040906@gould.diplodocus.org>

   [The design has evolved since then; the text below is not exactly
   the same as what Eric posted, but has the same general ideas.]

   I'd like to propose a new solution to this issue, and hopefully get
   it into 1.5.  What i'm really looking for is the kind of
   flexibility Perforce has with its client specs in which parts of a
   tree you check out.

   I don't think Ben Reser's proposal
   (http://svn.haxx.se/dev/archive-2005-07/0398.shtml) covers this.
   Using his first example, there is no way to avoid pulling in
   trunk/foo/images/another-big-dir when it is added.

   This is based on an idea from Karl Fogel.

           Implementing Incomplete Directory Support in SVN
          ==================================================

   Many users have very large trees of which they only want to
   checkout certain parts.  checkout -N is not today up to this task.
   This proposal introduces the --depth option to the checkout,
   switch, and update subcommands as a replacement for -N, which
   allows working copies to have very specific contents, leaving out
   everything the user does not want.

   This is similar to Perforce's client specs, but without the ability
   to have a repository entry have a different name in the working
   copy.  We actually already have this capability in switch.

   Depth:
   
   We have a new "depth" field in .svn/entries, which has (currently)
   four possible values: depth-empty, depth-files, depth-immediates,
   and depth-infinity.  Only this_dir entries may have depths other
   than depth-infinity.
   
      depth-empty ------>  Updates will not pull in any files or
                           subdirectories not already present.
   
      depth-files ------>  Updates will pull in any files not already
                           present, but not subdirectories.
   
      depth-immediates ->  Updates will pull in any files or
                           subdirectories not already present; those
                           subdirectories' this_dir entries will
                           have depth-empty.
   
      depth-infinity --->  Updates will pull in any files or
                           subdirectories not already present; those
                           subdirectories' this_dir entries will
                           have depth-infinity.  Equivalent to
                           today's default update behavior.
   
   The --depth option sets depth values as it updates the working
   copy, setting any new subdirectories' this_dir depth values as
   described above.
   
2. User interface
=================
   
   Affected commands:
   
       * checkout
       * switch
       * update
       * status
       * info
   
   The -N option becomes a synonym for --depth=files for these commands.
   This changes the existing -N behavior for these commands, but in a
   trivial way (see below).
   
   checkout without --depth or -N behaves the same as it does today.
   switch and update without --depth or -N behave the same way as
   today IFF the working copy is fully depth-infinity.  switch and
   update without --depth or -N will NOT change depth values
   (exception: a missing directory specified on the command line will
   be pulled in).
   
   Thus, 'checkout' is identical to 'checkout --depth=infinity', but
   'switch' and 'update' are not the same as 'switch --depth=infinity' and
   'update --depth=infinity'.  The former update entries according to
   existing depth values, while the latter pull in everything.
   
   To get started, run checkout with --depth=empty or --depth=files.
   If additional files or directories are desired, pull them in with
   update commands using appropriate --depth options.
   
   The 'svn status' should list the depth status of the directories, in
   addition to whatever statuses are being currently listed.
   
   The 'svn info' command should list the depth, iff invoked on a
   directory whose depth is not the default (depth infinity).
   
3. Examples
===========
   
   svn co http://.../A
   
       Same as today; everything has depth-infinity.
   
   svn co -N http://.../A
   
       Today, this creates wc containing only mu.  Now, this will be
       identical to 'svn co --depth=files /A'.
   
   svn co --depth=empty http://.../A Awc
   
       Creates wc Awc, but *empty*.
   
       Awc/.svn/entries                this_dir    depth-empty
   
   svn co --depth=files http://.../A Awc1
   
       Creates wc Awc1 with all files (i.e., Awc1/mu) but no
       subdirectories.
   
       Awc1/.svn/entries               this_dir    depth-files
       ...
   
   svn co --depth=immediates http://.../A Awc2
   
       Creates wc Awc2 with all files and all subdirectories, but
       subdirectories are *empty*.
   
       Awc2/.svn/entries               this_dir    depth-immediates
                                       B
                                       C
       Awc2/B/.svn/entries             this_dir    depth-empty
       Awc2/C/.svn/entries             this_dir    depth-empty
       ...
   
   svn up Awc/B:
   
       Since B is not yet checked out, add it at depth infinity.
   
       Awc/.svn/entries                this_dir    depth-empty
                                       B
       Awc/B/.svn/entries              this_dir    depth-infinity
                                       ...
       Awc/B/E/.svn/entries            this_dir    depth-infinity
                                       ...
       ...
   
   svn up Awc
   
       Since A is already checked out, don't change its depth, just
       update it.  B and everything under it is at depth-infinity,
       so it will be updated just as today.
   
   svn up --depth=immediates Awc/D
   
       Since D is not yet checked out, add it at depth-immediates.
   
       Awc/.svn/entries                this_dir    depth-empty
                                       B
                                       D
       Awc/D/.svn/entries              this_dir    depth-immediates
                                       ...
       Awc/D/G/.svn/entries            this_dir    depth-empty
       ...
   
   svn up --depth=empty Awc/B/E
   
       Remove everything under E, but leave E as an empty directory
       since B is depth-infinity.
   
       Awc/.svn/entries                this_dir    depth-empty
                                       B
                                       D
       Awc/B/.svn/entries              this_dir    depth-infinity
                                       ...
       Awc/B/E/.svn/entries            this_dir    depth-empty
       ...
   
   svn up --depth=empty Awc/D
   
       Remove everything under D, and D itself since A is depth-empty.
   
       Awc/.svn/entries                this_dir    depth-empty
                                       B
   
   svn up Awc/D
   
       Bring D back at depth-infinity.
   
       Awc/.svn/entries                this_dir    depth-empty
                                       ...
       Awc/D/.svn/entries              this_dir    depth-infinity
                                       ...
       ...
   
   svn up --depth=immediates Awc
   
       Bring in everything that's missing (C/ and mu) and empty all
       subdirectories (and set their this_dir to depth-empty).
   
       Awc/.svn/entries                this_dir    depth-immediates
                                       B
                                       C
       Awc/B/.svn/entries              this_dir    depth-empty
       Awc/C/.svn/entries              this_dir    depth-empty
       ...
   
4. Implementation Strategy
==========================
   
   It would be nice if all this could be accomplished with just simple
   tweaks to how we drive the update reporter (svn_ra_reporter2_t).
   However, it looks like it's not going to be that easy.
   
   Handling 'checkout --depth=empty' would be easy.  It should get us
   an empty directory at depth-empty, with no files and no subdirs,
   and if we just report it as at HEAD every time, the server will
   never send updates down (hmmm, this could be a problem for getting
   dir property updates, though).  Then any files or subdirs we have
   explicitly included we can just report at their respective
   revisions, and get proper updates; at least that'll work for the
   depth infinity ones.
   
   But consider 'checkout --depth=immediates'.  The desired state is a
   depth-immediates directory D, with all files up-to-date, and with
   skeleton subdirs at depth empty.  Plain updates should preserve this
   state of affairs.
   
   If we report D as at its BASE revision, files at their BASE
   revisions, and subdirs at HEAD, then:
   
      - When new files appear in the repos, they'll get sent down (good)
      - When new subdirs appear, they'll get sent down in full (bad)
   
   But if we don't report subdirs as at HEAD, then the server will try to
   update them (bad).  And if we report D at HEAD, then the working copy
   won't receive new files that have appeared in the repository since D's
   BASE revision (note that we *can* get updates for files we already
   have, though, by continuing to report them at their respective BASEs).
   
   The same logic applies to subdirectories at depth-files or
   depth-immediates.
   
   So, I think this means that for efficient depth handling, we'll
   need to have the client directly reporting the desired depth to the
   server; i.e., extending the RA protocol.
   
   Meanwhile, legacy servers will send back a bunch of information the
   client doesn't want, and the client will just ignore it, and
   everything will be slower than it needs to be, and people will
   complain on the users@ list, and we'll tell them to upgrade their
   servers, and they'll say they can't because they don't have control
   over the server, and we'll say "So?  This ain't no Grand Hotel!"

5. Compatability Matters
========================

   This feature introduces two new concepts into the RA protocol which
   will not be understood by older servers:

      * Reported Depths -- the depths associated with individual paths
        included by the client in the description (via the
        svn_ra_reporter_t) of its working copy state.  

      * Requested Depth -- the single depth value used to limit the
        scope of the server's response to the client.
        
   As such, it's useful to understand how these concepts will be
   handled across the compatability matrix of depth-aware and
   non-depth-aware clients and servers.

   NOTE: in the sections below, it is not necessarily that case that a
   value or state which is said to be "transmitted" literally has a
   presence in the RA protocol.  Some such bits of state have default
   values in the protocol and can therefore be effectively transmitted
   while not literally identifiable in a network trace of the
   client-server traffic.

   Depth-aware Clients (DACs)

      DACs will transmit reported depths (with "infinity" as the
      default) and will transmit a requested depth (with "unknown" as
      the default).  They will also -- for the sake of older,
      non-depth-aware servers (NDASs) -- transmit a requested recurse
      value derived from the requested depth:
   
         depth        recurse
         -----        -------
         empty        no
         files        no
         unknown      yes
         immediates   yes
         infinity     yes

      When speaking to an NDAS, the requested recurse value is the
      only thing the server understands , but is obviously more
      "grainy" than the requested depth concept.  The DAC, therefore,
      must filter out any additional, unwanted data that the server
      transmits in its response.  (This filtering will happen in the
      RA implementation itself so the RA APIs behave as expected
      regardless of the server's pedigree.)

      When speaking to a depth-aware server (DAS), the requested
      recurse value is ignored.  A requested depth of "unknown" means
      "only send information about the stuff in my report,
      depth-aware-ily".  Other requested depth values are honored by
      the server properly, and the DAC must handle the transformation
      of any working copy depths from their pre-update to their
      post-update depths and content as described in `3. Examples'.

   Non-depth-aware Clients (NDACs)

      NDACs will never transmit reported depths and never transmit a
      requested depth.  But they will transmit a requested recurse
      value (either "yes" or "no", with "yes" being the default).  (A
      DAS uses the presence of a requested depth in the actual protocol
      to distinguish DACs from NDACs, and knows to ignore the
      requested recurse value transmitted by a DAC.)

      When speaking to an NDAS, what happens happens.  It's the past,
      man -- you don't get to define the interaction this late in the
      game!

      When speaking to a DAS, the not-reported depths are treated like
      reported depths of "infinity", and the reported recurse values
      "yes" and "no" map to depths of "infinity" and "files",
      respectively.

6. Current Status
=================

   The sparse-directories code is merged to trunk in revision r23994.

   A new enum type 'svn_depth_t depth' is defined in svn_types.h.
   Both client and server side now understand the concept of depth,
   and the basic update use cases handle depth.  See depth_tests.py
   for what is known to be working.  (Many cases are not yet tested,
   and almost certainly some of them will fail right now.)

   On the client side, most of the svn_client.h interfaces that
   formerly took 'svn_boolean_t recurse' now take 'svn_depth_t depth'.
   (The -N option is deprecated, but it still works: it simply maps to
   --depth=files, which results in the same behavior as -N used to.)

   Some of this recurse-becomes-depth change has propagated down into
   libsvn_wc, which now stores a depth field in svn_wc_entry_t (and
   therefore in .svn/entries).  The update reporter knows to report
   differing depths to the server, in the same way it already reports
   differing revisions.  In other words, take the concept of "mixed
   revision" working copies and extend it to "mixed depth" working
   copies.

   On the server side, most of the significant changes are in
   libsvn_repos/reporter.c.  The code that receives update reports now
   receives notice of paths that have different depths from their
   parent, and of course the overall update operation has a global
   depth, which applies whenever not shadowed by some local depth for
   a given path.

   The RA code on both sides knows how to send and receive depths; the
   relevant svn_ra_* APIs now take depth arguments, which sometimes
   supersede older 'recurse' booleans.  In these cases, the RA layer
   does the usual compatibility dance: receiving "recurse=FALSE" from
   an older client causes the server to behave as if "depth=files"
   had been transmitted.

   Work remaining:

   The list of outstanding issues is shown by this issue tracker query
   of Summary prefixed with [sparse-directories]:

<http://subversion.tigris.org/issues/buglist.cgi?component=subversion&issue_status=NEW&issue_status=STARTED&issue_status=REOPENED&short_desc=%5Bsparse-directories%5D&short_desc_type=casesubstring>