notes/entries-caching

                    "I have a cunning plan"

                             or

             Entries Caching in the Access Batons



0. Preamble
   --------

Issue 749 provides some history.  The access batons now cache the
parsed entries file, as repeatedly reading, parsing and writing the
file proved to be a bottleneck.


1. Caching Interface
   -----------------

The basic functions to retrieve entries are svn_wc_entries_read and
svn_wc_entry.  The function svn_wc__entries_write is used to update
the entries file on disk.  The function svn_wc__entry_modify is
implemented in terms of entries_read and entries_write.

1.1 Write Caching Overview

An overview of the update process.

   1. Lock the directory
   2. Read the entries file and cache in memory
   3. Start the wc update
      3.1  Start a directory update
         3.1.1 Start file update
            3.1.1.1 Write a log file specific to this item
         3.1.3 Finish file update
      3.2. Finish directory update
      3.3. Run log files
         3.3.1. Log file commands modify entries in memory
      3.4  Finish log files
      3.5. Flush entries to disk
      3.6. Remove log files
   4. Finish update
   5. Unlock directory

Each directory update may contain multiple file updates so when the
directory update is complete there may be multiple log files.  While
the log files are being run the entries modifications are cached in
memory and written once when the log files are complete.  The reason
for accumulating multiple log files is that flushing the entries to
disk involves writing the entire entries file, if it were done after
each file then the total amount of entries data written would grow
exponentially during a checkout.


2. Interface Enhancements
   ----------------------

2.1 Entries Interface

A lot of the entries interface has remained unchanged since the
pre-caching days, and it shows.  Of particular concern is the
svn_wc_entries_read function, as this provides access to the raw data
within the cache.  If the application carelessly modifies the data
things may go wrong.  I would like to remove this function.

One use of svn_wc_entries_read is in svn_wc__entry_modify, this is
"within the entries code" and so is not a problem.

Of the other uses of svn_wc_entries_read the most common is where the
application wants to iterate over all the entries in a directory. I
would like to see an interface something like

  typedef struct svn_wc_entry_iterator_t svn_wc_entry_iterator_t;

  svn_wc_entry_iterator_t *
  svn_wc_entry_first(svn_wc_adm_access_t *adm_access,
                     apr_pool_t *pool);

  svn_wc_entry_iterator_t *
  svn_wc_entry_next(svn_wc_entry_iterator_t *entry_iterator);

  const svn_wc_entry_t *
  svn_wc_entry_iterator_entry(svn_wc_entry_iterator_t *entry_iterator);

Note that this provides only const access to the entries, the
application cannot modify the cached data.  All modifications would go
through svn_wc__entry_modify, and the access batons could keep track
of whether modifications have been made and not yet written to disk.

The other uses of svn_wc_entries_read tend to extract a single entry.
I hope these can be converted to use svn_wc_entry.  One slight problem
is the use of svn_wc_entries_read to intentionally extract a
directory's entry from its parent.  This is done because that's where
the "deleted" state is stored.  I think the entry returned by
svn_wc_entry could contain this state.  Why doesn't it?  I don't know,
possibly it's an accident, or possibly it's intentional as in the past
parsing two entries files would have been expensive.

2.2 Access Baton Interface

I would also like to modify the access baton interface.  At present
the open function detects and skips missing directories when opening a
directory hierarchy.  I would like to record this information in the
access baton set, and modify the retrieve functions to include an
svn_boolean_t* parameter that gets set TRUE when a request for a
missing directory is made.  The advantage of doing this is that the
application could avoid making svn_io_check_path and svn_wc_check_wc
calls when the access baton already has the information.  The function
prop_path_internal looks like a good candidate for this optimisation.


3. Access Baton Sets
   -----------------

Each access baton represents a directory.  Access batons can associate
together in sets.  Given an access baton in a set, it possible to
retrieve any other access baton in the set.  When an access baton in a
set is closed, all other access batons in the set that represent
subdirectories are also closed.  The set is implemented as a hash
table "owned" by the one baton in any set, but shared by all batons in
the set.

At present in the code, access batons are opened in a parent->child
order.  This works well with the shared hash being owned by the first
baton in each set.  There is code to detect if closing a baton will
destroy the hash while other batons are using it, as far as I know it
doesn't currently trigger.  If it turns out that this needs to be
supported it should be possible to transfer the hash information to
another baton.


4. Access Baton Conversion
   -----------------------

Given a function
  svn_error_t *foo (const char *path);
if PATH is always a directory then the change that gets made is usually
  svn_error_t *foo (svn_wc_adm_access_t *adm_access);
Within foo, the original const char* can be obtained using
  const char *svn_wc_adm_access_path(svn_wc_adm_access_t *adm_access);

The above case sometimes occurs as
  svn_error_t *foo(const char *name, const char *dir);
where NAME is a single path component, and DIR is a directory. Conversion
is again simply in this case
  svn_error_t *foo (const char *name, svn_wc_adm_access_t *adm_access);

The more difficult case is
  svn_error_t *foo (const char *path);
where PATH can be a file or a directory.  This occurs a lot in the
current code. In the long term these may get converted to
  svn_error_t *foo (const char *name, svn_wc_adm_access_t *adm_access);
where NAME is a single path component.  However this involves more
changes to the code calling foo than are strictly necessary, so
initially they get converted to
  svn_error_t *foo (const char *path, svn_wc_adm_access_t *adm_access);
where PATH is passed unchanged and an additional access baton is
passed.  This interface is less than ideal, since there is duplicate
information in the path and baton, but since it involves fewer changes
in the calling code it makes a reasonable intermediate step.


5. Logging
   -------

As well as caching the other problem that needs to be addressed is the
issue of logging.  Modifications to the working copy are supposed to
use the log file mechanism to ensure that multiple changes that need
to be atomic cannot be partially completed.  If the individual changes
that may need to be logged are all forced to use an access baton, then
the access baton may be able to identify when the log file mechanism
should be used.  Combine this with an access baton state that tracks
whether a log file is being run and we may be able to automatically
identify those places that are failing to use the log file mechanism.


6. Status
   ------

Entries caching has been implemented.

The interface changes (section 2) have not been started.

The access baton conversion is complete in so far as passing batons is
concerned.  The path->name signature changes (section 4) have not been
made.

Automatic detection of failure to use a log file (section 5) has not
been started.