notes/wc-ng/externals - subversion - Git at Google

                                                                 -*- Text -*-

  Externals handling in wc-ng
  ===========================

 A long time ago, we introduced svn:externals in Subversion as a solution to
 using multiple independent components.

 When updating we process svn:externals changes, by walking over all directories
 that have (or had) svn:externals properties defined on them.

 Then we take all the old and the new definitions and put them in a list. And
 then we start comparing them, following this schema

   *  For all directories that have and/or had svn:externals
      +  For all definitions that were and/or are defined
         -  If it does not exist locally
            o  If the definition is not removed: Checkout
         -  If it is unchanged: update it
         -  If the definition is just removed, remove the external
         -  If the definition was modified
            o  If it is switchable within a repository: switch
            o  Else: remove and then reinstall the new definition
         -  If the definition is new: checkout

 (There are some small variations around relative externals, unmatched externals
  and revision locked externals, but this is the primary idea)

 This schema allows a read-only working copy to fully follow svn:externals
 changes without any user interaction as long as any intermediate update is
 followed.

 For an incoming update on 'svn update' this is easy, as we have the old and the
 new versions of the property and we can just run the external update schema.


 Ok, pretty straightforward, but things get much harder once you start looking
 at how to change svn:externals definitions locally.


  Handling changes to svn:externals
  =================================

 The easiest way to handle this, is how we do it today (r1102626). We only apply
 svn:externals changes when they are applied to the BASE layer. In this case we
 can follow the original external update schema.

 But this is not what any normal subversion user likes, because you have to
 commit a potentially invalid external description just to test it. To avoid
 this hassle you would like to prefer changes before committing, but then the
 external update schema doesn't work any more.

 The external update schema needs the difference between an old and a new
 definition to update your working copy, but with local property changes you
 only have access to the last version and probably some older version, but not
 to the previous version.

 [[
 > If you do
 > Assuming you committed
 > $ svn propset svn:externals A .
 > $ svn up .
 > (updates externals to match definition nothing->A)
 >
 > $ svn ci -m""
 > $ svn propset svn:externals B .
 > $ svn update .
 >
 > Then currently this updates the directory to the definition A (just to be
 > safe), while you (and many others) would like it to go to B.
 >
 > So assume the preferred scenario this would do the update A->B
 >
 > But then you have the even likely case
 > $ svn propset svn:externals C
 > $ svn update .
 >
 > This would then apply A->C, so this could potentially break your working
 > copy, by leaving traces of B.
 >
 > We really need the step B->C here, but there is no way to access B, because
 > it wasn't committed. It just lived in a local property change.
 ]]

 Before Subversion 1.6, it wasn't such a big problem when the external update
 schema broke down on the working copy where svn:externals were edited. You just
 deleted the subdirectory and ran svn update again. But then we introduced file
 externals....

 "We really need some store with old (applied) svn:externals definitions, so we
  would only have to update externals from that to the latest definition"
 (Issue #2267, #3351, #3823)


  Wc-1.0 File Externals
  =====================

 Just before we released Subversion 1.6 somebody noticed that you could switch
 an added file (really a bug) and that you could use that to do svn:external
 like things with that. Within a few days some support to manage these externals
 was added and the file externals as we know them were born.

 These file externals were added as part of the working copy where they are
 located with a special registration in their svn_wc_entry_t. (Note: this didn't
 have to be the same working copy as where the definition lived).
 One noticable limitation was that they had to be from the same repository as
 the directory they are placed in.

 The existing externals update schema handled did this really well and file
 externals were released as part of Subversion 1.6.


 But when you mix the feature of editing svn:externals and file externals,
 things get ugly. If you add a file external to svn:external; then perform an
 update and then revert the definition of the file external before committing,
 you will find that this file external will still be in your working copy.
 (And there is no real way to fix this).


  The 'file externals' problem
  ============================

 When we introduced file externals, we expected this to be easy to maintain,
 as essentially we already supported switched files.

 But then somebody noticed that you could delete file externals via 'svn rm',
 and then the external was removed... but not where it was placed, but at
 the place where it was added from. So we added some tests to detect that
 specific case. We never had that problem with directory externals.

 (Same problem with moving files)

 Then somebody noticed that merge was recording more merge information when
 you had file externals. So we added some tests to detect and work around that.

 (And I think about 20 similar cases in different places of our code).

 We never expected any of this when we introduced file externals as an
 easy feature.

 [[
 <@cmpilato> A file external -- like any other external -- shouldn't be an add at all in a copy situation.
 <@Bert> cmpilato: But a file external is not like 'any other external'... It is like 'any other switch'
 <@cmpilato> That mindset is 90% of the problem with file externals.
 <@cmpilato> They were never intended to behave as switches.
 <@cmpilato> Switch was simply the low-hanging mechanism that was used to shoehorn them in.
 <@cmpilato> They were always intended to behave like dir externals.
 <@cmpilato> hrm.  we don't appear to be honoring the --ignore-externals option to 'svn cp WC WC' either...
 ]]

 "Why can't file externals be more like 'normal' externals?"
 (Issue #3589, #3518, #3351, #3665, #3816, #3843)


  The WC-NG Externals store
  =========================

 After holding back several changes to more file externals corner cases on
 "we should really design file externals before adding more features" these
 three questions got me thinking:

     "We really need some store with old (applied) svn:externals definitions, so
      we would only have to update externals from that to the latest definition"

     "Why can't file externals be more like 'normal' externals?"

 And later (via private mail):
     "Will this allow to exclude externals from WC? (Use-case: watching repo
      of public project with number of externals)"

 So we have three separate requests to store some information on externals,
 which we couldn't store before.

 - Handling externals changes -

 For every applied external we would like to have:
   * Where it was defined
   * What is its definition
   * If it has a fixed revision or not.
 This allows applying svn:externals changes from any previous state to the last
 state by just comparing the actual propery values against what is stored.

 - Moving file externals into their own storage -

 If we want to make file externals like 'normal externals, we should remove
 their presence from their parent working copy and just handle them as
 independent filesystem objects.

 For that we need some storage location of 'all the relevant information it
 would have in the past'.
   * Repository, repos_relpath, revision.
   * Presence (always status normal. Can't be deleted, moved, etc.)
   * Kind (always file or symlink)
   * Properties
   * Checksum
   * Changed date, author, revision
   * Recorded size and mod time

 - Excluding externals -

 To allow excluding externals we need some kind of presence flag containing
 'normal' and 'excluded' per external.


 ### Combine these three ideas and you have the EXTERNALS table for format 29
	-- Text --

	Externals handling in wc-ng
	===========================

	A long time ago, we introduced svn:externals in Subversion as a solution to
	using multiple independent components.

	When updating we process svn:externals changes, by walking over all directories
	that have (or had) svn:externals properties defined on them.

	Then we take all the old and the new definitions and put them in a list. And
	then we start comparing them, following this schema

	* For all directories that have and/or had svn:externals
	+ For all definitions that were and/or are defined
	- If it does not exist locally
	o If the definition is not removed: Checkout
	- If it is unchanged: update it
	- If the definition is just removed, remove the external
	- If the definition was modified
	o If it is switchable within a repository: switch
	o Else: remove and then reinstall the new definition
	- If the definition is new: checkout

	(There are some small variations around relative externals, unmatched externals
	and revision locked externals, but this is the primary idea)

	This schema allows a read-only working copy to fully follow svn:externals
	changes without any user interaction as long as any intermediate update is
	followed.

	For an incoming update on 'svn update' this is easy, as we have the old and the
	new versions of the property and we can just run the external update schema.


	Ok, pretty straightforward, but things get much harder once you start looking
	at how to change svn:externals definitions locally.


	Handling changes to svn:externals
	=================================

	The easiest way to handle this, is how we do it today (r1102626). We only apply
	svn:externals changes when they are applied to the BASE layer. In this case we
	can follow the original external update schema.

	But this is not what any normal subversion user likes, because you have to
	commit a potentially invalid external description just to test it. To avoid
	this hassle you would like to prefer changes before committing, but then the
	external update schema doesn't work any more.

	The external update schema needs the difference between an old and a new
	definition to update your working copy, but with local property changes you
	only have access to the last version and probably some older version, but not
	to the previous version.

	[[
	> If you do
	> Assuming you committed
	> $ svn propset svn:externals A .
	> $ svn up .
	> (updates externals to match definition nothing->A)
	>
	> $ svn ci -m""
	> $ svn propset svn:externals B .
	> $ svn update .
	>
	> Then currently this updates the directory to the definition A (just to be
	> safe), while you (and many others) would like it to go to B.
	>
	> So assume the preferred scenario this would do the update A->B
	>
	> But then you have the even likely case
	> $ svn propset svn:externals C
	> $ svn update .
	>
	> This would then apply A->C, so this could potentially break your working
	> copy, by leaving traces of B.
	>
	> We really need the step B->C here, but there is no way to access B, because
	> it wasn't committed. It just lived in a local property change.
	]]

	Before Subversion 1.6, it wasn't such a big problem when the external update
	schema broke down on the working copy where svn:externals were edited. You just
	deleted the subdirectory and ran svn update again. But then we introduced file
	externals....

	"We really need some store with old (applied) svn:externals definitions, so we
	would only have to update externals from that to the latest definition"
	(Issue #2267, #3351, #3823)


	Wc-1.0 File Externals
	=====================

	Just before we released Subversion 1.6 somebody noticed that you could switch
	an added file (really a bug) and that you could use that to do svn:external
	like things with that. Within a few days some support to manage these externals
	was added and the file externals as we know them were born.

	These file externals were added as part of the working copy where they are
	located with a special registration in their svn_wc_entry_t. (Note: this didn't
	have to be the same working copy as where the definition lived).
	One noticable limitation was that they had to be from the same repository as
	the directory they are placed in.

	The existing externals update schema handled did this really well and file
	externals were released as part of Subversion 1.6.


	But when you mix the feature of editing svn:externals and file externals,
	things get ugly. If you add a file external to svn:external; then perform an
	update and then revert the definition of the file external before committing,
	you will find that this file external will still be in your working copy.
	(And there is no real way to fix this).


	The 'file externals' problem
	============================

	When we introduced file externals, we expected this to be easy to maintain,
	as essentially we already supported switched files.

	But then somebody noticed that you could delete file externals via 'svn rm',
	and then the external was removed... but not where it was placed, but at
	the place where it was added from. So we added some tests to detect that
	specific case. We never had that problem with directory externals.

	(Same problem with moving files)

	Then somebody noticed that merge was recording more merge information when
	you had file externals. So we added some tests to detect and work around that.

	(And I think about 20 similar cases in different places of our code).

	We never expected any of this when we introduced file externals as an
	easy feature.

	[[
	<@cmpilato> A file external -- like any other external -- shouldn't be an add at all in a copy situation.
	<@Bert> cmpilato: But a file external is not like 'any other external'... It is like 'any other switch'
	<@cmpilato> That mindset is 90% of the problem with file externals.
	<@cmpilato> They were never intended to behave as switches.
	<@cmpilato> Switch was simply the low-hanging mechanism that was used to shoehorn them in.
	<@cmpilato> They were always intended to behave like dir externals.
	<@cmpilato> hrm. we don't appear to be honoring the --ignore-externals option to 'svn cp WC WC' either...
	]]

	"Why can't file externals be more like 'normal' externals?"
	(Issue #3589, #3518, #3351, #3665, #3816, #3843)


	The WC-NG Externals store
	=========================

	After holding back several changes to more file externals corner cases on
	"we should really design file externals before adding more features" these
	three questions got me thinking:

	"We really need some store with old (applied) svn:externals definitions, so
	we would only have to update externals from that to the latest definition"

	"Why can't file externals be more like 'normal' externals?"

	And later (via private mail):
	"Will this allow to exclude externals from WC? (Use-case: watching repo
	of public project with number of externals)"

	So we have three separate requests to store some information on externals,
	which we couldn't store before.

	- Handling externals changes -

	For every applied external we would like to have:
	* Where it was defined
	* What is its definition
	* If it has a fixed revision or not.
	This allows applying svn:externals changes from any previous state to the last
	state by just comparing the actual propery values against what is stored.

	- Moving file externals into their own storage -

	If we want to make file externals like 'normal externals, we should remove
	their presence from their parent working copy and just handle them as
	independent filesystem objects.

	For that we need some storage location of 'all the relevant information it
	would have in the past'.
	* Repository, repos_relpath, revision.
	* Presence (always status normal. Can't be deleted, moved, etc.)
	* Kind (always file or symlink)
	* Properties
	* Checksum
	* Changed date, author, revision
	* Recorded size and mod time

	- Excluding externals -

	To allow excluding externals we need some kind of presence flag containing
	'normal' and 'excluded' per external.


	### Combine these three ideas and you have the EXTERNALS table for format 29