When you use softwares which requires a username and password to be keyed in every time you access some resource, it becomes a pain. On the other hand if that software is capable of remembering your username and password, then it is a great advantage. But what if the username and password which is remembered by this software is stored in plaintext at some location in your system? Isn't it a security risk? Of course yes, specially when you don't know that your passwords are stored in plaintext. This was the case with subversion till 1.6.0, but now we have greater security improvements to subversion 1.6 which aids us with a lot of features to avoid such a scenario.

Warn caching of plaintext passwords

From the past subversion had capabilities of caching passwords, but in systems which does not have a good method of storing these passwords in encrypted form, subversion silently cached passwords in plaintext, which was bad, since the user is not aware of this fact, specially the new users of subversion. On one fine day when they come to know about this fact, they are disappointed. So we thought of solving this in the subversion community since this was a common problem reported by many users in the mailing list. Subversion 1.6 behaves in a different way when it is about to cache passwords in plaintext, as you can see from the following sample run,

 
 $ svn co http://localhost/svn/repos wc 
 Authentication realm: <http://localhost:80> TEST SVN repository 
 Password for 'stylesen': 
 ----------------------------------------------------------------------- 
 ATTENTION! Your password for authentication realm: 

 <http://localhost:80> TEST SVN repository 

 can only be stored to disk unencrypted! You are advised to configure 
 your system so that Subversion can store passwords encrypted, if 
 possible. See the documentation for details. 

 You can avoid future appearances of this warning by setting the value 
 of the 'store-plaintext-passwords' option to either 'yes' or 'no' in 
 '/home/stylesen/.subversion/servers'. 
 ----------------------------------------------------------------------- 
 Store password unencrypted (yes/no)? yes 
 Checked out revision 0. 
 $

Thus the user is aware that his password is cached in plaintext. What if the user decides not to store the passwords in plaintext, but don't want to get prompted each time? In such a case the user can play around with the following options in subversion servers file, ie., '~/.subversion/servers'

Globally,

 [global] 
 store-passwords = yes
 store-plaintext-passwords = yes

Per server basis, 

 [groups] 
 group1 = *.collab.net 
 othergroup = *.example.com
 
 [group1] 
 store-passwords = yes
 store-plaintext-passwords = yes
 [othergroup]
 store-passwords = no
 store-plaintext-passwords = yes 

Oh wait, all the above is specific to *NIX users, we already have mechanisms built in Subversion to cache passwords in encrypted form using wincrypt API in windows machines and Keychain services in Mac OS.

Okie, that is cool, but yet *NIX users like me are not happy, since we don't have a proper mechanism in place which stores passwords in an encrypted form. That is not true anymore, since 1.6 comes with support to cache passwords/passphrases in an encrypted form in GNOME Keyring or Kwallet depending upon the desktops they use. The password store could also be chosen with the following parameter in the subversion config file ie., '~/.subversion/config' as follows,

 [auth] 
 ### Set password stores used by Subversion. They should be 
 ### delimited by spaces or commas. The order of values determines 
 ### the order in which password stores are used. 
 ### Valid password stores: 
 ### gnome-keyring (Unix-like systems) 
 ### kwallet (Unix-like systems) 
 ### keychain (Mac OS X) 
 ### windows-cryptoapi (Windows) 
 password-stores = gnome-keyring , kwallet

GNOME Keyring

In order to enable Subversion to cache passwords in GNOME Keyring we need to pass the following parameter to the “configure” script while compiling Subversion source.

--with-gnome-keyring

The above requires GNOME Keyring libraries available in the operating system, failing which Subversion falls back to caching passwords unencrypted. Once you have Subversion binary compiled with GNOME Keyring support, the password is automatically cached in the Keyring, provided it is unlocked. CollabNet subversion binaries are compiled with GNOME Keyring support which you can use right away, to get this feature.

What if my GNOME Keyring is locked? Subversion provides a way to solve that too,

 $ svn co http://localhost/svn/repos wc 
 Password for 'default' GNOME keyring: 
 Authentication realm: <http://localhost:80> TEST SVN repository 
 Password for 'stylesen': 
 Checked out revision 0. 
 $ svn co http://localhost/svn/repos wc 
 Checked out revision 0. 
 $

KWallet

KDE users are not left alone, you can make use of KWallet in order to store passwords in encrypted form. In order to use KWallet the Subversion binaries must be compiled with the following option

 --with-kwallet

SSL client certificate passphrase caching

As we know, subversion was good at caching passwords, but it didn't had any mechanism to cache SSL client certificate passphrases, may be this was never thought, since the users were limited. The only way to avoid entering client certificate passphrases each time was to hard code it in the servers file with the parameter ssl-client-cert-pp, which is ugly! But now in 1.6 we use the same infrastructure as above to cache SSL client certificate passphrases.

 store-ssl-client-cert-pp = (yes/no )
 store-ssl-client-cert-pp-plaintext = (yes/no)

Aren't you curious to watch this in action? Here we go,

 $ svn co https://localhost/svn/repos wc 
 Authentication realm: https://localhost:443 
 Client certificate filename: /home/stylesen/stylesen.p12 
 Passphrase for '/home/stylesen/stylesen.p12': 
 ----------------------------------------------------------------------- 
 ATTENTION! Your passphrase for client certificate: 

  /home/stylesen/stylesen.p12 

 can only be stored to disk unencrypted! You are advised to configure 
 your system so that Subversion can store passphrase encrypted, if 
 possible. See the documentation for details. 

 You can avoid future appearances of this warning by setting the value 
 of the 'store-ssl-client-cert-pp-plaintext' option to either 'yes' or 
 'no' in '/home/stylesen/.subversion/servers'. 
 ----------------------------------------------------------------------- 
 Store passphrase unencrypted (yes/no)? yes 
 Checked out revision 0. 

Thus Subversion 1.6.x brings in lot of security improvements which enhances and gives a better user experience.

At CollabNet we have been discussing the differences and similarities between the Open Source community, and the Enterprise community. We want to be sensitive to that fine line between information awareness and marketing, without falling to one side or the other. Some people at CollabNet are hesitant to point to anything that costs money to the Open Source community for fear of offending, yet much of the content could indeed be very useful in shortening that learning curve.

As someone who is also helping to maintain the CollabNet site, and as Community Manager, I needed to learn what I needed to know as quickly as possible. I am not only learning to use Subversion, but I’m also learning to use the collaborative tools provided by TeamForge.You can see the tools of CEE if you look at any project on openCollabNet. We will be upgrading the site to TeamForge in the future.

Yes, we eat our own dog food, so to speak, and I’m glad. I have been impressed with the functionality and the ease of use of this platform compared to others I have used. To shorten my learning curve I have found some great training through articles, white papers, webinars, and web courses.

It is my hope that the resources I list below help to shorten your learning curve, no matter whether you want to stick with only the free materials or the ones that we charge for.

For those of you who are Subversion experts, you can stop reading here, though I value your opinion and experience and would appreciate any comments or suggestions you’d like to make. Also, some of you may not be aware of the variety collaboration tools that TeamForge provides.

Newsletters

Many bypass this option when registering for CollabNet, but the newsletter can point to useful information about learning Subversion, as well as other content you might not want to miss. You can subscribe by clicking on your user name at the top of the CollabNet site. Once in your profile, scroll down and click the Newsletter box. Or you can read the archived newsletters.

Webinars

Webinars are a great way of seeing software in action, and hearing information about the tools. Be sure to be logged into the site so you don't need to fill in any forms.

• What's New in Subversion 1.6
• Subversion Best Practices
• CollabNet TeamForge The Power of Centralization

Discussion Forums

Discussion forums are an excellent place to ask questions that fellow developers and users can answer. Be sure to return the favor by answering the questions when you can provide solutions based on your own experience. You'll need to be logged in to participate.

• Subversion for Admins
• Subversion End Users & Developers

Release Notes, Data Sheets, and Articles

We also have some good notes, data sheets, and articles. Be sure you are logged in so you needn't fill out contact information.

• Subversion 1.6
• Browsing a Subversion or CVS Repository
• CollabNet Subversion Datasheet
• CollabNet Training for Subversion

Training Courses

CollabNet offers a complete, role-based training curriculum across the entire CollabNet product line. Courses are delivered in a number of formats to best meet your education needs.

Subversion 1.5 - Individual Modules:

• Subversion 1.5 for Developers - Section 01 - Introduction to Version Control (10 min) - $28
• Subversion 1.5 for Developers - Section 02 - Introducing Subversion (19 min) - $28
• Subversion 1.5 for Developers - Section 03 - Global Revisioning and Working Copies (21 min) - $38
• Subversion 1.5 for Developers - Section 04 - Standard Work Cycle (33 min) - $38
• See More . . .

Instructor Led Courses

• Subversion 1.6 for Developers - Standard
• Subversion 1.6 for Developers - Enterprise
• Subversion 1.6 for the 1.x Developer
• Subversion 1.6 for Administrators
• See More . . .

FAQs

FAQs are probably one of the best free resources you can read for learning about Subversion.

• Subversion Client FAQ
• Subversion Server FAQ
• Subversion Migration FAQ

As I discover more resources for learning Subversion and TeamForge, I will write new blogs, and add to the newsletter as I discover.

Enjoy your Subversion learning path!

I added a post on my personal blog yesterday about support for Subversion being available for the just released Eclipse 3.5/Galileo release. Today I am going to show just how easy it is to install support for Subversion in Eclipse 3.5.

While I think that installing plugins in Eclipse is generally a pretty easy process, my views on this are colored by the fact that I have been doing it since before Eclipse 1.0 came out. That said, the Eclipse Mylyn team has made the process super-easy in the Eclipse 3.5 release.

I started by downloading one of the Eclipse packages from the download site. I chose the Eclipse IDE for Java Developers, just make sure you get one of the options that includes Mylyn.

After unzipping the bundle and starting Eclipse, look for the Mylyn Task List view and click on the New Task button as shown here:



This brings up the following Mylyn wizard. Note the new option to install more connectors:

After clicking the "Install More Connectors" button you are presented with a list of connectors that Mylyn can install. In the list of connectors that has been certified by Tasktop, you will see the CollabNet Desktop for SVN. The CollabNet Mylyn connectors, including Subclipse and our graphical merge client, have been certified by the team at Tasktop.  That means that these plugins play well with others and can be properly integrated into a number of Eclipse-based IDE's. Subclipse is the only Subversion plugin to have been certified by Tasktop.



I checked the box for the CollabNet Desktop for SVN and click Finish.

Three clicks, and I have started the process for installing Subversion support into Eclipse, it does not get much easier than that!

At this point, the rest is handled by the install magic from the Mylyn team. They run through the Eclipse install mechanism to verify and install your selections. I was doing this just after the launch of Eclipse 3.5 and I am sure the mirrors were getting hammered. Whatever the process does initially to validate the dependencies to verify the install took a few minutes, but eventually the following dialogs came up and it is just a matter of clicking through the wizard to complete the install. For completeness, here are those dialogs:

Confirm your selections and click Next:

I expanded the selection, so that you can see all of the components that will be installed.  You get the CollabNet Desktop, which allows you to connect to any CollabNet hosted site.  Subclipse, and its required components.  The Subversion revision graph feature that I have blogged about previously, and the CollabNet Merge client, which makes merging easy and powerful from the Eclipse environment.

Accept the licenses. All of these products are free and open-source, licensed under the EPL.

And finally, just click Yes to restart Eclipse.  That is it, not only do you have a working Subversion client in Eclipse, you have one with all of the bells and whistles as well as one that has been certified by Tasktop.

One Caveat!

The above install was done on Windows Vista 32-bit.  If you are running on another operating system, you must have the Subversion 1.6.x native libraries, including the JavaHL library available.  CollabNet provides an installer for OSX that includes this library, as well as a Linux client RPM that should work on any Linux distro.  Windows 64-bit users can install the SlikSVN package.  All of this information and more is available in this FAQ about JavaHL.

We've been publishing the releases for the Subversion Server and Client v1.6.2 for various platforms over the past week, and now the release is ready for multiple platforms:

• Windows
• Solaris
• Red Hat
• MAC

There are many exciting fixes and updates in this release, a few of which I've listed below:

• Improvement to memory usage with

  svn merge

• Resolve naming collisions with

  static stat() function in svnserve

• Avoid unnecessary server query during reverse merges

See all changes specific to each version.

Subversion 1.6 is a superset of all previous Subversion releases, and is considered the current "best" release. Any feature or bugfix in 1.0.x through 1.5.x is also in 1.6, but 1.6 contains features and bugfixes not present in any earlier release.

Download Subversion 1.6.2

Binaries for CollabNet Subversion 1.6.1 are now available for:

• Windows
• Red Hat Linux (32 and 64 bit)
• Solaris (SPARC and x86)

Check it out: certified CollabNet Subversion downloads

Also, from our community area, Universal SVN 1.6.1 binaries for Mac OS X (32 and 64 bit):

Unsupported Mac OS X download

As always, questions and general comments about using CollabNet Subversion go in the discussion forums.

"cmpilato ❤ sparse directories"

If I had a dollar for every time I've typed that… well, you and I could at least spring for some Fazoli's fast-food Italian. Okay, I admit that the emotion doesn't always drive me to public expression, but that in no way diminishes my fondness for this feature.

Introduced in Subversion 1.5, sparse directory support is one of a few features from that release (besides merge tracking and foreign repository merges) that I've fully integrated into my day-to-day activities. I dig organization. I tend to keep a pretty neat home directory. But I routinely work on several different pieces of software, and at any given time, I'm tracking several different development branches in each of those pieces of software. Were I not using sparse directories, my "projects" directory would look something like this:

$ ls ~/projects
subversion/		      svnbook/		 viewvc-1.0.7/
subversion-1.5.x/	      thotkeeper/	 viewvc-1.0.x/
subversion-1.6.x/	      thotkeeper-0.3.0/  viewvc-1.1.0-beta1/
subversion-http-protocol-v2/  viewvc/		 viewvc-1.1.x/
$

On the positive side of things, I could quickly update all my working copies by simply running svn update ~/projects/*.

But those of you who have command-line completion as hard-wired into your habits as I do will immediately notice that so many common directory prefixes does a useless completion environment make. And not using common prefixes? Well that's just barbaric.

Fortunately, sparse directories has given me a whole new perspective on working copy organization. Now, my projects directory contains (gasp!) only projects, and looks like this:

$ ls ~/projects
subversion/  svnbook/  thotkeeper/  viewvc/
$

Those directories are sparse checkouts of the root directories of their respective project repositories. Beneath them are (at least) "trunk" directories, probably "branch" and some of its children, and maybe even "tags" with some of its children in certain cases. svn up ~/projects/* still works, and my working copy topology matches that of the repositories.

I won't go into the details of how sparse checkouts works here — I've already documented the Subversion 1.5 implementation of them in the second edition of Version Control With Subversion (which you can read at http://svnbook.red-bean.com/en/1.5/svn.advanced.sparsedirs.html). The point of this blog post is to tell you about how Subversion 1.6 improves this feature in a key way.

In Subversion 1.6 (slated for release Any Day Now), the --set-depth parameter to svn update has grown a new value — exclude. This value tells Subversion to exclude the target from the working copy, immediately and until further notice. Prior to Subversion 1.6, if a branch I was working on was no longer of interest to me, I couldn't easily remove it from my working copy. If I simply deleted it, it would return the next time I updated the working copy. If I svn delete'd it, the branch remained as a local modification forever. (Unless, of course, I accidentally committed it, which brought a whole different sort of trouble to my doorstep. Angry peers. Pitchforks and torches. It was a mess — you don't want to go there.) The new exclusion mechanism in Subversion 1.6 is the Right Way To Do It.

Say I no longer care about what's going on some directory of one my project working copies. Maybe I don't care about the Subversion project's website any more. Well, with this new exclusion feature, I can tell Subversion to remove that directory:

$ cd ~/projects/subversion/trunk
$ svn update --set-depth=exclude www
D         www
$ ls www
ls: cannot access www: No such file or directory
$

Done deal. When I update my working copy in the future, I will not receive any changes aimed at that www directory. If I later decide that I once again care about that directory, I can "resubscribe" to it again:

$ svn update --set-depth=infinity www
A    www
A    www/links.html
A    www/testing-goals.html
…
A    www/tigris-permissions.html
A    www/webdav-usage.html
Updated to revision 36292.
$

Note that if you exclude a versioned directory that has some unversioned files in it, or some files with local modifications, Subversion handles this situation gracefully. All the files that aren't safe to delete, Subversion leaves around, and of course leaves any intermediate directories required to reach those files, too.

I hope this enhancement serves you as well as it has served me. What do you think? How are you using sparse directories to better organize your life?

The Subversion project released the first publicly available release candidate for the 1.6.0 release on Friday Feb. 20.  You can download the source for this release candidate from the Subversion project on tigris.org.  The release notes for the 1.6 release are still being assembled but you can follow the latest state of the document from the project website.

As we did with the Subversion 1.5.0 release, CollabNet is providing binary packages of this release candidate to make it as easy as possible for the community to evaluate the release and provide their feedback.  You can download binaries for Windows and Linux right now.  We will be adding Solaris and OSX binaries soon.

Follow this blog for more entries on specific features in Subversion 1.6.  There are several new features as you can see in the release notes.  The biggest new feature is arguably the support for tree conflicts.  This will need a separate post of its own to explain, perhaps more than one.  For those that cannot wait, I can only point to the folder with the notes the developers were keeping on the feature.  Not all of those files represent what finally went into the feature so keep that in mind and keep an eye out for more posts.

This past summer, via my role as project owner for the Subclipse project, I decided to participate as a mentor in the Google Summer of Code program.  Summer of Code is a great way to bring some visibility to your project as well as build relationships with other open source developers.  One of the first things you have to do when participating in Summer of Code as a mentor organization is to come up with some ideas for projects that students can work on for the summer.  It needs to be something interesting enough to attract talented students and at the same time it ought to be something where a measurable amount of work can be accomplished in just one summer.  I came up with a number of ideas, but the one I knew someone would go after, and the one I hoped would get done, was to add a revision graph feature.

One of the top requests I hear from new Subversion users is the desire for a good revision graph feature.  Unfortunately the way that Subversion stores information in its repository is not conducive to providing a feature like this.  Specifically, Subversion does not currently track "copy-to" information.  It can tell you the history of a given item and trace that history back through all of the places it was "copied-from", but for any given path/revision pair it cannot tell you if it has been copied to another location.  This is an essential feature to create a proper revision graph, so in order to draw a decent graph, you pretty much need to have the entire history of the repository available so that you can construct the "copy-to" information yourself.  Performance would be terrible if you had to get this history every time, so a good local cache of the information is essential.  This is a fairly challenging engineering task and therefore was one that the existing Subclipse team was unlikely to ever find the time to do.  At the same time, it was a great task for a motivated student.  Combined with the ability to visualize the information in cool ways via the rich graphical library that Eclipse provides, I knew we would attract some interest.

We were fortunate enough to attract a great student, Alberto Gimeno.  He dove right into the task when the Summer of Code program started and did a great job delivering the feature.  I was hoping we would come out of this with some interesting code that someone else might want to pick up and finish, but Alberto was able to drive this feature to a point that it was nearly ready to ship when the summer ended.  We have just spent a few months adding some Eclipse-polish to it and integrating it into Subclipse.  Today we release it, and this is what a simple graph looks like:

Notice that the graph is capable of showing merge information when used with Subversion 1.5.  It also contains a number of nice features made available via the Eclipse Graphical Editing Framework.  Namely, the ability to zoom in/out, export to an image file as well as the ability to navigate a large graph via the Outline view.

This feature will be included in an upcoming release of the CollabNet Desktop - Eclipse Edition.  You can get it today via the "dev-builds" update site for the CollabNet Desktop.  Download information for the CollabNet Desktop can be found at the project home here.

Additional details on the revision graph feature can be found at the Subclipse site.

I would like to close by once again thanking Alberto for the work he did on this feature.  I look forward to his continued involvement in this feature and the Subclipse project.  I would also thank Google for running Summer of Code and including Subclipse in the program for 2008.

This post covers several topics related to merging with Subversion 1.5.  The goal is to explain the problem of reflective merges, how the new reintegrate option helps, and also some problems that currently exist with that option.

Reflective/Cyclic Merges

This is easiest to explain with an example.  Suppose you are working on a feature branch copied from your trunk.  During the development process you regularly merge ''all'' new changes from trunk to your branch so that the branch stays in "synch" with the work occurring on trunk.  When you eventually merge your branch back to trunk, that is called a reflective (or cyclic) merge.  This type of merge is problematic for Subversion, let's look at why.

Revisions and Merge Tracking

Subversion is based on revisions.  Every commit to the repository creates a new revision and a merge is ultimately just a commit.  Subversion 1.5's merge tracking feature is all about recording which revisions were merged to which paths.  In the previous reflective merge example, this is too coarse of a solution to always yield the right results.  Recall that in our example we regularly merged changes from trunk to our branch.  Each of those merges winds up as a commit (revision) on our branch.  When it comes time to merge our branch back to trunk, all merge tracking can do is help decide whether to include or exclude those "synched" revisions.  The problem is that neither answer is always right.

If we exclude the revisions that were merges from trunk, then we also exclude any work we did to resolve conflicts as part of those merges.  Even worse, we might have carelessly committed unrelated changes as part of the merge (it happens) so those changes would also be excluded.

If we include those synched revisions, then we merge back changes that already exist in trunk.  This yields unnecessary and confusing conflicts.

The only way to truly solve this is to invent a new merge algorithm in Subversion that does not rely simply on revisions.  In discussing the problem it was suggested that this might require designing a new repository format as well.  As part of the 1.5 development cycle some preliminary work was done to explore a new algorithm within our existing design.  Ultimately we decided a new algorithm was not in scope for this release and to ship without it (the release cycle was long enough as is).  We hope to revisit this work for 1.6 or a future release to see if it pans out as a solution.

Reintegrate to the Rescue

Setting aside the issue of new algorithms for a moment, it turns out Subversion already has a technique that does the right thing with reflective merges.  It is the one we used prior to 1.5 and is called a 2-URL merge.  Going back to our original example, suppose the last time we merged trunk to our branch, we merged all the changes up to revision 100.  To properly merge our branch back to trunk, we start with a trunk working copy and run this command:

     svn merge url://trunk@100 url://feature-branch .


This tells Subversion to calculate the differences between trunk at r100 and the feature-branch at HEAD, and merge those differences into our working copy.  Since this is essentially just producing a diff, it includes our conflict resolution work, and does not include any changes that exist in both places.  In other words, we get the result we want.

The new reintegrate option is a shorthanded version of the 2-URL merge.  With it you can just run this command:

     svn merge --reintegrate url://feature-branch .


Internally, when you use this option, it calculates the url://trunk@100 part and then executes the EXACT SAME merge API that the 2-URL merge does.  This is important to remember when discussing some of the problems with this new option, because you can generally use the old 2-URL syntax to resolve the problem.  In other words, reintegrate is just a new syntax plus some safety checks (more on this later).  If these checks fail and cannot easily be corrected, then you can use the older syntax.

One big problem in general with the 2-URL merge process is that it will do whatever you tell it.  You can very easily make mistakes that the command won't catch.  Suppose we left off the @100 in our example and that r101 and r102 were committed to trunk since we last synchronized with it.  When the 2-URL merge runs, it calculates the differences between trunk@HEAD and branch@HEAD.  To the merge process, it would look like the branch removed the changes that happened in r101 and r102 and the merge would remove them from your working copy!  Depending on the change in those revisions, you might not notice this and wind up committing the removal of those changes as part of the merge.  There are a lot of other variations on this problem (such as getting the URL wrong), but most of those problems are at least more obvious and you could just revert the merge.

Because of this problem with the 2-URL merge process, when the reintegrate option was added we decided it should do some safety checks before it will run.  Some of these are fairly tame, such as making sure the working copy is at a single revision, with no switched children, and is not a sparse checkout (i.e. working copy depth is infinity).  If reintegrate errors out because of any of these problems, you can generally just use svn update or switch to "fix" your working copy.  Since reintegrate needs to calculate the base URL and revision for the merge, it also does an ancestry check to ensure the merge source is related to the merge target.

The most controversial reintegrate check is that the merge source does not have any subtree mergeinfo.  Mergeinfo (technically the versioned property svn:mergeinfo) stores merge tracking information.  Normally mergeinfo is set only on the merge target.  Subtree mergeinfo occurs when a merge target has some subtree that was previously a merge target itself (e.g. we merged from a file in trunk to a file in our branch, creating mergeinfo on that file.  This type of merge is fittingly called a subtree merge).  When reintegrating the branch back to trunk the reintegrate safety checks fail because the branch has subtree mergeinfo.  The reason for this check is that a 2-URL merge in this situation doesn't always give the right results.

Problems with Reintegrate and Renamed Files

Most of the problems with reintegrate stem from this check for subtree mergeinfo.  When this check fails, the error you see on the command line is something like this:

svn: Cannot reintegrate from 'url://feature-branch' yet:
Some revisions have been merged under it that have not been merged
into the reintegration target; merge them first, then retry.


The biggest problem is that, unlike when the other checks fail, both this problem itself and the way out of it are less obvious.  If you really did some subtree merges then these checks save you from making a mistake.  Unfortunately there is a very common case people are running into where it only ''appears'' like a subtree merge was performed: When you rename/move something, mergeinfo is created on the new path and this blocks reintegrate from working.  The reasons why copy/move create mergeinfo are beyond the scope of this post, but suffice it to say that we think in the majority of cases the mergeinfo needn't be created.  So the fix we will look at (hopefully in time for 1.5.1) is to be smarter about when copy/move creates mergeinfo.  If those subcommands minimize their creation of mergeinfo, then that would greatly reduce the occurrence of this specific problem when using reintegrate (at least for your future copy/move operations).

If you are running into this problem today, there are a couple of things you can do.

1. You might want to just manually remove the subtree mergeinfo.  You can do this by running the command:
   

   svn propdel svn:mergeinfo FOO


   You could run this command after the move, and before you commit, so that the problem never exists in the first place.  Or it can be run later to fix the problem after you realize you have it.

   Generally, the only time mergeinfo needs to be created is when copying/moving a path such that its nearest parent with mergeinfo is different in the path's source and destination.  This is because a path without explicit mergeinfo (when a path has the svn:mergeinfo property set on it it is said to have explicit mergeinfo) simply inherits the mergeinfo of its nearest parent with explicit mergeinfo.  In other words, the svn:mergeinfo property is inheritable and if a copied path inherits mergeinfo from the ''same'' place in both its source and destination, then there is nothing gained by setting explicit mergeinfo on the destination and you can safely delete that mergeinfo.

   In many cases, all mergeinfo exists on the root of your branch.  If you move a path within the branch, then its nearest parent with mergeinfo (the root of the branch) does not change and the path does not need new mergeinfo.  If however, you previously did a subtree merge and then move a path from outside the subtree into the subtree (or vice-versa) then the path inherits different mergeinfo.  In this case the destination path should keep its explicit mergeinfo.
   

2. You can also use the old 2-URL merge syntax in this situation.
   

Additional Reintegrate Problems

Closely related to the previous problem is when subtree mergeinfo is merged into branches or carried into branches when the branch is created (rather than being created in the branch as above).  If you rename a path in trunk (which currently creates mergeinfo), every branch you then create from trunk from then onward also has this subtree mergeinfo.  Alternatively, if you copy trunk to a branch, then copy a path within trunk (again creating mergeinfo), then synch up your branch, then the branch get's subtree mergeinfo merged into it.  In either case this effectively means you can never use reintegrate, at least not unless you can delete the svn:mergeinfo property from the subtree's that have it.

So in addition to being smarter about creating the mergeinfo, we need to examine whether reintegrate can safely ignore subtree mergeinfo in some cases and allow the reintegrate merge to proceed.  For example, what if we create a branch from trunk and some subtree mergeinfo comes into the branch when it is created.  As long as the subtree mergeinfo that exists on the branch remains equivalent with what is in trunk, it's probably safe to allow a reintegration merge.  Of course these sorts of checks start to get real complicated so we are less sure there are easy solutions here, but hopefully there are some that might make their way into 1.6.  One solution that might be worth considering is whether we ought to do this check in the first place?

Branch Management and Reintegrate

An important thing to point out, and I am not sure if the documentation currently does, is that once a branch is reintegrated, it should really be deleted.  If more work needs to happen, create it again with a fresh copy.  There are two reasons for this:

1. Remember our example with reflective merges?  Once you merge your branch to trunk, if you later want to synch all the changes with trunk back to the branch, you are now in this same reflective merge scenario with your branch!  You can no longer use the simple "svn merge url://trunk"  syntax you were using previously so you have to use 2-URL merges to get your trunk changes into your branch.
2. If you do not merge trunk changes back to your branch because of the previous point, then when you go to reintegrate again, it will still see r100 (to use our example), as the last trunk revision merged to the branch.  So when it diffs trunk@100 with your current branch, it will not "see" that in r103 you merged the branch to trunk and it will try to merge everything again.

The bottom line is that if you recreate your branches you will not run into this.  svn delete followed by svn copy runs fast, and does not take up any repository disk space.  You do not even have to do anything special to your working copy.  So just do it!

Future Solutions

This post has largely been focused on some problems we have been seeing and were aware of when we released 1.5.  I wanted to get this post out so that we can aid users in understanding the root of the problems.  That said, I do not want to leave you with an impression of "doom and gloom".  The fact is that overall the new 1.5 merge process works quite well and is a substantial improvement over previous versions.  And while the current reintegrate solution for reflective merges can only take us so far without a new merge algorithm, if you understand the process, you can develop a workflow that largely avoids the noted limitations.  There are also a few other other merge bugs we didn't cover here but fixes for these are targeted for the the upcoming 1.5.1 release (which is tentatively scheduled for release late this month).

It is also worth noting that now that merge tracking is "in the wild", the development community has real world data and use cases to work with, making it a lot easier to evaluate new algorithms and bug fixes.  This should help us get fixes out on the 1.5.x line more quickly.  The lack of large, real world repositories using merge tracking was a problem during the development of 1.5.0 as those simply did not exist (outside of Subversion's own repository).

Finally, I just want to point out that I have turned off comments on this post.  I suspect this post will raise a number of questions and I'd just prefer to handle these in a proper discussion forum.  I have created a thread to post comments and questions related to this post in the Subversion forum on openCollabNet.  Please post your comments and questions to that thread.

Chapter 4 of the forthcoming edition of the O'Reilly book "Version Control with Subversion" provides a good introduction to merge tracking with Subversion 1.5.  It describes the common scenario of feature branches copied from a trunk, which are then kept up to date with trunk via periodic synchronization, and at the end of their lifetime are reintegrated back to trunk (with the aptly named --reintegrate option).  If this model fits your merging needs, then after reading chapter 4 it is unlikely you will encounter any surprises.  But what if you don't follow this model?  What if you need to do some or all of the following:

• Merge to shallow working copies?

• Merge to targets with switched subtrees?

• Merge from sources which you don't have full access to because of authorization restrictions, or merge to targets missing subtrees for the same reason?

• Perform multiple merges to the same target without committing in-between?

• Merge to target, then merge to subtrees of that target?

Not to worry, in all of these cases Subversion attempts to "do the right thing", but you may encounter seemingly unusual behavior that doesn't fit within the 'synch and reintegrate' paradigm.  Key to understanding all of these cases is a knowledge of the new svn:mergeinfo property (frequently referred to simply as 'mergeinfo').

Mergeinfo is simply the history of merges made into a path.  But as with many things that can be described so simply, mergeinfo can become a bit more complex in practice.  Initially I had brave hopes of writing a reasonably sized blog post talking about mergeinfo; what it means and how it works.  Unfortunately as the page count of my draft post drifted past twenty (after cutting a lot of material out) and a co-worker began to refer to it as the "mergeinfo manifesto", we decided that it was better suited as an article - which you can find here.

So please take a look.  Even if you only ever do the simplest of merges, a quick read of the first few sections of the article and the "Parting Thoughts" section will help you in your move to Subversion 1.5.

Read the article: Subversion 1.5 Mergeinfo.