Chapter 4 of the forthcoming edition of the O'Reilly book "Version Control with Subversion" provides a good introduction to merge tracking with Subversion 1.5.  It describes the common scenario of feature branches copied from a trunk, which are then kept up to date with trunk via periodic synchronization, and at the end of their lifetime are reintegrated back to trunk (with the aptly named --reintegrate option).  If this model fits your merging needs, then after reading chapter 4 it is unlikely you will encounter any surprises.  But what if you don't follow this model?  What if you need to do some or all of the following:

• Merge to shallow working copies?

• Merge to targets with switched subtrees?

• Merge from sources which you don't have full access to because of authorization restrictions, or merge to targets missing subtrees for the same reason?

• Perform multiple merges to the same target without committing in-between?

• Merge to target, then merge to subtrees of that target?

Not to worry, in all of these cases Subversion attempts to "do the right thing", but you may encounter seemingly unusual behavior that doesn't fit within the 'synch and reintegrate' paradigm.  Key to understanding all of these cases is a knowledge of the new svn:mergeinfo property (frequently referred to simply as 'mergeinfo').

Mergeinfo is simply the history of merges made into a path.  But as with many things that can be described so simply, mergeinfo can become a bit more complex in practice.  Initially I had brave hopes of writing a reasonably sized blog post talking about mergeinfo; what it means and how it works.  Unfortunately as the page count of my draft post drifted past twenty (after cutting a lot of material out) and a co-worker began to refer to it as the "mergeinfo manifesto", we decided that it was better suited as an article - which you can find here.

So please take a look.  Even if you only ever do the simplest of merges, a quick read of the first few sections of the article and the "Parting Thoughts" section will help you in your move to Subversion 1.5.

Read the article: Subversion 1.5 Mergeinfo.

For many folks, the forthcoming Subversion 1.5 release gets billed as something like "merge tracking and some other stuff". It's probably true that the work put into the new merge tracking feature outweighs the investments in other individual areas. But of course, Subversion 1.5 isn't about merge tracking alone. There are several other major features coming in this release. And as you'd expect, the Subversion community has made countless bugfixes over the past year-and-a-half, too. But today I want to briefly talk about one half-bugfix, half-feature, that might have slipped in under your radar: merges from foreign repositories.

The Subversion project has issued a beta1 release on the path to the final GA. The next release will likely be rc1 but, as always, that depends on the feedback we get from testers.

CollabNet has created binaries for Linux, Windows and Solaris (with OS X coming soon) from this beta1.  A noteworthy change as we get closer to RC1 is that we are now providing our CollabNet Subversion packaging for all three platforms. Solaris and Windows are new with this beta release. In addition, for all three of these platforms the server installation can automatically install and configure ViewVC as part of the server installation.

You can download these binaries from openCollabNet. There are also updated versions of the CollabNet Merge Client, Subclipse and TortoiseSVN available on this site.

We have a forum where you can post questions, comments or problems that you encounter using this version. If after some discussion a report is indeed deemed a defect, we forward the bug to the Subversion community using their normal processes. Feedback on the GUI merge clients will go to CollabNet Engineering.

For those that missed the announcement of the Alpha2 release, here are a few things to be aware of:

• Subversion 1.5 client has an updated working copy format. Once an SVN 1.5 client updates a working copy it will be converted to the new format. This means that older SVN clients will no longer be able to read these working copies. So either be careful, or be sure to update all your clients to the same version. There is a Python script that you can use to downgrade your working copy to the 1.4 format.
• Subversion 1.5 server can be used to serve an existing repository, but you will not be able to use the new merge tracking features with this repository until it is upgraded. You have a few options here:
1. Dump existing repository, then create a new repository using the 1.5 binaries and load the dump file.
2. Create a new repository using 1.5 binaries and use svnsync 1.5 binary to transfer the contents to the new repository.
3. Run a new command svnadmin upgrade to upgrade an existing repository. This runs fast as it just updates the internal repository format. There are some downsides to this approach though:
• SVN 1.5 repositories maintain a new index called the node-origins index  This is needed to speed up certain merge operations. The index will be lazy-populated on upgraded repositories, but can be updated on demand by running a new tool named svn-populate-node-origins-index(.exe). If you have a large repository you will likely want to populate the index. New repositories, or ones that are dumped/loaded or svnsync'ed will not need this.
• For fsfs repositories, the node-origins index will be slightly faster and take up less disk space if the repository is reloaded or synced. For existing repositories, the index has to be maintained in a separate set of files that takes up extra disk space and I/O.
• For new 1.5 fsfs repositories, there is a new sharding mechanism for storing revisions on disk. Instead of storing all revisions in one single directory, they are now spread across multiple directories which will be better on certain filesystems.  There is a Python script you can run to shard an exising fsfs repository though.
4. If you were using an earlier version of the 1.5 binaries, note that the on-disk format of the repository has been changed in the most recent builds. You should dump your repository with your current binaries, then install the new binaries, create a new repository and load the repository.

Please take the opportunity to give these beta versions on the path to GA a try. Your feedback will lead to a better final release and likely also help us deliver Subversion 1.5 as soon as possible.

The Subversion project has issued an alpha2 release on the path to the final GA. The next release might be alpha3, it might be beta1, that depends on the feedback we get from testers. One thing is certain, and that is the sooner we can find any remaining bugs, the sooner the GA release will be available.  Likewise, hearing that you have used this version with some success will also help us make decisions.

CollabNet has created binaries for Linux and Windows (with OS X coming soon) from this alpha2. You can download them from openCollabNet. There are also updated versions of the CollabNet Merge Client, Subclipse and TortoiseSVN available on this site.

We have a forum where you can post questions, comments or problems that you encounter using this version. If after some discussion a report is indeed deemed a defect, we forward the bug to the Subversion community using their normal processes. Feedback on the GUI merge clients will go to CollabNet Engineering.

A few things to be aware of:

• Subversion 1.5 client has an updated working copy format.  Once an SVN 1.5 client updates a working copy it will be converted to the new format. This means that older SVN clients will no longer be able to read these working copies. So either be careful, or be sure to update all your clients to the same version. There is a Python script that you can use to downgrade your working copy to the 1.4 format.
• Subversion 1.5 server can be used to serve an existing repository, but you will not be able to use the new merge tracking features with this repository until it is upgraded. You have a few options here:
1. Dump existing repository, then create a new repository using the 1.5 binaries and load the dump file.
2. Create a new repository using 1.5 binaries and use svnsync 1.5 binary to transfer the contents to the new repository.
3. Run a new command svnadmin upgrade to upgrade an existing repository. This runs fast as it just updates the internal repository format. There are some downsides to this approach though:
• SVN 1.5 repositories maintain a new index called the node-origins index  This is needed to speed up certain merge operations. The index will be lazy-populated on upgraded repositories, but can be updated on demand by running a new tool named svn-populate-node-origins-index(.exe). If you have a large repository you will likely want to populate the index. New repositories, or ones that are dumped/loaded or svnsync'ed will not need this.
• For fsfs repositories, the node-origins index will be slightly faster and take up less disk space if the repository is reloaded or synced. For existing repositories, the index has to be maintained in a separate set of files that takes up extra disk space and I/O.
• For new 1.5 fsfs repositories, there is a new sharding mechanism for storing revisions on disk. Instead of storing all revisions in one single directory, they are now spread across multiple directories which will be better on certain filesystems.  There is a Python script you can run to shard an exising fsfs repository though.
4. If you were using an earlier version of the 1.5 binaries, note that the on-disk format of the repository has been changed in the most recent builds. You should dump your repository with your current binaries, then install the new binaries, create a new repository and load the repository.

Please take the opportunity to give these alpha versions on the path to GA a try. Your feedback will lead to a better final release and likely also help us deliver Subversion 1.5 as soon as possible.

Over the past months we blogged a lot about Subversion 1.5 and its new capabilities. People often ask: “Can you show us the changes from 1.4.x to 1.5 on the command line?”

I’ve created a PDF that answers this question by analyzing the built-in documentation of Subversion 1.5-dev and comparing it to the documentation of Subversion 1.4.x. The changes made to Subversion's  subcommands reflect most of the new features. 

The results of my investigation are in the Merge Tracking Early Adopter Program (the SVN 1.5 beta). Look for the document titled: "Subversion 1.5 Command Line Changes". All new features of Subversion 1.5-dev are highlighted in blue (when necessary, limitations of the AS400 port are mentioned as well).

I’m currently working on a document that describes and compares all new, obsolete and unchanged switches of the Subversion 1.5-dev subcommands. Please tell me what you think about the command-line-comparison approach and whether I should provide you a comparison for svnadmin, svnlook, svndumpfilter and svnsync as well.

A lot of people asked us about TortoiseSVN and its plans for merge tracking. For those of you who do not know, TortoiseSVN is a Subversion client for Windows users. It plugs into the Windows Shell and makes Subversion client commands available through right-click options within Windows Explorer and any of the open/save dialogues. TortoiseSVN is a great tool that rightfully won a Sourceforge.net Community Choice Award 2007. You can download it from openCollabNet and it is also part of CollabNet’s stack of supported integrations for Subversion.

I use TortoiseSVN myself all the time so I too was curious to find out about the TortoiseSVN plans for merge tracking. Some information is on their web site but I contacted the project lead for TortoiseSVN, Stefan Küng, and asked about the status. Here is his reply and some screenshots (thanks Stefan):

Merge tracking is usually done by the Subversion library without the user knowing about it. I.e., the user doesn't see that Subversion skips already merged revisions when merging from trunk to a branch. That's why there's not really much done in TSVN UI-wise. Where we thought we have to show the user more information, we did. Otherwise we tried to hide the specifics as well as possible.

What we did for now:

The log dialog, when started from the merge dialog to select revisions to merge shows the already merged revisions in grey instead of black. This indicates to the user that the grey revisions shouldn't be merged (again).

The log dialog shows merged revisions indented, with each indentation level being a merge level. This UI may change before the final 1.5 release because I'm not really happy with the way the indentation looks right now - it's hard to see the different levels. I may add some vertical lines there or do something else to improve the indentation.

When showing blames, merged lines are shown with an italic font, and the user can optionally show the merged paths of every line.

Nightly builds are on the TortoiseSVN web site but they might not always work with the Subversion 1.5 pre-release binaries currently downloadable from the Merge Tracking Early Adopter Program. Next week we will update the Subversion 1.5 binaries to the latest development status and we’ll post a TortoiseSVN build that works with our latest Subversion build.

One more thing: On October 30th CollabNet will do another webinar on Subversion 1.5, covering advanced branching and merging with Subversion 1.5. The webinar follows up on the one from a month ago about branching strategies and answers many of the questions people asked. Register here.

Two of the questions most asked during the webinar “Branching and Merging with Subversion 1.5” were:

• “What are the differences between Subversion 1.5's merge tracking features and svnmerge.py?”
• “Will there be a migration utility?”

I’m in Munich at the moment and today attended CollabNet’s Subversion Power Workshop that precedes SubConf, the first conference dedicated to Subversion. Answers to both questions were covered during the merge tracking sessions.

Before talking about the advantages of Subversion 1.5 merge tracking over svnmerge.py, I think it is important to say that svnmerge.py has been a very useful tool contributed and maintained by Subversion community members. We are all thankful for their contributions. That said, Subversion 1.5 does offer several advantages over svnmerge.py:

• SVN 1.5 tracks merge history at a per-path level of granularity and tracks file level merges. svnmerge.py only tracks merge info for the entire branch, it does not track finer levels (e.g.: files). This allows SVN to implement cherry picking, which svnmerge.py does not support.
• svnmerge.py doesn’t always properly merge merge-properties (like svnmerge-integrated and svnmerge-blocked), leading to property conflicts when merging across multiple branches. SVN 1.5 solves this.
• SVN 1.5 includes support for third-party tools, such as call-backs for conflict resolution. If a merge conflict arises, Subversion can call your GUI client and the client can then fire up a conflict resolution tool. This improves workflow and usability. You can see the integration between SVN 1.5 merging and GUI clients in action by trying CollabNet’s new GUI Merge Client.
• SVN’s merge tracking includes more comprehensive auditing, like querying the merge history to answer questions about what was merged when and where.

A utility to migrate svnmerge.py merge tracking history from a previous version to Subversion 1.5 was committed to the Subversion trunk just a few days ago. What can it do?

• Convert the latest merge history for multiple branches. In terms of history, information about the latest merges for each of your branches is the most important information because it allows for an easier merge the next time you merge. If you want to keep things simple and only migrate the latest merge history, you can.
• The migration utility can also scan your entire repo for merge history, or accept a series of path prefixes to look at: svnmerge-migrate-history.py /path/to/repos trunk tasks branches.

We’ve uploaded a copy of the script to the download section of the Merge Tracking Early Adopter Program. You can download it and test it out on your own repository (not for production please, Subversion 1.5 is still under development). It should work well but tell us if you run into any problems. We know of one limitation: don’t run it on a Subversion 1.5 repository that already has merge history, this has not been tested yet (then again: why would you want to do that?). Right now, only the .py script is available but the Subversion developers plan to release a Windows executable later.

One of the most common questions that occurs whenever a new release of Subversion (or any product) comes out, has to do with the considerations you need to take into account before upgrading.  I will try to cover some of those in this post.

Subversion's compatibility guidelines guarantee that any 1.x client can talk to any 1.x server.  So you do not have to upgrade the client and server at the same time.  The 1.5 client can talk to 1.0-1.4 servers and any 1.0-1.4 clients can talk to a 1.5 server.

Those same guidelines say that you never have to reload a repository when upgrading the server.  You can upgrade your server from 1.0-1.4 to 1.5 and you do not need to do anything to your repositories.  The new server will silently make any updates that are needed, when they are needed.  In the case of 1.5, a new SQLite database file will be added to the repository to store indexes that speed up the new merge tracking function.  Even though you do not have to reload your repository, there are often small efficiencies you can gain by doing a dump/load of your repository.  In the case of this release, there is a new storage mechanism for fsfs repositories.  Instead of storing all of the revisions in a single folder, we now "shard" the repository by creating a sub-directory for each 1000 revisions. This does not improve performance of Subversion, but it might improve performance of things like backup utilities or other tools you use to work with the file system on the server.  In this case, you do not have to dump/load to get this feature, there is a Python script you can run that will do the sharding for you.  This script will run relatively quickly, certainly orders of magnitude faster than a dump/load.

Some features of Subversion 1.5 will require that you have a 1.5 server and a 1.5 client to enjoy the full benefits of the feature.  For example, the merge tracking feature requires that both the client and server are running 1.5.  If one of them is running a lower version than you get the merge functionality that existed in that version of Subversion.  If you want to take advantage of merge tracking you really need to update your server and all of the clients that are doing merges.  Doing a merge with an old client will work, but will not set any of the merge tracking information when the change is committed.  This simply means that someone using a 1.5 client later will attempt to merge the same info again and get conflicts.  The svn merge --record-only command can be used to fix this after the fact.

When using the new sparse-checkouts functionality it really helps to have a 1.5 server.  The feature will work with an older server, but in most cases the server will still send the extra information (file contents) to the client and the client has to discard what it does not want.

There is an important client-side compatibility feature to take into account.  Some of the new features of 1.5 such as sparse-checkouts and changelists necessitate a new client working copy format number.  So this means as soon as you perform an operation that needs to re-write your working copy (checkout/update/switch/commit/propset etc.) then the internal format of the working copy will be upgraded to the new format for Subversion 1.5.  Once this happens, you will no longer be able to use a 1.4 client with that same working copy.  The same sort of change occurred with the 1.3 to 1.4 upgrade process.  We know that this really impacted many users that were stuck using an older client, so we are including a Python script that you can run to downgrade a working copy back to the 1.4 format.  We will distribute this as an EXE for Windows users.  Just to make this perfectly clear, you can have a 1.4 client and 1.5 client on the same system.  You just need to be careful to not mix their usage on the same working copy.  Also, pure-read operations like (status/info/ls) do not update the working copy.  So simply viewing a working copy with a new version of TortoiseSVN or SCPlugin installed does not modify the working copy format.  Conversely, however, an old version of one of these tools will not be able to read a 1.5 working copy.

Hopefully this answers the questions about upgrading to Subversion 1.5.  It is going to be a great release and I certainly hope people choose to upgrade when it comes out.  If you have additional questions that I did not cover, please post them in the comments or on the Subversion Users forum on openCollabNet.

Hopefully you have already read my post from last week, Subversion merge made easy.  In that post, I linked to information on the new CollabNet Merge client that we are building for Subversion 1.5.  There has been a lot of good feedback from users who have downloaded and tried that client.

This week, we are introducing a preview of an extension to the merge client that allows you to perform merges based on change sets.  With this extension to the merge client, users of CollabNet Enterprise Edition's Project Tracker can perform merges by specifying Project Tracker artifacts as the input to the merge process instead of having to specify a list of revisions.  This is a really easy way to perform certain types of merges and can lead to some interesting new workflows and business processes.

If you have already installed the merge client, you can just visit the same installation site and grab the change set extension.  I have also written an overview of the client that has some more details and screenshots.  Click here for the overview of the change set extension.

While you will need Subversion 1.5 on your server to get the full benefits of merge tracking, the bulk of the features of this change set extension can be used today with your version of CollabNet Enterprise Edition.  So give it a try and let us know what you think.

It has been a busy few months for the Subversion engineering team at CollabNet and the Subversion community in general. As we detailed in many posts this summer, the entire team is hard at work on Subversion 1.5, specifically the merge tracking feature. 

Back in the spring we started the Merge Tracking Early Adopter Program on openCollabNet. We wanted to make it as easy as possible to get people to try early versions of the merge tracking code and provide feedback to CollabNet and the Subversion community. The program has been pretty successful and provided some valuable feedback that was incorporated into the merge tracking deliverables for Subversion 1.5. Last week we refreshed the Subversion 1.5 Beta binaries on the merge tracking site to a more recent version in order to spawn a new round of testing. If you have not already downloaded them, please download them and give them a try. It is not too late to help the community deliver the 1.5 release.

The title of this post is "Subversion merge made easy" and that brings me to an announcement of sorts. While all of the work on Subversion 1.5 has been going on, another team at CollabNet has been working on a new GUI merge client for Subversion. Today we are releasing a beta version of this client as part of the Merge Tracking Early Adopter Program. A lot of work went into this client and there are many features. Rather than listing them here, let me instead link to the documentation on the merge tracking site.  The documents explain how to install the client and give an in depth overview of its features. I encourage you to download the client and give it a try. Use the forum of the merge tracking project to provide feedback, ask questions, report problems etc. If you try the client, you will automatically also test the Subversion 1.5 merge tracking feature, which can only help to accelerate its release.

Finally, since we are talking about merge and making it easier, I just want to also mention an upcoming webinar that is being presented by two of my colleagues at CollabNet -- Bob Jenkins and Auke Jilderda. They have both contributed a lot of ideas and feedback to the merge tracking feature as well as assisted in putting together the merge tracking early adopter program on openCollabNet. Their webinar is sure to be stocked with valuable insights on branching and merging in Subversion.

Subversion 1.4.5 was released today.  You can download the updated CollabNet Subversion binaries immediately.

Subversion 1.4.5 contains a fix for a security exploit on Windows clients. This exploit was discovered and reported by researchers at the Colorado Research Institute for Security and Privacy.

The only change from Subversion 1.4.4 is the patch for this security exploit.  Since the exploit only affects Windows clients, we decided to only release CollabNet Subversion 1.4.5 packages for Windows. There is no point for someone who is already running 1.4.4 on any other operating system to update to 1.4.5.

I am not going to give a lot of details about the exploit, you can find more information at various security reporting sites, such as CVE.  I will say that it was a legitimate exposure that made it possible for the Subversion client to write files outside the normal working copy.  That being said, there are a couple of points to make:

1. Creating the exploit requires commit access to the repository.  If you can trust the people who have write access to the repository, then you do not have too much to be concerned about. The keyword in that sentence is "trust". If you are checking out from a repository you cannot completely trust, such as on a public hosting service, then be careful and update to 1.4.5 first.
2. While the exploit itself is pretty easy to produce, it is also pretty difficult to use it in a way that would cause harm.
3. You can only create the exploit from a non-Windows platform.
4. There is nothing terribly secretive about the exploit.  If you send commit emails, or even just browse your repository using svn ls, this exploit would stand out as not normal.

If you are running a Subversion client on Windows, this would include the command line client as well as any graphical client such as TortoiseSVN or Subclipse, then you should definitely go ahead and install this version of Subversion.  I would recommend that users of earlier versions such as 1.3.2 or 1.2.3 also install this update immediately. The Subversion 1.4.5 client can talk to any 1.x version of the server, so there is no reason not to update your client (for compatibility: if you have the command line and a GUI client, update them both).

Subversion servers are not affected by this exploit.  That being said, a Windows server that uses the Subversion client in scripts would still be vulnerable and should be updated to 1.4.5.

Now that the Mac community at openCollabNet is providing complete, up-to-date installers for command-line Subversion, what about GUI access?  This is the Macintosh, after all!  There are, of course, several stand-alone GUI clients, a standard integration into Apple’s Xcode, and plugins for Eclipse, JDeveloper, NetBeans, and other applications, but what about basic Finder access?  Windows users have had TortoiseSVN for years, and another Tigris project, naughtysvn, is working toward similar capabilities for the Nautilus desktop environment on Linux.  Why not the Mac?

“Why not,” indeed!  Introducing SCPlugin: Subversion for the rest of OS X!

Over at tigris.org, the SCPlugin project has been working away steadily to bring you full Subversion capabilities as a plug-in to the Finder itself — your entire desktop is Subversion-enabled.  We’ve just released version 0.7, a beta release we believe is full-featured enough to support your day-to-day work. 

So, what’s an SCPlugin?

We add “badges” to all the Finder icons for files and directories that are under Subversion’s control.  The badges tell you the status of the file: freshly checked out, modified, in conflict, and so on.

We also extend the Finder “Contextual Menu” (you know … control-click or click-and-hold-for-a-real-long-time, or right-click if you have a multi-button mouse) to have Subversion commands.  The available commands are applied to the files and directories you have selected when you pop up the menu.

With this plug-in, your OS X Finder joins the world of Subversion.

Some technical notes: Works in OS X 10.3.9 and 10.4.x.  Uses Subversion 1.4.x (don’t mix with other clients still using Subversion 1.3.x).  This is a beta version: some things probably don’t work.  Join the users@scplugin.tigris.org mail list for help with problems or use notes.

Jack Repenning, CollabNet CTO and SCPlugin community lead

If you are a regular reader of this blog then you are undoubtedly aware that Subversion 1.5 will be coming out later this year and that the big feature is merge tracking. Technically, merge tracking is a largely internal feature with the exception of the ways in which it simplifies the overall UI of the merge process. Implementing this feature has caused us to spend a lot of time just thinking about the entire merge process and this has led to some additional improvements that are not technically part of the merge tracking feature. That brings us to what this post is about and that is how Subversion 1.5 will make it easier than ever for you to resolve conflicts.

We all make mistakes. Wouldn't it be great if life always gave you a second chance or an opportunity to correct those mistakes? Fortunately when you are using Subversion you can always correct a mistake, in most cases, you can correct those mistakes quite easily.

Here at CollabNet, we get a fairly constant stream of questions about Subversion from a wide variety of folks and via several different communication vehicles (webinar participation, training sessions, and so on). We try to answer them all as quickly as possible, but sometimes we wind up with a backlog of unanswered questions. So, as has been done on this blog before, I'll try to answer some of those queued up questions, focusing this time on questions related to Subversion's locking feature.

One quality of a typical engineer is a heightened sensitivity to potential problem areas. For software engineers, that often means spotting possible bugs in our peripheral vision while working on some nearby chunk of source code. This is both a blessing and curse. You want your bugs fixed, but unfortunately we sometimes get derailed from our primary task by these little excursions into Perfection.

Another fact of a developer's life is that there are always more things to work on than time to do so. And everything is of the utmost priority to someone. And so here again, we find ourselves working on multiple things at the same time, making slow-yet-incremental progress on each of them so at least we can say about them all with a clear conscience and a forced smile, "As it turns out, I'm working on that right now."

To my managerial readers, I regretfully admit that I have absolutely no solution for those problems. After all, this is a technical blog. But I can (and will) tell you about an upcoming Subversion feature that might reduce the context-switching costs for those developers: changelists.

As you probably know, Subversion 1.4.4 was recently released.  I wanted to let you know that the CollabNet Subversion team is hard at work putting together our packages for 1.4.4.  We will release them in a staged order: Linux first, Solaris second and Windows third.  Most of the time and effort in our release process is spent in QA and validation and the reason we do the Linux and Solaris packages first is that we can get those versions in the hands of our QA team the quickest.  Eventually, I plan on making the order Linux, Windows, Solaris, as there are more Windows than Solaris users who are seeking the packages.

The Linux package is in the final stage of our validation process and I expect it to be available on openCollabNet later this week.  The Solaris package should be available next week and Windows the following week.

It’s worth pointing out that the most significant fix in 1.4.4 is a potential (though very unlikely) repository corruption with BDB repositories.  Since CollabNet recommends FSFS and CollabNet Subversion comes pre-configured with that back-end, this BDB fix does not affect CollabNet Subversion users.

We had a really nice addition to the Subversion merge tracking feature committed to trunk last week.  Hyrum Wright is a student participating in the Google Summer of Code program.  He has signed up to provide "Merge Auditing" features for merge tracking.  You can see some details on what this means in the functional specifications, but essentially it is about adding intelligence and awareness of the merge tracking information to the svn log and blame output.  In the merge tracking planning, this has been slated as a post-1.5 feature.  It is a feature everyone really wants to see in the code-base, but it was just felt that shipping the release and the core functionality had to be the priority.

Anyway, in this case it appears we will be able to have our cake and eat it too.  Hyrum, whom I should add is already a full Subversion committer, has made great progress on adding this support to the svn log command and has committed those changes to trunk.  Additionally, it turns out that the sample repository we created for our Merge Tracking Early Adopter program really helped to work out the kinks in this feature.  Since we had a nicely documented sample repository, and matching graphic, that Hyrum could refer to, it made it easier for him to refine the feature and get it working properly.

The above picture is the history of our sample repository. If you take a close look, the r14 commit to trunk is an interesting example where this feature comes into play.  This commit is a merge from a branch, and the merge also contains additional merges from another branch.  If I run this command (note the new -g option):

svn log -g -r 14 $REPOS/trunk

Here is the new output:

------------------------------------------------------------------------
r14 | merger | 2007-05-30 15:48:11 -0400 (Wed, 30 May 2007) | 3 lines

Merge branch b - product roadmap and update about page

Command executed: svn merge $REPOS/branches/b
------------------------------------------------------------------------
r13 | buser | 2007-05-30 15:46:48 -0400 (Wed, 30 May 2007) | 1 line
Result of a merge from: r14

Update info about our company
------------------------------------------------------------------------
r12 | merger | 2007-05-30 15:45:19 -0400 (Wed, 30 May 2007) | 3 lines
Result of a merge from: r14

Merge branch a - product roadmap

Command executed: svn merge $REPOS/branches/a
------------------------------------------------------------------------
r11 | auser | 2007-05-30 15:43:00 -0400 (Wed, 30 May 2007) | 1 line
Result of a merge from: r14, r12

Add product roadmap
------------------------------------------------------------------------

Note how it includes information on the revisions that were merged in r14, and additionally how one of those revisions was the result of another merge.  Graphical clients like our CollabNet Desktop - Eclipse Edition and TortoiseSVN should be able to do really useful and interesting presentations of this information in their UI.

This is fantastic work Hyrum.  Keep it up.  I cannot wait to see the blame implementation come together.

Those of you who diligently watch your e-mail Inbox for those little blessings that come via announce@subversion.tigris.org recently got a pleasant surprise, though perhaps not the one you'd been wishing for. It's been nearly five months since Subversion's last release — nine months since the introduction of the 1.4 version line. "Where's the next big Subversion release?" "What's taking so long!?" "I want Merge Tracking, and I want it yesterday!"

Don't worry, Subversion 1.5 remains on schedule. Merge Tracking development is still ongoing, as is support for sparse directories and a number of other improvements and features. Those are, of course, in addition to all of the other goodies that 1.5 promises to bring which are already completed. But along the way of developing Subversion 1.5, we found enough important issues in 1.4.3 to justify another patch release on this line.

Subversion 1.4.4 delivers a number of bug fixes for various issues, including (but not limited to):

• an extremely rare directory property dataloss bug which, for BDB-backed repositories, also unfortunately results in repository corruption
• a race condition when changing revision properties in FSFS-backed repositories
• a low-risk security flaw in the 'svn prop* --revprop' subcommands
• problems with 'svn diff' when writing large diff hunks to the Windows console
• a path sorting bug in the output of 'svnadmin dump' for non-ASCII paths which can cause invalid dumpfile generation
• a hang bug triggered during character translation

Now, some of those items look sorta scary. Data loss, repository corruption, race conditions — that's the stuff a technologist's nightmares are made of! But without downplaying the significance of the fixes to those problems, allow me to assure you that the instances of them have thus far been extremely rare. Still, you are encouraged (as always) to upgrade to this latest release of Subversion.  You can see the full set of changes made in 1.4.4 in the Subversion CHANGES file.

And if you're one of those folks dying to get your hands on Merge Tracking functionality, let me encourage you to join the openCollabNet Merge Tracking Early Adopter Program. The more feedback the Subversion developers get on their work, the better — and more timely — the feature will be overall.

I thought I would take a break from my previous technical posts and give a general update about what has been going on in the land of CollabNet Subversion (our Subversion binaries).

We have been busy the last several weeks and the fruits of our labor are starting to see the light of day. First, a couple of weeks ago we released our Subversion 1.4.3 client and server RPM's for Red Hat Enterprise Linux. This was big news for us because it was our first 1.4.3 release and it had been bugging us that we were lagging behind. Producing the binaries is easy, but we created new packaging as well as additional scripts that we provide with the RPM. This took some extra time but now our build environment is complete and going forward it will be easier for us to stay in synch with the open source community.

Earlier this week, we released our Subversion 1.4.3 client and server packages for Solaris 10 SPARC. This was our first Solaris release and again it felt good to get it out the door. We spent some extra time making sure the build works the way we want it to but we sorted out any issues and the release sailed through our QA process. I know that I was pleased when I tested the Solaris version, installing and getting a server up and running is very easy. The extra time we spent paid off and like Linux we can now produce new releases fast.

Also this week, we helped kick-off a new area on openCollabNet that is providing "Community Binaries" for Subversion. The first binary release is for Mac OS X 10.4 PPC and Intel. Jeremy Whitlock, a contributor to this blog, deserves the credit for picking up the ball on this and running with it. He did a great job in both producing the binaries and in putting a lot of quality touches into the packaging. We will have more information on our plans and aspirations for the binaries project when we get the project launched on openCollabNet (within the next two weeks or so). For now, you OS X users, go download the binaries and let us know how it goes for you.

Finally, there is the Windows installer for CollabNet Subversion. Our current version is 1.4.2 and uses InstallAnywhere, a Java-based installer that adds a lot of heft to the download size because it includes a JRE. We are moving away from that to a Windows-specific installer. We are also adding a lot of new features to the installer such as installing the server as a Windows service as part of the install process. It is taking some time to get these features implemented correctly and QA-ed but in the end it will be worth it. We should have the final version out in a couple of weeks. Most likely it will be for Subversion 1.4.4 as that should be released between now and then and creating the binaries themselves is not the issue for us.

I see a lot of questions asked about "What is the recommended repository layout?", "What does trunk mean?", or: "What is the significance of trunk?". This post will try to answer those questions and more.

A Subversion repository implements the metaphor of a versioned filesystem. The repository is just a filesystem with folders and files. It so happens that modifications to this filesystem are versioned and there are implementation enhancements like "cheap" copies that make certain operations less expensive than they are in a traditional filesystem, but the repository itself still behaves like a filesystem: there are no special folders or names and Subversion itself has no knowledge of trunk or branches, they are just folders within the filesystem. It is up to you as the user to give those folders names and structure that are meaningful to you.

That said, there are several common layouts that have been adopted by the community as best practices and therefore one could think of these as recommendations. If your repository is accessible to the public, following these conventions might make it easier for users that have accessed other Subversion repositories to find what they are looking for.

There are two commonly used layouts:

trunk
branches
tags

This first layout is the best option for a repository that contains a single project or a set of projects that are tightly related to each other. This layout is useful because it is simple to branch or tag the entire project or a set of projects with a single command:

svn copy url://repos/trunk url://repos/tags/tagname -m "Create tagname"

This is probably the most commonly used repository layout and is used by many open source projects, like Subversion itself and Subclipse. This is the layout that most hosting sites like Tigris.org, SourceForge.net and Google Code follow as each project at these sites is given its own repository.

The next layout is the best option for a repository that contains unrelated or loosely related projects.

ProjectA
   trunk
   branches
   tags
ProjectB
   trunk
   branches
   tags

In this layout, each project receives a top-level folder and then the trunk/branches/tags folders are created beneath it. This is really the same layout as the first layout, it is just that instead of putting each project in its own repository, they are all in a single repository. The Apache Software Foundation uses this layout for their repository which contains all of their projects in one single repository.

With this layout, each project has its own branches and tags and it is easy to create them for the files in that project using one command, similar to the one previously shown:

svn copy url://repos/ProjectA/trunk url://repos/ProjectA/tags/tagname -m "Create tagname"

What you cannot easily do in this layout is create a branch or tag that contains files from both ProjectA and ProjectB.  You can still do it, but it requires multiple commands and you also have to decide if you are going to make a special folder for the branches and tags that involve multiple projects. If you are going to need to do this a lot, you might want to consider the first layout.

As for the names of the folders within the repository, again: they are just a convention. They have no special meaning to Subversion.

"trunk" is supposed to signify the main development line for the project. You could call this "main" or "mainline" or "production" or whatever you like.

"branches" is obviously supposed to be a place to create branches. People use branches for a lot of purposes. You might want to separate your release or maintenance branches from your feature branches or your customer modification branches etc. In this case, you could create a layer of folders beneath branches, or just create multiple branch folders at the top-level.

"tags" are not treated as special by Subversion either. They are a convention, perhaps enforced by hook script or authz rules, that indicate you are creating a point in time snapshot. Typically the difference between tags and branches is that the former are not modified once they are created. You might call your tag folders "releases" or "snapshots" or "baselines" or whatever term you prefer.

Remember, the significance of the name is for your benefit, not for Subversion. Finally, the architecture of Subversion, with its global revision number can often make the need for tags unnecessary. I do not think there is any reason to create tags just for the sake of creating them. If you find a need to recreate the software at a specific point in time, you can always do so by using svn log to determine the relevant revision number. Tags are best when there are "external" consumers of the repository. Maybe it is a QA/Release team that needs to perform builds, maybe it is an internal development team that wants to use releases of your code in another product, or maybe it is literally external users or customers who need to grab release snapshots from your repository. In these scenarios, creating tags is both a convenient way to be sure they get the right code, as well as a good communication mechanism to indicate the presence of release snapshots.

Hopefully this post clarifies some of these issues for you and makes it easier for you to understand how Subversion works.

I would like to finish by pointing out that a Subversion repository layout can be changed. You can always reorganize and restructure your layout after the fact. At worst, it just might create some short term pain as users adjust their working copies. It is not like you need to start over though. Just change names, move folders or do whatever to get the filesystem looking the way you want it.

If you have been reading this blog, then you know that we recently held a webinar on Subversion in the Enterprise. The attendance at this webinar was amazing and we received a lot of questions from attendees during the webinar. I was assigned to answer the questions that related to Subversion tools, since that is my particular area of expertise. So, without further ado...

Does Subversion have a source file differencing tool? Can a third-party differencing tool be supported?

Yes and yes. Subversion includes the diff command. The only limitation it has is that it will only work on files that are managed by Subversion. You cannot use it to difference two arbitrary files, it has to be files within Subversion. However, you can also use it to produce all the differences between two branches or the current trunk and a tag, or just the changes you have made locally in your working copy. The svn diff command can also launch a third-party diff tool if you have one that you prefer. Finally, the Subversion graphical clients I have used (Subclipse, TortoiseSVN etc.) include a graphical diff tool and can also be configured to work with external tools.

I had a recent post about the diff command on this blog.  You might take a look at that post to see some examples of how diff can be used and the syntax of the command.

Can you recommend a Web-based client tool for Subversion?  TortoiseSVN is great, but it requires us to load software on the user's desktop.

I assume you are talking about an actual web-based Subversion client. I am not aware of any that exist, although I have seen people express interest in writing one. So it is possible there are some that exist in some state of development. The truth is that it would be a difficult task to write a good web based client. You would have to recreate all of the Subversion features in the web client and then somehow implement those features on behalf of the user on the web server.

Does CollabNet offer anything to help integrate Subversion with a development environment such as Eclipse?

I am one of the project owners of the Subclipse project which provides support for Subversion inside of Eclipse. The Subclipse team (specifically me) works closely with the Subversion developers to help define the features and API we need to develop a good Eclipse client. There will be a number of enhancements to Subversion coming in the 1.5 release which were specifically added at the request of the Subclipse team and our needs. Of course these enhancement benefit everyone, but we helped define the "itch" so that the developers would "scratch" it.

What clients would you recommend for Eclipse and also Visual Studio?

As a project owner for Subclipse, I would obviously recommend Subclipse for Eclipse. Subclipse has been around for several years and is a mature product that implements all of Subversion's features and has a strong user community. On the Visual Studio side, I would look at two tools. AnkhSVN is an open source project that provides support for Visual Studio 2003 and later. I have heard lots of positive comments from people that use it.

VisualSVN is a somewhat recent entry on the commercial side. I think what they are doing is interesting and the price is very reasonable. They have taken the approach of building their client on top of TortoiseSVN which I think is a great idea because it gets them access to a very full featured user interface and lets them focus on adding the value that is needed to make a good Visual Studio integration.

I do not develop in Visual Studio, so I have never used either product.

Which Eclipse plug-ins support refactoring through the Eclipse context menus?

Subclipse has always supported refactoring within Eclipse. If you perform a refactoring operation that renames/moves a file, Subclipse sees this happening, intercepts the operation and performs it using Subversion so that Subversion knows about it. There are some limitations in Subversion itself that might give the perception that Subclipse does not support refactoring.

1. Subversion does not allow a file/folder to be renamed/moved twice without a commit in between. Subversion trunk code has been enhanced to remove this limitation, so it will be available when 1.5 is released. Subclipse refactoring support will be a lot better because of this change. I have already made the changes in Subclipse to leverage this Subversion change and refactoring now works really nicely. That being said, refactoring works nicely today if you are aware of the limitations and can work around them.
2. Subversion's support for move/rename has some limitations that can effect you when doing an update or merge. You can see a detailed explanation in this post on my personal blog. A short explanation is simply that if you have made some changes to a file, and someone else renames the file and commits the rename, when you do an update, Subversion does not transfer your changes to the renamed file. Subversion is smart enough to not delete the local file with your changes, so you can at least copy them over to the new file manually. Ideally, it would do this for you. The Subversion developers recognize the need to make this improvement, so hopefully it will come in a future release. It is a difficult problem for them to solve and is not planned for the next release (1.5).

A very common question asked on the Subversion mailing list is "How can I see the differences between two tags?" Of course there are a lot of variants of this question such as what are the differences between trunk and a branch or two branches etc. The person asking this question is almost always aware that they can run the svn diff command to get the differences, but usually they just want to know the list of differences at the file name level, not the complete line-level diff. Prior to Subversion 1.4, the answer to this question was usually that you had to parse the diff output and extract the file names.

With Subversion 1.4, however, a new option, --summarize, has been added to the diff command.  When this option is provided, the diff command just outputs the differences at the file level.  For example, to see the differences between the Subversion 1.3.1 and 1.3.2 tags I can run this command:

svn diff --summarize http://svn.collab.net/repos/svn/tags/1.3.1 http://svn.collab.net/repos/svn/tags/1.3.2
M      http://svn.collab.net/repos/svn/tags/1.3.1/STATUS
M      http://svn.collab.net/repos/svn/tags/1.3.1/build.conf
M      http://svn.collab.net/repos/svn/tags/1.3.1/configure.in
M      http://svn.collab.net/repos/svn/tags/1.3.1/build/ac-macros/aprutil.m4
M      http://svn.collab.net/repos/svn/tags/1.3.1/build/ac-macros/apr.m4
M      http://svn.collab.net/repos/svn/tags/1.3.1/build/ac-macros/swig.m4
M      http://svn.collab.net/repos/svn/tags/1.3.1/build/get-py-info.py
M      http://svn.collab.net/repos/svn/tags/1.3.1/build/generator/swig/external_runtime.py
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/include/svn_version.h
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/libsvn_wc/status.c
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/libsvn_wc/lock.c
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/bindings/swig/ruby/libsvn_swig_ruby/swigutil_rb.c
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/bindings/swig/INSTALL
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/bindings/swig/NOTES
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/mod_dav_svn/version.c
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/mod_dav_svn/repos.c
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/tests/clients/cmdline/stat_tests.py
A      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/tests/clients/cmdline/authz_tests.py
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/tests/clients/cmdline/svntest/actions.py
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/tests/clients/cmdline/svntest/main.py
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/tests/libsvn_repos/repos-test.c
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/libsvn_repos/commit.c
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/svnserve/serve.c
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/po/ja.po
M      http://svn.collab.net/repos/svn/tags/1.3.1/subversion/po/zh_TW.po
M      http://svn.collab.net/repos/svn/tags/1.3.1/contrib/client-side/svn_load_dirs.pl.in
M      http://svn.collab.net/repos/svn/tags/1.3.1/tools/hook-scripts/mailer/mailer.py
A      http://svn.collab.net/repos/svn/tags/1.3.1/tools/server-side/svnauthz-validate.c
A      http://svn.collab.net/repos/svn/tags/1.3.1/tools/server-side
M      http://svn.collab.net/repos/svn/tags/1.3.1/CHANGES
M      http://svn.collab.net/repos/svn/tags/1.3.1/packages/rpm/rhel-3/apr.patch
M      http://svn.collab.net/repos/svn/tags/1.3.1/packages/rpm/rhel-4/apr.patch
 M     http://svn.collab.net/repos/svn/tags/1.3.1

With the --summarize option provided to the diff command, the output shows the changes at the file level. The output is similar to what you would see with many of the other Subversion commands. The first column has a value to indicate if the file was Added, Modified, Deleted, the second column does the same for properties.  This makes it easy to parse the output in scripts. The --summarize option was a great enhancement to the diff command and solves this use case really well.