The Subversion project has issued an alpha2 release on the path to the final GA. The next release might be alpha3, it might be beta1, that depends on the feedback we get from testers. One thing is certain, and that is the sooner we can find any remaining bugs, the sooner the GA release will be available.  Likewise, hearing that you have used this version with some success will also help us make decisions.

CollabNet has created binaries for Linux and Windows (with OS X coming soon) from this alpha2. You can download them from openCollabNet. There are also updated versions of the CollabNet Merge Client, Subclipse and TortoiseSVN available on this site.

We have a forum where you can post questions, comments or problems that you encounter using this version. If after some discussion a report is indeed deemed a defect, we forward the bug to the Subversion community using their normal processes. Feedback on the GUI merge clients will go to CollabNet Engineering.

A few things to be aware of:

• Subversion 1.5 client has an updated working copy format.  Once an SVN 1.5 client updates a working copy it will be converted to the new format. This means that older SVN clients will no longer be able to read these working copies. So either be careful, or be sure to update all your clients to the same version. There is a Python script that you can use to downgrade your working copy to the 1.4 format.
• Subversion 1.5 server can be used to serve an existing repository, but you will not be able to use the new merge tracking features with this repository until it is upgraded. You have a few options here:
1. Dump existing repository, then create a new repository using the 1.5 binaries and load the dump file.
2. Create a new repository using 1.5 binaries and use svnsync 1.5 binary to transfer the contents to the new repository.
3. Run a new command svnadmin upgrade to upgrade an existing repository. This runs fast as it just updates the internal repository format. There are some downsides to this approach though:
• SVN 1.5 repositories maintain a new index called the node-origins index  This is needed to speed up certain merge operations. The index will be lazy-populated on upgraded repositories, but can be updated on demand by running a new tool named svn-populate-node-origins-index(.exe). If you have a large repository you will likely want to populate the index. New repositories, or ones that are dumped/loaded or svnsync'ed will not need this.
• For fsfs repositories, the node-origins index will be slightly faster and take up less disk space if the repository is reloaded or synced. For existing repositories, the index has to be maintained in a separate set of files that takes up extra disk space and I/O.
• For new 1.5 fsfs repositories, there is a new sharding mechanism for storing revisions on disk. Instead of storing all revisions in one single directory, they are now spread across multiple directories which will be better on certain filesystems.  There is a Python script you can run to shard an exising fsfs repository though.
4. If you were using an earlier version of the 1.5 binaries, note that the on-disk format of the repository has been changed in the most recent builds. You should dump your repository with your current binaries, then install the new binaries, create a new repository and load the repository.

Please take the opportunity to give these alpha versions on the path to GA a try. Your feedback will lead to a better final release and likely also help us deliver Subversion 1.5 as soon as possible.

It is an exciting time for Subversion as its adoption continues at a dizzying pace in enterprises. I'm out there helping that adoption so I'm a bit late in posting the questions and answers I promised around the three basic branching strategies that I covered in the last two webinars in which I presented (Branching and Merging Strategies for Subversion 1.5 and Advanced Merge Tracking and Branching with Subversion 1.5). Hopefully these will be useful to many of you.

First, a quick reminder on what the three branching strategies were:

1. The unstable trunk which mimics the way that Subversion itself is developed. In this model, development is done on the trunk and a release branch is created around the time of feature completion with the formal promotion process carried out on that branch.

2. The stable trunk where development is done on system version branches and the promotion process is also conducted on that branch. The trunk is the branch point where the production releases are merged in and parallel development efforts are branched out.

3. The agile release strategy where development is done on individual feature branches and a release branch is created late in the process with the feature branches merged to it that will define that release. The formal promotion process is conducted on the release branch and the production version merged to the trunk as well as to all remaining active feature branches.

So on to the questions, first general branching questions:

I've had the pleasure of participating in two recent webinars on the topic of branching and merging particularly in the context of the pending 1.5 release with merge tracking. The first webinar, entitled "Branching and Merging Strategies for Subversion 1.5", focused on three branch and merge models. The second, "Advanced Merge Tracking and Branching with Subversion 1.5", focused on questions from the audience of the first webinar. The intent of both was to provide some practical information on a topic that has great importance in enterprises, but isn't well covered by what's generally available around Subversion.

I wanted to post some of the questions and answers that I covered in the second webinar here in our blog. I hope that they will be useful and I encourage you to view the replays of the webinars to learn more (http://www.collab.net/webinar21 and http://www.collab.net/webinar22).

Two of the questions most asked during the webinar “Branching and Merging with Subversion 1.5” were:

• “What are the differences between Subversion 1.5's merge tracking features and svnmerge.py?”
• “Will there be a migration utility?”

I’m in Munich at the moment and today attended CollabNet’s Subversion Power Workshop that precedes SubConf, the first conference dedicated to Subversion. Answers to both questions were covered during the merge tracking sessions.

Before talking about the advantages of Subversion 1.5 merge tracking over svnmerge.py, I think it is important to say that svnmerge.py has been a very useful tool contributed and maintained by Subversion community members. We are all thankful for their contributions. That said, Subversion 1.5 does offer several advantages over svnmerge.py:

• SVN 1.5 tracks merge history at a per-path level of granularity and tracks file level merges. svnmerge.py only tracks merge info for the entire branch, it does not track finer levels (e.g.: files). This allows SVN to implement cherry picking, which svnmerge.py does not support.
• svnmerge.py doesn’t always properly merge merge-properties (like svnmerge-integrated and svnmerge-blocked), leading to property conflicts when merging across multiple branches. SVN 1.5 solves this.
• SVN 1.5 includes support for third-party tools, such as call-backs for conflict resolution. If a merge conflict arises, Subversion can call your GUI client and the client can then fire up a conflict resolution tool. This improves workflow and usability. You can see the integration between SVN 1.5 merging and GUI clients in action by trying CollabNet’s new GUI Merge Client.
• SVN’s merge tracking includes more comprehensive auditing, like querying the merge history to answer questions about what was merged when and where.

A utility to migrate svnmerge.py merge tracking history from a previous version to Subversion 1.5 was committed to the Subversion trunk just a few days ago. What can it do?

• Convert the latest merge history for multiple branches. In terms of history, information about the latest merges for each of your branches is the most important information because it allows for an easier merge the next time you merge. If you want to keep things simple and only migrate the latest merge history, you can.
• The migration utility can also scan your entire repo for merge history, or accept a series of path prefixes to look at: svnmerge-migrate-history.py /path/to/repos trunk tasks branches.

We’ve uploaded a copy of the script to the download section of the Merge Tracking Early Adopter Program. You can download it and test it out on your own repository (not for production please, Subversion 1.5 is still under development). It should work well but tell us if you run into any problems. We know of one limitation: don’t run it on a Subversion 1.5 repository that already has merge history, this has not been tested yet (then again: why would you want to do that?). Right now, only the .py script is available but the Subversion developers plan to release a Windows executable later.

One of the most common questions that occurs whenever a new release of Subversion (or any product) comes out, has to do with the considerations you need to take into account before upgrading.  I will try to cover some of those in this post.

Subversion's compatibility guidelines guarantee that any 1.x client can talk to any 1.x server.  So you do not have to upgrade the client and server at the same time.  The 1.5 client can talk to 1.0-1.4 servers and any 1.0-1.4 clients can talk to a 1.5 server.

Those same guidelines say that you never have to reload a repository when upgrading the server.  You can upgrade your server from 1.0-1.4 to 1.5 and you do not need to do anything to your repositories.  The new server will silently make any updates that are needed, when they are needed.  In the case of 1.5, a new SQLite database file will be added to the repository to store indexes that speed up the new merge tracking function.  Even though you do not have to reload your repository, there are often small efficiencies you can gain by doing a dump/load of your repository.  In the case of this release, there is a new storage mechanism for fsfs repositories.  Instead of storing all of the revisions in a single folder, we now "shard" the repository by creating a sub-directory for each 1000 revisions. This does not improve performance of Subversion, but it might improve performance of things like backup utilities or other tools you use to work with the file system on the server.  In this case, you do not have to dump/load to get this feature, there is a Python script you can run that will do the sharding for you.  This script will run relatively quickly, certainly orders of magnitude faster than a dump/load.

Some features of Subversion 1.5 will require that you have a 1.5 server and a 1.5 client to enjoy the full benefits of the feature.  For example, the merge tracking feature requires that both the client and server are running 1.5.  If one of them is running a lower version than you get the merge functionality that existed in that version of Subversion.  If you want to take advantage of merge tracking you really need to update your server and all of the clients that are doing merges.  Doing a merge with an old client will work, but will not set any of the merge tracking information when the change is committed.  This simply means that someone using a 1.5 client later will attempt to merge the same info again and get conflicts.  The svn merge --record-only command can be used to fix this after the fact.

When using the new sparse-checkouts functionality it really helps to have a 1.5 server.  The feature will work with an older server, but in most cases the server will still send the extra information (file contents) to the client and the client has to discard what it does not want.

There is an important client-side compatibility feature to take into account.  Some of the new features of 1.5 such as sparse-checkouts and changelists necessitate a new client working copy format number.  So this means as soon as you perform an operation that needs to re-write your working copy (checkout/update/switch/commit/propset etc.) then the internal format of the working copy will be upgraded to the new format for Subversion 1.5.  Once this happens, you will no longer be able to use a 1.4 client with that same working copy.  The same sort of change occurred with the 1.3 to 1.4 upgrade process.  We know that this really impacted many users that were stuck using an older client, so we are including a Python script that you can run to downgrade a working copy back to the 1.4 format.  We will distribute this as an EXE for Windows users.  Just to make this perfectly clear, you can have a 1.4 client and 1.5 client on the same system.  You just need to be careful to not mix their usage on the same working copy.  Also, pure-read operations like (status/info/ls) do not update the working copy.  So simply viewing a working copy with a new version of TortoiseSVN or SCPlugin installed does not modify the working copy format.  Conversely, however, an old version of one of these tools will not be able to read a 1.5 working copy.

Hopefully this answers the questions about upgrading to Subversion 1.5.  It is going to be a great release and I certainly hope people choose to upgrade when it comes out.  If you have additional questions that I did not cover, please post them in the comments or on the Subversion Users forum on openCollabNet.

Hopefully you have already read my post from last week, Subversion merge made easy.  In that post, I linked to information on the new CollabNet Merge client that we are building for Subversion 1.5.  There has been a lot of good feedback from users who have downloaded and tried that client.

This week, we are introducing a preview of an extension to the merge client that allows you to perform merges based on change sets.  With this extension to the merge client, users of CollabNet Enterprise Edition's Project Tracker can perform merges by specifying Project Tracker artifacts as the input to the merge process instead of having to specify a list of revisions.  This is a really easy way to perform certain types of merges and can lead to some interesting new workflows and business processes.

If you have already installed the merge client, you can just visit the same installation site and grab the change set extension.  I have also written an overview of the client that has some more details and screenshots.  Click here for the overview of the change set extension.

While you will need Subversion 1.5 on your server to get the full benefits of merge tracking, the bulk of the features of this change set extension can be used today with your version of CollabNet Enterprise Edition.  So give it a try and let us know what you think.

It has been a busy few months for the Subversion engineering team at CollabNet and the Subversion community in general. As we detailed in many posts this summer, the entire team is hard at work on Subversion 1.5, specifically the merge tracking feature. 

Back in the spring we started the Merge Tracking Early Adopter Program on openCollabNet. We wanted to make it as easy as possible to get people to try early versions of the merge tracking code and provide feedback to CollabNet and the Subversion community. The program has been pretty successful and provided some valuable feedback that was incorporated into the merge tracking deliverables for Subversion 1.5. Last week we refreshed the Subversion 1.5 Beta binaries on the merge tracking site to a more recent version in order to spawn a new round of testing. If you have not already downloaded them, please download them and give them a try. It is not too late to help the community deliver the 1.5 release.

The title of this post is "Subversion merge made easy" and that brings me to an announcement of sorts. While all of the work on Subversion 1.5 has been going on, another team at CollabNet has been working on a new GUI merge client for Subversion. Today we are releasing a beta version of this client as part of the Merge Tracking Early Adopter Program. A lot of work went into this client and there are many features. Rather than listing them here, let me instead link to the documentation on the merge tracking site.  The documents explain how to install the client and give an in depth overview of its features. I encourage you to download the client and give it a try. Use the forum of the merge tracking project to provide feedback, ask questions, report problems etc. If you try the client, you will automatically also test the Subversion 1.5 merge tracking feature, which can only help to accelerate its release.

Finally, since we are talking about merge and making it easier, I just want to also mention an upcoming webinar that is being presented by two of my colleagues at CollabNet -- Bob Jenkins and Auke Jilderda. They have both contributed a lot of ideas and feedback to the merge tracking feature as well as assisted in putting together the merge tracking early adopter program on openCollabNet. Their webinar is sure to be stocked with valuable insights on branching and merging in Subversion.

Subversion 1.4.5 was released today.  You can download the updated CollabNet Subversion binaries immediately.

Subversion 1.4.5 contains a fix for a security exploit on Windows clients. This exploit was discovered and reported by researchers at the Colorado Research Institute for Security and Privacy.

The only change from Subversion 1.4.4 is the patch for this security exploit.  Since the exploit only affects Windows clients, we decided to only release CollabNet Subversion 1.4.5 packages for Windows. There is no point for someone who is already running 1.4.4 on any other operating system to update to 1.4.5.

I am not going to give a lot of details about the exploit, you can find more information at various security reporting sites, such as CVE.  I will say that it was a legitimate exposure that made it possible for the Subversion client to write files outside the normal working copy.  That being said, there are a couple of points to make:

1. Creating the exploit requires commit access to the repository.  If you can trust the people who have write access to the repository, then you do not have too much to be concerned about. The keyword in that sentence is "trust". If you are checking out from a repository you cannot completely trust, such as on a public hosting service, then be careful and update to 1.4.5 first.
2. While the exploit itself is pretty easy to produce, it is also pretty difficult to use it in a way that would cause harm.
3. You can only create the exploit from a non-Windows platform.
4. There is nothing terribly secretive about the exploit.  If you send commit emails, or even just browse your repository using svn ls, this exploit would stand out as not normal.

If you are running a Subversion client on Windows, this would include the command line client as well as any graphical client such as TortoiseSVN or Subclipse, then you should definitely go ahead and install this version of Subversion.  I would recommend that users of earlier versions such as 1.3.2 or 1.2.3 also install this update immediately. The Subversion 1.4.5 client can talk to any 1.x version of the server, so there is no reason not to update your client (for compatibility: if you have the command line and a GUI client, update them both).

Subversion servers are not affected by this exploit.  That being said, a Windows server that uses the Subversion client in scripts would still be vulnerable and should be updated to 1.4.5.

If you are a regular reader of this blog then you are undoubtedly aware that Subversion 1.5 will be coming out later this year and that the big feature is merge tracking. Technically, merge tracking is a largely internal feature with the exception of the ways in which it simplifies the overall UI of the merge process. Implementing this feature has caused us to spend a lot of time just thinking about the entire merge process and this has led to some additional improvements that are not technically part of the merge tracking feature. That brings us to what this post is about and that is how Subversion 1.5 will make it easier than ever for you to resolve conflicts.

Here at CollabNet, we get a fairly constant stream of questions about Subversion from a wide variety of folks and via several different communication vehicles (webinar participation, training sessions, and so on). We try to answer them all as quickly as possible, but sometimes we wind up with a backlog of unanswered questions. So, as has been done on this blog before, I'll try to answer some of those queued up questions, focusing this time on questions related to Subversion's locking feature.

One quality of a typical engineer is a heightened sensitivity to potential problem areas. For software engineers, that often means spotting possible bugs in our peripheral vision while working on some nearby chunk of source code. This is both a blessing and curse. You want your bugs fixed, but unfortunately we sometimes get derailed from our primary task by these little excursions into Perfection.

Another fact of a developer's life is that there are always more things to work on than time to do so. And everything is of the utmost priority to someone. And so here again, we find ourselves working on multiple things at the same time, making slow-yet-incremental progress on each of them so at least we can say about them all with a clear conscience and a forced smile, "As it turns out, I'm working on that right now."

To my managerial readers, I regretfully admit that I have absolutely no solution for those problems. After all, this is a technical blog. But I can (and will) tell you about an upcoming Subversion feature that might reduce the context-switching costs for those developers: changelists.

On the 30th of May we did another “Subversion in the Enterprise Webinar”. Quite a few people asked about migrating from ClearCase® to Subversion (Bob blogged about that already) and some asked specifically about synchronizing the two systems. For example:

“I work for a place that does contract development. Often we find people that are deeply entrenched with ClearCase, which works fine as long as you are working within the local environment. Is there a tool set that allows moving to and from Clear Case? We far and away prefer to work with Subversion, and only push back major releases to the customer’s ClearCase repository.”

There is a solution for this: you can download a ClearCase <> Subversion connector right here from openCollabNet, for free!

The connector synchronizes changes between IBM Rational ClearCase and a Subversion working copy. It provides two modes to synchronize data: from ClearCase to Subversion, and vice versa.

From ClearCase to Subversion: For the connector to work in this mode, you need to have two ClearCase views to clearly distinguish between modifications made:

• A "ClearCase Reference View" which must match the Subversion working copy.
• A "ClearCase Current View" containing the most recent changes made to the ClearCase VOB.

The connector gets the difference between a ClearCase - Reference View and the ClearCase - Current View, and synchronizes it with the Subversion working copy.

When you export, the following will happen:

• The connector will detect the difference in data between the two ClearCase views.
• The file deleted (red), file added (blue) and file renamed (green) will be exported to the Subversion working copy and also directly committed to the Subversion repository.
• The file whose content has been modified (Orange) will be updated only to the Subversion working copy. You need to manually commit it to a Subversion repository to synchronize all data.

From Subversion to ClearCase: In this mode, the connector brings in the changes made in the CollabNet Subversion working copy to a ClearCase view of your choice.

When you import, the following will happen: The file deleted (red), directory added (blue), file renamed (green) and file whose content is modified (orange) in the Subversion working copy will be imported to the ClearCase view. You need to manually checkin the changes made in this view to the ClearCase VOB.

The connector can be used with stand-alone Subversion as well as with CollabNet Enterprise Edition and SourceForge Enterprise Edition. If you want more information about the connector, download it and read the included manual (most of the above is from there). To get community support for the connector, go to our connectors project on openCollabNet. Support is also available directly from CollabNet.

Next to the ClearCase connector we have two free connectors for HP Quality Center. Our Subversion connector allows you to version test plans in Subversion and the CollabNet Enterprise Edition connector bi-directionally synchronizes defect information to ensure that developers and testers work on the same data. You can download datasheets for all connectors from our corporate web site.

We had a really nice addition to the Subversion merge tracking feature committed to trunk last week.  Hyrum Wright is a student participating in the Google Summer of Code program.  He has signed up to provide "Merge Auditing" features for merge tracking.  You can see some details on what this means in the functional specifications, but essentially it is about adding intelligence and awareness of the merge tracking information to the svn log and blame output.  In the merge tracking planning, this has been slated as a post-1.5 feature.  It is a feature everyone really wants to see in the code-base, but it was just felt that shipping the release and the core functionality had to be the priority.

Anyway, in this case it appears we will be able to have our cake and eat it too.  Hyrum, whom I should add is already a full Subversion committer, has made great progress on adding this support to the svn log command and has committed those changes to trunk.  Additionally, it turns out that the sample repository we created for our Merge Tracking Early Adopter program really helped to work out the kinks in this feature.  Since we had a nicely documented sample repository, and matching graphic, that Hyrum could refer to, it made it easier for him to refine the feature and get it working properly.

The above picture is the history of our sample repository. If you take a close look, the r14 commit to trunk is an interesting example where this feature comes into play.  This commit is a merge from a branch, and the merge also contains additional merges from another branch.  If I run this command (note the new -g option):

svn log -g -r 14 $REPOS/trunk

Here is the new output:

------------------------------------------------------------------------
r14 | merger | 2007-05-30 15:48:11 -0400 (Wed, 30 May 2007) | 3 lines

Merge branch b - product roadmap and update about page

Command executed: svn merge $REPOS/branches/b
------------------------------------------------------------------------
r13 | buser | 2007-05-30 15:46:48 -0400 (Wed, 30 May 2007) | 1 line
Result of a merge from: r14

Update info about our company
------------------------------------------------------------------------
r12 | merger | 2007-05-30 15:45:19 -0400 (Wed, 30 May 2007) | 3 lines
Result of a merge from: r14

Merge branch a - product roadmap

Command executed: svn merge $REPOS/branches/a
------------------------------------------------------------------------
r11 | auser | 2007-05-30 15:43:00 -0400 (Wed, 30 May 2007) | 1 line
Result of a merge from: r14, r12

Add product roadmap
------------------------------------------------------------------------

Note how it includes information on the revisions that were merged in r14, and additionally how one of those revisions was the result of another merge.  Graphical clients like our CollabNet Desktop - Eclipse Edition and TortoiseSVN should be able to do really useful and interesting presentations of this information in their UI.

This is fantastic work Hyrum.  Keep it up.  I cannot wait to see the blame implementation come together.

Those of you who diligently watch your e-mail Inbox for those little blessings that come via announce@subversion.tigris.org recently got a pleasant surprise, though perhaps not the one you'd been wishing for. It's been nearly five months since Subversion's last release — nine months since the introduction of the 1.4 version line. "Where's the next big Subversion release?" "What's taking so long!?" "I want Merge Tracking, and I want it yesterday!"

Don't worry, Subversion 1.5 remains on schedule. Merge Tracking development is still ongoing, as is support for sparse directories and a number of other improvements and features. Those are, of course, in addition to all of the other goodies that 1.5 promises to bring which are already completed. But along the way of developing Subversion 1.5, we found enough important issues in 1.4.3 to justify another patch release on this line.

Subversion 1.4.4 delivers a number of bug fixes for various issues, including (but not limited to):

• an extremely rare directory property dataloss bug which, for BDB-backed repositories, also unfortunately results in repository corruption
• a race condition when changing revision properties in FSFS-backed repositories
• a low-risk security flaw in the 'svn prop* --revprop' subcommands
• problems with 'svn diff' when writing large diff hunks to the Windows console
• a path sorting bug in the output of 'svnadmin dump' for non-ASCII paths which can cause invalid dumpfile generation
• a hang bug triggered during character translation

Now, some of those items look sorta scary. Data loss, repository corruption, race conditions — that's the stuff a technologist's nightmares are made of! But without downplaying the significance of the fixes to those problems, allow me to assure you that the instances of them have thus far been extremely rare. Still, you are encouraged (as always) to upgrade to this latest release of Subversion.  You can see the full set of changes made in 1.4.4 in the Subversion CHANGES file.

And if you're one of those folks dying to get your hands on Merge Tracking functionality, let me encourage you to join the openCollabNet Merge Tracking Early Adopter Program. The more feedback the Subversion developers get on their work, the better — and more timely — the feature will be overall.

About a month ago, I provided a post in this blog that suggested that enterprises consider taking some of the processes that are used to produce Subversion (as an example of good open source processes): In addition to the code. That stirred some interesting discussions, more offline than online, so I wanted to revisit this topic.

I find it disappointing, if not surprising, that enterprises are so reticent to consider the less restrictive, but still process-driven configuration management procedures that the Subversion project utilizes. I get a lot of initial backlash when I talk about open source processes. The reality is the backlash is not really coming from knowledge, but uninformed impressions. It is almost funny to think of organizations, who depend upon open source technology (e.g., Apache, Linux, Subversion, etc.), rejecting the processes that produced it without true consideration. That's like saying "I love my house and find it to be of high quality, but I wouldn't consider building another one the same way." What happened to good old plagiarism of successful ideas and processes?

For example, visibility seems an obvious approach to better quality that should be enabled by our processes. First, one is going to be more cautious in their work knowing that it can potentially be seen by the broader project team. It is a strong deterrent to shortcutting the coding standards that the project (or the enterprise) has identified. Second, the code benefits by having more eyes reviewing the changes, which means a higher probability of issues being identified earlier in the lifecycle at a lower cost to their enterprise. I realize some people are reticent to having their work "exposed", but that shouldn't be an inhibitor to doing the right thing. Most of the people that you really want doing the work will welcome the chance to share their "brilliance" at a more detailed level. This visibility is achieved to some extent by general read access, but also by posting deltas to a mailing list and its associated archive. Visibility can also be enhanced through an association of the delta with a change request

If visibility is achieved, then it is logical to consider what the impact would be to other processes. Who would consistently violate accepted processes if they knew the team saw those violations and knew the team is impacted by those violations? How would the broader team react to these violations? Aren't people hired with the expectation that they will do the right thing for the enterprise and as an extension of that, for the project team?

Contrast that to implementing a hook that impacts everyone's productivity and, while forcing adherence, can be consistently required for some users versus correcting their behavior. Each user pays a price with every operation just in case they "might" be violating a process. Of course, since we're talking version control (aka a time machine) the changes can be rolled back if a violation occurs. And consider the fact that this won't be isolated to one type of violation, but will be repeated for many different types which will build on the impact to each individual.

I think enterprises need to become more open minded and consider the benefits to the approach the Subversion project takes versus the process enforcement that most lean towards today. Since you've built your house on Subversion (Apache, Linux, etc.), it seems reasonable to look to build other houses in the same way.

I must admit that I'm new to blogging and these lines have been written on plain old paper first - yes, I was a COBOL developer in the early days of my professional career and I still write a lot of things in upper case.

When I made the move to a small start-up named Atria Software back in 1996, the world of software development looked quiet different: small to mid-sized teams, typically co-located around a specially designated project server. Today's broadband connections are faster than most intranet lines then, and different sites are connected through GB-capacity leased lines? Back then, if you were lucky you had a doubled ISDN dial-up connection.

Back in these days companies started to ramp-up distributed teams to work on highly complex projects, and of course they needed an infrastructure that supported this. The answer of most commercial SCM systems was Replication and Synchronization of the repositories. You duplicate the archive, figure out the delta at the different sites from time to time, you exchange these deltas and apply it to the different replicas - done! The biggest advantage of this approach was that it worked with the thin connections. The biggest disadvantage: conflicts due to the time gap between change and sync. If the same object is changed at 2 sites, a potential conflict has to be resolved during the synchronization. You might argue that this still can be solved by mastership concepts to avoid that the same object is changed simultaneously. You are right, but this implies that you are introducing branches not because the project needs them, but just because of the replication.

Modern approaches try to solve this challenge like Raid systems with write-through technology. If you commit a change to an archive, the change is propagated immediately to the other replicas, without any mastership definition. This little time gap eliminates most of the risk - almost like a central archive!

But - just "almost". First, if the connection between the replicas is down the risk for conflict rises again. Second, it reduces the flexibility of organizations to react to change, because the setup of the infrastructure reflects the status quo (if you've ever moved replicas between different sites you know what I mean …). Third, it makes IP (Intellectual Property) Governance more complicated. I know of a ClearCase VOB that was replicated across 28(!) sites, most of them with read-only access. Imagine the cost to establish the same level of security to avoid unauthorized access at each site?

A common argument against a centralized approach in SCM is what to do if you (or your site) are offline and you need data that is not in your working copy right now. Well, in today’s world there is no longer the need for being offline (you can even go online in an airplane these days); the technology is there and it is way cheaper than bothering a large developer team with a highly complex branching strategy. And if you want to isolate work for a while before sharing it with the other project team members, Subversion (or an embedded Subversion) allows you to do that as well.

Don't get me wrong, there are still some scenarios for replication, like an extreme data volume. Then, and only then, it is the right approach. But don't let it be your silver bullet to meet the challenge of distributed development and do it because you always did it!

I must admit I have my fair share in seeding the concept of replication into organizations. But - replication as an SCM concept to support distributed teams is more than 15 years old; in our market space this is almost 2 generations of technology. It is time to try something new, although it is not that new at all because the mainframers have done it for least for 5 generations ...

This is posted by an excited NKOTB (New Kid on the Blog) coming from the pre-internet generation.

I thought I would take a break from my previous technical posts and give a general update about what has been going on in the land of CollabNet Subversion (our Subversion binaries).

We have been busy the last several weeks and the fruits of our labor are starting to see the light of day. First, a couple of weeks ago we released our Subversion 1.4.3 client and server RPM's for Red Hat Enterprise Linux. This was big news for us because it was our first 1.4.3 release and it had been bugging us that we were lagging behind. Producing the binaries is easy, but we created new packaging as well as additional scripts that we provide with the RPM. This took some extra time but now our build environment is complete and going forward it will be easier for us to stay in synch with the open source community.

Earlier this week, we released our Subversion 1.4.3 client and server packages for Solaris 10 SPARC. This was our first Solaris release and again it felt good to get it out the door. We spent some extra time making sure the build works the way we want it to but we sorted out any issues and the release sailed through our QA process. I know that I was pleased when I tested the Solaris version, installing and getting a server up and running is very easy. The extra time we spent paid off and like Linux we can now produce new releases fast.

Also this week, we helped kick-off a new area on openCollabNet that is providing "Community Binaries" for Subversion. The first binary release is for Mac OS X 10.4 PPC and Intel. Jeremy Whitlock, a contributor to this blog, deserves the credit for picking up the ball on this and running with it. He did a great job in both producing the binaries and in putting a lot of quality touches into the packaging. We will have more information on our plans and aspirations for the binaries project when we get the project launched on openCollabNet (within the next two weeks or so). For now, you OS X users, go download the binaries and let us know how it goes for you.

Finally, there is the Windows installer for CollabNet Subversion. Our current version is 1.4.2 and uses InstallAnywhere, a Java-based installer that adds a lot of heft to the download size because it includes a JRE. We are moving away from that to a Windows-specific installer. We are also adding a lot of new features to the installer such as installing the server as a Windows service as part of the install process. It is taking some time to get these features implemented correctly and QA-ed but in the end it will be worth it. We should have the final version out in a couple of weeks. Most likely it will be for Subversion 1.4.4 as that should be released between now and then and creating the binaries themselves is not the issue for us.

As I mentioned in my previous post, the Subversion project has identified many requirements based on input from both the open source community and from the enterprise data collected by CollabNet from its customers. The requirements as well as the functional and design specifications are available on the Subversion project site.

Subversion's upcoming 1.5 release is scheduled to deliver the core merge tracking feature with subsequent releases likely extending this. Fundamental to all merge tracking related functionality is the functionality to automatically record what change sets have been copied, or merged, where. On top of this, the 1.5 release will deliver the following functionality:

• Repeated Merge

One of the most frequent definitions of what merge tracking means is the concept of dealing with repeated merges. The requirement is to track which change sets have been applied where, so that users can repeatedly merge branch A to branch B without having to remember the last range of revisions copied. This would also track change set cherry picking by users, so that Subversion won’t accidentally re-merge change sets that are already applied.

• Cherry Picking

Cherry picking is the merging of one or more individual changes from branch A into branch B (as opposed to the merge of all previously unmerged changes). Subversion will provide a way to indicate that the change(s) have been merged into branch B. Such merging must be supported in multiple different directions, for instance if a release branch B is created by copying the trunk, both forward port changes made on branch B into the trunk and backport changes made on the trunk into branch B without confusing the merge tracking algorithm.

• Record Manual Merge

Recording manual merges allows change sets to be marked as merged without having used the ‘svn merge’ sub command to create the defined target revision. This could be to mark a situation when there is no merge tool for the file type and a user merged the changes manually. Also, it should support merge tracking of a change set which is sufficiently different when ported to a different branch that the use of ‘svn merge’ is no longer appropriate.

Example scenarios requiring this functionality include (a) the actual change you want to apply to the branch has no overlap with its incarnation on the source branch but is conceptually equivalent, (b) only a subset of a change set warrants application, and (c) the branch content has drifted far enough apart to make automatic merging impossible (for instance, because it would generate excessive merge conflicts).

• Rollback Merge

The ability to undo a merge and/or the associated tracking meta data, regardless whether the merge is the result of an automated or manual merge.

• Block/Unblock Change Set

Subversion will be able to protect revisions from accidentally being propagated elsewhere.

• Automated Merge

Subversion will have the ability to automate merges and support interfaces for resolving conflicts and other errors.

Any branching scheme requires the consideration of merging. Up until now, Subversion facilitates merges but does not record them as such in its history information. That is, Subversion does not automatically record, or track, what change sets have been merged where and what revisions were created as a result of merging.  To work around the lack of merge tracking, current best practice is to properly document merges in commit log messages.

Merge tracking is a multi-faceted set of functionality and no single definition exists that is commonly agreed upon. Most users and organizations tend to think of only the subset of features that addresses particular issues they face in their development organizations. Examples of features that are commonly part of merge tracking are to automatically record what change sets exist on what branches, prevent duplicate merges, handle bi-directional merging, support for manual recording of tracking information, block certain change sets from ever being propagated elsewhere, and various reporting capabilities.

Little over a year ago, the Subversion project started to define, design, and implement merge tracking. CollabNet contributed by gathering enterprise requirements for the open source project. In February 2006, we organized a workshop that brought select customers together with the Subversion development team to make an inventory of required functionality.  The Subversion team has taken these requirements and driven the definition, design, and implementation. CollabNet employees continue to take a key role in developing the feature and addressing open issues related to merge tracking.

Subversion’s Merge Tracking requirements, functional specification, design documents, and the current status of the project are available on the Subversion project's merge tracking pages.

In my next post, I will provide an overview of the functionality that the Subversion team intends to deliver in their upcoming 1.5 release.

My previous blog entry discussed the issue of repository layout. This entry will try to answer the question of whether you should have one repository per project or a single repository that houses all your projects. There is not going to be a single right answer to this question. Hopefully this post will help you understand the tradeoffs so you can make the right decision that suits your requirements. These are some of the advantages of the single repository approach.

1. Simplified administration. One set of hooks to deploy. One repository to backup. etc.
2. Branch/tag flexibility. With the code all in one repository it makes it easier to create a branch or tag involving multiple projects.
3. Move code easily. Perhaps you want to take a section of code from one project and use it in another, or turn it into a library for several projects. It is easy to move the code within the same repository and retain the history of the code in the process.

Here are some of the drawbacks to the single repository approach, advantages to the multiple repository approach.

1. Size. It might be easier to deal with many smaller repositories than one large one. For example, if you retire a project you can just archive the repository to media and remove it from the disk and free up the storage. Maybe you need to dump/load a repository for some reason, such as to take advantage of a new Subversion feature. This is easier to do and with less impact if it is a smaller repository. Even if you eventually want to do it to all of your repositories, it will have less impact to do them one at a time, assuming there is not a pressing need to do them all at once.
2. Global revision number. Even though this should not be an issue, some people perceive it to be one and do not like to see the revision number advance on the repository and for inactive projects to have large gaps in their revision history.
3. Access control. While Subversion's authz mechanism allows you to restrict access as needed to parts of the repository, it is still easier to do this at the repository level. If you have a project that only a select few individuals should access, this is easier to do with a single repository for that project.
4. Administrative flexibility. If you have multiple repositories, then it is easier to implement different hook scripts based on the needs of the repository/projects. If you want uniform hook scripts, then a single repository might be better, but if each project wants its own commit email style then it is easier to have those projects in separate repositories

This is just a sampling of the pros and cons of each approach. Hopefully it gives you something to go on to make a decision. I tend to prefer the one repository per project approach with the caveat that if I have many projects that are related to each other, I would put those all in the same repository. I also tend to break up repositories by group or team, although in reality this is just a variation of the project concept.

For example, I had one repository for the Documentation department to use for their projects. Of course in the case of on-line help that was often located in the same project as the application code, but the Documentation team also had other materials they produced and I gave them a repository for that. Likewise, the Marketing department had a repository to store the things they worked on, including the company web site. As was the case with the layout of the repository, this is really just a decision of what will work best for you. That  said, it is a little more difficult to change your repository setup after the fact.

So, it is worth it to take some time to understand your requirements and which approach best suits them.

One of my jobs at CollabNet has me consulting with enterprises that are planning to adopt Subversion either for specific projects or as a standard for the organization. These enterprises understand the value of Subversion for meeting their requirements for version control and software configuration management. They appreciate the work that the open source community has invested in this state-of-the-art tool and are excited to leverage that investment. What they often struggle with is their predisposition to use the tool as a "hammer" to enforce processes versus taking a cue from the Subversion developers themselves to improve their own processes. They are open to using an open source tool, but can't see adopting some of the open source processes.

I participated in a webinar two weeks ago, Subversion in the Enterprise, in which we had a limited chance to respond to questions. We’ll answer some of the remaining questions here.

One topic that came up multiple times was enterprises migrating from Rational ClearCase. Since I spent 5 years helping sell, implement and support ClearCase, and now 6 years at CollabNet, I understand both worlds pretty well.

Before focusing on ClearCase though, I encourage you to read some very good migration articles that can be found in the Subversion migration project. I talk to big companies every day about Subversion and many of them are considering to migrate or have although done so. These articles are a good start to prepare for a migration.

When planning a migration, the most important thing you need to do is realistically determine what data you really will benefit from migrating, rather than just assuming you want to migrate everything. For one thing, you don’t want to clutter your new repository. If you move from one house to another, you have garage sales, donate items to charity, and throw things away. The same concept should be applied to version control migrations. What really do you need in your new "house"? Realize that you have to pay the moving costs for whatever you migrate so be sure that the value you realize is worth more than the cost.

I always suggest leaving historic data in the legacy tool and just taking the latest from the live branches or perhaps snapshots of key milestones from current development branches. You should minimize your investment in the legacy tool both in terms of licenses and hardware, but retain it as the historical archive. A potential alternative is to migrate the historical data to its own Subversion repository while migrating limited data into the “production” repository. That way, history is retained (with the clutter stored elsewhere) and the legacy tool is eliminated along with its licensing and support costs.

Any migration will have limits based on the differences between the legacy system and Subversion as well as the extent to which the migration tool’s author(s) decided to map functionality and metadata between the two tools. With ClearCase, it is not a question of "will there be limits?", but rather how many limits. ClearCase doesn’t really support data migration so the development of migration tools is very challenging. That said, I am aware of at least a couple of attempts at providing such a tool. The first was a tool developed by an enterprise specifically for their needs and situation. It had limitations that they were willing to accept but the tool will unlikely be acceptable to most other companies.

Most of the focus these days is on the SVN Importer tool which purports to support migrations from many different version control tools. Unfortunately, a brute force approach - limited to a full migration - is taken to ClearCase and to most of the other systems. Such an approach requires a lot of system resources (e.g., disk and memory) to execute. That has proven itself to limit the size of a source VOB that can be migrated using a relatively “average” machine. One customer has seen that limit at a 150MB VOB, but the limit will vary depending on the hardware used for the migration. This customer had one VOB larger than the tool could support but fortunately they wanted to split the VOB contents anyway so they split it and executed two migrations resulting in two Subversion repositories.

As for case studies of enterprises successfully migrating from ClearCase, I certainly see it happening, particularly with companies that are truly doing distributed development and those who don’t need the unique features of ClearCase. Some companies have migrated history but most are leaving it behind. In my experience, the biggest key to success is not to recreate ClearCase processes in Subversion, but rather revisit your organization’s version control needs and map Subversion’s features and proven best practices to meet those needs.

Keep in mind that companies usually run a pilot before making a commitment to something as critical as a new version control tool. When they commit, they do so based on their own documented experiences. For a migration from ClearCase, I highly recommend you do a pilot first. After that you can join the companies that are enjoying the many benefits Subversion brings over ClearCase, such as atomic commits, real support for globally distributed development and consistent performance with tags and branches.

If you need help with your migration, our Professional Services team has a lot of experience helping customers move to Subversion. Check out our corporate web site for more information.