The last few weeks we blogged a lot about the Merge Tracking feature in Subversion 1.5.  Of course there are several other great new features coming. Let’s look at what else is new.

The Subversion 1.5 release notes (which are not final of course) mention these new features for 1.5:

• Merge Tracking
• Sparse checkouts
• WebDAV transparent write-through proxy
• Cyrus SASL support for ra_svn and svnserve
• Copy/move improvements: peg revisions, 'svn mv file1 file2; svn mv file2 file3', 'svn cp *.c dir'
• Cancellation improvements
• Changelist support
• FSFS sharding
• Command-line client improvements
• JavaHL bindings improvements
• Many improved APIs

If you are new to this blog and want to find out more about Merge Tracking, check out our Merge Tracking Early Adopter Program. Over the next few weeks we’ll blog about some of the other new Subversion features. Not about everything though, other people are blogging about 1.5 as well and we’ll link to them. For example: Malcolm Rowe blogged about mod_dav_svn improvements, tree-structured FSFS repositories and backing up FSFS repositories, Subversion 1.5 style.

In this post we’ll talk about sparse directories.

Sparse Directories

When you first checkout a Subversion repository, or a directory within that repository, you get the whole directory with everything underneath it. In large projects that can be a problem because the files are copied over the network and that can take some time. This is very contrary to how Subversion subsequently sends small deltas over the network with minimal use of network resources. Also, do you want all these files cluttering your disk?

Subversion 1.5 introduces Sparse Directories, giving you more control over what to checkout and how svn update works. You can read the specs here. But I don't learn from reading, I need to play. So let's play (but let’s be mindful of the fact that SVN 1.5 is not feature complete yet).

I started with downloading the Subversion 1.5 pre-release binaries from the Merge Tracking Early Adopter Program and setup a repository:

• Download the binaries (Windows in my case).
• Copy .exe and .dll files to c:\svn and add c:\svn to %PATH%.
• Create repository (svnadmin create repo) at c:\svn (repository is c:\svn\repo).
  Download the repository dump file that comes with the Merge Tracking beta.
• Load the dumpfile (svnadmin load c:\svn\repo < c:\svn\mergetracking.dump).
• Create directory for working copy.
• Checkout the repository (svn checkout file:///c:/svn/repo/trunk)

The main directory of the trunk of the repository contains one file and a few sub-directories:

index.html
about
jobs
news
products
support

Suppose I'm another developer (I mimicked that with working copy wc2) and I have no need for the subdirectories. With current releases of SVN, I can use the -N switch (for: non-recursive) to checkout only the main directory of the trunk and not the sub-directories:

I now have the topmost directory of trunk and the file in it, but not the sub-directories. For a big repository, this will reduce the amount of time for the checkout, and keep your working copy cleaner and subsequent updates will only pull in files added to the directory you checked out. But –N is all the control you have.

SVN 1.5 will be more flexible by making the -N switch redundant and replacing it if for a --depth option. According to the spec, the possible values for --depth are:

• --depth=empty: Updates will not pull in any files or subdirectories not already present.
• --depth=files: Updates will pull in any files not already present, but not subdirectories.
• --depth=immediates:  Updates will pull in any files or subdirectories not already present; the subdirectories will have depth=empty.
• --depth=infinity:  Updates will pull in any files or subdirectories not already present; the subdirectories will have depth=infinity.  Equivalent to today's default update behavior.

Now I am developer number 3 with wc3 as working copy. Let’s checkout the repository with -–depth=empty:

As you probably know, Subversion 1.4.4 was recently released.  I wanted to let you know that the CollabNet Subversion team is hard at work putting together our packages for 1.4.4.  We will release them in a staged order: Linux first, Solaris second and Windows third.  Most of the time and effort in our release process is spent in QA and validation and the reason we do the Linux and Solaris packages first is that we can get those versions in the hands of our QA team the quickest.  Eventually, I plan on making the order Linux, Windows, Solaris, as there are more Windows than Solaris users who are seeking the packages.

The Linux package is in the final stage of our validation process and I expect it to be available on openCollabNet later this week.  The Solaris package should be available next week and Windows the following week.

It’s worth pointing out that the most significant fix in 1.4.4 is a potential (though very unlikely) repository corruption with BDB repositories.  Since CollabNet recommends FSFS and CollabNet Subversion comes pre-configured with that back-end, this BDB fix does not affect CollabNet Subversion users.

On the 30th of May we did another “Subversion in the Enterprise Webinar”. Quite a few people asked about migrating from ClearCase® to Subversion (Bob blogged about that already) and some asked specifically about synchronizing the two systems. For example:

“I work for a place that does contract development. Often we find people that are deeply entrenched with ClearCase, which works fine as long as you are working within the local environment. Is there a tool set that allows moving to and from Clear Case? We far and away prefer to work with Subversion, and only push back major releases to the customer’s ClearCase repository.”

There is a solution for this: you can download a ClearCase <> Subversion connector right here from openCollabNet, for free!

The connector synchronizes changes between IBM Rational ClearCase and a Subversion working copy. It provides two modes to synchronize data: from ClearCase to Subversion, and vice versa.

From ClearCase to Subversion: For the connector to work in this mode, you need to have two ClearCase views to clearly distinguish between modifications made:

• A "ClearCase Reference View" which must match the Subversion working copy.
• A "ClearCase Current View" containing the most recent changes made to the ClearCase VOB.

The connector gets the difference between a ClearCase - Reference View and the ClearCase - Current View, and synchronizes it with the Subversion working copy.

When you export, the following will happen:

• The connector will detect the difference in data between the two ClearCase views.
• The file deleted (red), file added (blue) and file renamed (green) will be exported to the Subversion working copy and also directly committed to the Subversion repository.
• The file whose content has been modified (Orange) will be updated only to the Subversion working copy. You need to manually commit it to a Subversion repository to synchronize all data.

From Subversion to ClearCase: In this mode, the connector brings in the changes made in the CollabNet Subversion working copy to a ClearCase view of your choice.

When you import, the following will happen: The file deleted (red), directory added (blue), file renamed (green) and file whose content is modified (orange) in the Subversion working copy will be imported to the ClearCase view. You need to manually checkin the changes made in this view to the ClearCase VOB.

The connector can be used with stand-alone Subversion as well as with CollabNet Enterprise Edition and SourceForge Enterprise Edition. If you want more information about the connector, download it and read the included manual (most of the above is from there). To get community support for the connector, go to our connectors project on openCollabNet. Support is also available directly from CollabNet.

Next to the ClearCase connector we have two free connectors for HP Quality Center. Our Subversion connector allows you to version test plans in Subversion and the CollabNet Enterprise Edition connector bi-directionally synchronizes defect information to ensure that developers and testers work on the same data. You can download datasheets for all connectors from our corporate web site.

We had a really nice addition to the Subversion merge tracking feature committed to trunk last week.  Hyrum Wright is a student participating in the Google Summer of Code program.  He has signed up to provide "Merge Auditing" features for merge tracking.  You can see some details on what this means in the functional specifications, but essentially it is about adding intelligence and awareness of the merge tracking information to the svn log and blame output.  In the merge tracking planning, this has been slated as a post-1.5 feature.  It is a feature everyone really wants to see in the code-base, but it was just felt that shipping the release and the core functionality had to be the priority.

Anyway, in this case it appears we will be able to have our cake and eat it too.  Hyrum, whom I should add is already a full Subversion committer, has made great progress on adding this support to the svn log command and has committed those changes to trunk.  Additionally, it turns out that the sample repository we created for our Merge Tracking Early Adopter program really helped to work out the kinks in this feature.  Since we had a nicely documented sample repository, and matching graphic, that Hyrum could refer to, it made it easier for him to refine the feature and get it working properly.

The above picture is the history of our sample repository. If you take a close look, the r14 commit to trunk is an interesting example where this feature comes into play.  This commit is a merge from a branch, and the merge also contains additional merges from another branch.  If I run this command (note the new -g option):

svn log -g -r 14 $REPOS/trunk

Here is the new output:

------------------------------------------------------------------------
r14 | merger | 2007-05-30 15:48:11 -0400 (Wed, 30 May 2007) | 3 lines

Merge branch b - product roadmap and update about page

Command executed: svn merge $REPOS/branches/b
------------------------------------------------------------------------
r13 | buser | 2007-05-30 15:46:48 -0400 (Wed, 30 May 2007) | 1 line
Result of a merge from: r14

Update info about our company
------------------------------------------------------------------------
r12 | merger | 2007-05-30 15:45:19 -0400 (Wed, 30 May 2007) | 3 lines
Result of a merge from: r14

Merge branch a - product roadmap

Command executed: svn merge $REPOS/branches/a
------------------------------------------------------------------------
r11 | auser | 2007-05-30 15:43:00 -0400 (Wed, 30 May 2007) | 1 line
Result of a merge from: r14, r12

Add product roadmap
------------------------------------------------------------------------

Note how it includes information on the revisions that were merged in r14, and additionally how one of those revisions was the result of another merge.  Graphical clients like our CollabNet Desktop - Eclipse Edition and TortoiseSVN should be able to do really useful and interesting presentations of this information in their UI.

This is fantastic work Hyrum.  Keep it up.  I cannot wait to see the blame implementation come together.

When I first announced the "Universal OS X Subversion Binary" nearly a month ago, I had no idea that the binary would be such a hit (more than 2700 downloads in four weeks).  Sure, Subversion is an excellent source management system and the Mac is a killer platform but knowing this didn't prepare me for the impact this has apparently made.  So, how can things get even better? 

Well..... we've upgraded the binary to Subversion 1.4.4 for starters and that is just the beginning.  The latest version of the binary now includes the Subversion Apache modules for Apache 2.2.x and all Subversion language bindings maintained by the Subversion developers.  This includes bindings for Java, Perl, Python and Ruby.  The 1.4.4 release of the binary includes everything you need for using Subversion as an end-user, developing Subversion-related applications and even for installing a Subversion server.  Nothing has been left out this time around.

The last piece of information to bring to you is related to a new project on openCollabNet where the collaboration and efforts behind this release are handled.  The SVNbinaries project on openCollabNet is a project where you and the rest of the Subversion community can get involved in building and releasing Subversion-related binaries.  The Mac OS X Universal Subversion binary is the first finished product of this project.  The purpose of this project is to allow community members to contribute binaries and information necessary to reproduce the binaries so they can be improved by others.

The reason I bring this project up is not only because it is where the Universal Subversion Binary is developed but also because this is where you will submit bugs, feature requests, new binaries and resources related to the binaries developed and distributed in the project.  If you need a specific binary, a specific binding or even specific Apache modules, the SVNbinaries project is a great place to make those requests or help create them.  The SVNbinaries project is a true community project with the sole purpose to create easy-to-use Subversion installations for the masses. 

Those of you who diligently watch your e-mail Inbox for those little blessings that come via announce@subversion.tigris.org recently got a pleasant surprise, though perhaps not the one you'd been wishing for. It's been nearly five months since Subversion's last release — nine months since the introduction of the 1.4 version line. "Where's the next big Subversion release?" "What's taking so long!?" "I want Merge Tracking, and I want it yesterday!"

Don't worry, Subversion 1.5 remains on schedule. Merge Tracking development is still ongoing, as is support for sparse directories and a number of other improvements and features. Those are, of course, in addition to all of the other goodies that 1.5 promises to bring which are already completed. But along the way of developing Subversion 1.5, we found enough important issues in 1.4.3 to justify another patch release on this line.

Subversion 1.4.4 delivers a number of bug fixes for various issues, including (but not limited to):

• an extremely rare directory property dataloss bug which, for BDB-backed repositories, also unfortunately results in repository corruption
• a race condition when changing revision properties in FSFS-backed repositories
• a low-risk security flaw in the 'svn prop* --revprop' subcommands
• problems with 'svn diff' when writing large diff hunks to the Windows console
• a path sorting bug in the output of 'svnadmin dump' for non-ASCII paths which can cause invalid dumpfile generation
• a hang bug triggered during character translation

Now, some of those items look sorta scary. Data loss, repository corruption, race conditions — that's the stuff a technologist's nightmares are made of! But without downplaying the significance of the fixes to those problems, allow me to assure you that the instances of them have thus far been extremely rare. Still, you are encouraged (as always) to upgrade to this latest release of Subversion.  You can see the full set of changes made in 1.4.4 in the Subversion CHANGES file.

And if you're one of those folks dying to get your hands on Merge Tracking functionality, let me encourage you to join the openCollabNet Merge Tracking Early Adopter Program. The more feedback the Subversion developers get on their work, the better — and more timely — the feature will be overall.

I often talk to developers at large companies that are on a version control system dictated from the top but disliked by developers and administrators. The legacy system is too hard to use, not a good fit for distributed development and often too difficult and expensive to deploy. What happens is that developers start to implement their own Subversion servers for their teams or departments. CIOs and VPs of development see this happen all over the company and wonder what to do: continue to enforce a system that does not work for the company or jump on the Subversion train? Before that can happen, they need to be convinced that Subversion is scalable, secure and ready for the enterprise and they want to be convinced by more than their own people; they look at what other companies do, where the industry as a whole is moving and what external experts have to say.

If you are one of those people working for a company that uses a legacy system that you would like to see replaced with Subversion, here is some good help for you: Forrester just released the report “The Forrester Wave™: Software Change and Configuration Management, Q2 2007”, which cites Subversion as the sole leader in standalone SCM.

The way these things work, I am actually not allowed to quote much of the report in this blog. I guess Forrester wants you to read the actual report and ensure that nobody places things out of context. Fair enough, they did the hard work and objectivity is important. If you want to read this independent report, you can get a copy from our web site.

There are a couple of things I can share in this blog. For instance: “The Forrester Wave™, which is a graphical representation of Forrester's call on the market. As you can see, for standalone SCM Subversion is indeed the sole leader:

I can also share what Carey Schwaber wrote about Subversion. Ms. Schwaber is Senior Analyst at Forrester Research and her research area is Application Development with a special focus on SCM tools:

"Subversion’s strengths are scalability, administration, and geographical distribution. Subversion’s ability to scale to meet enterprise needs is well established, with single instances managing 7,500 users... Subversion is also easy to implement and administer: Subversion customer references reported initial implementation times of less than a month and administrator-to-user ratios of better than 1:1,000".

So, if you need help convincing your management that Subversion is ready for your company, send them to http://www.collab.net/forrester_wave_report.

              Get this great news out, Digg this story. 

About a month ago, I provided a post in this blog that suggested that enterprises consider taking some of the processes that are used to produce Subversion (as an example of good open source processes): In addition to the code. That stirred some interesting discussions, more offline than online, so I wanted to revisit this topic.

I find it disappointing, if not surprising, that enterprises are so reticent to consider the less restrictive, but still process-driven configuration management procedures that the Subversion project utilizes. I get a lot of initial backlash when I talk about open source processes. The reality is the backlash is not really coming from knowledge, but uninformed impressions. It is almost funny to think of organizations, who depend upon open source technology (e.g., Apache, Linux, Subversion, etc.), rejecting the processes that produced it without true consideration. That's like saying "I love my house and find it to be of high quality, but I wouldn't consider building another one the same way." What happened to good old plagiarism of successful ideas and processes?

For example, visibility seems an obvious approach to better quality that should be enabled by our processes. First, one is going to be more cautious in their work knowing that it can potentially be seen by the broader project team. It is a strong deterrent to shortcutting the coding standards that the project (or the enterprise) has identified. Second, the code benefits by having more eyes reviewing the changes, which means a higher probability of issues being identified earlier in the lifecycle at a lower cost to their enterprise. I realize some people are reticent to having their work "exposed", but that shouldn't be an inhibitor to doing the right thing. Most of the people that you really want doing the work will welcome the chance to share their "brilliance" at a more detailed level. This visibility is achieved to some extent by general read access, but also by posting deltas to a mailing list and its associated archive. Visibility can also be enhanced through an association of the delta with a change request

If visibility is achieved, then it is logical to consider what the impact would be to other processes. Who would consistently violate accepted processes if they knew the team saw those violations and knew the team is impacted by those violations? How would the broader team react to these violations? Aren't people hired with the expectation that they will do the right thing for the enterprise and as an extension of that, for the project team?

Contrast that to implementing a hook that impacts everyone's productivity and, while forcing adherence, can be consistently required for some users versus correcting their behavior. Each user pays a price with every operation just in case they "might" be violating a process. Of course, since we're talking version control (aka a time machine) the changes can be rolled back if a violation occurs. And consider the fact that this won't be isolated to one type of violation, but will be repeated for many different types which will build on the impact to each individual.

I think enterprises need to become more open minded and consider the benefits to the approach the Subversion project takes versus the process enforcement that most lean towards today. Since you've built your house on Subversion (Apache, Linux, etc.), it seems reasonable to look to build other houses in the same way.