Most Subversion users are familiar with text conflicts. The classic case: You have a locally edited file in your working copy, an svn update brings a change to the same file from the repository, that incoming change cannot be merged cleanly into your local change, and the result is a text conflict. Subversion 1.6.0 expands this concept to cover conflicts at the directory level, e.g. you locally delete a file then an update tries to bring a text change down on that file. These new types of conflicts are called tree conflicts.

Let's look at a simple example of what you can expect with tree conflicts, but first we'll look at the "old" behavior under Subversion 1.5.6.

Like text conflicts, tree conflicts can occur during updates, switches, or merges (and technically you can get them on checkouts too). In this example we will look at a merge from our 'trunk' into our current working directory which is a working copy for a branch of trunk.

1.5.6

In our working copy we have a file with some text edits:

1.5.6> svn st
M notes\obliterate\obliterate-functional-spec.txt

Checking what is available to be merged from trunk we find r36680:

1.5.6> svn mergeinfo --show-revs eligible %URL%/trunk .
r36680

Consulting the log for r36680 we see that Barry renamed the very file that we have local edits on:

1.5.6> svn log -v -r36680 %URL%
------------------------------------------------------------------------
r36680 | Barry | 2009-03-19 11:21:25 -0400 (Thu, 19 Mar 2009) | 1 line
Changed paths:
A /trunk/notes/obliterate/obliterate-func-spec.txt (from
/trunk/notes/obliterate/obliterate-functional-spec.txt:36679)
D /trunk/notes/obliterate/obliterate-functional-spec.txt

Just making a file name a bit shorter
------------------------------------------------------------------------

Recall that Subversion performs renames as a combination of a copy and a delete. When we merge r36680 the addition half of the rename occurs, but the deletion half does not because the file to be deleted has local modifications and Subversion generally endeavors to not remove unversioned modifications:

1.5.6> svn merge %URL%/trunk . -c36680
--- Merging r36680 into '.':
A notes\obliterate\obliterate-func-spec.txt
Skipped 'notes\obliterate\obliterate-functional-spec.txt'

After the merge we are left with two versioned copies of our spec file, the one added by Barry as part of the rename and our original local edit. We could commit this as-is, but it is quite unlikely this is what we really want. Getting the correct result in this case, namely our text edits in the renamed file, requires jumping through a few hoops due to Subversion's method of treating renames as separate copy and delete actions:

First we need to copy our modified file onto its new location with an OS copy:

1.5.6> copy notes\obliterate\obliterate-functional-spec.txt notes\obliterate\obliterate-func-spec.txt
Overwrite notes\obliterate\obliterate-func-spec.txt? (Yes/No/All): y
 1 file(s) copied.

Then use Subversion to revert our original locally modified file:

1.5.6> svn revert notes\obliterate\obliterate-functional-spec.txt
Reverted 'notes\obliterate\obliterate-functional-spec.txt'

Which now allows us to delete it:

Alternatively you could simply use the --force option with delete and combine the first two steps. IMHO that is a bad habit though, as it is a good way to wipe out all your local changes when you don't mean to. Recall how I said Subversion endeavors preserve unversioned local modifications? svn delete --force is one of the cases outside of the revert subcommand that does just that, so use with caution.

1.5.6> svn del notes\obliterate\obliterate-functional-spec.txt
D notes\obliterate\obliterate-functional-spec.txt

Leaving us with what we likely want, 'notes\obliterate\obliterate-func-spec.txt' added but with our original local changes.

1.5.6> svn st
 M .
D notes\obliterate\obliterate-functional-spec.txt
A + notes\obliterate\obliterate-func-spec.txt

Now in the above example it is fairly obvious what happened. But what if we were merging in hundreds of revisions and changing hundreds of paths? It would be quite easy to miss those 'Skipped' messages and commit the merge. Our mistake might not be caught until much later.

1.6.0

Now let's look at what happens with tree conflict handling in 1.6.0. Given the same starting branch working copy:

1.6.0> svn st
M notes\obliterate\obliterate-functional-spec.txt

We perform the same merge. Notice the first significant change wrought by tree conflicts, in the notifications we see a tree conflict reported rather than a skip:

1.6.0> svn merge %URL%/trunk . -c36680
--- Merging r36680 into '.':
A notes\obliterate\obliterate-func-spec.txt
C notes\obliterate\obliterate-functional-spec.txt
Summary of conflicts:
 Tree conflicts: 1

Checking the status of our working copy we see a second difference compared with 1.5.6. The addition half of the rename occurs as before, but now we see a tree conflict reported on 'notes\obliterate\obliterate-functional-spec.txt' (the 'C' in the 7th column). Also, there is the additional information about the nature of the tree conflict, specifically that we have a'local edit' with an 'incoming delete' when we performed a 'merge'.

1.6.0> svn st
 M .
M C notes\obliterate\obliterate-functional-spec.txt
A + notes\obliterate\obliterate-func-spec.txt

Let's assume for a moment we don't care about the tree conflict and want to commit anyway. Here is a third significant change, Subversion will not allow you to commit a working copy with unresolved tree conflicts:

1.6.0> svn ci -m "I don't care what happened! Commit away" .
svn: Commit failed (details follow):
svn: Aborting commit: 'C:\SVN\1.6.0.WC\my_branch_WC\notes\obliterate\obliterate-functional-spec.txt' remains in conflict

To commit this merge we need to decide what we want. If for some reason we wanted both files to remain we could simply resolve and commit the change. Chances are we want to apply our changes to the new file. In that case we would resolve the conflict:

1.6.0> svn resolve --accept working -R .
Resolved conflicted state of 'notes\obliterate\obliterate-functional-spec.txt'

1.6.0> svn st
 M .
M notes\obliterate\obliterate-functional-spec.txt
A + notes\obliterate\obliterate-func-spec.txt

Then follow the same steps we did in 1.5.6.

Note: In 1.6.0 the various --accept options to the svn resolve subcommand have no differing effects on tree conflicts. As far as tree conflicts go, the result is the same as if you used the deprecated svn resolved command. Only with traditional text conflicts does the --accept option matter.

It is worth remembering that the concept of tree conflicts is not new. You could have a tree conflict in any version of Subversion prior to 1.6. The 1.5.6 example above is a "tree conflict", it simply isn't flagged as such nor handled in a very useful way. 1.6.0's tree conflict features are primarily about identifying tree conflicts and preventing them from getting into the repository without first being resolved.

As with any new feature, more remains to be done in future releases. In this case making command line resolution of tree conflicts easier is a high priority. If you are a user of the Eclipse based Collabnet Desktop however, then you are in luck. The 1.8 release of the Desktop, scheduled for March of 2009, includes a host of tree conflict resolution features. These features do much of the "heavy lifting" of tree conflict resolution for you. In our previous example you could easily merge your local changes from 'obliterate-functional-spec.txt' to 'obliterate-func-spec.txt' and remove the former from version control and from disk, all via one dialog. So if you are interested in 1.6 for its tree conflict features, but are not using the Collabnet Desktop, it might be a good time to check it out.

Subversion 1.5 introduced that idea of sharding for FSFS-backed repositories. For every commit to a FSFS repository, Subversion creates a single file which describes all the changes in that revision. Prior to 1.5, all of those files were stored in a single directory, which had several drawbacks: incremental backups took a long time, the repository could not be dymnically grown across different filesystems, and some filesystems have degraded performance when the number of directory entries grows too large. With sharding, these revision files were split into separate subdirectories, eliminating a large number of these problems.

Even with sharding, the filesystem still has some inefficiencies. For instance, due to the block size of the underlying filesystem, having many files can still lead to wasted space on disk, especially with many small commits. Subversion can open and read data from many revisions over the course of an operation, and using a large number of files means that Subversion can not exploit various operating system-level caches. Backing up and restoring a repository, although quicker, can still take a long time because of the large number of files spread across the repository.

One of the great ideas that came out of the 2008 Subversion Developers' Summit was the notion that FSFS filesystems could be packed, that is, all the files in a completed shard could be glued together to create a single monster revision file.  This pack file would save space on disk, give the operating system a chance to do some caching, and generally improve the snappiness of the system.

In order to use FSFS packing, you simply need to ensure that target repository has been upgraded to the latest format, and then pack the repository using svnadmin.  Note that repositories do not automatically pack themselves, so for heavily used repositories, you may want to install a cron job or post commit hook to do the packing.  Users can continue to use the repository while it is being packed:

$ svnadmin upgrade repo
Repository lock acquired.
Please wait; upgrading the repository may take some time...

Upgrade completed.
$ svnadmin pack repo
Packing shard 0...done.
Packing shard 1...done.
Packing shard 2...done.
Packing shard 3...done.
Packing shard 4...done.
...
Packing shard 36...done.
$

To give an idea of the potential space savings, on my local 1.5-era copy of Subversion's own repository I get the following results:

$ du -sh svnrepo-1.5/
659M	svnrepo-1.5/
$ 

While on a packed 1.6 copy of the same repository, with rep-sharing enabled, I see the following:

$ du -sh svnrepo-1.6/
593M	svnrepo-1.6/
$ 

That's more than a 10% decrease in space, at no cost in performance. These space savings will vary depending upon your own repository and use habits, but we're excited about the improvements in the FSFS backend in Subversion 1.6.

"cmpilato ❤ sparse directories"

If I had a dollar for every time I've typed that… well, you and I could at least spring for some Fazoli's fast-food Italian. Okay, I admit that the emotion doesn't always drive me to public expression, but that in no way diminishes my fondness for this feature.

Introduced in Subversion 1.5, sparse directory support is one of a few features from that release (besides merge tracking and foreign repository merges) that I've fully integrated into my day-to-day activities. I dig organization. I tend to keep a pretty neat home directory. But I routinely work on several different pieces of software, and at any given time, I'm tracking several different development branches in each of those pieces of software. Were I not using sparse directories, my "projects" directory would look something like this:

$ ls ~/projects
subversion/		      svnbook/		 viewvc-1.0.7/
subversion-1.5.x/	      thotkeeper/	 viewvc-1.0.x/
subversion-1.6.x/	      thotkeeper-0.3.0/  viewvc-1.1.0-beta1/
subversion-http-protocol-v2/  viewvc/		 viewvc-1.1.x/
$

On the positive side of things, I could quickly update all my working copies by simply running svn update ~/projects/*.

But those of you who have command-line completion as hard-wired into your habits as I do will immediately notice that so many common directory prefixes does a useless completion environment make. And not using common prefixes? Well that's just barbaric.

Fortunately, sparse directories has given me a whole new perspective on working copy organization. Now, my projects directory contains (gasp!) only projects, and looks like this:

$ ls ~/projects
subversion/  svnbook/  thotkeeper/  viewvc/
$

Those directories are sparse checkouts of the root directories of their respective project repositories. Beneath them are (at least) "trunk" directories, probably "branch" and some of its children, and maybe even "tags" with some of its children in certain cases. svn up ~/projects/* still works, and my working copy topology matches that of the repositories.

I won't go into the details of how sparse checkouts works here — I've already documented the Subversion 1.5 implementation of them in the second edition of Version Control With Subversion (which you can read at http://svnbook.red-bean.com/en/1.5/svn.advanced.sparsedirs.html). The point of this blog post is to tell you about how Subversion 1.6 improves this feature in a key way.

In Subversion 1.6 (slated for release Any Day Now), the --set-depth parameter to svn update has grown a new value — exclude. This value tells Subversion to exclude the target from the working copy, immediately and until further notice. Prior to Subversion 1.6, if a branch I was working on was no longer of interest to me, I couldn't easily remove it from my working copy. If I simply deleted it, it would return the next time I updated the working copy. If I svn delete'd it, the branch remained as a local modification forever. (Unless, of course, I accidentally committed it, which brought a whole different sort of trouble to my doorstep. Angry peers. Pitchforks and torches. It was a mess — you don't want to go there.) The new exclusion mechanism in Subversion 1.6 is the Right Way To Do It.

Say I no longer care about what's going on some directory of one my project working copies. Maybe I don't care about the Subversion project's website any more. Well, with this new exclusion feature, I can tell Subversion to remove that directory:

$ cd ~/projects/subversion/trunk
$ svn update --set-depth=exclude www
D         www
$ ls www
ls: cannot access www: No such file or directory
$

Done deal. When I update my working copy in the future, I will not receive any changes aimed at that www directory. If I later decide that I once again care about that directory, I can "resubscribe" to it again:

$ svn update --set-depth=infinity www
A    www
A    www/links.html
A    www/testing-goals.html
…
A    www/tigris-permissions.html
A    www/webdav-usage.html
Updated to revision 36292.
$

Note that if you exclude a versioned directory that has some unversioned files in it, or some files with local modifications, Subversion handles this situation gracefully. All the files that aren't safe to delete, Subversion leaves around, and of course leaves any intermediate directories required to reach those files, too.

I hope this enhancement serves you as well as it has served me. What do you think? How are you using sparse directories to better organize your life?

My previous blog entry discussing Subversion, Apache and LDAP is nearing two years old. It was written when Apache 2.0.x was still the mainstream and when Apache 2.2.x was released, changes in the LDAP modules and their respective configuration directives has left my previous entry very confusing for those wanting to use Apache 2.2.x. The purpose of the Definitive Guide is to provide a single location for questions for Apache 2.0.x and 2.2.x, while also providing more depth about things to consider when building your Apache-based Subversion server using LDAP for authentication.

The Configuration

For those of you that just want to get to the point, where you can copy and paste and move on, here you go:

Example Apache 2.2.x Configuration Snippet

# Load Apache LDAP modules
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

# Load Subversion Apache Modules
LoadModule dav_svn_module     modules/mod_dav_svn.so # Use full path to SUBVERSION_HOME/bin/mod_dav_svn.so on Windows
LoadModule authz_svn_module   modules/mod_authz_svn.so # Use full path to SUBVERSION_HOME/bin/mod_authz_svn.so on Windows

# Work around authz and SVNListParentPath issue
RedirectMatch ^(/repos)$ $1/

# Enable Subversion logging
CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION

<Location /repos/>
  # Enable Subversion
  DAV svn

  # Directory containing all repository for this path
  SVNParentPath /subversion/svn-repos

  # List repositories colleciton
  SVNListParentPath On

  # Enable WebDAV automatic versioning
  SVNAutoversioning On

  # Repository Display Name
  SVNReposName "Your Subversion Repository"

  # Do basic password authentication in the clear
  AuthType Basic

  # The name of the protected area or "realm"
  AuthName "Your Subversion Repository"

  # Make LDAP the authentication mechanism
  AuthBasicProvider ldap

  # Make LDAP authentication is final
  AuthzLDAPAuthoritative on

  # Active Directory requires an authenticating DN to access records
  AuthLDAPBindDN "CN=ldapuser,CN=Users,DC=your,DC=domain"

  # This is the password for the AuthLDAPBindDN user in Active Directory
  AuthLDAPBindPassword ldappassword

  # The LDAP query URL
  AuthLDAPURL "ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)"

  # Require a valid user
  Require valid-user

  # Authorization file
  AuthzSVNAccessFile /subversion/apache2/auth/repos.acl
</Location>

Example Apache 2.0.x Configuration Snippet

# Load Apache LDAP modules
LoadModule ldap_module modules/mod_ldap.so
LoadModule auth_ldap_module modules/mod_auth_ldap.so

# Load Subversion Apache Modules
LoadModule dav_svn_module     modules/mod_dav_svn.so # Use full path to SUBVERSION_HOME/bin/mod_dav_svn.so on Windows
LoadModule authz_svn_module   modules/mod_authz_svn.so # Use full path to SUBVERSION_HOME/bin/mod_authz_svn.so on Windows

# Work around authz and SVNListParentPath issue
RedirectMatch ^(/repos)$ $1/

# Enable Subversion logging
CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION

<Location /repos/>
  # Enable Subversion
  DAV svn

  # Directory containing all repository for this path
  SVNParentPath /subversion/svn-repos

  # List repositories colleciton
  SVNListParentPath On

  # Enable WebDAV automatic versioning
  SVNAutoversioning On

  # Repository Display Name
  SVNReposName "Your Subversion Repository"

  # LDAP Authentication is final
  AuthLDAPAuthoritative on

  # Do basic password authentication in the clear
  AuthType Basic

  # The name of the protected area or "realm"
  AuthName "Your Subversion Repository"

  # Active Directory requires an authenticating DN to access records
  AuthLDAPBindDN "CN=ldapuser,CN=Users,DC=your,DC=domain"

  # This is the password for the AuthLDAPBindDN user in Active Directory
  AuthLDAPBindPassword ldappassword

  # The LDAP query URL
  AuthLDAPURL "ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)"

  # Require authentication
  Require valid-user

  # Authorization file
  AuthzSVNAccessFile /subversion/apache2/auth/repos.acl
</Location>

(The configurations above were for pointing to an Active Directory (AD) server.

Understanding the Configuration

So...the above Apache configurations are what I personally use when building an Apache-based server. Obviously there are changes that need to be made depending on the environment in but for now, it's a great start. To make the best of this opportunity, let's talk about the miscellaneous parts of the configuration.

SVNListParentPath and Subversion's authz

One of the first problems people run into when building an Apache-based Subversion server is when they want to have mod_dav_svn serve a list of repositories. Everything works fine until they enable Subversion's authorization (authz) support. What happens is the server will be configured properly and secured properly but when you go to the repository collection list, which in our case is http://localhost/repos, you are forbidden to view the collection even if you have access. Well, with the RedirectMatch closer to the top of the configuration, you fix this issue. How you might be asking and the reason is that when you enable authz, you must have a trailing slash at the end of the collection url. With the RedirectMatch, we automatically redirect urls to the collection listing when there is no trailing slash. Problem solved.

Custom Subversion Logging

Subversion uses Apache's WebDAV support for providing access to its repositories when using Apache. Unfortunately, when you look at Apache's access logs to try and see your Subversion usage, you end up with a lot of WebDAV communication being logged and you only see a portion of the actual client/server communication. This is because mod_dav_svn uses Apache subrequests and Apache does not log subrequests. Even if it did, turning the Subversion communication in the Apache access log into something meaningful would be nearly impossible. That being said, the configuration above has been setup to use one of Subversion's features: Apache Logging which takes the guess work out.

Subversion Configuration

The other Subversion-specific parts of the Apache configuration are pretty self-explanitory. To summarize what is enabled with the above:

• SVNListParentPath: Enables the ability to browse the location root and get a list of repositories being served by that url base
• SVNAutoversioning: Enables the use of WebDAV clients to make changes to the repository contents without using a Subversion client
• SVNParentPath: Enables serving N number of repositories for the url base
• SVNReposName: Enables you to put in your own text to be visible in the web browser when browsing your repository contents via the built-in repository browser provided by mod_dav_svn
• AuthzSVNAccessFile: Tells Subversion's mod_authz_svn module where to find the authz file.

For more details about the Subversion-specific Apache directives, and a list of even more ways you can configure your Apache-based Subversion server, view the mod_dav_svn and the mod_authz_svn documentation.

LDAP Configuration

The LDAP portion of the Apache configuration is where most people run into problems. That being said, we'll spend a little more time explaining the Apache LDAP configuration. The most important thing to note is the subtle differences between Apache 2.0.x and Apache 2.2.x:

Apache 2.0.x           | Apache 2.2.x
-----------------------------------------------
AuthLDAPAuthoritative  | AuthzLDAPAuthoritative
AuthLDAPBindDN         | AuthLDAPBindDN
AuthLDAPBindPassword   | AuthLDAPBindPassword
AuthLDAPURL            | AuthLDAPURL
                       | AuthBasicProvider

You should note that the Apache LDAP module names have also changed between Apache 2.0.x and 2.2.x. Now that we see the naming changes, let's talk about how to properly use these Apache directives to get the LDAP-based authentication you're looking for. (I will be using the Apache 2.2.x names for the Apache directives. If you're still using Apache 2.0.x, please refer to the table above for how to take my documentation and apply it to Apache 2.0.x.)

• AuthzLDAPAuthoritative: Tells Apache whether or not a failed authentication request can be passed to other Apache modules
• AuthLDAPBindDN: The distinguished name of the user account that Apache will use to connect to the directory system to perform its user authentication
• AuthLDAPBindPassword: The password for the user account configured via the AuthLDAPBindDN directive
• AuthLDAPURL: This is a url that tells where the directory server is, where to look for users at, what user attribute is used to identify a user and other miscellaneous things specific to the LDAP query syntax (More on this later.)
• AuthBasicProvider: This tells Apache which authentication module you want to use for Basic authentication

All of the directives above are pretty straight forward except for the AuthLDAPURL directive. This directive we will discuss in more detail below. For any other Apache configuration questions, please resort to the Apache Documentation for your respective Apache version.

The LDAP Query URL

For most, the AuthLDAPURL directive is the most challenging to understand. There is good reason for this. That one directive actually consists of 6+ pieces of information that will be different for each Subversion server. Let's break our example AuthLDAPURL into its pieces and discuss the importance, and nuances, of each.

For simplicity, here is the url again, in its entirety: ldap://your.domain:389/DC=your,DC=domain?sAMAccountName?sub?(objectClass=*)

• Url scheme: [ldap] This is nothing more than a url scheme. It will usually be either 'ldap' or 'ldaps' in the event that you're using SSL for accessing your directory server.
• Hostname: [your.domain] This is the ip address or hostname of your directory server.
• Port: [389] This is the port the server is listening on for directory server communication.
• Search Base: [DC=your,DC=domain] This is the distinguished name to the path in the directory tree that you want to search for users.
• Username attribute: [sAMAccountName] This is the attribute contains the login name being used.
• Query scope: [sub] This tells the directory server what type of query to perform.
• Filter: [(objectClass=*)] This tells the directory server to filter the query for objects matching a particular filter

For more details on constructing an ldap url, which is a standard and not specific to Apache, view RFC 2255.

Working with Active Directory

Active Directory is known as a Multi-Master Directory System. This being said, each directory server in AD does not always have all the necessary information to perform all directory server requests. The best way to handle this is to have Apache query a Global Catalog. A Global Catalog server has the ability to search at the whole forest for users. This means if you want to do domain-wide searches or larger, you need to point to a Global Catalog and you need to update your Apache configuration accordingly. When using a Global Catalog, you should be using port 3268 when performing your queries.

Searching for Users

In the example url above, the sAMAccountName attribute is used to identify the username. This attribute is Windows/Active Directory specific so for those of you using OpenLDAP or another option, that attribute probably will not exist. Change your attribute accordingly. An example is if you wanted to use the Common Name to login, you could specify "CN" as the attribute.

LDAP Query Tuning

The last thing we will talk about is the ability to use filters to make your LDAP query a little more specific. In the example url above we used "(objectClass=*)", which will search for all objects. If you know that you only want to search for a particular object type, like the "user" type, you could use "(objectClass=user)" instead.

Conclusion

Building an Apache-based Subversion server with LDAP as the authentication mechanism can be daunting for some. I hope this has made things easier for you.