My previous blog entry discussed the issue of repository layout. This entry will try to answer the question of whether you should have one repository per project or a single repository that houses all your projects. There is not going to be a single right answer to this question. Hopefully this post will help you understand the tradeoffs so you can make the right decision that suits your requirements. These are some of the advantages of the single repository approach.

1. Simplified administration. One set of hooks to deploy. One repository to backup. etc.
2. Branch/tag flexibility. With the code all in one repository it makes it easier to create a branch or tag involving multiple projects.
3. Move code easily. Perhaps you want to take a section of code from one project and use it in another, or turn it into a library for several projects. It is easy to move the code within the same repository and retain the history of the code in the process.

Here are some of the drawbacks to the single repository approach, advantages to the multiple repository approach.

1. Size. It might be easier to deal with many smaller repositories than one large one. For example, if you retire a project you can just archive the repository to media and remove it from the disk and free up the storage. Maybe you need to dump/load a repository for some reason, such as to take advantage of a new Subversion feature. This is easier to do and with less impact if it is a smaller repository. Even if you eventually want to do it to all of your repositories, it will have less impact to do them one at a time, assuming there is not a pressing need to do them all at once.
2. Global revision number. Even though this should not be an issue, some people perceive it to be one and do not like to see the revision number advance on the repository and for inactive projects to have large gaps in their revision history.
3. Access control. While Subversion's authz mechanism allows you to restrict access as needed to parts of the repository, it is still easier to do this at the repository level. If you have a project that only a select few individuals should access, this is easier to do with a single repository for that project.
4. Administrative flexibility. If you have multiple repositories, then it is easier to implement different hook scripts based on the needs of the repository/projects. If you want uniform hook scripts, then a single repository might be better, but if each project wants its own commit email style then it is easier to have those projects in separate repositories

This is just a sampling of the pros and cons of each approach. Hopefully it gives you something to go on to make a decision. I tend to prefer the one repository per project approach with the caveat that if I have many projects that are related to each other, I would put those all in the same repository. I also tend to break up repositories by group or team, although in reality this is just a variation of the project concept.

For example, I had one repository for the Documentation department to use for their projects. Of course in the case of on-line help that was often located in the same project as the application code, but the Documentation team also had other materials they produced and I gave them a repository for that. Likewise, the Marketing department had a repository to store the things they worked on, including the company web site. As was the case with the layout of the repository, this is really just a decision of what will work best for you. That  said, it is a little more difficult to change your repository setup after the fact.

So, it is worth it to take some time to understand your requirements and which approach best suits them.

One of my jobs at CollabNet has me consulting with enterprises that are planning to adopt Subversion either for specific projects or as a standard for the organization. These enterprises understand the value of Subversion for meeting their requirements for version control and software configuration management. They appreciate the work that the open source community has invested in this state-of-the-art tool and are excited to leverage that investment. What they often struggle with is their predisposition to use the tool as a "hammer" to enforce processes versus taking a cue from the Subversion developers themselves to improve their own processes. They are open to using an open source tool, but can't see adopting some of the open source processes.

I see a lot of questions asked about "What is the recommended repository layout?", "What does trunk mean?", or: "What is the significance of trunk?". This post will try to answer those questions and more.

A Subversion repository implements the metaphor of a versioned filesystem. The repository is just a filesystem with folders and files. It so happens that modifications to this filesystem are versioned and there are implementation enhancements like "cheap" copies that make certain operations less expensive than they are in a traditional filesystem, but the repository itself still behaves like a filesystem: there are no special folders or names and Subversion itself has no knowledge of trunk or branches, they are just folders within the filesystem. It is up to you as the user to give those folders names and structure that are meaningful to you.

That said, there are several common layouts that have been adopted by the community as best practices and therefore one could think of these as recommendations. If your repository is accessible to the public, following these conventions might make it easier for users that have accessed other Subversion repositories to find what they are looking for.

There are two commonly used layouts:

trunk
branches
tags

This first layout is the best option for a repository that contains a single project or a set of projects that are tightly related to each other. This layout is useful because it is simple to branch or tag the entire project or a set of projects with a single command:

svn copy url://repos/trunk url://repos/tags/tagname -m "Create tagname"

This is probably the most commonly used repository layout and is used by many open source projects, like Subversion itself and Subclipse. This is the layout that most hosting sites like Tigris.org, SourceForge.net and Google Code follow as each project at these sites is given its own repository.

The next layout is the best option for a repository that contains unrelated or loosely related projects.

ProjectA
   trunk
   branches
   tags
ProjectB
   trunk
   branches
   tags

In this layout, each project receives a top-level folder and then the trunk/branches/tags folders are created beneath it. This is really the same layout as the first layout, it is just that instead of putting each project in its own repository, they are all in a single repository. The Apache Software Foundation uses this layout for their repository which contains all of their projects in one single repository.

With this layout, each project has its own branches and tags and it is easy to create them for the files in that project using one command, similar to the one previously shown:

svn copy url://repos/ProjectA/trunk url://repos/ProjectA/tags/tagname -m "Create tagname"

What you cannot easily do in this layout is create a branch or tag that contains files from both ProjectA and ProjectB.  You can still do it, but it requires multiple commands and you also have to decide if you are going to make a special folder for the branches and tags that involve multiple projects. If you are going to need to do this a lot, you might want to consider the first layout.

As for the names of the folders within the repository, again: they are just a convention. They have no special meaning to Subversion.

"trunk" is supposed to signify the main development line for the project. You could call this "main" or "mainline" or "production" or whatever you like.

"branches" is obviously supposed to be a place to create branches. People use branches for a lot of purposes. You might want to separate your release or maintenance branches from your feature branches or your customer modification branches etc. In this case, you could create a layer of folders beneath branches, or just create multiple branch folders at the top-level.

"tags" are not treated as special by Subversion either. They are a convention, perhaps enforced by hook script or authz rules, that indicate you are creating a point in time snapshot. Typically the difference between tags and branches is that the former are not modified once they are created. You might call your tag folders "releases" or "snapshots" or "baselines" or whatever term you prefer.

Remember, the significance of the name is for your benefit, not for Subversion. Finally, the architecture of Subversion, with its global revision number can often make the need for tags unnecessary. I do not think there is any reason to create tags just for the sake of creating them. If you find a need to recreate the software at a specific point in time, you can always do so by using svn log to determine the relevant revision number. Tags are best when there are "external" consumers of the repository. Maybe it is a QA/Release team that needs to perform builds, maybe it is an internal development team that wants to use releases of your code in another product, or maybe it is literally external users or customers who need to grab release snapshots from your repository. In these scenarios, creating tags is both a convenient way to be sure they get the right code, as well as a good communication mechanism to indicate the presence of release snapshots.

Hopefully this post clarifies some of these issues for you and makes it easier for you to understand how Subversion works.

I would like to finish by pointing out that a Subversion repository layout can be changed. You can always reorganize and restructure your layout after the fact. At worst, it just might create some short term pain as users adjust their working copies. It is not like you need to start over though. Just change names, move folders or do whatever to get the filesystem looking the way you want it.

Usually compiling JavaHL is pretty simple: just follow the directions mentioned in the source code. Not on OS X though, there is an anomaly on OS X when running "make javahl" or "make install-javahl". When you run these make targets on OS X, you get output like this:

ld: multiple definitions of symbol ___divdi3
/usr/lib/gcc/powerpc-apple-darwin8/4.0.1/libgcc.a(_divdi3.o) private external definition of 
___divdi3 in section (__TEXT,__text)
/usr/lib/libgcc_s.10.4.dylib(_divdi3_s.o) definition of ___divdi3
ld: multiple definitions of symbol ___udivdi3
/usr/lib/gcc/powerpc-apple-darwin8/4.0.1/libgcc.a(_udivdi3.o) private external definition of 
__udivdi3 in section (__TEXT,__text)
/usr/lib/libgcc_s.10.4.dylib(_udivdi3_s.o) definition of ___udivdi3
ld: multiple definitions of symbol ___umoddi3
/usr/lib/gcc/powerpc-apple-darwin8/4.0.1/libgcc.a(_umoddi3.o) private external definition of
___umoddi3 in section (__TEXT,__text)
/usr/lib/libgcc_s.10.4.dylib(_umoddi3_s.o) definition of ___umoddi3
/usr/bin/libtool: internal link edit command failed
make: *** [subversion/bindings/java/javahl/native/libsvnjavahl-1.la] Error 1

After digging around, I finally joined the #fink irc channel on irc.freenode.net and started asking questions (I heard they had a working script to compile the JavaHL bindings.) After talking to a few people in #fink, they passed me this nice little snippet that fixed the problem:

perl -pi -e "s/-all_load//g" libtool

The above Perl call removes the "-all_load" flag from libtool to work around a libtool issue on the Mac. Going into deeper detail is really a libtool matter and I'll omit that here but let me give you a little cheat sheet for building the JavaHL bindings from scratch. Here are the brief steps:

1. Download Subversion source.
2. Change directory to where you downloaded the Subversion source.
3. Run "./autogen.sh".     
4. Run "./configure <your_flags>". 
5. Run "make". 
6. Run "perl -pi -e  "s/-all_load//g" libtool". 
7. Run "make javahl".     
8. Run "make install-javahl".

If you follow the steps above, you'll have a working JavaHL installation on your OS X system.  Enjoy.

Today we took Ohloh live on openCollabNet. Ohloh is a reporting engine for public projects. Subversion is one of the projects they have a dashboard for. You can see the history of the code base, who the committers are, the history of their commits, languages used and other things. Check out the Subversion dashboard at http://opencollabnet.ohloh.net/projects/1 (note that Subversion is project number 1 in Ohloh’s index, pretty cool).

You can find Ohloh on the home page of openCollabNet and on some of the community sites we host, like Tigris.org.

One of the cool things you can do, is add a “badge” to your own public project that links directly to your dashboard. Here is an example for Subversion:

Find out how to do this for your project at: http://ohloh.open.collab.net/

In the last few blog posts we have mentioned the webinar "Subversion in the Enterprise". A replay of this webinar is availabe from CM Crossroads. The webinar covers:

• A quick introduction of CollabNet (real quick)
• What is Subversion?
• Why Subversion is right for the Enterprise
• What companies are using it for
• A Reuters Case Study (very interesting to hear a large company talk about Subversion)
• Support and Training Options

If you have been reading this blog, then you know that we recently held a webinar on Subversion in the Enterprise. The attendance at this webinar was amazing and we received a lot of questions from attendees during the webinar. I was assigned to answer the questions that related to Subversion tools, since that is my particular area of expertise. So, without further ado...

Does Subversion have a source file differencing tool? Can a third-party differencing tool be supported?

Yes and yes. Subversion includes the diff command. The only limitation it has is that it will only work on files that are managed by Subversion. You cannot use it to difference two arbitrary files, it has to be files within Subversion. However, you can also use it to produce all the differences between two branches or the current trunk and a tag, or just the changes you have made locally in your working copy. The svn diff command can also launch a third-party diff tool if you have one that you prefer. Finally, the Subversion graphical clients I have used (Subclipse, TortoiseSVN etc.) include a graphical diff tool and can also be configured to work with external tools.

I had a recent post about the diff command on this blog.  You might take a look at that post to see some examples of how diff can be used and the syntax of the command.

Can you recommend a Web-based client tool for Subversion?  TortoiseSVN is great, but it requires us to load software on the user's desktop.

I assume you are talking about an actual web-based Subversion client. I am not aware of any that exist, although I have seen people express interest in writing one. So it is possible there are some that exist in some state of development. The truth is that it would be a difficult task to write a good web based client. You would have to recreate all of the Subversion features in the web client and then somehow implement those features on behalf of the user on the web server.

Does CollabNet offer anything to help integrate Subversion with a development environment such as Eclipse?

I am one of the project owners of the Subclipse project which provides support for Subversion inside of Eclipse. The Subclipse team (specifically me) works closely with the Subversion developers to help define the features and API we need to develop a good Eclipse client. There will be a number of enhancements to Subversion coming in the 1.5 release which were specifically added at the request of the Subclipse team and our needs. Of course these enhancement benefit everyone, but we helped define the "itch" so that the developers would "scratch" it.

What clients would you recommend for Eclipse and also Visual Studio?

As a project owner for Subclipse, I would obviously recommend Subclipse for Eclipse. Subclipse has been around for several years and is a mature product that implements all of Subversion's features and has a strong user community. On the Visual Studio side, I would look at two tools. AnkhSVN is an open source project that provides support for Visual Studio 2003 and later. I have heard lots of positive comments from people that use it.

VisualSVN is a somewhat recent entry on the commercial side. I think what they are doing is interesting and the price is very reasonable. They have taken the approach of building their client on top of TortoiseSVN which I think is a great idea because it gets them access to a very full featured user interface and lets them focus on adding the value that is needed to make a good Visual Studio integration.

I do not develop in Visual Studio, so I have never used either product.

Which Eclipse plug-ins support refactoring through the Eclipse context menus?

Subclipse has always supported refactoring within Eclipse. If you perform a refactoring operation that renames/moves a file, Subclipse sees this happening, intercepts the operation and performs it using Subversion so that Subversion knows about it. There are some limitations in Subversion itself that might give the perception that Subclipse does not support refactoring.

1. Subversion does not allow a file/folder to be renamed/moved twice without a commit in between. Subversion trunk code has been enhanced to remove this limitation, so it will be available when 1.5 is released. Subclipse refactoring support will be a lot better because of this change. I have already made the changes in Subclipse to leverage this Subversion change and refactoring now works really nicely. That being said, refactoring works nicely today if you are aware of the limitations and can work around them.
2. Subversion's support for move/rename has some limitations that can effect you when doing an update or merge. You can see a detailed explanation in this post on my personal blog. A short explanation is simply that if you have made some changes to a file, and someone else renames the file and commits the rename, when you do an update, Subversion does not transfer your changes to the renamed file. Subversion is smart enough to not delete the local file with your changes, so you can at least copy them over to the new file manually. Ideally, it would do this for you. The Subversion developers recognize the need to make this improvement, so hopefully it will come in a future release. It is a difficult problem for them to solve and is not planned for the next release (1.5).

Here are some more answers to questions asked during the “Subversion in the Enterprise Webinar”. I  picked a few that seem to come from people who are starting to familiarize themselves with Subversion.

What is meant by global revision numbers? Are individual files not controlled?

Users of most other version control systems are used to version numbers changing at the file level (the revision number changes when the file changes). As a result, if there are 5 files in a tree, each file can have a different version number.

The problem with this approach is that it does not give a unique version number to the set of files that makes the entire tree. This complicates communication because when you talk about version 4 of a file you have to look up what the version numbers are of related files before you can talk about the complete set.

When you commit changes to Subversion, the revision of the whole repository is increased by one, even if only one file was changed.  This has a key advantage: you can now talk about the revision number for the entire tree. It takes a mental change that you’ll soon love: when you switch to Subversion the question “What is revision 5 of the file?” is not really correct anymore. You should ask: “What did the file look like in revision 5 of the tree?”

You said a Branch is a copy of a Directory. How is it accomplished without taking disk space?

Subversion does not technically have branches or tags, it has copies of directories. Subversion handles copies in a smart way. When you “branch” a directory, Subversion does not make copies of the files (why make identical copies in a version control system?). Instead, a branch starts as a reference to the original directory, a very quick operation. After the branch is made, Subversion only tracks changes relative to the original files. Disk space to create a branch is therefore almost zero and the branch can be created in a very short time. This is a key advantage over most version control systems because they create redundant copies of files, which takes a long time and leads to performance problems as their repositories grow.

In Subversion, branches and tags are essentially the same. Any suggestions/recommendations on when to use one or the other?

As explained above, Subversion does not really know about branches, instead it makes “cheap” copies of directories. Branches and tags are names for these copies and we use two terms to describe different use cases.

Typically you make a branch so that you can control and isolate the changes from other change activity that is happening in the repository. Think about maintenance of a shipping product on a branch, while at the same time working on a new release on the trunk. Another use case is working on a large feature that can destabilize a build. Doing this on a separate branch means that other developers can create builds that won’t be affected by the work on that branch.

Tags have another use case. They are used to “set aside a revision” and keep it unchanged. One use case is to keep a version of the code of a shipping product; corporate governance might dictate that and you’ll need it to hunt down defects.

Technically you do not need tags because the exact state of each revision is already in Subversion of course. You could just remember that revision r8810 was used for Release 1.0. But will you remember this? Isn’t it easier to create a tag and call it “Release_1.0”?  Tags are also convenient if you have external consumers of your project code and you want to give them an easy way to grab a specific published version of the code.

Does Subversion define the technology of its repository? Is it file based? DBMS based?

You have a choice. Originally Subversion’s backend used the Berkeley Database system. This is still supported but since version 1.1 Subversion can also use the operating system’s file system. Subversion calls that FSFS (Filesystem Filesystem, as opposed to BDB Filesystem). The Subversion development community has put a lot of energy into FSFS and since 1.2 it is the default backend. CollabNet recommends FSFS over BDB because it is more stable (less chance of wedges) and is easier to maintain and support. Read more about FSFS pros in Subversion’ own repository.

Is there documentation on both best and bad practices with Subversion?

We have a very popular document right here on openCollabNet: Subversion Best Practices. As for bad practices, great idea for a future blog post.

Scalability

There were quite a few questions on scalability. Like: “Can I use it with 200 developers?” Or: “Can Subversion handle half a million files?”

We get these questions all the time because scalability is an issue with many other systems out there; specially some of the big commercial ones. Competitive systems often fall over when you scale your deployment. They require or need one administrator for every few dozen developers, a heavy server for every 50 developers or the load corrupts your repository again before you fixed the last problem.

A typical Subversion deployment uses one central server and that’s probably why people worry: can one server really handle a thousand developers?” Yes, it can! Subversion does not need the server much. Developers check out files and work locally. Only in a few cases do they need to establish contact with the server; for instance to commit a change or to update their local working copy.  Because of its architecture Subversion scales very well. CollabNet has customers with thousands of developers using Subversion on inexpensive server hardware and with only one or a few administrators.  The Apache, KDE and GCC projects are examples of very large repositories with a lot of developers (5000+ for Apache).

Also, Subversion keeps the size of the repository small. We discussed that in an earlier post. But to be fair, the question: “Can Subversion handle half a million files?” needs context because it depends what these files are. I’m sure Subversion will behave differently with half a million large videos than with half a million typical source files.  Assuming that we are talking about the latter, half a million files is not a problem;  CollabNet has customers with more files than that.

Where can I get the Subversion Book?

There are a number of books on Subversion. The only English language book updated for version 1.4 is “Practical Subversion” by Dan Berlin and Garrett Rooney. If you read German, there is a Subversion 1.4 book for you. Another publication: "Version Control with Subversion" is considered the standard book on Subversion but the printed edition is not current with release 1.4. However, the authors are updating this book and because it is an open source project, you can always see the latest and greatest. Visit http://svnbook.red-bean.com  to read or download the original book or refer to nightly builds of the 1.4 update.

I participated in a webinar two weeks ago, Subversion in the Enterprise, in which we had a limited chance to respond to questions. We’ll answer some of the remaining questions here.

One topic that came up multiple times was enterprises migrating from Rational ClearCase. Since I spent 5 years helping sell, implement and support ClearCase, and now 6 years at CollabNet, I understand both worlds pretty well.

Before focusing on ClearCase though, I encourage you to read some very good migration articles that can be found in the Subversion migration project. I talk to big companies every day about Subversion and many of them are considering to migrate or have although done so. These articles are a good start to prepare for a migration.

When planning a migration, the most important thing you need to do is realistically determine what data you really will benefit from migrating, rather than just assuming you want to migrate everything. For one thing, you don’t want to clutter your new repository. If you move from one house to another, you have garage sales, donate items to charity, and throw things away. The same concept should be applied to version control migrations. What really do you need in your new "house"? Realize that you have to pay the moving costs for whatever you migrate so be sure that the value you realize is worth more than the cost.

I always suggest leaving historic data in the legacy tool and just taking the latest from the live branches or perhaps snapshots of key milestones from current development branches. You should minimize your investment in the legacy tool both in terms of licenses and hardware, but retain it as the historical archive. A potential alternative is to migrate the historical data to its own Subversion repository while migrating limited data into the “production” repository. That way, history is retained (with the clutter stored elsewhere) and the legacy tool is eliminated along with its licensing and support costs.

Any migration will have limits based on the differences between the legacy system and Subversion as well as the extent to which the migration tool’s author(s) decided to map functionality and metadata between the two tools. With ClearCase, it is not a question of "will there be limits?", but rather how many limits. ClearCase doesn’t really support data migration so the development of migration tools is very challenging. That said, I am aware of at least a couple of attempts at providing such a tool. The first was a tool developed by an enterprise specifically for their needs and situation. It had limitations that they were willing to accept but the tool will unlikely be acceptable to most other companies.

Most of the focus these days is on the SVN Importer tool which purports to support migrations from many different version control tools. Unfortunately, a brute force approach - limited to a full migration - is taken to ClearCase and to most of the other systems. Such an approach requires a lot of system resources (e.g., disk and memory) to execute. That has proven itself to limit the size of a source VOB that can be migrated using a relatively “average” machine. One customer has seen that limit at a 150MB VOB, but the limit will vary depending on the hardware used for the migration. This customer had one VOB larger than the tool could support but fortunately they wanted to split the VOB contents anyway so they split it and executed two migrations resulting in two Subversion repositories.

As for case studies of enterprises successfully migrating from ClearCase, I certainly see it happening, particularly with companies that are truly doing distributed development and those who don’t need the unique features of ClearCase. Some companies have migrated history but most are leaving it behind. In my experience, the biggest key to success is not to recreate ClearCase processes in Subversion, but rather revisit your organization’s version control needs and map Subversion’s features and proven best practices to meet those needs.

Keep in mind that companies usually run a pilot before making a commitment to something as critical as a new version control tool. When they commit, they do so based on their own documented experiences. For a migration from ClearCase, I highly recommend you do a pilot first. After that you can join the companies that are enjoying the many benefits Subversion brings over ClearCase, such as atomic commits, real support for globally distributed development and consistent performance with tags and branches.

If you need help with your migration, our Professional Services team has a lot of experience helping customers move to Subversion. Check out our corporate web site for more information.