Bitkeeper and tossing crates of money out the windows (vistas)

wish I was like Mr. Gates, all my money in big crates – Bruce Springstein

One of the many peculiar features of the software industry is the stubborn manner in which important innovations are ignored. If you read this account of Microsoft’s source code managment methods and this more believable response, and you know about BitKeeper, you can’t help but be struck by how much money Microsoft was throwing away in order to keep using obsolete technology.

A source management system is essentially a data base recording changes to a program. When a problem is found in running a program under development, the source management system allows developers to peel back changes to find the culprit. Without such a system, developers are reduced to the kind of painstaking line by line comparison of differences that should be reserved for obsessive James Joyce scholars and many projects have sunk because of poor record keeping. A complex program can be almost impossible to debug during the development process, but source management allows us to think of the program a collection of more manageble components – the changes that have been added over time.

Things become more complicated when the program is so complex that several developers need to work on it concurrently. If groups A, B, and C are all adding changes at the same time, these changes can clash and tracing which change caused which problem is harder. The traditional way to manage such problems is to create development “branches” – copies of the program that can be worked on independently. The primary copy is called the “root”. Each group starts out by with a “check out” of a copy of the root, works on it until they have something that meets some quality criteria (hopefully) and then “commits” their changes back to the root. Unfortunately, while one group is programming away, other groups will “commit” so that the root is no longer identical to what it was, and a combination of smart commit software in the source management program and effort by the programmers is needed to resolve incompatibilities.

When a development effort is so big that branches have sub-branches and sub-branches have sub-sub-branches and so on, ad infinitum the process can become daunting. If group A uses a module developed by group B and finds a bug, then A needs to wait for B to fix the bug and commit it and then for the fix to percolate up the sub branches and so on until it gets to the root – at which point A may find the fix breaks something else and so on.

In 1733, Johnathan Swift, in his On Poetry: A Rhapsody, wrote:
So Nat’ralists observe, a Flea/Hath smaller Fleas that on him prey,
And these have smaller Fleas to bite’ em
And so proceed ad infinitum.
In the 19th century, an unknown author, possibly Oliver Wendell Holmes, parodying Swift, wrote
Big bugs have little bugs
Upon their backs to bite them.
And little bugs have littler bugs,
And ad infinitum.

Well you can see the problem. Or if you can’t, go back and read the more believable account of Vista’s problems.

This problem of branching source management is not new, but it was solved a couple of years back in BitKeeper. One of the brilliant insights behind Bitkeeper was that if you kept track of information in the right way you could throw out the tree structure. So instead of waiting for group A to merge back into main, group B can just “pull” from A to get its changes directly. Then if group C needs fixes from A and B it will pull from B getting both the B and A fixes which can then be merged back into the main. Thanks to some smart software and a number of “obvious in retrospect” ideas about how to record change data, Bitkeeper can turn a tree into a network. Instead of waiting for changes to wend their way up and down tree branches, groups can pull in what they need and let the source code management software take care of keeping it all straight. The effects on software productivity are remarkable – one of the 3 or 4 reasons why FSMLabs has been able to blow away software organizations 20 times our size is our fanatical use of Bitkeeper.

In a rational market, the kind that economists like to fantasize about, the immediate response would be to give the Bitkeeper engineers a lot of money and incorporate the technology into most software projects. But the market is not rational. Engineers and engineering management resist change that makes them learn something new or admit that someone else was smart. “That’s obvious, I did something like that at my old company” and “we could reimpliment that easily” are standard refrains of this business. (Isn’t it about time that “something like that” was regarded skeptically and the difference between “obvious” and “obvious in retrospect” was commonly understood in software engineering?). Company managers resist change that makes them take a risk on something new – “but X is the standard solution”. And accounting procedures in many companies make spending money on buying software less acceptable than spending much more money on working around the absence of that software. We run into the same phenomena with RTCore and it complicates life. For example, we see that our customers projects succeed at a much higher rate than industry norm, but is how much of this is due to the quality of our products and how much of it is due to the self selected nature of a customer group smart enough to make sensible purchases of innovation?