Stewart Brand showed “How Buildings Learn”, So Why Can’t Software?

(Or, why Johnny’s development project sucks.)

Stewart Brand interviewed Christopher Alexander, the creator of A Pattern Language and one of the movers in The Oregon Project.

‘Things that are good have a certain kind of structure,’ he told me. ‘You can’t get that structure except dynamically. Period. In nature you’ve got continuous very-small feedback-loop adaptation going on, which is why things get to be harmonious. That’s why they have the qualities that we value. If it wasn’t for the time dimension, it wouldn’t happen. Yet here we are playing the major role in creating the world, and we haven’t figured this out. That is a very serious matter.’ [pp. 21.]

Alexander has some good points. In the end, you can’t know what you will need to have at the onset of your project. You can only get the optimized environment after you’ve lived with it and changed it.

And, of course, it follows that you can’t get an optimized environment for tomorrow’s needs until you change it for tomorrow. So, you’ll never be done; you will always need to make adjustments to your environment, or in response to your environment.

At the same time, Brand’s call for full reconfigurable space is somewhat naive. Think of Cathedral and the Bazaar: Raymond gets it only half right. We need both, at the same time, in opposition to each other. We need the vision of a Burnham with the freedom of anarchy in the small spaces; broad, sweeping visions that have both the power to stir men’s souls and to give us a path for years to come. Burnham’s plan for Chicago is still being followed, still being called up, now some hundred years after he published the advisory master plan for America’s Windy City. Yet this master plan has wide empty spaces — Burnham and Co. did not feel the need to determine every small element in the plan.

This is arguably where we get software architecture wrong: we believe that either everything must be specified for projects or we must live in full uncertainty. Most of the big project people fall into the requirements fully specified camp. Fergus O’Connell even subtitled the third edition of his mostly excellent How To Run Successful Projects, “The Silver Bullet”, meaning that you can use project management techniques that he lays out to properly specify a project’s requirements. But many of the great in Software Development’s pantheon of heroes have come out against such a notion. Even the true vision behind the Capability Maturity Model, Watts Humphrey, comes out against the idea that requirements can be fully known prior to implementation:

There is a basic principle of most systems the involve more than minor evolutionary change: The system will change the operational environment. Since the users can only think in terms of the environment they know, the requirements for such systems are always stated in the current environment’s terms. These requirements are thus necessarily incomplete, inaccurate, and misleading. The challenge for the system developer is to devise a development process that will discover, define, and develop to real requirements. [pp. 167]

The reaction seems to be to throw knowledge out and adopt a rigorous methodology of not knowing, such as Extreme Programming (XP) and its many clones. XP is pretty rigorous: the XP gods are insistent that you cannot do just part of the methodology. It works in an iterative process that uses direct interaction by the development group with the users.

Monolithic requirements produce dead software that meets the requirements of yesterday but may be completely wrong for today’s market situation. Making it up as you go means changing designs and uncoordinated change. The Allies wouldn’t have won the war against Germany’s murderous National Socialism by winging it.

Maybe there is a middle way, a way that incorporates the best of both worlds. The glory of massive planning is that it allows you to do something like Paris remodeled under Napoleon, OS 390, or the D-Day invasion of France. Winging it allows you to do something like Linux, free markets, or the D-Day invasion of France. Wait a second: didn’t I already mention that?

The American forces had a strength that their enemies didn’t. The Germans had better generals, far better technologies and the advantage of having held France for five years. The Allies had a massive planning effort done over months to plan the invasion. They also had the advantage of inventiveness within the ranks, of regular Joes coming up with innovations that turned the tide of battles. Tank commanders always tell the story of having gone to France only to meet the French hedgerows, 3 m monsters that no one bothered to tell them about. If they went over them, they go stuck. If they went through the holes in the hedgerows, the Nazis mowed them down. The solution: clever yanks retrofitted their Sherman tanks to plow right through the hedges. It was in this, and in the inventiveness in their repairs of the ever-failing Shermans that gave the Americans the edge. (It certainly wasn’t the design of the Shermans, a wretched device.)

The Americans then had somewhat decent planning and organization (arguments can be made elsewhere about its actual quality), and a willingness to improvise and innovate when confronted with something unusual. This is a pretty good model for any large software project.

As Matta and Ashenas observe about the World Bank’s projects, you need long-term projects to accomplish these massive shifts. Even the fairly successful American revolution against British power took several years, and then the government had to be redesigned significantly the next decade at the Constitutional Convention of 1789. Long term planning and projects are, quite simply, necessary. However, you need short-term projects that actually deliver something to the end-users so that your staff’s morale can make it through. At the World Bank, they created several 120 day projects that would implement a piece of the ten-year project’s overall goals. During these guerilla projects, the staff learned many things that might derail a large project. They chose to innovate and adapt to the circumstances instead. These small projects are low threat, deliver real value to the end user, get the staff revved up and teach you what will really happen in implementation. Not a pilot but a real delivery. [I recommend their article to all IT managers.]

Zhou has a similar idea. He takes the Capability Maturity Model, that monolith of written process, and adds the taiji of Chinese philosophy (commonly called “that yin-yang thingy” in the Midwest America). You need chaos of going off and creating something new, and you need ever increasingly procedural codification. Not one or the other: both, and.

How do you think this would work in a typical conservative software development group?

Cited works

• Brand, Stewart, How Buildings Learn: What Happens After They’re Built Penguin Books, 1995 [1994].
• Donaldson, Scott E. & Siegel, Stanley G. (2001) Successful Software Development (2nd Edition), Prentice Hall.
• Humphrey, Watts (1990). Managing the Software Process, Addison-Wesley.
• Zhiying, Zhou (2003), “CMM in uncertain environments“, Communications of the ACM. 46(8) [August 2003], pp. 115-119.
• Matta, Nadim F. & Ashkenas, Ronald N. (2003), “Why Good Projects Fail Anyway“, Harvard Business Review, 81(9) [September 2003], pp. 109-114.

Image Credit: Bix sits in confusion (Detail), by Jimmy Thompson. From “Heaven’s to Betsy!”, Club “16” comics