If you dig a ditch slowly, you make progress anyway. But when programming, even if hacking away a dirty solution might seem quick, it might backfire in hindering your progress in the long run. And when this happens, it wrecks the best intentions of agile project management to predict reliably.

Scrum is the best thing that happened to software project management since the invention of sweets in meetings. Scrum frees developers from excessive meetings and paperwork, and helps them to communicate their problems and feelings. And most importantly, the sprint burndown chart provides a good, empirical estimate on whether the project is on schedule or not.

It’s excellent that big software companies are publicising their use of Scrum.

Scrum is not a silver bullet, but rather like a good pair of scissors that does its limited job well. To succeed, Scrum needs several more things such as

  • good communication
  • good communication (worth repeating, don’t you think?)
  • a hard-working product owner that tells the developers about the wanted features and their relative priorities
  • a motivated, self-organising team
  • a scrum master that coaches and supports, without commanding and controlling
  • adequate tools for handling the product and sprint backlogs
  • good programmers

A real-world experience with Scrum with good “lessons learned” material can be found in the presentation (PDF) of Timo Lukumaa in the September 2005 Agile Finland session.

I will now dwell on my last point about Scrum — or indeed any agile project management — needing good programmers. This is by no way a novel nor original idea. For example Pekka Abrahamsson from the VTT of Oulu underlined that agile software development is not for beginners in his excellent “Agile Software Development” presentation in the inaugural Finland Agile Seminar of March 2005. But I must confess that I haven’t deeply accepted this view until recently.

I used to think that agile methods might help just about any project or programming feat, and specifically that even junior developers could benefit from them. And surely they can. But I will now try to explain why I conclude that an agile project needs skilled programmers to succeed.

At the heart of agile project management is the idea of linear, quantifiable progress: the velocity of the project can be measured by the rate of completing the wanted features, and this is used to predict the schedule. This is exactly the function of the sprint burndown chart of Scrum, mentioned above. The developers break each feature down to measurable tasks, estimate the relative sizes of the tasks, and the process tracks the rate of completing the tasks. And this is how you get data on the velocity of completing the features: if making the full text search is a task twice the size of making the login screen, and making the login screen took 17 hours, by all probability making the full text search will take 34 hours. Scrum rocks because it gives this kind of constantly-improving data with minimal effort.

But creating software is tricky. Craig Larman says that creating software is new product development rather than predictable or mass manufacturing, and this goes well in line with the famous thought of Jack Reeves that source code is design. Creating software is not like cutting a plank, where you could safely say that you are 50 % ready when you have reached the middle of the plank with your saw. It’s not like building a toy cottage where having the walls standing firmly and lacking the roof would show that you’re well on the way.

Instead, what we software developers develop is actually a collection of source code listings that do not serve for the end purpose of our work until they are built into machine code that is being executed in a computer. And to get there, the software must compile, run, and when running do useful things.

This is where the non-linearity comes in. When making more features, it is all too easy to break the features that were supposedly completed earlier. It’s very easy to break the source code in such a way that it does not even compile to executable form anymore. But most importantly, it’s too easy to grow the system in a way that increases the cost of changes by time.

At least since Kent Beck’s first 1999 edition of Extreme Programming Explained: Embrace Change the idea that the cost of change does not need to increase by time has been a central agile tenent. And Scrum seems to assume that the cost of change within the project remains constant, because there is not any kind of a coefficient for the velocity that would be a function of time. In my login screen / full text search example, it should not matter in which iteration of the project the full text search would be implemented.

Robert C. Martin (“Uncle Bob”) has written a great analysis of the problem of the increasing cost of change in his book Agile Software Development: Princples, Patterns, and Practices (2002). You can find what seems to be more or less the same text in his article “Design Principles and Design Patterns” (PDF). Here Uncle Bob neatly slices the awkward feeling of “this code is difficult to work with and expensive to change” to four distinct symptoms of rotting design: rigidity, fragility, immobility and viscosity. He then presents the principles of object-oriented design and demonstrates how they contribute to keeping the design of the software such that the cost of change is being controlled.

The more I learn about object-oriented programming, the less I feel I know. Reading Uncle Bob’s book was an experience similar to studying logic at the university: After the first couple of logic courses, I felt that I had mastered all that there was special to logic, being able to work with both syntactic and semantic methods of the first-order predicate logic, and vaguely aware that “some people do weird things with logic such as leaving out the proof by contradiction.” It was only when I understood the alternative and higher-order logic, proof theory and model theory a bit deeper, when I realised that I had just been climbing a little foothill, from the top of which one saw a whole unknown mountain range with a variety of peaks to climb, from Frege’s appalling syntax of predicate calculus via Hintikka’s modalities to Martin-Löf’s constructive type theory.

Making good software design — which, if you believe what Reeves writes, is the same thing as programming well — is difficult. It takes a lot of time, practice, reflection and study to advance in it. And there will be a lot of moments when a programmer overestimates her capabilities and writes something that she sees good but which turns out the be bad design, the debt of which the software incurs to be paid later.

Hence Scrum does not work in a vacuum: if your team does not program well, the progress is not linear as Scrum assumes. The design rots as the codebase grows, this slows down the development of new features and changing old features, and the predictions that Scrum provides start to err on the optimistic side.

I used to think that Uncle Bob’s Agile Software Development had a bad title despite being one of the most valuable programming books I know of. I thought that the subtitle Principles, Patterns, and Practices was a lot more indicative of the content. But with the line of thinking expressed here, I am coming to the conclusion that in the end, good software design is an intrinsic part of deloping software in the agile way, because it is necessary for controlling the cost of change over time.

I would be delighted if someone, possibly knowing more about Scrum, could show holes in my thinking. Meanwhile, I see no other option than practising to be a better programmer 🙂

Advertisement