programming


If you dig a ditch slowly, you make progress anyway. But when programming, even if hacking away a dirty solution might seem quick, it might backfire in hindering your progress in the long run. And when this happens, it wrecks the best intentions of agile project management to predict reliably.

Scrum is the best thing that happened to software project management since the invention of sweets in meetings. Scrum frees developers from excessive meetings and paperwork, and helps them to communicate their problems and feelings. And most importantly, the sprint burndown chart provides a good, empirical estimate on whether the project is on schedule or not.

It’s excellent that big software companies are publicising their use of Scrum.

Scrum is not a silver bullet, but rather like a good pair of scissors that does its limited job well. To succeed, Scrum needs several more things such as

  • good communication
  • good communication (worth repeating, don’t you think?)
  • a hard-working product owner that tells the developers about the wanted features and their relative priorities
  • a motivated, self-organising team
  • a scrum master that coaches and supports, without commanding and controlling
  • adequate tools for handling the product and sprint backlogs
  • good programmers

A real-world experience with Scrum with good “lessons learned” material can be found in the presentation (PDF) of Timo Lukumaa in the September 2005 Agile Finland session.

I will now dwell on my last point about Scrum — or indeed any agile project management — needing good programmers. This is by no way a novel nor original idea. For example Pekka Abrahamsson from the VTT of Oulu underlined that agile software development is not for beginners in his excellent “Agile Software Development” presentation in the inaugural Finland Agile Seminar of March 2005. But I must confess that I haven’t deeply accepted this view until recently.

I used to think that agile methods might help just about any project or programming feat, and specifically that even junior developers could benefit from them. And surely they can. But I will now try to explain why I conclude that an agile project needs skilled programmers to succeed.

At the heart of agile project management is the idea of linear, quantifiable progress: the velocity of the project can be measured by the rate of completing the wanted features, and this is used to predict the schedule. This is exactly the function of the sprint burndown chart of Scrum, mentioned above. The developers break each feature down to measurable tasks, estimate the relative sizes of the tasks, and the process tracks the rate of completing the tasks. And this is how you get data on the velocity of completing the features: if making the full text search is a task twice the size of making the login screen, and making the login screen took 17 hours, by all probability making the full text search will take 34 hours. Scrum rocks because it gives this kind of constantly-improving data with minimal effort.

But creating software is tricky. Craig Larman says that creating software is new product development rather than predictable or mass manufacturing, and this goes well in line with the famous thought of Jack Reeves that source code is design. Creating software is not like cutting a plank, where you could safely say that you are 50 % ready when you have reached the middle of the plank with your saw. It’s not like building a toy cottage where having the walls standing firmly and lacking the roof would show that you’re well on the way.

Instead, what we software developers develop is actually a collection of source code listings that do not serve for the end purpose of our work until they are built into machine code that is being executed in a computer. And to get there, the software must compile, run, and when running do useful things.

This is where the non-linearity comes in. When making more features, it is all too easy to break the features that were supposedly completed earlier. It’s very easy to break the source code in such a way that it does not even compile to executable form anymore. But most importantly, it’s too easy to grow the system in a way that increases the cost of changes by time.

At least since Kent Beck’s first 1999 edition of Extreme Programming Explained: Embrace Change the idea that the cost of change does not need to increase by time has been a central agile tenent. And Scrum seems to assume that the cost of change within the project remains constant, because there is not any kind of a coefficient for the velocity that would be a function of time. In my login screen / full text search example, it should not matter in which iteration of the project the full text search would be implemented.

Robert C. Martin (“Uncle Bob”) has written a great analysis of the problem of the increasing cost of change in his book Agile Software Development: Princples, Patterns, and Practices (2002). You can find what seems to be more or less the same text in his article “Design Principles and Design Patterns” (PDF). Here Uncle Bob neatly slices the awkward feeling of “this code is difficult to work with and expensive to change” to four distinct symptoms of rotting design: rigidity, fragility, immobility and viscosity. He then presents the principles of object-oriented design and demonstrates how they contribute to keeping the design of the software such that the cost of change is being controlled.

The more I learn about object-oriented programming, the less I feel I know. Reading Uncle Bob’s book was an experience similar to studying logic at the university: After the first couple of logic courses, I felt that I had mastered all that there was special to logic, being able to work with both syntactic and semantic methods of the first-order predicate logic, and vaguely aware that “some people do weird things with logic such as leaving out the proof by contradiction.” It was only when I understood the alternative and higher-order logic, proof theory and model theory a bit deeper, when I realised that I had just been climbing a little foothill, from the top of which one saw a whole unknown mountain range with a variety of peaks to climb, from Frege’s appalling syntax of predicate calculus via Hintikka’s modalities to Martin-Löf’s constructive type theory.

Making good software design — which, if you believe what Reeves writes, is the same thing as programming well — is difficult. It takes a lot of time, practice, reflection and study to advance in it. And there will be a lot of moments when a programmer overestimates her capabilities and writes something that she sees good but which turns out the be bad design, the debt of which the software incurs to be paid later.

Hence Scrum does not work in a vacuum: if your team does not program well, the progress is not linear as Scrum assumes. The design rots as the codebase grows, this slows down the development of new features and changing old features, and the predictions that Scrum provides start to err on the optimistic side.

I used to think that Uncle Bob’s Agile Software Development had a bad title despite being one of the most valuable programming books I know of. I thought that the subtitle Principles, Patterns, and Practices was a lot more indicative of the content. But with the line of thinking expressed here, I am coming to the conclusion that in the end, good software design is an intrinsic part of deloping software in the agile way, because it is necessary for controlling the cost of change over time.

I would be delighted if someone, possibly knowing more about Scrum, could show holes in my thinking. Meanwhile, I see no other option than practising to be a better programmer 🙂

Advertisements

I was a fan of the stories of Thor Heyerdahl — I read Kon-Tiki, Amon-Ra and also a book on the mystery of the huge statues on Easter Island. The scientific-adventurous style of the books is great. But what I will write about now is a little curiosity that stuck to my mind in the Ra story.

Ra was the Papyrus reed boat that served in one of Heyerdahl’s famous feats of experimental archaeology. He wanted to test the hypothesis that the ancient people would have had vessels capable of crossing the oceans. Earlier, with the balsa raft Kon-Tiki he could show that the ancient South Americans could have travelled all the way to Easter Island. And with Ra he intended to show that the old Egyptian reed boat design was capable of crossing the Atlantic ocean.

Heyerdahl had the reed boat built following the designs from the old Egyptian paintings. Many of the paintings showed a rope between the bottom and the upward-tilted back tip of the boat. Heyerdahl thought that the function of the rope was to force the back end of the boat curve upwards, but when he observed that the boat stayed in good shape even without the rope, he decided to leave it out. This proved to be a bad idea:

A decision to cut what proved to be a crucial line between the prow and the deck caused the stern to sink and the ship to increasingly list to one side. After almost two months afloat, and with a hurricane approaching, the Ra was abandoned at sea frustratingly short close to its goal, still floating but nearly uninhabitable.

(Donald P. Ryan, THE RA EXPEDITIONS REVISITED.)

Heyerdahl learned valuable lessons, built a new reed boat Ra 2, and crossed the Atlantic with it. In the realm of commercial software production, this might be equivalent for example with a quick and easy bug fix, or with a major financial failure, depending on various circumstances.

Obviously you would not be reading even this far if you did not find some interest in fabricated parallels between software and the real world, such as this: the mistake of Heyerdahl is like leaving the programmer testing out of XP. More generally, even good advice can turn against you, if you make a misinterpretation or false assumption, and end up not doing quite what the advisor was thinking about.

The extreme programming example goes to the heart of what I want to convey. If you take such a broad and complex holity as the extreme programming, and do not realise how the separate parts need the support of each other, you might inadvertently cut off a crucial line. This might well be provoked by an illusion of resource scarcity or other types of laziness, for which we have some great expressions in the Finnish language: rusinat pullasta (“only the raisins from the sweet buns“) and yli siitä, mistä aita on matalin (“crossing the fence from where it’s lowest”). We Finns pride ourselves on eating the boring dry doughy (domestic) part as well as the sweet (imported) raisins, and on crossing the fences at their highest parts to not look like slackers. The kind of slackery of not jumping that high when it’s not exactly necessary for getting on the other side is nowadays fashionable as effectiveness, and I think that it is exactly the pursuit of this kind of effectiveness in which you might go astray.

Already in the first edition of the “white” extreme programming book, Kent Beck underlined the fact that the different legs of XP support each other and that the process would fall down like a house cards if you left one wall out. (Without still having gotten around to read the second edition, I should mention that it’s most probably well worth a read.) Obviously he did not describe his process like a house of cards, but that’s the feeling that stuck in me. Later on, I think that Beck pretty much confessed that of the XP practices, at least test-driven development could stand alone, so even if it might not save your project, you might be able to do good work practicing TDD even in a project that does not embrace other XP practices. Though if refactoring would be limited politically, this might make it very diccicult to do TDD.

Of the other XP practices, I think that perhaps pair programming might fly alone as well. Without proper first-hand experience, my guess is not educated. But something I have been through is what happens when you take the XP path of leaving out the big design up front or indeed almost any design up front (which is already getting too extreme to be proper XP), but fail to implement automatic testing, merciless refactoring and pair programming. If this doesn’t result in a mess for you, you better play a round of lottery. It is exactly this kind of faulty shortcuts that give “XP” the bad reputation as glorified hacking.

Another, more subtle example of the intersupport of XP practices, is the crucial but easily overlooked third step of test-driven development: red – green – refactor. If you leave in the repository the first kludge that seems to work, you’re not actually doing TDD but causing trouble:

Once the tests pass the next step is to start over (you may first need to refactor any duplication out of your design as needed, turning TFD into TDD).

Scott W. Ambler

To keep your TDD boat afloat, you need to refactor both the production and test code continuously. It’s not enough to just write a test always before writing code. Another necessity is that all tests are run often and showing green, as maintaining an exhaustive suite of tests is a part of TDD.

Another example still from the XP / agile realm, but from a bit higher point of view, would be overconfidence on developer tests as a measure of progress. For me so far, customer-defined automatic tests or indeed any predefined customer tests have been but a dream. And to measure whether the software is close to complete, it is exactly the customer / user tests that matter. Even if you have good code coverage by programmer tests and you think you have implemented a good part of the required features, you must fight to get the feedback from the customer, or whoever in your environment would be closest to the real user of the system.

What I still saw, even with TDD, are misunderstandings between the project’s customers (also known as business experts or product owners) and the software developers. Even if the deployed code was almost bug-free, it didn’t do everything the customer had expected. We can enlarge on the concept of TDD and reduce the risk of delivering the ‘wrong’ code with what I call customer test-driven development.

— Lisa Crispin: “Using Customer Tests to Drive Development”

To make all these long stories one short one, I claim that you must not adopt a process or a method, and then leave out a crucial part, because then you might fail horribly. Am I then saying that you should adopt any good-looking process blindly and without criticism?

If you got this far and think so, I hope that you will come up with a general counterexample in the next few seconds 🙂

Well, of course I don’t advocate following processes blindly, because cargo-cult programming can be just as bad a problem as taking the faulty shortcuts. In fact I think that the problems are related — omitting crucial parts of a method can be seen as doing cargo-cult on the remaining parts, without fully understanding their dependencies and interactions.

So in the end my point here is not really about being lazy in adapting practices, but being lazy in thinking about your work. The steps on the journey of adapting a method or a process effectively can be seen as the stages in reading texts on a subject with which one becomes more familiar, and can thus move from restatement through description and interpretation to critical reading. To master a skill, you must have the passion, perseverance and ability to learn, so that you will build a couple more reed boats before you get one to cross an ocean with.

« Previous Page