The Pessimist Pamphlet

When I first heard of Agile development, I got a feeling of catch-22. That is, I agreed with most of the Agile principles and the underlying reasoning. However, I felt that the reason why I agreed is because of my personal experience, and having already learnt (often the hard way) to approach software development in a way very similar to Agile. I figure that most experienced developers would feel the same. The catch-22 then, is in how you would implement Agile development. For many of the ideas they promote, you need at least one or two of these very experienced developers in your team, who have this sort of Fingerspitzengefühl. Because if you look at the 12 principles, they name esoteric things like ‘sustainable development’, ‘technical excellence’, ‘good design’, ‘simplicity’ and reflecting on their own functioning. If you don’t have a certain level of experience and insight in software development, you will not be able to properly recognize and identify such things within the team and its software.

What I think is the most important of the 4 main values in the Agile manifesto is this:

Individuals and interactions over processes and tools

To me, this statement says that you should always trust the ‘Fingerspitzengefühl’, that is, the insight and experience of developers over some ‘rules’ on how you should or shouldn’t approach a particular area of software development.

In practice however, we mainly see one type of ‘Agile’ development being implemented, which is Scrum. The irony of it all is that Scrum is a very rigid set of rules. While it is considerably more ‘Agile’ than traditional hierarchical/waterfall-like business processes, and allows for more flexible changes in requirements and such, in practice it is often not quite as ‘Agile’ as I believe the people of the Agile Manifesto had envisioned. Scrum has become more of a Cargo Cult thing than something actually Agile.

One of my personal objections against Scrum is that people seem to interpret it as “the team can solve everything”, “anything can be cut up into smaller pieces”, “we’ll cross that bridge when we get there”, and “architecture is a dirty word”.

Sometimes, there are things that you simply cannot solve iteratively. If you have a single component that has various highly dependent requirements, and you choose to solve them iteratively, you can shoot yourself in the foot quite badly. For example, I have seen a project where items could come in from various sources, and had to be processed via a web-based client. The initial design used queues for the items, so that they could be processed first-in, first-out. Prioritization was possible by assigning different weights to the queues of each of the sources.

However, there were two more requirements, that had been postponed until later, which threw a huge spanner in the works: The client wanted to assign a ‘priority time’ to each item, and the client wanted an item to be processed (as soon as possible) by a specific web-client user. This meant that FIFO queues were completely inadequate. Since each item can now have a completely unpredictable priority time, it is no longer a queueing problem, it is a sorting problem. Likewise, routing an item directly to a particular client is not something you can do with just FIFO queues.

I believe this is a case where Scrum failed horribly: You never know exactly where you are going, you are just solving the problem one sprint at a time. You don’t realize that you took a wrong decision at the start until you’ve already built a large part of the application, and you need to redesign and reimplement a large part of the application. Nobody in the team recognized how much of a technical impact these particular requirements would have. They thought they were Scrumming away quite nicely. I suppose the reason is that the main form of interaction that the Scrum-method proposes is the daily standup. But that only discusses things that you are currently working on, you are not looking beyond the current sprint.

The Fingerspitzengefühl here should have said “Wait, this is a very complex application, we need to analyze enough of the requirements, and do enough preparation and design work, with perhaps some Proof-of-Concept to at least know that we are starting our sprints in the right direction.” You have to look before you leap. If nothing else, the ancient and outdated waterfall-methodology at least made sure you did that much.

My idea then, is that the above value of Agile should be re-formulated into a stronger version:

It is the wrong way until you have determined the right way

This is what I would like to call the Pessimist Pamphlet. It is stronger because it starts from the negative, somewhat similar to the philosophy of science where you have verification and falsification. The idea is to inspire more interactions between individuals, think about the problem you’re solving, and how you can best solve it, and apply critical thinking to the processes and tools you’re using. Are they really the right ones in this particular case? What other options are there? What experienced developers (either inside your own team or not) could I ask for their input? Instead of thinking “I’m probably doing things right” until you find out that you didn’t, you should question yourself beforehand and try to avoid problems that you could not see, but others could have, if you just had the interaction. Are you absolutely sure? Don’t assume you’re sure, make sure you’re sure.

Hopefully that will guard you from some of the pitfalls of Scrum, such as that you do not try to solve any problem whole, and that you assume that there is always someone in the team who will come up with the correct solutions and insights. Some problems need to be solved as a whole, not one requirement at a time. The key is to properly identify the dependencies between the requirements, and group all dependent requirements into a single ‘problem’ to solve. I am still not talking about designing the entire application at once obviously, merely identifying the parts that need to be designed as a whole, because the set of dependent requirements push you into a certain minimum featureset that your data model, choice of algorithms etc. need to support (as in this example, don’t choose queues when you might need more complex queries or sorting at a later stage, so eg a relational database may be more appropriate). And sometimes you really do need to consult someone from outside the team, who may have relevant insights and knowledge that your team does not have.

It would seem that I am not the only one who has their doubts about Scrum and how it is applied in practice. This is a very nice article discussing some pitfalls and shortcomings of Scrum:

I believe this also brings us back to that valuable lesson that Fred Brooks taught us in the 1980s: There is no silver bullet for software development. I suppose what I have discussed here comes down to determining what he calls the ‘essential complexity’ of a problem. In his famous article, he talks about creating working prototypes and developing them incrementally towards a final product. One of the arguments he gives is that it is almost impossible for the client to give a complete and accurate specification upfront, so the requirements are subject to change. You could say he laid the foundation for Agile development in that article. However, he also makes a clear point that there’s a difference between good and great designers. You should cherish these great designers, because they can simply do things that good designers cannot, no matter what methodology you apply. These great designers can make the biggest difference.


This entry was posted in Software development and tagged , , , , , , , , , , . Bookmark the permalink.

5 Responses to The Pessimist Pamphlet

  1. I wanted to thank you for your post.
    For the past two years I’ve experienced pretty much exactly what you (and Aaron Gray) are describing: so much so that I’m going to leave my job very soon because of the stress buildup this whole situation caused (worked on this big project since 2011, something I really care about – or used to up until recently).
    For a long period of time I thought it was my fault, due to my weird patterns/schedules and the way I work in general; after reading your experiences, though, I realized I’m not alone in thinking maybe the issue is something else!
    Going to bookmark this, as I’m sure it will be useful again in the future.


  2. Pingback: Experience and skill in software development | Scali's OpenBlog™

  3. Pingback: Scali on Agile Development | Scali's OpenBlog™

  4. Pingback: Bugs you can’t fix | Scali's OpenBlog™

  5. Pingback: Read any good books lately? | Scali's OpenBlog™

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s