Software Engineering seemed like a good idea at the time, and the analogy was further extended to Software Architecture around the 1990s, by first designing a high-level abstraction of the complex system, trying to reduce the complexity and risks, and improving quality by dealing with the most important design choices and requirements first.
And while proper Software Engineering and Software Architecture can indeed deliver very high-quality software, there is a huge practical problem: time. It takes a lot of time to do everything ‘right’, also, things can get very bureaucratic with lots of documentation and rules and such. A related problem is that requirements may change during the design, so by the time the design is done, the problem it was designed to solve may have changed into an entirely different problem, and the design is no longer adequate.
So, Software Engineering/Architecture, once again not the Silver Bullet that people were hoping for. Which led to new approaches, such as Design Patterns, and slogans such as ‘design for change’, ‘best practices’ and all that. One thing that strikes me when I read books or articles about such topics, or object-oriented programming in general, is that everyone seems to use different terminology. They may use different names for the same concepts, or even worse, they use the same names for different concepts. And we are talking about very basic concepts such as ‘encapsulation’, ‘implementation’, ‘aggregation’ and things like that.
This shows at the very least, that Software Engineering is still a very new and immature field. It might also show a different aspect, namely that the people within the Software Engineering community are not communicating and working together very well.
If you read David Parnas’ work on Software Aging, he already mentioned this back in 1994:
Software is used in almost every industry, e.g. aircraft, military, automotive, nuclear power, and telecommunications Each of these industries developed as an intellectual community before they became dependent upon software. Each has its own professional organisations, trade organisations, technical societies and technical journals. As a result, we find that many of these industries are attacking their software problems without being aware of the efforts in other industries. Each industry has developed its own vocabulary and documents describing the way that software should be built. Some have developed their own specification notations and diagraming conventions. There is very little cross-communication. Nuclear Industry engineers discuss their software problems at nuclear industry meetings, while telecommunications engineers discuss very similar problems at entirely different meetings. To reach its intended audience, a paper on software engineering will have to be published in many different places. Nobody wants to do that (but promotion committees reward it).
This intellectual isolation is inappropriate and costly. It is inappropriate because the problems are very similar. Sometimes the cost structures that affect solutions are different, but the technical issues are very much the same. It is costly because the isolation often results in people re-inventing wheels, and even more often in their re-inventing very bumpy and structurally weak wheels. For example, the telecommunications industry and those interested in manufacturing systems, rarely communicate but their communication protocol problems have many similarities. One observes that the people working in the two industries often do not realise that they have the same problems and repeat each other’s mistakes. Even the separation between safety-critical and non safety-critical software (which might seem to make sense) is unfortunate because ideas that work well in one situation are often applicable in the others.
We need to build a professional identity that extends to people in all industries. At the moment we reach some people in all industries but we don’t seem to be reaching the typical person in those industries.
The paper itself ironically enough proves his very point: the term Software Aging has since taken on a different meaning. Parnas meant aging of code/applications because of maintenance, adding features, and needs and expectations changing:
There are two, quite distinct, types of software aging. The first is caused by the failure of the product’s owners to modify it to meet changing needs; the second is the result of the changes that are made. This “one-two punch’ can lead to rapid decline in the value of a software product.
These days, the term Software Aging instead is used for software that exhibits problems after running for an extended period of time, such as resource leaks or data corruption. So the term was ‘reinvented’ by other ‘software engineers’ to mean something entirely different than what Parnas meant by it.
When is a crisis not a crisis
Parnas also points out that the so-called ‘software crisis’ had been going on for some 25 years already, at the time of writing. And despite advancements in ‘software engineering’, apparently the ‘crisis’ has not been solved. So this is not really a crisis, it is a long-term problem. And there are apparently other problems than just the ones that ‘software engineering’ has tried to address so far.
He goes on to explain that ‘engineering’ is something quite different from what people understand in terms of ‘software engineering’. It is also about a certain professional ethic. A certain professional/academic education, professional standards, a professional identity. In that sense it doesn’t really help that many people who develop software aren’t actually formally educated in computer science, but rather some other field, such as for example physics or electrical engineering, and happen to be writing software related to their field.
I personally would describe this as something like “cheapening of the trade”. It can be annoying to have to work with people who haven’t ‘read the classics’, and aren’t familiar with various concepts or customs that proper computer science people take for granted. It can be very difficult to communicate with these people, because they are not working from the same common basis of knowledge. Yet, they are seen as ‘software engineers’ as much as those of us who studied CS at a university.
So is Agile Development the answer?
In recent years, there has been more criticism on Software Engineering/Architecture, mainly from the realm of Extreme Programming and Agile Development. Their philosophy argues that proper Software Engineering/Architecture is too ‘rigid’, has a lot of overhead, and cannot deal with change effectively. So instead, a more lightweight and ‘agile’ way of development is proposed.
But is that the answer? Well, not entirely, as I have argued before. In fact, I would go as far as to say that Agile Development has mostly been adopted by people outside of Computer Science, such as project managers. To them it is attractive that Agile Development, and in particular Scrum, appears to give a very good overview of progress and use of resources. I say ‘appears to give’, because it is a false sense of accuracy: all the ‘progress’ you see is based on estimates, which may or may not be realistic.
While I agree that Extreme Programming and Agile Development make certain good points, and provide some useful methodologies, the obvious problem is that they tend to completely ignore the process of developing, testing and debugging software itself.
In the next part I want to go into these areas, and introduce some movements in software development that focus more on these topics.