Software Engineering seemed like a good idea at the time, and the analogy was further extended to Software Architecture around the 1990s, by first designing a high-level abstraction of the complex system, trying to reduce the complexity and risks, and improving quality by dealing with the most important design choices and requirements first.
And while proper Software Engineering and Software Architecture can indeed deliver very high-quality software, there is a huge practical problem: time. It takes a lot of time to do everything ‘right’, also, things can get very bureaucratic with lots of documentation and rules and such. A related problem is that requirements may change during the design, so by the time the design is done, the problem it was designed to solve may have changed into an entirely different problem, and the design is no longer adequate.
So, Software Engineering/Architecture, once again not the Silver Bullet that people were hoping for. Which led to new approaches, such as Design Patterns, and slogans such as ‘design for change’, ‘best practices’ and all that. One thing that strikes me when I read books or articles about such topics, or object-oriented programming in general, is that everyone seems to use different terminology. They may use different names for the same concepts, or even worse, they use the same names for different concepts. And we are talking about very basic concepts such as ‘encapsulation’, ‘implementation’, ‘aggregation’ and things like that.
This shows at the very least, that Software Engineering is still a very new and immature field. It might also show a different aspect, namely that the people within the Software Engineering community are not communicating and working together very well.
If you read David Parnas’ work on Software Aging, he already mentioned this back in 1994:
Software is used in almost every industry, e.g. aircraft, military, automotive, nuclear power, and telecommunications Each of these industries developed as an intellectual community before they became dependent upon software. Each has its own professional organisations, trade organisations, technical societies and technical journals. As a result, we find that many of these industries are attacking their software problems without being aware of the efforts in other industries. Each industry has developed its own vocabulary and documents describing the way that software should be built. Some have developed their own specification notations and diagraming conventions. There is very little cross-communication. Nuclear Industry engineers discuss their software problems at nuclear industry meetings, while telecommunications engineers discuss very similar problems at entirely different meetings. To reach its intended audience, a paper on software engineering will have to be published in many different places. Nobody wants to do that (but promotion committees reward it).
This intellectual isolation is inappropriate and costly. It is inappropriate because the problems are very similar. Sometimes the cost structures that affect solutions are different, but the technical issues are very much the same. It is costly because the isolation often results in people re-inventing wheels, and even more often in their re-inventing very bumpy and structurally weak wheels. For example, the telecommunications industry and those interested in manufacturing systems, rarely communicate but their communication protocol problems have many similarities. One observes that the people working in the two industries often do not realise that they have the same problems and repeat each other’s mistakes. Even the separation between safety-critical and non safety-critical software (which might seem to make sense) is unfortunate because ideas that work well in one situation are often applicable in the others.
We need to build a professional identity that extends to people in all industries. At the moment we reach some people in all industries but we don’t seem to be reaching the typical person in those industries.
The paper itself ironically enough proves his very point: the term Software Aging has since taken on a different meaning. Parnas meant aging of code/applications because of maintenance, adding features, and needs and expectations changing:
There are two, quite distinct, types of software aging. The first is caused by the failure of the product’s owners to modify it to meet changing needs; the second is the result of the changes that are made. This “one-two punch’ can lead to rapid decline in the value of a software product.
These days, the term Software Aging instead is used for software that exhibits problems after running for an extended period of time, such as resource leaks or data corruption. So the term was ‘reinvented’ by other ‘software engineers’ to mean something entirely different than what Parnas meant by it.
When is a crisis not a crisis
Parnas also points out that the so-called ‘software crisis’ had been going on for some 25 years already, at the time of writing. And despite advancements in ‘software engineering’, apparently the ‘crisis’ has not been solved. So this is not really a crisis, it is a long-term problem. And there are apparently other problems than just the ones that ‘software engineering’ has tried to address so far.
He goes on to explain that ‘engineering’ is something quite different from what people understand in terms of ‘software engineering’. It is also about a certain professional ethic. A certain professional/academic education, professional standards, a professional identity. In that sense it doesn’t really help that many people who develop software aren’t actually formally educated in computer science, but rather some other field, such as for example physics or electrical engineering, and happen to be writing software related to their field.
I personally would describe this as something like “cheapening of the trade”. It can be annoying to have to work with people who haven’t ‘read the classics’, and aren’t familiar with various concepts or customs that proper computer science people take for granted. It can be very difficult to communicate with these people, because they are not working from the same common basis of knowledge. Yet, they are seen as ‘software engineers’ as much as those of us who studied CS at a university.
So is Agile Development the answer?
In recent years, there has been more criticism on Software Engineering/Architecture, mainly from the realm of Extreme Programming and Agile Development. Their philosophy argues that proper Software Engineering/Architecture is too ‘rigid’, has a lot of overhead, and cannot deal with change effectively. So instead, a more lightweight and ‘agile’ way of development is proposed.
But is that the answer? Well, not entirely, as I have argued before. In fact, I would go as far as to say that Agile Development has mostly been adopted by people outside of Computer Science, such as project managers. To them it is attractive that Agile Development, and in particular Scrum, appears to give a very good overview of progress and use of resources. I say ‘appears to give’, because it is a false sense of accuracy: all the ‘progress’ you see is based on estimates, which may or may not be realistic.
While I agree that Extreme Programming and Agile Development make certain good points, and provide some useful methodologies, the obvious problem is that they tend to completely ignore the process of developing, testing and debugging software itself.
In the next part I want to go into these areas, and introduce some movements in software development that focus more on these topics.
Scali's OpenBlog™ 
In your opinion what are the classics that every developer worth its salt should read?
Well, I don’t have a clearcut list, but I did mention a few ‘classics’ over time… Such as “No Silver Bullet”, “The Mythical Man-Month”, “Go-To Statement Considered Harmful”… In general, works of Brooks, Dijkstra, Knuth, Parnas etc are well-recommended. Then there’s of course also stuff from Turing, about Turing machines, the Turing Test, and the response about the Chinese Room thought-experiment… and well, that sort of thing.
Sometimes I wonder if the term “software engineering” is misleading to the point of being harmful. It implies that an engineer knows how to solve some well understood problem in a well researched field (how to build a bridge, how to lay out a PCB, how to build a car engine). My experience with software development is that it has much more in common with research, doing things that have not been done before and therefore are not well understood and not well researched. Maybe other developers have a completely different experience, I don’t know.
Yes, I still have a ‘part 3’ coming, where I want to get into this sort of thing some more… You raise exactly the kind of point I want to discuss. I think the problem with software is that it depends on who you ask. I mean, if you consider someone who builds websites a ‘software engineer’, and he or she is building the umpteenth website using the same basic technology, then I would say it is a well-understood problem, and not much, if any, research will come into play.
But my personal experience is indeed that pretty much every project I have worked on in my professional career had a significant part of ‘new’ things… Things that I and my colleagues at least had never built before, and in many cases wasn’t just something that was so common that tons of books and articles were published on the subject. You indeed had to do research, build prototypes, and try to gain understanding of the problem, and find the best solution within your capabilities.
I think the danger is that one group of software developers doesn’t really understand that there is another group of developers that works in an entirely different way, and they mix up their methodologies, because everything is treated as the same thing: ‘software engineering’.
Yes, an example I had thought of was someone building the 10th web store. Then it’s probably more like building the 10th residential house. Which is not to say there isn’t a lot of experience and skill that needs to go into that, making things faster/better/cheaper/more reliable.
But the projects I’ve worked on were more of the variety that maybe there’s some open source project doing something similar, but probably only vaguely, and maybe there’s some commercial product that only serves as an “it can be done” guidepost. All you have is a pile of documents and a debugger. In the worst case, trying to build something that might not even be doable, but you won’t know until you try.
So yes, my experience is much closer to “The Mythical Man-Month” than to some cookie cutter methodology.
Yes, perhaps that is an indication of the biggest underlying difference between software and ‘regular’ engineering: If you want to build 10 houses, you need to build 10 houses. You can re-use the design of a house 10 times, but you still need to actually build it 10 times.
With software, making multiple instances of software is trivial, to the point where actual software engineering is not required. For example, for the first website, a group of engineers has created WordPress. But this tool gives people with no programming knowledge whatsoever the means to build their own websites.
So perhaps there is a much sharper divide in software engineering between re-use and building new than in other areas of engineering.
Re-use of entire systems is more trivial, but building something new may not allow you to re-use much, if anything.
I mean, if I look at re-usable stuff, I can find things like Design Patterns on one side, and common datastructures and algorithms on the other. But the difficult part is finding the right patterns, datastructures and algorithms, and putting them together in the right way to solve your problem.
For software development, there are different models are available “V-model” is the best. it is the best model to develop a perfect software development.
Well, allow me to disagree 🙂
Firstly, I don’t think there is something like ‘perfect software development’ in practice. There are so many factors involved that it’s highly unrealistic that you get all of them exactly right during any software development project.
Secondly, I don’t think there is any ‘one size fits all’ model for software development. There are various models, each have their advantages and disadvantages. What you need is some expert developers who understand that ‘it depends’ on the context of your specific problem domain and the capabilities of your team. Experts that are capable of mixing and matching development ideas and methodologies from various different models, and using what works best for their specific situation.
Pingback: What is software development? An art? Craft? Trade? Occupation?… Part 3 | Scali's OpenBlog™
Pingback: Who is a software architect? What is software architecture? | Scali's OpenBlog™