What do you do when someone else’s evidence and your logic collide? You might:
- Use ad hominem “logic” to cast aspersions on the someone else.
- Not waste time bothering with ad hominem logic – just deny the other side’s evidence.
- Create a strawman version of what you’re trying to refute – something that resembles it, only with a few fatal flaws added to let you “prove” it can’t work.
- Embrace your confirmation bias. Find someone on the internet … anyone on the internet … who asserts evidence supporting whatever position you like best. Cite them as the only authority worth paying attention to.
- Redefine the criteria for being right.
- Find a historical case of a scientist whose colleagues ridiculed their theories, only to be vindicated in the end. Claim they’re your ideological brethren. Whether the historical scientist was actually subjected to ridicule or not doesn’t matter.
- Or, reverse the last stratagem: Find a theory that was popular once upon a time, only to ultimately be proven wrong. Those who disagree with you would have agreed with it.
Which brings us to the sad, sad case of why Waterfall methodologies persist.
In case you’re unfamiliar with the term, a Waterfall methodology is one that designs a project plan in terms of a linear progression. For application development these would typically start with a feasibility study, followed by requirements definition, specifications, coding, testing, training, and deployment.
With Waterfall methodologies, design has to be complete before development can begin, and if anything has been left out, or a design assumption has changed (as in, “Oh, you wanted wings on that airplane?”) whoever is championing the change goes through a change control process, which includes the project manager rippling the change through the entire project plan.
Agile, in contrast, is a family of methodologies, not a methodology. What its variants have in common is that they build software iteratively and incrementally. The list of Things the Application is Supposed to Do is called the backlog.
Projects still start with some form of feasibility study, one of whose work products is the initial backlog; another is defining the “minimum viable product” – the irreducible core of the planned application that everything else hooks onto.
From that point forward there is a process for deciding which items in the backlog have the highest priority. As for anything left out of the backlog or a change in design assumptions, these are pushed into the backlog where they are subjected to the same priority-setting process as everything else.
This will do as a barebones sketch. If you want more, please refer to chapter 3 of There’s no such thing as an IT project, Fixing Agile.
The best available evidence, from the widely respected Standish Group, reports that Agile projects are fully successful at nearly twice the rate as Waterfall projects, which fail completely about two and a half times as often as their Agile counterparts.
Case closed? If only.
Some Waterfall proponents counter with one or more of the denial strategies that started this article. For example a popular strawman is that Agile can’t work because in order to optimize the whole you have to suboptimize the parts, which supposedly can’t happen because in Agile, each developer does whatever he or she wants to build a capability that accretes onto the accumulating application.
This is a strawman. Agile projects build to a well-defined architecture, to user-interface standards, and to an overall coherent plan: Anything added to the backlog has to be consistent with the rest of the backlog.
Meanwhile, the implication is that in Waterfall projects, designers can optimize the whole. This assertion is, in a way, accurate. Designers can optimize the whole, so long as the target “the whole” is shooting at doesn’t change over the life of the project.
Good luck with that. The specifics of what a business needs to succeed within a given application’s domain change all the time.
So by the time a Waterfall-developed application is done, the criteria for “done” have changed.
Bob’s last word: The specifics of what a business needs to succeed within a given application’s domain changes all the time in successful businesses. Leaders of successful businesses know that their success depends on continuous adaptation to changing circumstances.
Success, that is, depends on agility.
Bob’s sales pitch: “To optimize the whole you must sub-optimize the parts” is a well-recognized principle here in KJR-land. If you’d like some guidance on how to apply it to your own real-world situations, you’ll find it in chapter 2 of Keep the Joint Running: A Manifesto for 21st Century Information Technology, which explores this concept in depth.
Actually Bob, when I want to prove a point I do what many “journalists” are now doing. I show tweets that support my point. I find that random comments from people I don’t know really pins things down.
One thing about Agile that I’ve heard more about lately as a negative. A couple of CIOs asked me about the issues around refactoring, something I ran into years ago as my teams started using Agile. Refactoring isn’t something new, version 2 has always been better than version 1 and new ideas pop all the time.
But refactoring was a problem with Waterfall also. Yes we tried to get the framework right the first time, but we rarely did. Once a project gets into recursive refactoring, you are headed for the failed project scrap heap.
My answer was that refactoring in Agile is really like any feature and a project always has to remember that developers get better the longer they work on something. So admitting that some early work isn’t as good as later work is reality. There can always be a version 2 to go back and make some changes. Just avoid getting in the trap of looking for perfection in version 1 and make sure you have someone with a great vision of architecture.
I’ve worked on waterfall projects and agile projects and on many projects that used both. I don’t think it’s an either/or proposition. Larger, more complex projects should lean towards waterfall but that doesn’t mean that parts of the project can’t be done from the agile side. Smaller less complex projects lean towards agile, but could use waterfall if a function within the project is complex. I see the biggest problem with waterfall to be expertise. If the project doesn’t have extremely sharp, knowledgeable, motivated, and personable project members (from the user side, the management side and the development side) then all the sins you mention for waterfall become evident.
Or you set up a false strawman and make fun of it, while thinking that IT is the entire universe and must be given the most of everything as it is the most important thing because it is the only thing.
There is a place for agile. It is for small changes to existing projects. To some extent also for duplicating small simple projects.
There is a place for a double helix (NOT a waterfall!!) to do new unique large complex unprecedented projects. The SYSTEMS architecture is composed of over a dozen other supporting architectures. They must coordinate and compromise to make the SYSTEMS architecture the best it can be.
It is far better to take the time to get it right than to waste even more time trying to fix the problems caused by just doing it. Lincoln said if he needed to cut down a tree in an hour he would spend 50 minutes sharpening his ax first.
The problem is always in the middle. Like with physics where Einstein’s view holds for the universe, and quantum stuff seems to hold for ultra tiny stuff, somewhere in the middle they both fall apart.
The same thing happens with writing as witness the same arguments between the pantsers who just sit down and write and the planners who are the successful professionals who hit word count on deadline while being compatible with previous episodes of a TV show or movie sequence by planning in advance.
Personally I agile anything under 1500 words. After that I start doing more planning. For my book I followed Lincoln’s advice as noted above.
Agile people deny the fundamental law of systems which says that optimizing subsystem(s) will sub optimize the total system. The corollary is that to optimize the total system you must suboptimize the sub systems.
You can build a shed without a plan but trying to agile a massive cathedral would be a fail without a proper architecture to guide the tradesmen building it.
@ Pat La Pella:
It depends. Agile has a place. A VERY SMALL place.
It is okay for applying modest changes to existing systems that were architected properly to allow such changes to be done well.
Agile is not a viable method to ARCHITECT design and then build larger systems while ignoring all the other factors that need to be considered such as security being built in integrally not tacked on later as an afterthought so it will always be failing to the next hacker who comes along.
And like you note, the people will determine how wide the gray area is before double helix (NOT Waterfall !!) is the necessary approach to ensure that you do not fail miserably.
Full disclosure: I come to this problem from the perspective of a SYSTEMS ARCHITECT whose experience is on new large complex unprecedented systems that are pushing technology to the limits to be feasible.
If your world consists only of IT considerations for small changes to existing small simple commodity systems then your view may well be different. Especially if you do not really care about security like so many existing systems have shown to be the case.
At least 4 major companies have been hacked so that my personal data has been stolen by the black hatters. Naturally I pay the cost of what happens next and the company writes it off as being a cheaper way of doing business than to worry about security up front.
As long as speed of releasing something takes precedence over security, then security will always be vulnerable.
I had a development manager tell me a while ago that all projects become waterfall when (a) you’ve had the contract for a while, and (b) the external customer’s go-live is a week out.
It’s “worse” now that I’ve been moving into the copywriting space: The contract has milestones and deadlines, and if you want to see the money, you’ll be meet them