Is Agile agile enough?
I’ve been an Agile proponent since before we knew to call it Agile (although curiously, I don’t seem to have written much about it over the years). I’m not sure it’s going to survive its own success, though, for three reasons.
In third place is the inherent conflict between Agile and software engineering. The issue is familiar to anyone who has ever had to support an aging legacy system: The accumulation of small enhancements and patches eventually turns even the best-designed system into a mess.
As all forms of Agile turn big development into a swarm of small enhancements, they all (except, perhaps, eXtreme) put you at risk of starting out that way.
The most common solution is to use the word “refactoring” just as if it actually means something. Usually it means “leaving it up to someone else to clean up the mess.”
Bad software engineering isn’t inevitable with Agile, just more likely.
In second place is the increasing triumph of form over substance.
More than anything else, Agile is a cultural change — a redefinition of the relationship between developers and business users. Waterfall methodologies maximized their separation: Users defined their requirements, approved the specifications, then went away while the developers created software which, so long as it met the now-frozen specs, was considered perfect.
Agile, in contrast, emphasized informality and high levels of contact throughout the development effort — there should be no actual need for User Acceptance Testing because end-users have seen the software in action so often, and have provided so much feedback throughout its creation, that there’s nothing new to react to.
That’s the idea. Sadly, many of Agile’s current crop of advocates are process-izing it excessively, focusing too much on following the steps and not enough (sometimes not at all) on the essential change in how developers and business users collaborate.
The first and most important issue, though, is the fundamental incompatibility between Agile’s high-interaction mode of operation and the seemingly unquenchable desire on the part of many U.S. organizations to place a minimum of ten time zones, plus significant cultural differences and linguistic challenges, between developers and business users.
Which isn’t fair: These barriers aren’t what companies are looking for. They’re buying the drug to get lower hourly wages; all the negatives are like the list of unfortunate side effects rapidly narrated at the end of pharmaceutical ads (such as, for example, the obsessive need to place two bathtubs on the beach, out in the woods, or at the edge of a cliff).
Given a choice between Agile’s clear advantages over Waterfall methodologies on the one hand (when done right; see above) and the lower price of offshore developers, it isn’t hard to predict which will win: Bet on cheap.
And so, with regret, we need an alternative to Agile that provides as many of its advantages as possible while still being workable with offshore development.
What’s emerging to address this challenge is something we might decide to call “Rapids Development.” No, not Rapid Application Development (RAD). While also speedy, Rapids Development gets its name from what rapids are in a river — miniature Waterfalls.
Rapids Development more or less follows the Waterfall model of separating requirements definition, specification, coding, testing, and production into separate sequential phases, but it makes them very, very short phases. Ideally, the entire cycle occupies three months or so; alternatively, following advice given in this space a very long time ago, two teams, each following six-month cycles, work three months out of phase with each other to still provide a new release every three months.
Rapids Development (there really should be a TM superscript after it, shouldn’t there?) has a number of attractive qualities. It solves the software engineering issue (to the extent any practice can solve it). It provides the same fast delivery and the opportunity to adjust to changing circumstances Agile delivers.
And, because it preserves Waterfall’s separation of duties, Rapids Development easily adapts to offshore development.
Which isn’t a plus for American developers, but is a plus for their soon-to-be-former employers.
What Rapids Development doesn’t solve for is Agile’s culture change … the shift from formal interviews, documentation of requirements, business-user sign-off of specifications they don’t understand in the slightest but have to sign off on anyway or the project stalls and all the rest of it, to actual day-to-day collaboration.
Rapids Development offers nearly everything a business could want … everything except what matters most.
What do you want from me, perfection?
Here, Here, on agile! My leadership is supporting it; however just months ago made me work on a project where I had no interaction with the endusers – I think we are lossing something in the process1!! People can have the buzzwords down but not the heart of the matter.
Bob,
Thanks for reflecting on agile. We need a lot more discussion of this topic at the executive level. I’m assuming you were joking about refactoring (which has very specific meaning to agile developers and certainly isn’t about leaving anything behind for anybody else to clean up) and were joking about software “engineering.” You don’t still use that term do you? Otherwise I might think that you really don’t understand agile at all, and that would be a heart-breaker because, as I said, this is a topic that is deeply misunderstood and feared at levels above development teams. Say it isn’t so!
Do I still use “software engineering” as a term? Sure. What’s the alternative – architecture used as a verb? I’m talking about the consistent, professional structuring of software so it adheres to solid engineering principles. I’m open to a different vocabulary, not to a different concept.
And I’ve watched Agile projects that were otherwise admirably managed result in poorly and inconsistently structured systems, specifically because Agile makes no explicit provision for the engineering aspects of development. Some forms describe an “architectural spike,” which might have actual meaning but sure sounds like hand-waving.
And some depend on refactoring, which is defined as re-writing poorly structured code without changing its external behavior. Sounds like kicking the can down the road to me. The only question is how far.
Just to make sure we’re clear: Solid engineering isn’t the same as the absence of bugs. Truly awful code can still be bug-free.
Understand, I’m a big fan of Agile, and especially its culture-change aspect. I just think its proponents need to take software engineering (substitute your preferred term if you like) more seriously and explicitly.
As someone who is experiencing the “process” of Scrum, your comments are spot on. But back to the 1996 column you referenced. In 1996 we had high productivity tools available such as PowerBuilder and VB (before .NET). Lacking those types of tools, development has become more complicated for typical business applications. With the former tools a project team of five worked well. With C#.net, five seems to just get you going. While we claim to be getting more productive in IT, it seems we take one step forward and two steps back.
Pingback: Tweets that mention Rapids Development | IS Survivor Publishing -- Topsy.com
I use agile methodology, and often work with offshore programmers. As you say, it’s not the best solution. Agile works best when folks are in the same physical room (much less the same time zone). In our current project we had to insist on bringing the programmers here. Bye-bye “savings”.
For the past several years I’ve advocated what I call “boondocking” as opposed to “offshoring”. If you want cost savings, put your programmers in rural areas where the cost of living is low instead of the usual high-cost locations. Here in SC I make less than my associates in CA yet maintain an arguably higher standard of living. I’m not as cheap per hour as a cookie-cutter resource in Bangalore, but given the additional inconvenience, delays and errors introduced by the inefficiency of offshoring, I’d compare my project costs to anyone’s. Boondocking gives you what offshoring only promises.
I believe Agile can work with Offshore by having Scrum meetings via Video. My concern with Agile is having an architecture that can handle the constant changes so that the product looks like a system that has been engineered rather than a bunch of mods to a prototype piece of software.
The thought of going back to Waterfall, even in short cycles, is VERY scary. Why? Because users are terrible at writing specifications, and developers are terrible at the skills it takes to pull good specifications out of the users. If we go back to Waterfall, there will be opportunities for consulting businesses to provide education on these topics!
Been there done that. My experience is similar. The idea that you would have developers speak directly to end users and the wallet is neither new nor unique. The team I was on did it 15 years ago. It was all the rage at the time and was codified in Steve McConnell’s book “Rapid Development”. When we knew what the final business function looked like we used “Staged Delivery” and when we were less sure we used “Evolutionary Delivery”. Both models had enough architecture at the beginning to keep us from ending up with dreck after several release cycles. And we had to fight with marketing for time to do “refactoring” every 3-4 releases but we won those arguments in the end. We still created requirements and documentation but as Bob discusses in his Manifesto it’s always easier to add and fix than to do a huge project. Our company was a great test of that hypothesis as we had 3 divisions 2 of which used the waterfall while we used our more iterative method (now relabeled as “agile”)
We built twice a day, they built once a month. We used automated testing they used ad hoc manual testing. We delivered running code to marketing and sales every week, they delivered once a quarter. Our software engineer to QA ration was 2:1 theirs was 1:2. And on the business side Our 25 development personnel generated the same revenue as the 250 in the other 2 divisions combined. I guarantee you we weren’t 10 times smarter. So there are still reasons to do everything from pure waterfall to evolutionary protyping. It depends on the problem you’re solving, risk tolerance, urgency, etc. For most projects that means enhancements to existing systems so the fat part of the curve fits some or all of the traits of what’s now termed “agile”.