First there was “frequent informal conversations between business people and programmers about what the computer could do.” The result: Useful systems that worked, onto which programmers and their business counterparts added enhancements that gradually accumulated into large systems that worked.

Then came formal methodologies — structured, sequential steps that completely specified a system’s design, functionality, and user interface before programmers wrote the first line of code … in a word, waterfall. It was more elegant, which caused most experts to ignore its 70%+ failure rate for a decade or two.

In 2001, via the Agile Manifesto, we rediscovered the value of frequent informal conversations, only now, in self-defense, we call it a methodology: Agile. It’s actually a family of methodologies, but let’s not quibble. Also not to quibble: Yours truly described much the same thing in this space two years before that, for all the thanks I get.

But never mind all that. Patting myself on the back makes my shoulder hurt, and oversimplifying Agile this much does it a disservice.

There are those who think they’re practicing Agile when all they’re doing is flailing away at a problem. They’re fooling themselves, trying to fool everyone else, or both. Agile requires just as much discipline as waterfall, just a different sort of discipline.

Waterfall projects start with a complete specification, with “complete” defined as “what we’re going to build no matter what the business actually needs.”

Then someone has a good idea and such bad manners that they share it, suggesting the team add the idea to the specification. In retaliation, all good project managers excoriate these miscreants, calling them, to their faces, creeps.

Scope creeps.

Well-run waterfall projects have a way to deal with scope creeps — the change-control committee. It’s a gauntlet all good ideas that happen at the wrong time must run. The committee reviews proposed scope changes to determine which ones provide enough value to warrant a change in the project plan.

Let’s be honest. It’s control-oriented bureaucracy. Control is necessary because after all, we can’t trust the creeps who want to change our scope to have the good judgment God gave rice.

Agile teams, in contrast, expect a steady flow of good ideas. Agile also recognizes that project teams … more accurately, release teams … have a fixed bandwidth, unless, of course, the business changes the team size.

Rather than pretend good ideas are really bad ideas, the business people who actually use the system add the good ideas to the good-idea queue (called the “backlog”). The team estimates how much effort will be required to code each good idea, and the business people decide which good ideas the team should work on, and in what order.

It’s called Agile because it really is agile, in the sense that when new ideas arise and new business situations change priorities, project (sorry, release) teams can change their priorities just as fast.

And it’s just as disciplined as having a change committee review proposed scope changes, assuming, that is, that the release team can be trusted to make reasonably accurate estimates and that the “system owners” … the business people who use the system to help get work done … are appropriate judges of what’s more important than what.

In case the point isn’t clear: They’d better be and if they aren’t it’s time to replace them, and to fix the hiring process so such irresponsible people aren’t put in positions of responsibility any more.

To be fair, in businesses that have devolved into rival organizational siloes (most of them), Agile priority setting will only be accurate for within-silo decisions, because for the most part release teams and the software they support will be focused inside one organizational silo.

Also, like it or not, siloes are more often a constraint than a problem, which is to say siloes are something to deal with — they aren’t something that ever gets fixed.

Fortunately, there’s a simple way for the company’s top leadership to set priorities across siloes, and that’s to adjust the size of the release teams to reflect the company’s overall business priorities.

Of course, this takes discipline too, in the form of an executive leadership team that provides executive leadership — that thinks in terms of the enterprise’s overall priorities and not just how to jockey for position so their silo gets its fair share or more.

If your company has an ELT like this it can achieve organizational agility.

If it doesn’t, the best you can hope for is that the executive who sits at the top of the silo you live in is adept at jockeying for position.

* * *

Sixteen years ago in KJR I revealed a leadership secret I’d used for years – never carry a pager.

Speaking of leaders, in 2003, KJR divided the leadership world into Steven Spielberg and Roger Ebert. Here’s why you need to be Steven Spielberg.

And in 2007, an object lesson in why organizational listening is among the most important leadership skills.

Jimmy Dean should never have recorded “Big Bad John.” His squeaky tenor just doesn’t fit the lyrics — they demand a Johnny Cash baritone.

Still, Dean got some things right — his sausages, for example.

And, he gave us a useful quote: “I can’t change the direction of the wind, but I can adjust my sails to always reach my destination.” It will never make it on despair.com, but that’s okay. Cynicism is more fun, but figuring out how to make things work is what pays the bills.

My last two columns have talked about the direction of the wind (“An imperfect storm,” 2/24/2014 and “More storm warnings,” 3/4/2014) — trends you can’t do much to affect, and do have to respond to:

  • Cloud 3.0 — enterprise-class computing that includes cloud-based applications.
  • Shadow IT — the growing amount of information technology implemented without IT’s involvement.
  • The digital enterprise — a grab-bag; right now its most important elements are smart products, the so-called “internet of things,” and all of the opportunities possible when you put the two together.
  • Income disparity — and the rising demand for luxuries and uniqueness by those on the lucky end of this trend.
  • The rise of business practices over processes — practices being the way to organize how work gets done so as to be able to deliver uniqueness.

Your challenge: Adjusting your sails so IT can at least survive these trends and maybe even enjoy the outcome. Suggestions:

Get the relationship right. I know you’re tired of hearing me rant and rave about moving beyond the supplier/internal-customer relationship model to a fully collaborative alternative. I also know I talk to IT leaders all the time who haven’t made the transition.

It matters in this context because in your brave new world of embracing shadow IT, a collaborative relationship is what will stop shadow IT from become rogue IT.

Automated regression testing. Take this to the limit, and beyond.

Cloud 3.0 means multi-cloud plus inside-the-firewall infrastructure provisioning. Multi-cloud, and especially multiple cloud solutions managed directly by the lines of business, means patch management and version management move outside IT’s control.

With automated regression testing you might be able to persuade the lines of business that IT should test cloud-vendor-induced configuration changes before they’re put into production. Without it, IT will once more be positioning itself as a bottleneck rather than an enabler.

Redefine the “I” in “IT.”

Except for shadow IT, all of these trends mean more work for IT, not less. Even cloud computing doesn’t mean IT has less work to do. You’ll be managing multi-cloud systems. Think monitoring for availability and performance. Think more reliance on your WAN. Think about what restoring from backup now means. Especially, think about integration.

This is the redefinition of “I” — from “information,” which never truly encompassed IT’s responsibilities anyway, to “integration,” which completely describes where IT is essential.

Look, like it or not, sales managers everywhere understood the difference between cloud-based shadow IT and the installed alternative. Installed software meant asking IT to unlock sales reps’ laptops so they could install Act! and asking IT to provide a server so their laptops could synchronize to a shared database.

The Cloud meant buying licenses from Salesforce, doing everything through the browser, and getting the shared database too, all with no IT involvement.

So encourage Shadow IT. Get rid of as much responsibility for the applications portfolio as you can. For individual applications, shadow IT’s drawbacks are diminishing, and this also eliminates the “This application doesn’t do what I need” vs “The specs were wrong” arguments that now dominate many business/IT relationships.

But integrating the applications? Only IT … “Integration Technology” … can make this happen.

Which gets us to enterprise technical architecture management (ETAM). This is a long-running personal favorite, and it’s only going to be more important in the future.

A multi-cloud environment with lots of quasi-independent line-of-business and departmental IT departments adding to the application layer is akin to a bunch of developers adding buildings to a community without building codes or well-designed water purification, electricity-delivery and sewage treatment systems to connect to.

In particular, the ETAM function should choose the company’s integration technology system and define the company’s data integration engineering requirements.

Data integration is what causes the most trouble when it comes to accidental architecture. Without a clean, clear, well-engineered approach, shadow IT will exacerbate the situation exponentially.

Okay, okay. “Polynomially” is more accurate, but who’s counting?

* * *

15 years ago in KJR’s predecessor, InfoWorld’s “IS Survival Guide”: An app dev methodology that looks a lot like Agile, two years before the Agile Manifesto.

Way back in 1996, I recommended viewing yourself as a product, not an employee.

And ten years ago, how to avoid the proximity trap — the tendency to pay more attention to those who have access than to those who have answers.