HomeApp Dev Methodologies

DevOps is a whole lot harder than Dev+Ops

Like Tweet Pin it Share Share Email

If everyone did everything right from the beginning, it would be easy.

“It,” in this case, is automated regression testing. “Did everything right” is building a strong test plan every time IT put new software into production, and then added it to the existing regression test suite.

Easy peasy.

Easy peasy if, that is, you’re a management consultant like me and not CIO of an actual IT organization. Easier peasier than the alternatives, perhaps, but easy? There’s nothing about software quality assurance that’s easy.

But very few IT disciplines rank as high in importance when it comes to succeeding as a modern IT organization.

The IT pundit class has discovered the need for speed, and about time. Speed, more than any other single factor, is what lets businesses outmaneuver their competitors, thereby profitably selling more products to more customers.

And … in most businesses, most of the time, delivering needed information technology is what limits the pace of change. Speed up IT, speed up the business.

Not there aren’t any number of other factors waiting in the wings to keep things slow, because slow quickly becomes a habit, not a matter of critical path planning. But I digress …

For business to be faster, IT has to be faster at delivering changes to the applications portfolio.

Which in turn means DevOps is in your future, which in its turn means automated regression testing is in your future, if it isn’t an important part of your past and present.

Yes, DevOps. Call me a converted skeptic. Back when folks thought the lead DevOps story was that Dev and Ops were now collaborating, it earned a gigantic ho hum from yours truly.

But as it turns out, DevOps is the least interesting aspect of DevOps. What’s most interesting: DevOps blows up our old understanding of how to move software out of development and into production.

The Standard Model involves bundling software changes into major releases, which then are subjected, not only to the full range of test protocols (unit, integration, regression, stress, and user acceptance), but then have to run the Change Advisory Board (CAB) gauntlet.

The assumption behind this bulky approach to software implementation is that each new release carries with it the potential for blowing up production … either by sucking up server or network capacity so as to slow everything to a crawl; by corrupting one or more corporate databases; by opening up a gaping, easily exploited security hole; or by some other nasty consequence software defects can cause.

Not that these concerns are unfounded. Software defects can cause any or all of these problems, and the bigger the release, the more opportunities there are for bugs to be hiding that can cause them.

What DevOps does that’s truly interesting is stand this equation on its head: Instead of bundling changes into the major releases that create so much risk that drastic measures are called for, it puts changes into production in large numbers of small doses.

And because each release is small, and has been … and this is crucial … subjected to automated regression and stress testing, the risk of it blowing things up is so small that the whole CAB process becomes a fifty buck solution to a five buck problem, as it were.

The magic buzz-phrase is “continuous delivery,” and to give you an idea as to whether “continuous” is an exaggeration or not, way back in 2011 Amazon was making production changes every 11.6 seconds.

This incredibly rapid pace of change lets Amazon test different selling approaches, fine-tuning its merchandising to an astonishing, and, if you’re a competing retailer intimidating extent.

In your case?

Here’s where it gets interesting (or, depending on your level of interest, more interesting): As you know because you’re a regular reader here, there’s no such thing as an IT project, which means there’s no such thing as a software implementation.

This is something even the most advanced DevOps practitioners get wrong. When Amazon deploys its website changes, what’s changing is its selling approach to a large enough fraction of its online customers to provide a valid statistical comparison to its current practices.

When a DevOps team working on an internal application releases changes, for the most part they’re changes to internal business practices.

Which leads to this question: Should DevOps teams just slipstream changes into production as Amazon does on its website? If not …

* * *

Tell you what — I’m not going to do all the work on this. Post your thoughts on how IT should issue changes to internal business systems and the business processes and practices they support as Comments this week, and we’ll continue the conversation in next week’s KJR.

Comments (7)

  • … or a permalink.

    I forgive you – still worth what I pay for it 🙂

  • I wanted to point out a new trend on the horizon which is feature flags. Here individual features can be turned off or downgraded after release removing the need for rollback. One person who is an expert in this area is Edith Harbaugh of LaunchDarkly.com I interviewed her about six months ago in Founder Story: Edith Harbaugh of Launch Darkly and I think what they are working on is on point for your exploration of the implications of DevOps for the “Change Advisory Board” model. I think it’s actually going to change the way we do release planning in the sense that the feature flags are going to look like a line item bill of materials for the features in the release.

    Anyway, I always enjoy our insights and thought I would pass this along as food for thought.

    • But … what if there’s a bug in the feature-flag processing module? Do we need a feature-flag feature flagger to turn feature flagging on and off?

  • While your article was great, as usual, I found the link about John Boyd fascinating. Never heard of the guy, but he sounds like he was my kind of person.

  • Bob – I got the link for the comments but not the message. – Mike

  • The big thing that DevOps brings to the table as far as I can see is agility allowing us to iterate business processes. I can have a meeting with our warehouse manager, get a process change idea from him, run it by a couple of stakeholders for approval and then implement the change in software. If it turns out not to work as he hoped, roll it back or tweak the software.

    Major changes still require a more formal process, but minor changes become quick and easy because DevOps gives us an “undo” button.

Comments are closed.