All Posts in Metrics

|In: Industry Commentary|Last Updated:

DevOps is the least interesting feature of DevOps.

In case this is the first you’ve run across the term, DevOps is what you get when you add IT Operations to Agile development. This is a Good Thing because when the team needs computing resources a team member can provide them instead of the request waiting in a FIFO queue until it floats to the top.

For the record: I recommended this to a client in 2006. It didn’t seem important enough to name, let alone promote as the Next Big Thing.

But DevOps has become a lot more than just adding Ops to Dev.

First and foremost, it’s about establishing a culture of collaboration that includes all stakeholders in a project and the business change it’s been chartered to engender.

Which makes this yet another opportunity to say I told you so: There is no such thing as an IT project.

You’re welcome.

Second: DevOps is more Agile than Agile: Instead of bundling lots of small changes into a big release that requires extensive testing and change management, DevOps deploys changes so often that each individual change is too small to worry about.

The magic buzzphrase is “continuous testing and integration.” In practical terms this means DevOps makes a wonderful excuse … sorry, change that, business case … for an IT shop to do what it should have done a long time ago, namely, implement automated regression and stress testing.

So … DevOps = Agile + Collaborative business change + Automated regression and stress testing + Continuous implementation of small changes.

Sounds outstanding. But does it pass the so-what test?

Sadly, many of the answers are little more than hand-waving.

For example: “DevOps is a powerful revenue driver. About 34 percent of respondents working at organizations with greater than average profit growth said they had already adopted DevOps while only 17 percent of those with less than average profit growth had done so,” says Andi Mann, vice president, CA Technologies.

I don’t want to pick on Andi Mann. He’s been a Force for Good in this area, providing no shortage of excellent insights.

But as correlation doesn’t prove causation, this isn’t one of them. More likely, the best-run companies — those whose strategy, overall execution and business culture drive exceptional profit growth — are more likely to adopt DevOps than their brethren.

Then there are metrics that are just plain … well, you be the judge. Puppet Labs doesn’t really deserve to be singled out but it is the source of one such metric, namely, that Amazon Web Services claims a mere 0.001% of its deployments cause outages. “0.001%?” I hear the metrically naive exclaim. “Wow! That’s fantastic!”

Uh … no. If you take a release that caused an outage because it contained a defect and break it up into 100,000 micro-releases, one of which contains the same defect, you end up with the exact same number of defects released into deployment, even though only 0.001% of your releases contain them.

To be fair, finding and fixing the bugger would be a lot easier with 100,000 micro-releases.

Then there’s my favorite — DevOps lets you release new features to customers more quickly. It’s a fair claim, so long as the software you’re DevOps-ing is visible to your customers, which it is if it’s part of your eCommerce site but isn’t if it’s something you use for, say, property management.

What’s that? Internal customers? Just a guess: If you change the screens they work with several times a day without warning, they aren’t going to thank you.

Okay, fun’s fun, and fun as it is to debunk hand-waving business cases, providing valid ones is probably more useful. With that in mind, here are three:

  • Automated testing: Yes, setting up and maintaining an automated test environment is far from easy. The payoff: Orders of magnitude fewer bugs and system failures make it through to production. DevOps gives you a convenient context for it.
  • Integrated business change: There really isn’t any such thing as an IT project. The industry momentum behind DevOps gives you a convenient context for this as well.
  • Whether correlation results from causation doesn’t matter: Making DevOps work in a business requires changes in mental habits outside IT that will make the business more agile. And not only more agile, but more focused on whether what it’s doing is something Real Paying Customers will care about.

That is: While the companies that have already implemented DevOps already have other characteristics that make them more successful than their competitors, for those that haven’t, implementing DevOps could help transform them into companies that have these characteristics as well.

It’s a classic case of the fringe benefits outweighing the planned benefits.

Comments

|In: Industry Commentary|Last Updated:

Evidence-based decision-making is superior to intuition-based decision-making. If you disagree, please feel free to build a bridge or skyscraper on footings designed by engineers who prefer gut feel to empirically tested formulas.

And then come all the caveats, because as much as KJR has been a strong proponent of evidence-based decision-making, there are plenty of ways to go about it that are far inferior to, not to mention far more expensive than your average Magic 8 Ball®.

The most obvious (and not this week’s topic) is the popular pastime of solving for the number — of hiding intuition-based decision-making inside evidence-oriented clothing. Before big-data analytics became popular, Excel was the preferred tool for this job.

The Hadoop ecosystem includes far more sophisticated ways to reach the same foregone conclusions. Apply the right filters and shop around among the different statistical tests available to you in even the sparsest of statistical packages and if you can’t come up with the answer you want, you aren’t using your imagination.

But even with the best of intentions and no desire to distort, conscious or otherwise, statistical analysis holds plenty of pitfalls, even for professionals.

Take this recent correction request, filed by CNN with the Pew Research Center. As reported by the Washington Post’s Erik Wemple, a recent Pew study concluded that last January, Foxnews.com had more unique visitors than CNN.com.

CNN’s complaint: Pew’s analysis …

Uses a custom entity, [E] Foxnews.com, for Fox News against raw site-level property metrics, [S] for CNN.com. This is not an apples-to-apples comparison since a custom entity may contain a collection of other URLs that remain hidden. As it turns out, we learned from our inquiry to comScore that Fox News’ custom entity is also comprised of a variety off-site traffic assignment letters (TALs) and, as such, is not truly the audience of foxnews.com but instead is assigned traffic from other sites that is reallocated back to Fox News even though the visitor did not consume said content on foxnews.com.

I won’t comment as to whether the use of TALs is legitimate or not, on the grounds that I’m not remotely qualified to do so. If you’re interested, here’s a link for more on the topic.

Presumably, Pew’s analysts are properly qualified, but (1) might not have been aware that comScore included TALs in its Foxnews.com tallies; or (2) might have concluded that including them in web traffic statistics is legitimate.

Which gets us to your big-data repository. One of the attractions of NoSQL technologies like Hadoop is that you can pretty much dump data into them without worrying too much about how the data are organized. That’s addressed during the analysis phase, which is why another descriptor for this family of technologies is “schema on demand.”

It’s reasonably well-known that this also means a lot of the data being dumped into these “data lakes” has not been subjected to much in the way of cleansing. That’s almost the point of it: Hadoop and its brethren are adept at storing huge streams of inbound data (hence “big data”). They wouldn’t be so adept at it if some pre-processor had to cleanse it all first.

You have to pay the piper sometime, of course. In this case, it means you’ve shifted work from those who program data-loading into traditional data warehouses to those who analyze data stored in non-traditional NoSQL data lakes.

What’s less-well recognized is what Pew’s analysts either did or didn’t address with the TAL question: With traditional data warehouses, professional analysts make decisions like this as a conscious part of designing their extract, translate and load (ETL) processes.

They might miss something subtle too, of course … there never are any guarantees when those pesky human beings are involved … but at least there’s a defined task and assigned responsibility for taking care of the problem.

Not that this means it’s all taken care of: Whatever filtering decisions data warehouse analysts might consciously make while implementing the system will usually turn into hidden assumptions inherited by those who analyze the data stored there later on.

At least with schema-on-demand, analysts have to make these decisions consciously, and so are aware of them.

If, that is, they’re knowledgeable enough to be aware of the need to make them all.

Which is why, whether your analytics strategy is built on a traditional data warehouse or a schema-on-demand data lake, you need the services of a professional data scientist.

Or, as we used to call them, statisticians.

Comments