Want to be part of the Digital Transformation? You’ll need a new capability. I’ve dubbed it Apologies at Scale.

Our story starts with Wink. My wife and I bought one of its hubs and a couple of Wink-compatible light bulbs for our lakeside cottage, to create the impression the place is occupied when there’s no one around. It worked fine, until one day when it didn’t.

Astonishingly, there in my inbox was an email, letting me know Wink had goofed. It had sent out a firmware upgrade that took Wink hubs offline. And because the Wink hubs were offline it couldn’t send out a revised upgrade to fix the problem.

The email apologized and offered to send me a pre-paid shipping box to send them my hub in so they could fix it.

Wink’s mistake was pretty dumb. Its recovery, though, was brilliant.

Unlike how Sling Media handled its decision to kill the Slingcatcher.

Sling Media’s Slingbox is a gadget that sends television signals across the Internet, where you can play them on a Slingplayer app, of which there are versions for (at least) Windows, Mac OS, iOS, and Android.

Or, once upon a time, you could use a Slingcatcher instead — a closed box with a Slingplayer app inside it that could connect to a standard television. This was handy for non-technical folks like my in-laws, who want to watch the news or Antiques Roadshow on a television when they’re visiting our lakeside cottage, not on an app on a computer, tablet, or smartphone.

Then, Sling Media upgraded its system, making the Slingcatcher incompatible, not that the company let its customers know so we could avoid wasting hours of effort in pointless troubleshooting.

Enter Amazon’s Fire TV, for which Sling Media provides a Slingplayer app. We got a Fire TV Stick, plugged it into the cottage TV, ran the Slingplayer app, and everything was groovy.

Until one day when it wasn’t. Out of nowhere the Fire TV Slingplayer app just wouldn’t run, even though the Slingplayer Apps for my Android phone and Windows PC worked just fine.

After an hour of screwing around I contacted Amazon tech support, which told me Slingbox had “withdrawn support” for the Fire TV. Adrenaline in my arteries, I contacted Slingbox tech support, which told me, no, it hadn’t withdrawn support. Amazon had changed something in its system that prevented the Fire TV Slingplayer app from working, their customer service rep told me.

I’m inclined to believe Slingbox on this one, as it had a new version available within a week that worked on my Fire TV Stick, not that it let anyone know it had fixed the problem it hadn’t let anyone know about.

We in IT spend a lot of our time and creative energy thinking about scale. If you’ve read anything at all about the Internet of Things and the smart products it connects to you’ll have read about the daunting scaling challenges it represents.

And these articles aren’t wrong. Sell, say, a million smart products. Figure each of them sends out a 1 KB data packet once a minute. The 20 MBPS of bandwidth that takes isn’t completely unmanageable, but the half terabyte per year of data that one product generates might need some attention.

And that’s just technical scale.

As products become smarter, they also become more bug-prone. Not only that — modern products have more in common with the Portuguese Man of War than with the jellyfish it physically resembles, jellyfish being independent organisms where the Portuguese Man of War is a colony.

Apps are colonies — at a minimum, in addition to their own code, they’re composed of the Internet, an ISP, external data, and a host OS, any of which can fail or become incompatible.

Then there’s Wink. Its whole reason for being is that it’s a colony — it unifies control of lots of different company’s home automation devices.

Anyway, Portuguese Man of War or not, your company’s name is on the product.

I’m willing to cut Amazon some slack. As of 2015 it was selling 488 million SKUs in the U.S. alone. Knowing one of them has developed a problem and which customers bought it is a non-trivial task, which is why I call it Apologies at Scale.

I have, on the other hand, no sympathy for Sling Media. It only has 18 SKUs to manage — four devices + 14 Slingplayer apps. Managing a QA lab to spot problems and sending out emails to registered users when there is one shouldn’t be all that challenging.

And compared to the cost of handling disgruntled customers dialing into its call center, a simple proactive email would have been cheaper, too.

Last week I enjoyed my freedom from political correctness by ridiculing New Jersey for the state of its roads (D+ grade from the American Society of Civil Engineers), its correlatively low gas tax, and the consensus among its governor, legislature and citizens that tax increases are off the table.

Who to ridicule this week? I know … most of the private sector, because if you think this sort of behavior is limited to public governance, you aren’t paying attention to your own backyard.

For example …

I know of an insurance company that’s grown through acquisition to the point that it now has eleven functionally equivalent underwriting/policy administration systems. And no, it isn’t run as a holding company.

Whenever there’s a change to business logic, it has to make that change eleven different times in eleven different ways.

From all reports, the company’s IT department has become quite good at coordinating and implementing business logic changes. As it should, because when it comes to deciding what capabilities your organization needs it’s good to concentrate on what the business is likely to need from you.

The only choice that would be better would be to retire ten of the eleven systems.

Except that by just about every reasonable measure, the company in question is extraordinarily successful.

This is the sort of thing that keeps management consultants awake at night. If you’re metaphorically inclined, it’s as if, by policy, a state limited road maintenance to dumping asphalt into potholes and still became one of the nation’s primary transportation hubs.

But that isn’t what this week’s column is about. This week it’s about planning for the obvious and how organizational dynamics so easily prevents it.

Planning for the obvious first: If you add something to your fund of stuff, whether it’s a house or car for your household, or a machine, information system or facility if you’re a business, you’re going to have to spend money in the future to maintain it.

Otherwise your fund of stuff (from here on in, FoS) will deteriorate, losing its value or performance over time.

It’s a simple, inarguable equation: FoS increases, maintenance costs increase too, or else FoS value steadily decreases.

As a nation, during the Eisenhower administration we built our interstate highway system, and, in 1956, created the Highway Trust Fund, supported by a penny per gallon federal gas tax increase (to 3 cents, which, in case you care, is equivalent to 24 cents today, compared to the current federal gas tax rate of 18.4 cents).

Meanwhile, I’d bet good money (enough to pay five gallons worth of New Jersey gas tax) most KJR readers work in companies that, when they implement new information systems, don’t increase the IT budget by enough to cover the easily predicted need for ongoing maintenance.

Why not, given that any meteorologist would kill to be able to make predictions with this level of confidence?

Based on my exposure to and experience in quite a few businesses over the course of my career, it’s because of:

  • ROI computation: Proposed projects are usually evaluated on their financials. Include the cost of maintenance and the financials look worse, making the business case less attractive. Better to conveniently forget about them.
  • Tradition: This isn’t just the opening number for Fiddler on the Roof. If nobody else had to include maintenance costs in the past, why should my pet project be burdened with them now?
  • No good deed going unpunished: We haven’t increased the IT budget to support maintenance yet, and yet IT has managed to maintain everything so far. What’s changed?
  • Wishful thinking: Maintenance is a separate spending bucket. If IT needs to maintain a system, the maintenance will have to be cost-justified on its own merits.
  • Baumal’s Cost Disease: Everyone else is expected to continuously improve. Instead of increasing IT’s budget, IT should continuously improve enough to cover the difference.
  • Reality Distortion Fields: We can’t increase IT’s maintenance budget because we’re going to need this money to invest in new strategic initiatives.
  • Distrust: If we increase IT’s budget by enough to cover maintenance of this system, how do we know IT will actually spend the money on this system? (Several correspondents from New Jersey explained their anti-gas-tax-increase position on this basis.)
  • Siloes: If we increase IT’s budget by x to cover the cost of maintenance for someone else’s system, that will leave less on the table to cover the cost of the new systems and system enhancements I’m going to want.

Against these forces, the CIO is armed with nothing beyond logic and maybe a Gartner study or two.

It’s time to buy more asphalt.