Apologies at Scale

Want to be part of the Digital Transformation? You’ll need a new capability. I’ve dubbed it Apologies at Scale.

Our story starts with Wink. My wife and I bought one of its hubs and a couple of Wink-compatible light bulbs for our lakeside cottage, to create the impression the place is occupied when there’s no one around. It worked fine, until one day when it didn’t.

Astonishingly, there in my inbox was an email, letting me know Wink had goofed. It had sent out a firmware upgrade that took Wink hubs offline. And because the Wink hubs were offline it couldn’t send out a revised upgrade to fix the problem.

The email apologized and offered to send me a pre-paid shipping box to send them my hub in so they could fix it.

Wink’s mistake was pretty dumb. Its recovery, though, was brilliant.

Unlike how Sling Media handled its decision to kill the Slingcatcher.

Sling Media’s Slingbox is a gadget that sends television signals across the Internet, where you can play them on a Slingplayer app, of which there are versions for (at least) Windows, Mac OS, iOS, and Android.

Or, once upon a time, you could use a Slingcatcher instead — a closed box with a Slingplayer app inside it that could connect to a standard television. This was handy for non-technical folks like my in-laws, who want to watch the news or Antiques Roadshow on a television when they’re visiting our lakeside cottage, not on an app on a computer, tablet, or smartphone.

Then, Sling Media upgraded its system, making the Slingcatcher incompatible, not that the company let its customers know so we could avoid wasting hours of effort in pointless troubleshooting.

Enter Amazon’s Fire TV, for which Sling Media provides a Slingplayer app. We got a Fire TV Stick, plugged it into the cottage TV, ran the Slingplayer app, and everything was groovy.

Until one day when it wasn’t. Out of nowhere the Fire TV Slingplayer app just wouldn’t run, even though the Slingplayer Apps for my Android phone and Windows PC worked just fine.

After an hour of screwing around I contacted Amazon tech support, which told me Slingbox had “withdrawn support” for the Fire TV. Adrenaline in my arteries, I contacted Slingbox tech support, which told me, no, it hadn’t withdrawn support. Amazon had changed something in its system that prevented the Fire TV Slingplayer app from working, their customer service rep told me.

I’m inclined to believe Slingbox on this one, as it had a new version available within a week that worked on my Fire TV Stick, not that it let anyone know it had fixed the problem it hadn’t let anyone know about.

We in IT spend a lot of our time and creative energy thinking about scale. If you’ve read anything at all about the Internet of Things and the smart products it connects to you’ll have read about the daunting scaling challenges it represents.

And these articles aren’t wrong. Sell, say, a million smart products. Figure each of them sends out a 1 KB data packet once a minute. The 20 MBPS of bandwidth that takes isn’t completely unmanageable, but the half terabyte per year of data that one product generates might need some attention.

And that’s just technical scale.

As products become smarter, they also become more bug-prone. Not only that — modern products have more in common with the Portuguese Man of War than with the jellyfish it physically resembles, jellyfish being independent organisms where the Portuguese Man of War is a colony.

Apps are colonies — at a minimum, in addition to their own code, they’re composed of the Internet, an ISP, external data, and a host OS, any of which can fail or become incompatible.

Then there’s Wink. Its whole reason for being is that it’s a colony — it unifies control of lots of different company’s home automation devices.

Anyway, Portuguese Man of War or not, your company’s name is on the product.

I’m willing to cut Amazon some slack. As of 2015 it was selling 488 million SKUs in the U.S. alone. Knowing one of them has developed a problem and which customers bought it is a non-trivial task, which is why I call it Apologies at Scale.

I have, on the other hand, no sympathy for Sling Media. It only has 18 SKUs to manage — four devices + 14 Slingplayer apps. Managing a QA lab to spot problems and sending out emails to registered users when there is one shouldn’t be all that challenging.

And compared to the cost of handling disgruntled customers dialing into its call center, a simple proactive email would have been cheaper, too.