Can metrics be saved?

The most disturbing business book of the past ten years has a pretty dull title: Measuring and Managing Performance in Organizations (Robert D. Austin, 1996). It isn’t exactly an easy read either. It’s chock-full of economic and behavioral theory, and bibliographic references to more economic and behavioral theory.

Reading the book is work. Read it anyway. It asks the musical question, does instituting a system of performance metrics ever do more good than harm? Its answer: Yes, but not often. Usually, metrics create organizational dysfunction. The problem isn’t easily fixed, because four separate challenges stand between you and a system of measures that helps your organization succeed.

The simplest challenge is bad formulas. The business metrics landscape abounds with them. You can fix them, but doing so requires both ingenuity and mathematical insight. Most organizations have plenty of the former but only limited amounts of the latter. For example: Often, the right metric requires multivariate statistical analysis (multiple regression analysis or analysis of variance). If your staff lacks expertise in, for example, testing data for stationarity and heteroskedasticity, the right metric is unavailable to you.

And this is the easiest problem to fix.

The second challenge is completeness. It’s critical, because you get what you measure. If the creation of value depends on several factors and you can only measure one or two of them, you’ll still get what you measure.

Take an organization that develops and deploys software. Software value depends on fit to function, performance, absence of defects (bugs), and ease of use. One at a time: (1) Performance is (relatively) easy to measure objectively, so you do, and improve it. (2) Users will help you accurately count bugs, so you and they do, and you fix them. (3) Users will also help you understand fit to function, but not in a reliable, quantitative way (“This software is worthless!” is a typical data point). (4) Ease of use, being a matter of aesthetics, is in the eye of the beholder.

Since no two users will agree on either fit to function or ease of use, you’ll appoint an official owner, and adjust the software to his/her satisfaction. Value to the business improves very little compared to how much you’ve invested to improve it.

And that’s a simple measurement situation, which also serves to illustrate a third metrics challenge: Reliance on proxy measures.

The happiness of the appointed owner is a proxy measure for both fit to function and ease of use — it’s indirect and unreliable. Using proxy measures isn’t a wrong thing. Especially in subjective situations it’s the only option available to you. But every time you use a proxy measure you increase your uncertainty. Think of it as the difference between interviewing a driver about a car’s braking ability and measuring stopping distance with a tape measure.

Which leaves the final and most intractable challenge to making metrics useful for last — the pesky human beings who both collect and report the raw data. They expect to be measured by the results, and they’re right — the numbers “objectively” determine raises, bonuses and promotions. This creates a powerful incentive to shade, spin, distort, game, or simply falsify the Garbage In half of the system. Not being fools, employees at all levels, up to and including the CEO, will do exactly that. Don’t believe me? Review the recent histories of MicroStrategy, WorldCom, Tyco, and Enron.

It’s what gets in the way of the cockpit or dashboard analogy used by many (including yours truly not long ago) to illustrate the value of organizational metrics. As Austin explains:

“[The] analogy would be more accurate if it included a multitude of tiny gremlins controlling wing flaps, fuel flow, and so on of a plane being buffeted by winds and generally struggling against nature, but with the gremlins always controlling information flow back to the cockpit instruments, for fear that the pilot might find gremlin replacements. It would not be surprising if airplanes guided this way occasionally flew into mountainsides when they seemed to be progressing smoothly toward their destinations.”

The situation isn’t hopeless. Instituting useful metrics isn’t impossible, nor is it a worthless endeavor. It is, however, hard work, and calls for considerable finesse.

You’ll also need a culture of honest inquiry, which means relinquishing a cherished assumption on the part of most metrics advocates: That you can use metrics as a tool to manage and motivate the performance of individuals and workgroups.

Doing that lets the gremlins in.