“If it’s not if you win or lose, it’s how you play the game – then why do we keep score?”
– Ed Carnes
Year: 1999
Potholes in the road to IT measurement (first appeared in InfoWorld)
One of the most enjoyable experiences of my career was an upgrade.
It wasn’t the upgrade itself that was so much fun. It was the financial justification. Usually, getting capital approved for IS projects is about as much fun as being one of Sigfried and Roy’s cats – you get to jump through a bunch of flaming hoops while someone you’d enjoy clawing to death cracks a whip in your general direction.
This time, since we needed to upgrade our interactive voice response (IVR) system, we calculated the impact of unplugging the existing one. Since it was a 24-line system that was saturated during peak hours, this was simple: We just toted up the cost of adding another twenty-four customer service representatives to the call center and compared it to our original sub-$100,000 investment. Upgrade approved.
Occasionally, it’s possible to calculate the return on information technology through a clear and unambiguous calculation like this one. Usually, as pointed out in the last two columns, the value we provide is harder to quantify. Except for the rare process we can fully automate (an IVR is an example), the value we deliver comes in the form of capabilities used by the rest of the organization … capabilities that let the company achieve its strategic goals or that enable more effective marketing efforts, core processes, internal and external communications, and individual employees.
Capabilities are enablers: Each is necessary; none by itself is sufficient. How do you measure the value of a capability? How much does the fuel pump contribute to winning the Indianapolis 500?
To begin, you need to understand the value of achieving the business goal itself. Take a core business process. Process redesigns reduce unit costs and cycle times while improving the quality of whatever product the process creates.
To implement the new process a company probably needs new technology, new skills, a shift in culture, and perhaps a reorganization (even if it doesn’t, it will probably reorganize anyway because that’s what you do when you’re a big company undergoing change). What contribution does IT make to the tangible and intangible benefits?
The new process is completely different from the old one, so you can’t simply measure how much technology lowered costs, sped things up and improved quality. The best you can do is to determine the optimal process design possible without any technology changes … a good idea when redesigning a process in any event … and compare it to the one you plan to implement. You can plausibly define the difference as the value created by IT. (Caveat: This method isn’t mathematically rigorous; skeptics can challenge it on several grounds. Since nothing better is available, though, call it an approximation and be happy.)
While in principle, you can use this technique to measure the value of your legacy systems, too – by determining how the business would run if you unplugged them – in practice there’s little point. They deliver value – they’re in use, after all – so just measure unit costs and overhead, driving them as low as you can.
Another pothole in the road to IT measurement: Not every business goal is defined quantitatively. When a company redefines its strategy, for example, the usual reason is that the old one has run out of steam. The value of change is survival, not a numeric target. If the company needs new information technology to achieve its strategic goals, what value does it contribute? There’s no answer. Your contribution, while important, can’t be quantified. Don’t try.
And then there’s the hardest job of all: Measuring the value of a capability you deliver that’s unused or under-used. Lots of companies have thrown out gigabytes of customer information over the past decade, for example, because they didn’t have an immediate use for it. It had no value then. Now, with new strategies based on having extensive knowledge of customers, it would be a gold mine … but it’s gone, because the future value of the capability wasn’t understood at the time.
That’s just one reason measurement has its limits, but the ability to envision the future does not.