HomeIndustry Commentary

An ounce of metrics can’t be weighed

Like Tweet Pin it Share Share Email

Cast your mind back to November of 2008.

Unlike many political commentators I suspect your memory of the time is vivid, filled as it was with layoffs, plummeting investments, layoffs, negative profits, layoffs, the failure of huge financial institutions and layoffs.

One might almost think something important went on back then.

In its last months the Bush administration, and in its first months the Obama administration intervened intensely in the private-sector-driven fiasco. Optimists figured we were in for a replay of 1973 and the ensuing decade of stagflation. Pessimists considered Wiley Coyote the better metaphor — we had already run off the cliff and as soon as we noticed … kaboom.

We’re now on the verge of a Sesame Street recovery: Economists (and the Economist) aren’t sure if it will be brought to us by the letter V, U, or W.

Now that the worst appears to be over, the controversy rages: Was all that intervention a Good Thing or a Bad Thing? Predictably, those who distrust all things government are certain government action had no positive effect, while those who view government in constructive terms are equally certain it’s what saved us from falling off the economic cliff.

Which side is right, and why does the question belong in Keep the Joint Running?

The answers: (1) Neither side is right, and (2) leaders of all stripes spend a lot of their lives dealing with their wrongness.

Neither side is right because when the question is “What would have happened if?” only those who, like the protagonists of some Keith Laumer novels can move among parallel universes, can be certain.

The rest of us are left either with the “certainty” that accompanies evidence-free belief, or the intelligent conclusion that we’ll never be certain.

The Bush and Obama administrations lived with possibilities: Of another Great Depression, and that various interventions might reduce the risk. Their only certainty was that once the crisis passed they would be subject to intense second-guessing.

Sound familiar? If not, you’re too young to have lived through the Y2K crisis. The only difference is that with Y2K, everyone who knew the situation in any depth at all knew that large numbers of companies relied on large numbers of systems that assumed the first two digits of the year were “19.” Without lots of hard work, after December 31, 1999 they would have assumed wrong, with disastrous consequences for the companies that relied on them.

If you are too young to have lived through Y2K as a professional, you’ve still lived through plenty of other risk/prevent/second-guess cycles yourself. Consider, for example, these business situations and recommended courses of action:

  • “We should replace our aging server-based data storage with a RAID Level 5 Storage Area Network (SAN). If we don’t, we run the risk of a business outage that will cost us days in which we are effectively out of business.”
  • “We never brought our systems into PCI compliance – we store unencrypted credit card information in our databases. We need to fix this, or one day we’ll find we’ve exposed our customers to identity theft … and we’ll face some serious fines and loss of business, too.”
  • “At the current rate of expansion we’ll need more A/C power in our data center than our circuits will support. We need to either rewire it, or make a serious investment in energy-conserving technologies. Otherwise, by this time next year we’ll have hit the wall.”
  • “We need to spend the time and money in each of our projects to implement our systems in an architecturally sound way. If we don’t, in five years or so we’ll find ourselves expending more effort in each project keeping our kludges from breaking than in new functionality.”

Imagine you’re persuasive enough that the Executive Committee approves the new SAN, column-level encryption, green technology, and solid architecture.

The result: Disaster never strikes.

Then, the old CEO retires, replaced by a new one who believes in S.M.A.R.T. (Specific and strategic, Measurable, Attainable, Realistic, and Time-bound) goals and nothing but S.M.A.R.T. goals. “We spent a lot of money,” he complains. “Where are our measurable results?”

Not to pick on the dear departed Dr. Drucker (who at least had the good sense to include “Strategic” in his formulation — good sense that subsequent adopters of the formula, who have omitted it, lack) but …

Yes, “but” means what it usually does. I am going to pick on him, because the obsession with measurability Dr. Drucker instigated has done a great deal to discredit acts of prevention.

And to encourage self-serving second-guessing among those who lack both the courage to lead and the good sense to avoid self-righteous certainty.

Comments (11)

  • You fell off the boat. You don’t measure prevention. You do risk analysis. BTW Drucker was often big on risk analysis.

    I have worked in several state agencies. An eye opener was working as Acting CIO of Corrections. We spent the time of 10 or more programmers daily keeping up prison time calculations in line with sentencing and laws. It isn’t easy to figure out if a prisoner was picked up on parole violation and also committed a crime when is the earliest time he can get out based on multiple factors including amount of time each judge sentenced and the laws in effect at that sentencing. We spent all of this time not as prevention but because the RISK associated with an incorrect calculation could have massive lawsuit value. Figure $1 million for each year served beyond the legal limit. After about 5 such lawsuits, it seemed cheaper just to make sure that each prisoner’s record (including added time or “good” time) was accurate.

    Now I work for a social service agency that also has the duty to provide unemployment insurance benefits to those out of work. It is a nightmare and one of those events where risk analysis would have correctly predicted that in any given 5 year scenario, the worst will happen and we should have been prepared. That has nothing to do with prevention and everything to do with risk analyis.

    Companies don’t roll out new products without a thorough ROI and risk analysis. Why do IT departments think that unless it is customer-initiated request for something, risk analysis of the hardware and software and people used to supply IT services, is not a requirement?

    I’m throwing you a life jacket for the boat you just left untimely. Re-focus the question you are asking and the answer is pretty darn clear. Prevention is the same thing as insurance. An insurance company analyzes risk before providing a policy and setting the fee for same. Stop trying to measure prevention and start thinking about risks and insurance. That’s something that can be sold.

    Best Regards,
    Paulette Lowe

    • Paulette … I’m not going to reply in full here, because this subject is what I plan to cover next week. Briefly, though, since you brought it up:

      Risk analysis works when you’re dealing with a statistical universe and testable models that can be perfected with data. Your prison-population models are an excellent example.

      The economic meltdown challenged the validity of current macroeconomic models, so we can’t use their predictions to cross-check how things are actually turning out. And, as a one-time event it isn’t part of a statistical universe, either.

      I agree with you that when possible, risk analysis is how organizations ought to assess the impact of preventive measures.

      A minor point: Prevention (improving the odds) isn’t a form of insurance. They are alternative ways of managing risk, with a third being mitigation (reducing the impact).

  • Two quotations seem relevant here:

    Doubt is not a pleasant condition, but certainty is absurd. (Voltaire)

    Soviet Joke

    Student: “Comrade teacher, can you explain, in the simplest terms, the essential difference between communism and capitalism.”

    Teacher: “Of course, comrade student. Under capitalism Man oppresses Man, whereas under communism, it’s just the opposite!”

    (Anon. Soviet Comrade)

  • Paulette accurately describes how you can sell such solutions, but that is slightly off topic. What Bob (I think) is discussing is how you measure the outcomes to assess the effectiveness of your solution.

    Sometimes you may have high-impact high-frequency risks (for some definition of ‘high’). You implement suitable control measures and the frequency falls from several times per year to zero (read: it hasn’t recurred yet). That is a measurable outcome – if the cost of implementing your control is less than the reduced costs arising from the risk, you’re saving money.

    The problem arises for risks that are similar to fires – high-impact low-frequency. You may be successful in persuading senior managers to implement controls either to reduce the frequency (e.g. smoke detectors) or the impact (e.g. extinguishers or firewalls). It’s measuring the effectiveness of this type of control that is the difficulty.

    The attitude of far too many of today’s C-level execs is that of the guy who doesn’t buy fire insurance for his house and then congratulates himself on saving money each year it doesn’t burn down.

  • Along the same vein: There are several interpretations of S.M.A.R.T. If you are using “Attainable,” then I recommend “Relevant”. If you are using “Realistic” (a pretty good synonym for “Attainable”), I recommend “Aligned”. The point is to add strategic alignment with the greater corporate vision to your goals.

    Which brings us back to “Measurable.” I take issue with your straw man who argues that prevention results aren’t measurable. I concur with Ms. Lowe (if I may paraphrase) that insurance is a fine way to quantify risk. If the new guy comes in and starts trashing the old guy for wasting money, I think we can prove the value of the previous efforts. Based on the old technology and the actual computing requirements, what would the power requirements have been a year later had the server room not gone green? Would they have exceeded the circuitry capacity? Were the estimates correct? Show you were right and try not to be too smug about it. If requirements didn’t grow at the expected rate, or if normal rotation of equipment would have reduced power consumption, straw man might have a point. Review the forecasting model, power consumption curves over time, etc. and do better next time.

    What has been the cost to companies that have had identity theft issues? The simplest formula takes the probability of a breach, multiplied by the cost, and you end up with a number bean counters can live with. If 3% of your competitors have taken a $10MM hit, you have a $300,000 risk. Refine that figure by noting that 20% of the competitors without improved security have taken a hit, and your budget goes up to $1.99MM (I’d say $2MM, but there has to be some ROI in order to get approval).

    If the risk isn’t measurable, then prevention is like meteor insurance. (Only one person has ever been hit, and she suffered a broken leg). I suspect no one buys personal meteor insurance… and it would be pretty hard to calculate the value of it.

  • I didn’t know there was anyone else left who had read Keith Laumer! Great stuff;, keep it coming.

  • There is a slight flaw in the analogy between world economies and company strategies when it comes to prevention. The flaw is that there is only one world economy, and the actions of the US carry such a big influence on this economy that there is no room for the comparisons that would be possible between different businesses. The way to measure prevention is not within a single business but among a group of similar businesses that responded to prevention challenges in different ways. If you have enough businesses and access to the data, you should be able to draw reasonable inferences of the relative values of different prevention strategies. This was effectively the approach taken by Jim Collins in his book “Good to Great.”

  • I appreciate your point that there is now a lot of time being invested second guessing what has worked or not. However, the risk analysis point is where everything has been and is still currently falling apart.The lack of risk assessment in that past:Giving loans to people who could never afford to pay them.Turning junk loans into risky securities (while claiming they were low risk).3) Allowing Fannie and Freddie to grow so huge, but forgoing good oversight to advance a political agenda.
    Then we had the “broken” fix of the stimulus where there was no risk management in the bill, but the vice president would be in charge of punishing people after the fact when is it discovered that they pilfered the money.There is no second guessing needed here – this was a design for failure.

  • Bob,

    I understand your reasoning and have been in the position to sell preventative measures myself.

    On thing that bugs me to this day was the fact that while Y2k needed serious time and resources spent on it, where I worked during the Y2k times IT used Y2k to drive everything. The panic that was sold to the management was palpable end of the world stuff.

    I remember getting into an argument with one of the corporate Y2k coordinators who was arguing that your car may not even start because of Y2k. My argument was, “why the bleep would an automotive engineer even consider using date based functions when programming a cars computer to run the car.” He said, “well we can’t be sure.” Of course we can’t be sure, but in my mind any engineer that would create a control system for a car that would fail due to Y2k, shouldn’t be an engineer.

    So at the company I was at during Y2k got a lot of money, IT bought a lot of servers. IT went through a lot of code (most of it rightfully so, but some not). Management saw the spending go up and up, and didn’t like it.

    Sure the prevention worked, disaster was avoided. But there was a nasty side effect, management lost faith in IT because they knew Y2k was oversold. I had previous to Y2k instituted a plan for server refreshes that had me replacing about 1/3 of my machines every 3 years. I used this for PCs and for Servers. The plan made sense, it was strategic, it was stopped.

    Where I in 1999 bought some new hardware and installed new Oses on older hardware, in keeping with my plan. Others replaced everything. Three years later, IT spending was frozen. The reasoning was that you could eek out another year from everyone’s 3 year old equipment. But a few years into this freeze 1/3 of my equipment was now 5 years old and eeking out more time for it would be relatively hard. Planning on my part was punished, and lack of planning by some of my counterparts in IT was not.

    C level interests have since Y2k, and through any downturn, looked to cut IT to the bone. I believe that a real reason for this was that IT asked for money to fend off Y2k by saying “the sky is falling!”, instead of coming up with real logical plans.

  • Another insightful column, thanks. I remember as a young person laughing out loud when President Ford, hearing of an impending flu epidemic, stepped up programs to make and distribute flu vaccine. Ford did the right thing, and of course there was no epidemic, but we’ll never be able to prove that his actions averted it.

    One of IT’s biggest challenges: if anything goes wrong, everybody asks what you’re doing to earn your pay. If nothing goes wrong, everybody asks what you’re doing to earn your pay…

Comments are closed.