Leadership – Page 84

Data-informed decisions are alive and well

Don’t make data-driven decisions. Make data-informed decisions, or so my friend Hank Childers advises.

It’s a nice distinction that recognizes both the value of evidence and its limitations.

For those who like to rely on evidence we live in tricky times. The increasing availability of evidence about just about any topic is accompanied by an at-least-equally increasing supply of disinformation, in direct proportion to the profit to be made by biasing things in the profit-maker’s favor.

Which is one reason I’m skeptical of the long-term reliability of IBM’s Watson as a medical diagnostician.

There’s a program called Resumix. What it does is scan resumes so as to find the best skill-to-task matches among the applicants. It’s popular among a certain class of recruiter because reading resumes is an eye-blearing chore.

Worse, the recruiter might, through fatigue, miss something on a resume. Worse still, the recruiter might inadvertently practice alphabetical discrimination, if, for example, the resumes are sorted in name order: Inevitably those at the front and back of the stack will receive more attention than those in the middle.

But on the other side of the Resumix coin is this: Most applicants know how to play the Resumix game. Using techniques similar to how those who write websites learn how to get the attention of search engines, job-seekers make sure Resumix sees what it’s supposed to see in their resumes.

If Watson becomes the diagnostician of choice, do you think there’s any chance at all that those who stand to profit from the “right” diagnosis won’t figure out how to insert what Watson is looking for in the text of the research papers they underwrite and pharmaceutical ads they run?

It’s one thing for those who developed and continue to refine IBM’s Watson division … for whom, by the way, I have immense respect … to teach it to read and understand medical journals and such. That task is merely incredibly difficult.

But teaching it to recognize and discard utter horse pucky as it does so? Once we try to move beyond the exclamation (“That’s a pile of horse pucky!”) to actual definition, it isn’t easy to find one that isn’t synonymous with “I don’t want that to be true!”

Well, not, that isn’t right. A useful definition is easy: Horse pucky is a plausible-sounding narrative that’s built on a foundation of misinformation and bad logic.

Defining horse pucky? Easy. Demonstrating that something is horse pucky, especially in an age of increasingly voluminous disinformation? Very, very hard. It’s much easier to declare that something is horse pucky and move on … easier, but intellectually bankrupt.

So imagine you’re leading a team that has to make an important decision. You want the team to make a data-informed decision — one that’s as free from individual biases as possible; one that’s the result of discussion (solving shared problems), not argument (one side winning; the other losing).

Is this even possible given the known human frailties that come into play when it comes to evaluating evidence?

No, if your goal is perfection. Absolutely if your goal is improvement.

While it’s fashionable to disparage the goal of objective inquiry because of “what we now know to be true about how humans think,” those doing the disparaging are relying on evidence built on the philosophical foundations of objective inquiry … and are drawing the wrong conclusions from that evidence.

Here’s the secret:

The secret of evidence-informed decision-making: Don’t start by gathering evidence.

Evidence does, of course, play a key role in evidence-informed decision-making (and what would you do without KJR to give you profound insights like this?). But it isn’t where you start, especially when a team is involved.

Starting with evidence-gathering ensures you’ll be presiding over an argument — a contest with winners and losers — when what you want is collaboration to solve a shared problem.

Evidence-gathering follows two essential prerequisite steps. The first is to reach a consensus on the problem you’re trying to solve or opportunity you’re trying to chase. Without this, nothing useful will happen. With it, everyone agrees on what success will look like when and if it eventually happens.

The second prerequisite step is consensus on the process and decision-making framework the team will use to make its decision. This means thinking through the criteria that matter for comparing the available alternatives, and how to apply evidence to evaluate each alternative for each of the criteria.

Only then should the team start gathering evidence.

Informed decisions take hard, detailed work. So before you start, it’s worth asking yourself — is this decision worth the time and effort?

Or is a coin-toss good enough?

Evidence-based decision-making’s limitations

Evidence has its limits.

Regular readers, good friends, casual acquaintances and just about anyone else I can trick into a conversation on the subject know I’m a strong proponent of evidence-based decision-making. Most of the time, compared to most of the alternatives, evidence should be a tool of choice when making an important decision.

But … (please don’t snicker) it’s a big but … evidence does have limitations. It isn’t always the right tool for the job:

Evidence here but not there: Sometimes you have to choose among alternatives when reliable evidence is only available for one of them.

Insistence on evidence is why large enterprises so often favor cost-cutting over revenue enhancement. Cut the cost of a business process and it’s [relatively] easy to follow the savings to the bottom line. Invest in revenue enhancement, on the other hand, and there’s rarely a reliable and trustworthy way to provably connect investment and results.

This is one of the biggest challenges with the M in so-called SMART goals (specific, measurable, achievable, realistic, and time-bound). SMART causes organizations to prefer goals that are measurable … for which it’s possible to collect numerical evidence … over goals that are important.

Of course that’s Gus: As Daniel Kahneman explained in his don’t-even-think-about-not-reading-it book, Thinking, Fast and Slow, when you run into your friend Gus, you don’t need evidence that it’s Gus. Unless you’re a schmuck or trapped in a horror movie, you see his face and that’s that.

It’s people who don’t know Gus who might ask to see his driver’s license.

The future: Those who plan for the future have definitely chosen the best period of time to plan for. Planning for the past is way too late; meanwhile, the present turns into the past before you’ve finished planning for it.

When you plan (for the future), evidence should be something you take into account, but cautiously. Your evidence is about the past, after all. When the future turns out to be like the past only more so, evidence is just the ticket. When the future turns out some other way, the evidence will have pointed you in the wrong direction.

Will the future look like the past? Good luck finding evidence to help you figure that out.

Confirmation bias: This is the formal term for when they accept without question evidence that supports what they want to be true while nitpicking to death evidence that supports your position.

They’re using evidence for ammunition, that is, not for illumination.

What’s harder is knowing when we are they. Here’s one clue: If you find yourself memorizing a point so as to win a future argument with one of them, you’re probably succumbing to confirmation bias. More to the point: If you’re reading about a subject and your goal isn’t to understand it more thoroughly, you’re turning into one of them.

Citing other people’s opinions: In the Internet age, someone else’s opinion often counts as evidence, and if not, someone else’s opinion about someone else’s opinion does.

An expert’s opinion is useful, when it’s based on original research or research the expert has carefully reviewed.

But often a “consensus of the experts” is little more than aggregating a bunch of dumb looks. The averaged opinion of people who are experts in other subjects isn’t evidence. It’s just what a bunch of folks who know something about something else think.

Survey monkeys: Some surveys provide useful evidence. The rest just tell you what a bunch of anonymous respondents say they think about a subject.

We’re talking about people whose qualifications you don’t know and most likely the surveyors don’t know either, other than the “qualification” that they’re willing to take the time to respond to a survey.

Even for honest practitioners, opinion research is a complex field fraught with evidentiary landmines. And they’re a vanishing breed compared to a growing population of push-pollers who do everything possible to get the results their clients have asked for. So accept survey results with caution.

Are these cautionary limitations enough to persuade you evidence-based decision-making is a bad idea? I hope not, because they shouldn’t be. When evidence that passes these tests is available, take maximum advantage of it when making an important decision.

But don’t pretend when it isn’t. Sometimes the best you can do is make explicit assumptions and apply careful logic to them. When that’s the situation, do your best with it.

And don’t worry that it’s the best you can do.