“If you torture data long enough, it will confess.”
– Economist Ronald Coase
“If you torture data long enough, it will confess.”
– Economist Ronald Coase
MY SIDE WON!!!
No, not President Obama. Not the Democratic party either and not any ballot initiatives, state constitutional amendments, or other specific candidates either.
Nate Silver won this election, and it matters to all of us.
Silver, in case you somehow managed to ignore all political polling this election season, is the polling class’s version of Moneyball’s Paul DePodesta (Peter Brand in the movie) — a hard data and analytics guy whose results proved superior to gut feelings and personal preferences.
Longtime readers will recall that KJR is affiliated with the Competence Party — a mythical organization built on eleven principles, three of which are:
Nate Silver is our kind of guy, and this last election just might turn out to be a turning point (given Malcolm Gladwell’s advocacy of “the power of thinking without thinking” in Blink I refuse to say “tipping point”) … where was I? Oh, yes, Silver’s dead-on predictions provide a reason for cautious optimism that in the contest for business decision-making’s soul, evidence and analysis might regain (or maybe just gain) primacy.
If they do, there’s a lesson to be learned from comparing how Nate Silver did so well to how Wall Street’s equally nerdy “quants” fared so poorly.
It’s a simple lesson, too. Here’s what it isn’t — that Nate Silver was of better character and integrity than the quants. I’m not saying he is and I’m not saying he isn’t. I’m saying that if we place our bets based on character, we’re likely to place them quite poorly.
No, it’s a matter of incentive. Silver won by having the most accurate predictions. He’s a celebrity because he called 49 states correctly. Or maybe 50, depending on whether Florida ever does declare a winner, and, if it does, who it is. In spite of all the gas he took during the run-up to the election about being biased (and really, does anyone think anyone decides who to vote for based on opinion poll results?), his only bias was in needing to be the most accurate analyst out there.
Compare that to the quants. Their financial rewards were front-end loaded — they made their money when their financial instruments sold out. They had every reason to understate risk and overstate payoff.
This doesn’t mean they did so deliberately. It’s much more likely they “knew” the picture was great (and knew the answer management wanted to hear). They sold themselves, and so didn’t dig deeply enough into the underlying assumptions of the models.
They rationalized, that is, and did so with surpassing sophistication.
Nate Silver, in contrast, didn’t just aggregate poll results. He dug into the methodologies followed by each poll to determine their relative trustworthiness, and weighted them accordingly.
Business leaders who want Nate Silver results will need to be very careful to avoid rewarding analysts who follow the quant model. They’ll be making decisions based on what the analysts tell them, so if there’s even a whiff of a preferred answer in the wind, some analysts will smell it and unconsciously fiddle things to get it.
And most managers, faced with conflicting analyses, will unfailingly choose the analysis that tells them what they want to hear.
Which is why KJR has made such a fuss, so often, about the critical role the business culture plays in all this, and in particular establishing a culture of honest inquiry.
If your “quants,” analysts, “data scientists,” or … call them by their proper name and background, statisticians … operate within a culture of honest inquiry, they will give you unbiased answers, because what you care about is whether they turn out to be right or not. Otherwise, I, Silverishly, predict their predictions won’t be reliable.
There’s a danger in the Nate Silver model, by the way, not that I’m particularly worried about it. It’s the impact of relying on aggregators rather than front-line researchers. If the lesson learned here is that aggregating the work of multiple polls is what led to Silver’s best-in-class results, that will give us fewer source polls and more aggregation of those polls. That would be bad, because without reliable polls to draw on, aggregation is nothing more than garbage in, garbage out.
On the other hand, “garbage” might be just the right metaphor for most pre-election coverage.