MY SIDE WON!!!
No, not President Obama. Not the Democratic party either and not any ballot initiatives, state constitutional amendments, or other specific candidates either.
Nate Silver won this election, and it matters to all of us.
Silver, in case you somehow managed to ignore all political polling this election season, is the polling class’s version of Moneyball’s Paul DePodesta (Peter Brand in the movie) — a hard data and analytics guy whose results proved superior to gut feelings and personal preferences.
Longtime readers will recall that KJR is affiliated with the Competence Party — a mythical organization built on eleven principles, three of which are:
- Our decisions will always begin by examining the evidence. And we will recognize that when our cherished principles collide with the evidence, the evidence wins. Every time.
- With new evidence we will reconsider old decisions. Without it, we won’t.
- We will never mistake our personal experience for hard evidence. Personal experience is the evidence we know best. It’s also a biased sample.
Nate Silver is our kind of guy, and this last election just might turn out to be a turning point (given Malcolm Gladwell’s advocacy of “the power of thinking without thinking” in Blink I refuse to say “tipping point”) … where was I? Oh, yes, Silver’s dead-on predictions provide a reason for cautious optimism that in the contest for business decision-making’s soul, evidence and analysis might regain (or maybe just gain) primacy.
If they do, there’s a lesson to be learned from comparing how Nate Silver did so well to how Wall Street’s equally nerdy “quants” fared so poorly.
It’s a simple lesson, too. Here’s what it isn’t — that Nate Silver was of better character and integrity than the quants. I’m not saying he is and I’m not saying he isn’t. I’m saying that if we place our bets based on character, we’re likely to place them quite poorly.
No, it’s a matter of incentive. Silver won by having the most accurate predictions. He’s a celebrity because he called 49 states correctly. Or maybe 50, depending on whether Florida ever does declare a winner, and, if it does, who it is. In spite of all the gas he took during the run-up to the election about being biased (and really, does anyone think anyone decides who to vote for based on opinion poll results?), his only bias was in needing to be the most accurate analyst out there.
Compare that to the quants. Their financial rewards were front-end loaded — they made their money when their financial instruments sold out. They had every reason to understate risk and overstate payoff.
This doesn’t mean they did so deliberately. It’s much more likely they “knew” the picture was great (and knew the answer management wanted to hear). They sold themselves, and so didn’t dig deeply enough into the underlying assumptions of the models.
They rationalized, that is, and did so with surpassing sophistication.
Nate Silver, in contrast, didn’t just aggregate poll results. He dug into the methodologies followed by each poll to determine their relative trustworthiness, and weighted them accordingly.
Business leaders who want Nate Silver results will need to be very careful to avoid rewarding analysts who follow the quant model. They’ll be making decisions based on what the analysts tell them, so if there’s even a whiff of a preferred answer in the wind, some analysts will smell it and unconsciously fiddle things to get it.
And most managers, faced with conflicting analyses, will unfailingly choose the analysis that tells them what they want to hear.
Which is why KJR has made such a fuss, so often, about the critical role the business culture plays in all this, and in particular establishing a culture of honest inquiry.
If your “quants,” analysts, “data scientists,” or … call them by their proper name and background, statisticians … operate within a culture of honest inquiry, they will give you unbiased answers, because what you care about is whether they turn out to be right or not. Otherwise, I, Silverishly, predict their predictions won’t be reliable.
There’s a danger in the Nate Silver model, by the way, not that I’m particularly worried about it. It’s the impact of relying on aggregators rather than front-line researchers. If the lesson learned here is that aggregating the work of multiple polls is what led to Silver’s best-in-class results, that will give us fewer source polls and more aggregation of those polls. That would be bad, because without reliable polls to draw on, aggregation is nothing more than garbage in, garbage out.
On the other hand, “garbage” might be just the right metaphor for most pre-election coverage.
The quote I like is ‘torture data long enough and it will tell you what you want to hear.”
The reason poll aggregation worked better than the Republican approaches is because Rove relied on land-line opinion polls. I don’t know anyone under 40 who has a land line at their house. Thus calling folks with land-lines no longer provides a random sampling of the entire population.
Mistaking large amounts of numbers for ‘data’ is a management problem of the first degree. Silver realized that enough polls from _different_ self-selected groups is much better than ones from the same group.
I am pleased for Nate Silver and all his recent positive acclaim. I’ve been following Andrew Tanenbaum (the “Votemaster”) for his data-driven predictions of the electoral vote count since 2004, however. I think Professor Tanenbaum’s work in the field anticipates that of Nate Silver and is worth a mention. At a minimum, I thought he should receive some notice here given Professor Tanenbaum’s close connection to information technology. (Read more, including about his status as a “grandfather of Linux,” at http://www.electoral-vote.com.) By the way, great column, Bob, as usual. I haven’t kept a joint running since 2010, but I still enjoy your work.
Pingback: Fresh Links Sundae from David Lowe of Actionable ITSM | Actionable ITSM