Announcing the Great KJR 2016 Election Real-time Fact Checking Challenge!
But first an assurance: This won’t be a political column. And a disclaimer: What follows is probably a pipedream. Still …
A confession: I’m addicted to fact-checking websites. The granddaddy of them all is, to the best of my knowledge, factcheck.org, and it’s arguably the most thorough. The Washington Post’s Glenn Kessler runs its Fact Checker blog, which Kessler livens up by summarizing each fact-check with a rating that runs from Gepetto (completely accurate) through 4 Pinocchio’s.
My favorite is PolitiFact, with a rating scale of True, Mostly True, Half True, Mostly False, False, and for truly egregious falsehoods, Pants on Fire.
What I most appreciate about all of these is that they are (1) entirely non-partisan (or, more accurately, they’re partisans for accuracy and against fictional accounts of events, not for a political party or philosophy); and (2) sufficiently explanatory that their scoring rationales are clear and clearly reliable.
But … their value suffers from a limitation familiar to IT professionals, namely, data latency. In the world of IT, data latency is the result of overnight batch processing. That is, transactions come in throughout the course of a day, but the information available from the databases they post to isn’t an accurate reflection of the state of things until the next morning.
In the case of the fact-checking sites, politicians and pundits make speeches or debate one day, but the fact checkers don’t catch up until the next morning at best. Even worse, many of those who listen to the speeches aren’t addicted to fact-checking sites as I am, and so, like managers who trust their guts so much they don’t bother reading computer-generated reports, they won’t ever read the fact checkers’ findings.
Which is why I’m announcing the Great KJR 2016 Election Real-Time Fact Checking Challenge.
What it will take — the winning entrant will listen to political speech, and, in near-real-time:
- Recognize statements of purported fact (as opposed to opinions, which will not be scored).
- Evaluate them using criteria similar to what human fact-checkers use.
- Evaluate quickly enough to flash the phrase in question along with a red, yellow, or green indicator before the speaker is too far into the next subject.
- Pass an accuracy test by matching or improving on the judgment of the current crop of human fact checkers, with “improving on” defined as the fact checkers reading the machine’s explanation and saying, “Gee, I never thought of that.”
I figure this isn’t all that different or more difficult than what it took for Watson to learn to play Jeopardy, let alone its having read and assimilated the meaning of ream upon ream of medical literature so as to become a premiere diagnostician.
Which means IBM is one logical entrant for our little contest. Meanwhile, Google is the undisputed master of Internet search, and has invested heavily in machine learning technology besides, so Google is another logical entrant.
Who else? Beats me. I don’t see Siri, Cortana, or Echo reaching this level any time soon, but I’m sure there are other potential entrants that would jump at the opportunity.
And while I’d be happy to adjudicate, KJR’s involvement isn’t really necessary. Any company that could win, place, or show in the Great KJR 2016 Election Real-Time Fact Checking Challenge could more easily market the result directly to whichever networks will carry the pre-election debates between (unless something seriously unexpected happens) Donald Trump and Hillary Clinton.
What network wouldn’t want to include this feature, especially if all the others did?
Regrettably, while everyone reading these words can see the inevitability of this capability becoming reality, it probably won’t be in time for this presidential election. The next one, though, is likely, and if not then certainly the one after that.
When it is available, real-time fact-checking will be a complete game changer for electioneering. Even if some news networks prove so partisan that they create their own phony fact checking AIs, dual-screening is increasingly prevalent, so many voters would bring up an independent fact checking AI on their tablet or mobile device while watching a speech or debate.
Okay, fun’s fun. Imagining candidates who know that the moment they utter a false statement it would immediately be flagged Pants on Fire for all to see is a lovely day dream.
The larger point is that unlike previous tries at commercializing artificial intelligence technologies, which have proven useful but not transformational, we’re on the verge of capabilities whose impact will be nothing short of dramatic.
It isn’t too soon to start your strategic planning engines, to figure out how the new wave of artificial intelligence might affect your company’s marketplace, and its competitive position in it.
I think you have a super idea. I think it would have to be administered by a mostly recognized neutral non-profit like the League of Women’s Voters or Stanford University.
Technology to search the net for the veracity of a given statement already exists with Google search as the back end. But, some candidates are quite adept at not answering the question they have been asked. I analyzed the transcripts from 5 debates in the last 8 months and found one candidate that had 9 different ways of not answering 131 questions out of 150 questions asked of that candidate. Only 2 or 3 times did this particular candidate tell a lie. Skillful non-answers work better than lies because they confuse and mislead, but it’s very hard be held accountable for your behavior.
One of the 9 techniques this candidate used was to say something that was true, but would be the answer to a different question. Another technique was to answer a question with a truism, a statement that was true but not an actual answer to the question posed.
So showing, in real time, whether the question posed was actually answered, then, if it was, doing a Google search for it veracity, would great.
But, IMHO, also showing the non-answers and what kind of non-answer was given could also be done in real time. You could have a small panel of speech experts vote on each answer, even showing the results of each vote. It should be no harder than real-time translating at the UN. My guess is that if some volunteer group of programmers, like Source Forge, really wanted to do it, it could get done in 3 to 4 weeks.
If the will is there, this could actually get done before the conventions start. Thanks, again, for sharing your idea.
Nice analysis – interesting statistics. Thanks.
I recall watching Richard Nixon use this exact technique – sounding like he was answering a question but really answering a different question that sounded like the one that had been asked.
Works like a charm, every time … the only defense against this are more aggressive debate moderators.
I had the same concern..
Bob, how about adding a requirement such as:
“Detects non-answers and evasions”
Well, human fact-checkers restrict their work to assessing statements of fact for a reason, and I think it’s a good one. A statement either is about the facts (or the evidence; there’s a subtle difference) or it isn’t. It’s pretty much one or the other.
But evasion is a matter of degree. As for non-answers, the level of semantic analysis that would be required to objectively identify these would be, even if achievable, so controversial that it wouldn’t be worth the effort.
Just my opinion.
GREAT use of Watson-like technology! Unfortunately, as existing fact checking services (and Snopes) can attest, there are always believers, facts be damned.
ManagementSpeak: Thanks for the input. We’ll definitely include that in our analysis.
Translation: Get outta here you mook. This is too important for facts to get in the way of our ideology
Just a thought on the non-answers issue. I see concerns about the potentially controversial, for some, results of such an analysis as substantial and well considered.
But the way IBM’s Watson was able to beat the world’s chess champion and the best Jeopardy players was because it know more correct answers than any human could know and was very well “trained” to find them in real time, not because it out reasoned or was smarter than the human it was competing against.
It seems to me that if you had several academic speech experts; profession politicians from the Democrats, Republican, socialists, Tea Party, communists, and Green Party; men and women; blacks, whites, Native American, and Latino; inherited wealthy, new wealthy, middle income non-Bourgeoisie, Bourgeoisie, poor, and destitute, all selected with statistically valid means.
And, if you then have small groups from each of these groups go through 5 debate transcripts from each major party over a 5 or 6 month period, and then turn their responses over to IBM for training, Watson could actually do a better job of identifying non-answers than humans could, and with less controversy.
Training Watson would take more than 3 to 4 weeks, but the ultimate rewards could save our bacon.
Anyway, thanks for your column and the follow up dialogue.
Why stop at elections — any transactional interchange (sale, hire, lease, etc.) could be subject to the same real-time fact-checking. We now have apps that act a micro-lie detectors for tells like blood pressure, micro-expressions and the like. But those only help if the speaker knows he or she is lying. Real-time factchecking could also work with the ignorant or misinformed even if they sincerely believe what they are saying.
Pickup artists and con artists would object, as well as the super polite white liars, and such machines would ruin good fiction in plays, movies, and the like, but these are small issues relative to exposing the “truth” whenever and wherever technology can be employed.
“Staying on message” is another term for the above discussion. Every candidate is given (or comes up with) the bullet points they want to run on, and is coached/trained/ordered to answer every question with one of the bullet points, relevant or not.
Campaign staff HATE it when you go off message. They spent all that time crafting a winning message (theme, bullet points, background facts and research), and you the candidate just ignore it all and actually answer the question?!?
Of course, really good campaign staffs prepare you for the questions that aren’t part of your message. Really REALLY good staffs prepare you to answer the question AND shape the answer to reflect your theme. But, not everyone has a really good staff (or takes the time to prep).
I love the whole idea, although I worry about the loss of nuance and adult dialogue. But, considering some of what I’ve seen already this year and in past elections, most of that is already lost. (Still a goal, though.)
As much as I would like to see it, I’m convinced most decisions have little to do with facts. (This year’s election should have convinced anyone watching it.)
But what I’d love to see is making this real-time analysis available to a moderator with a spine and the mandate to use it. Imagine Jon Stewart with this system grilling candidates.