Checking how fact-checkers check?

That title made me think of the woodchuck…anyway, here is the abstract:

Fact-checking has gained prominence as a reformist movement to revitalize truth-seeking ideals in journalism. While fact-checkers are often assumed to code facts accurately, no studies have formally assessed fact-checkers’ performance. I evaluate the performance of two major online fact-checkers, Politfact at Tampa Bay Times and Fact Checker at Washington Post, comparing their interrater reliability using a method that is regularly utilized across the social sciences. I show that fact-checkers rarely fact-check the same statement, and when they do, there is little agreement in their ratings. Approximately, 1 in 10 statements is fact-checked by both fact-checking outlets, and among claims that both outlets check, their factual ratings have a Cohen’s κ of 0.52, an agreement rate much lower than what is acceptable for social scientific coding. The results suggest that difficulties in fact-checking elites’ statements may limit the ability of journalistic fact-checking to hold politicians accountable.

That paper (pdf) is by Chloe Lim, political science at Stanford.  For the pointer I thank Andrew Hall, some interesting political science papers on his home page.  Here is his very interesting book manuscript on how the devaluing of political offices drives polarization, worthy of a top publisher…


anon BTFO

Markus has a good answer below.

More generally, I don't think Tyler means this as "who can ever know?"

As Markus says, in a proper fact-check the facts and logic are laid out. The proper sneer is to dissect those, and not adopt a rote, partisan, position that X or Y fact-check is always wrong.

Derp. You have used these unreliable and biased sites to argue from authority.

"The proper sneer is to dissect those"

Just look at your diction, you hack. Fact-checking fact-checkers is accurately described as "sneering"? Maybe to someone who depends on biased institutions for ideological spoon feeding.

To be completely honest, I was on my tablet, and somehow mangled "answer." Google thought that I meant "sneer." I noticed too late, but I decided I could go with it.

After all, "haha" was trying to sneer up there, without "facts" or "checking"

'on how the devaluing of political offices drives polarization'

Sounds like mission accomplished for the Virginia School, actually.

I gave up on these guardians of truth after I followed a link to Snopes about the truthiness of the claim that Madonna offered BJs to anyone who voted for Hillary. Despite a video in which the crone said exactly that Snopes twisted itself into a pretzel to reject the claim on the grounds that she couldn't have been serious because it was a practical impossibility. If Trump were to say he'd give Putin the Moon for a hug you can bet Snopes would judge the text in strict Scalian fashion.

Are you talking about this article:

They didn't say the claim about Madonna was "false" -- indeed, they quote her while also noting that her statements were made on stage while introducing comedian Amy Schumer.

It's been awhile but this bit I do recall: "What many viewers who didn’t follow the whole story missed was that Madonna wasn’t seriously making such an offer..."

Don't bring up source material - it confuses those with alternative facts.

Politifact: Trump's claim that Hillary bleached her emails

Finding: FALSE, she used a program to zero out her drives!

Here is another example of an uncharitable disambiguation

"Last Thursday evening, self-described fact-checkers for Politifact, the Los Angeles Times, and the Washington Post mobilized as soon as the text of Donald Trump's nomination acceptance speech was leaked to challenge a series of immigration facts cited by the candidate. In their haste, or maybe it was their zeal, to try to de-bunk Trump's statements on immigration, they put out sloppy work and sometimes downright false evaluations."

It's good that they did this research, but any non-partisan familiar with these sites and their work could have told you the same immediately. Fact checking politics is a great deal of motivated reasoning.

Pretty much, which is why it will be so refreshing to be able to access source material, such as Comey's notes and Trump's 'tapes' without needing the media in any way, shape, or form

"Truth" is boring and the competition for online ad revenue is fierce. Fact checkers exist because they generate page views, not to engage in boring but scientifically rigorous, objective, emotion free "truth" seeking. You economists of all people should understand the incentives in the industry. The public intuitively understands this, hence the decline in public trust in journalists as the importance of the online page view business model has grown.

post of the year candidate

>post of the year candidate

True. But it was kind of boring, don't you think?


This is a bit misleading. In social science, high interrater agreement is achieved by training and detailed instructions on what is a codeable unit, how it is to be coded etc. This doesn't apply here. (Even then you see kappas down to 0.5 being published, usually with the acknowledgement that this is less than what one would hope to get and an blaming it on the subject matter. The latter is sometimes valid, as it is much easier to agree whether someone touched their nose than whether someone made a disparaging remark.)
I just googled 'kappa in radiology' and got e.g. this:
"There was moderate agreement between the two readings of one MR radiologist (kappa=0.482); the other three radiologists had kappa values that were good to excellent (0.716, 0.691, and 0.829). There was a clinically acceptable level of interobserver variability between all radiologists. The agreement was fair to moderate between the MR radiologist and the other observers (kappa range: 0.362-0.519) and moderate to good between the other three radiologists (0.508-0.730)."

More importantly however, the thing fact checkers do, unlike your typical interrater rater is they provide detailed explanations and references to evidence for their conclusions. People can check those. Oddly enough, with only 14 disagreements among fact checkers (in the binary version) the author of the paper did not bother to determine what caused the two fact checkers to come to different conclusions which would have been worthwhile IMHO.

I have seen some of the others. The biggest problem is that, as is indeed obvious by looking at their detailed explanations and references, they don't really limit themselves to checking "facts," albeit for an understandable reason. Most politicians (not Trump) often make technically true statements which are misleading, or imply further facts which are not technically contained in the statement. Even more subjectively, a statement may be entirely true but be about something which isn't very important at all (but the politician is implying is very important), which can include things like talking about a tiny increase year on year in a statistic while ignoring that it's still down a very significant number from 10 years ago. A great deal of what these "fact checkers" do is distinguish between "mostly true," "mostly false," and so forth on this basis. That is why there is less difference in the binary version, though sometimes this is enough that they do affect things even in a binary version. (Trump makes this easy by blatantly, outrageously lying on basic facts.)

This is where many of the differences arise, as well as claims of bias.

To take one example, President Obama was initially rated as being completely true when he said that "if you like your doctor, you can keep it." Now, it's certainly true that there's nothing in the PPACA that would ban people from keeping their doctor. On the other hand, it surely must be admitted that the PPACA, like any legislation making enormous regulatory changes, would certainly upset the status quo (indeed, that's part of its intent) and would certainly make changes that would cause financial incentives making doctors drop patients, or insurance companies drop doctors or hospitals, or so on. So there's also a very good case that the statement was at the very least misleading. (It was also very difficult to judge before the full effects of the law were felt, or even afterwards since changes to health care unrelated to the law certainly occurred.)

To take a relevant example, Politifact itself gave different ratings to a common claim about the income tax when it was uttered by Democratic Senator Jim Webb (Mostly True) as when it was said by Republican Ron Paul (Half True). After this was pointed to as an example of bias, they changed the former to Half True so that they matched and, to their credit, left up a correction notice.

My experience is that their bias, often unconscious, results in rating things said by Democrats as "Mostly True" or "True" when they are technically true but misleading, but rate Republicans as "Mostly False" or "Half True" for the same.

I think those two pages Webb/Paul show the utility of the effort. It only takes a glance to see that, yes they should both have the same answer.

It's working the claim in public, for review. It's "open source" fact checking, and not a pundit's bellow of "Wrong!!!" and then moving on.

It demonstrates the utility of the full explanation, yes. However, it does mean that cute little charts of "see, this proves that X politician / X party is more honest than Y" because X got lots of Mostly True while Y averaged a Half True are almost entirely useless. It's magnified by the fact that exactly which assertions to fact check (and how often, as some statements have been repeatedly fact checked) is a choice of the organization.

Indeed, such charts can be "true but utterly misleading" in the same way as the typical political utterance.

Perhaps, like some movie and game reviewers have tried, it would be more beneficial with just the long explanation but without the letter grade or star rating. However, people like boiling down to a simple statistic that can then be misused.

I took a quick look at the book manuscript. I thought that perhaps Mr. Hall would cite Alan Ehrenhalt's 1992 The United States of Ambition: Politicians, Power, and the Pursuit of Office, might be worth a look:

Politifact gave its worst rating ("pants on fire") to a Trump claim that "Crime is rising." Then, after after the crime statistics showed Trump was correct, Politifact responded with a bunch of this: "Trump’s statement was broad, without qualifiers, and it came amid comments that painted an overarching image of a nation in decline. Trump didn’t say that crime was rising 'recently' or 'in recent months' or 'over the past year' or 'in some places.'"

They look correct to me. Here was the claim:

"Hard to imagine what's happened to our country," Trump said. "America is being taken apart piece by piece … just rapidly auctioned off to the highest bidder. We're broke. We're broke. (Our debt is) $19 trillion, going quickly to $21 trillion. Our infrastructure is a disaster. Our schools are failing. Crime is rising. People are scared. The last thing we need is Hillary Clinton in the White House or an extension of the Obama disaster."

Our country, meaning nationally. Crime is rising.

Got data? I do:

From your data: "It’s important to note, however, that the FBI reported a 3% increase in the violent crime rate between 2014 and 2015, including a 10% increase in the murder rate." We can argue over whether people ought to worry over a recent 10% increase in the murder rate, when the trend over a longer series was still down. But we can't argue over whether the violent crime rate was rising around the time Trump made his statement. It was.

I really don't think that is what Trump meant or was trying to say. That paragraph is a saga of disaster, American ruin. It is meant to appeal to the false-but-common belief that crime is generally-really getting worse in America.

When ~60% of voters believe a thing, it is easier to tell them that you believe it too, rather than correct them.

Now, why do so many believe this false thing? It might start with the headline effect. News tells us of crimes every day, without giving us perspective on their distribution. The whole country watches the news of one kidnapping and feels it as if it is local news.

On a second order though, I think politicians and some media profit from continuing the lie. If people believe the thing, reinforce it. Profit by it.

In that sense the fact-check is still helping you. You should have learned the big picture, and joined the ~40% who know crime is falling.

All of this is reasonable. The problem is calling it a "fact check". Trying to interpret what Trump was "trying to say", or whether something is misleading, incomplete, etc. is a matter of opinion, and it's inevitably influenced by political biases. It's fine to do that, but Politifact and others, by calling their work fact checking, make a claim that they are definitive, authoritative sources, rather than merely pundits.

What Trump actually said is "Crime is rising". Given the fact that it is a general statement that didn't specify time frame, etc., one ought to simply ask "Is there a plausible interpretation of this statement which is true?" And yes there is, since there was recent data showing a rise in crime. So you can't credibly rate this statement as a lie.

If you think that Trump is misleadingly implying doom and gloom, given overall trends, it's perfectly fine to state that opinion. I would agree, actually. But that's not a fact check.

"I really don’t think that is what Trump meant or was trying to say."

Of course you don't. Just like your fact-checking Gods, you are choosing to analyze whatever it is that you wanted Trump to mean, instead of what he actually says. You do realize this is because you are incredibly biased, right?

I award this comment a +1

My +1 was directed at dan1111, not that i disagree with Thomas. Everyone can relax now that ive cleared that up.

I just went and re-read Trump's convention speech.

You might claim that he finally got his "facts" lined up, but I certainly think it was to tell the big lie that:

These are the facts:

Decades of progress made in bringing down crime are now being reversed by this Administration’s rollback of criminal enforcement.

Do you, dan1111 and MOFO, really think that added up to true, enduring, facts about America?

I mean it's silly if you know anything about crime and enforcement in America. Most criminal enforcement is locally funded and organized .. but I'll let you answer.

And yeah I think the words "fact" and "checking" can be applied to a claim that "progress made in bringing down crime are now being reversed by this Administration’s rollback of criminal enforcement."

The alternative, to not "check" that "fact" is to give it free reign, to mean anything.

@anon, no, I don't think the statement about Obama reversing progress on crime is fact checkable. It's a broad, vague statement that is not even remotely falsifiable. It's not clear what sort of "progress" Trump is referring to, nor what actions he think are "reversing" it. And given that he is talking about stuff happening "now", the evidence of what effects these policies will have probably doesn't even exist yet. Predictions of the future are inherently opinions.

Trump's statement on Obama's legacy on crime is opinion, not fact.

I guess we can agree to differ then. I see "crime" as a quantifiable thing, etc.

But more than that I really fear the opposite course, where everything becomes opinion. Is climate changing? There are lots of numbers, but maybe it's all "opinion" of what "climate" even is?

A bad path IMO.

A simpler way to say it: when Trump said "crime is rising" was he telling a big, enduring, truth about America?

The crime he was referring to is the $21 Trillion stolen from my sons?

You omit crucial details that are recounted in the very article you link to. Politifact did its analysis through 2014 because 2014 was the last year for which official, finalized statistics were available at the time they wrote their piece in the middle of 2016. Preliminary statistics for 2015 showed crime might have increased and so they amended their article noting that fact. However, as of September 2016, the FBI has released what appear to be final numbers for 2015 showing a slight increase in violent crime between 2014 and 2015 but a decrease in property crime. The overall trend for both crime rates remains down.

What matters for evaluating whether Trump was lying is the data that was known at the time. When he made the statement, the preliminary data was available, but not the final data.

Re: Politifact gave its worst rating (“pants on fire”) to a Trump claim that “Crime is rising.” Then, after after the crime statistics showed Trump was correct,

Um, no. Or at least please document this rise. It is true that murder rates have risen a tiny amount nationally because of increases in a handful of large cities. I've seen nothing which suggests that other crimes are increasing (on a per capita basis that is). Long term, crime rates have been falling since the 90s, if only because the population is aging and crime in mostly a young man's thing, and that slow decline continues.

"The FBI’s preliminary 2015 data indicates that violent crime rose by 1.7 percent, with murder going up by 6.2 percent and rape going up by 1.1 percent or 9.6 percent, depending on the definition that is used. (Property crime continued to decline, falling in total by 4.2 percent.) And the increase appears to be continuing in early 2016. The Major Cities Chiefs Association reports — based on data from 63 reporting agencies — that violent crime in the first quarter of 2016 was up over the first quarter of 2015. In particular, homicide was up 9 percent in the reporting jurisdictions, and rape was up 4 percent. The AEI post has more. (Note that the yearly population growth in the United States is only about 0.7 percent, so not just the raw amount of violent crime but also the per-person violent crime rate is increasing.)"

What is the right fact-check when 57% of registered voters are getting it wrong?

"Public perceptions about crime in the U.S. often don’t align with the data. Opinion surveys regularly find that Americans believe crime is up, even when the data show it is down. In 21 Gallup surveys conducted since 1989, a majority of Americans said there was more crime in the U.S. compared with the year before, despite the generally downward trend in both violent and property crime rates during much of that period. In a Pew Research Center survey in late 2016, 57% of registered voters said crime had gotten worse since 2008, even though BJS and FBI data show that violent and property crime rates declined by double-digit percentages during that span.

Maybe "pants on fire" was an unkind way to correct a common misconception. "False, but commonly believed?"

Like the idea that the living standard of the average American hasn't risen for 40 years, right? Everybody knows that.

'Facts' about how rotten everything is have remarkable staying power under a political system where contradicting the vague natural human notion that someone is ripping me off somehow is political suicide.

"Like the idea that the living standard of the average American hasn’t risen for 40 years, right?"

Politifact likely wouldn't "fact-check" this coming out of a Democrat's mouth. This is basically a Democrat party philosophy and it is wrong and Politifact is too biased to "fact-check" their team and partisan hacks like anon don't care.

Here's a simple way to look at it Brian. People say they don't like "fact-checking." Which is it that they don't like, facts? Or checking? You say "the living standard of the average American hasn’t risen for 40 years" ... are you saying that isn't a fact at all, or it can't be checked at all?

I believe he's saying it is often parroted by D's frequently to the point it is taken as gospel by their base and yet never shows up on the fact-checker's dockets.

Google can fact check that statement for us. Top substantive entries from Politifact:

I think people don't like the checking, because it is cited by many as an end-all-be-all argument. Really, most statements, as we have seen in the comments on this site, have at least some argument on both sides. To have somebody state their argument on these fact-checking websites with finality, then allow it to be religiously cited by people who treat these websites as Gods of truth, is well...foolish. If bias i possible, which it always is with humans as the interpreters, then official "fact-checking" has every opportunity to be pure bullshit.

It's not a fact, it's a falsity, but it sure as hell ain't just Democrats saying it's a fact, either. Some of the biggest bellyachers around here who like to cite stuff like this are way off to the right.

In answer to you all, I agree that "living standard" is pretty all-encompassing. No one knows how to balance better health care with higher costs. But, in defense of fact-checkers, if they do it the right way they lay out what they are saying, including that it is probably more narrow than "everything." As an example, Pew:

But it's so much worse than that.

For one thing, why are you citing 2014 wages here in 2017?

If you were interested in facts, you would be less lazy.

I'll assume you can find the relevant CPI-U numbers, but I've done the math for you:

The $2.50 in 1964, adjusted to 2017, is $19.73. Average hourly wages in January 2017 were $21.83. That's an 11% increase. Barely budged? Feel free to take an 11% pay cut and tell me that your pay has barely budged.

But is CPI-U a heaven sent, sure fire way of comparing dollars at different points in time? What about CPI-W? That shows a 15% increase. Or how about PCE? The Fed likes PCE. Using PCE gives us a 44% increase.

And good luck factoring in all the stuff that didn't exist at any price in 1964, or all the stuff that exists for basically free today.

Here's Pew's headline: "For most workers, real wages have barely budged for decades", which I'm sure you rate as an unassailable fact.

The worst sin of all in the headline is in the wording. It is much as says that most workers have been slogging away for decades without a real wage increase. Beyond the issues I raise above, this statement mistakes percentiles for people. Can you show me any individuals who have moved through time according to this pattern over a span of decades? I reckon there may be some, but very few.

The American people have no trouble swallowing hard luck stories about themselves. And no politician will dare argue the point. The result is a proliferation of bullshit accepted as fact by all and sundry.

To be fair Brian, an 11% pay cut over 53 years is barely noticeable. Which is the point, an 11% pay raise over the same period is effectively zero.

I'm not one of the moaners and complainers, as you know from my posts I'm a Ridleyan rational optimist who thinks things are generally much better now than 50 years ago. But it's not unfair to say wages have stagnated.

I will admit the laziness of taking the top search result. LINK.

I'll have to think about the rest, and how much I think a 11-15% CPI based growth over 50 years is good or bad. I think I might be JonFraz in that I'm not sure.

What counterfactual is there?


I reject your frame. You say that an 11% increase works out to tiny annual increases (0.2%).

I, like Einstein, marvel at the miracle of compounding. 11% is 11%. That's an additional $4,000 per year. If you're gonna talk about wages in 1964 versus wages today, this is the relevant frame.

And, as I suggest, there is plenty of reason to be skeptical of the 11% number anyway.

This is just stupid.

Anytime someone uses wages or "cash wages" or "monetary wages" stop reading.

Benefits soak up a larger and larger % of income for the median worker. If you intentionally don't Include benefits then you are a lying liar who is telling lies.

And measures of inflation are largely bullshit unless it's the same basket of goods. And the same quality of goods. And it's not. Unless it's food and gas. I wonder where we could find one for food and gas.....

Comparing "phone" to "phone"....idiots.

Re: Like the idea that the living standard of the average American hasn’t risen for 40 years, right? Everybody knows that.

What makes that hard to calculate is the change (and improvements) is the available technologies. I make less in inflation adjusted dollars than my father made at my age (this is especially annoying as I am college educated and in tech; my father was a truck driver back when that was a good unionized job). But I have access to an I-phone and the whole Internet, and better healthcare too. So who has lived better, in material terms? I honestly don't know.

The Andrew Hall book is very good. I have felt, and even argued here, that there are under-counted moderate voters.

Fact-check confirmed.

Who checks the checkers? Other checkers, I guess.
"Approximately, 1 in 10 statements is fact-checked by both fact-checking outlets, and among claims that both outlets check, their factual ratings have a Cohen’s κ of 0.52, an agreement rate much lower than what is acceptable for social scientific coding." America has become hopelessly divided.

Tense political atmosphere wrecking markets in Brazil

What fact-checkers can't (and don't attempt to) do is provide context. A statement can be literally true or literally false depending on the context. Indeed, a statement that is literally true can be interpreted the opposite of the literal meaning depending on the context and, hence, be false. Many of Trump's statements are impossible to fact-check accurately because of both his unusual syntax and habit of omitting the context for a statement. Then there is the cultural phenomenon of irony: the intentional use of words to have the opposite of their literal meaning. What's missing here is the poison that pervades communication when clarity is passe. The Strunk and White generation understood that vague, obscure, and unintelligible communication were the tools of the tyrant, that clarity in communication is the antidote to the tyrant.The use of fact-checkers has the ironic effect of normalizing vague, obscure, and unintelligible communication.

"Fact-checking" was an attempt to coat left-wing opinion with a layer of pretend objectivity. I don't think very many people were fooled.

I don't think there's any reason to assume bad faith. To me it looks like good intentions + bias + lack of clear thinking about the task and methodology.

"I don’t think there’s any reason to assume bad faith."

People on one side of the newsroom are hard core liberals who constantly lean left, those {sometimes the same people} on the other side are un-biased truth tellers who merely "check facts".

The whole concept is bad faith.

In other words, Bob and Rich, there is no truth only partisanship, right?

The rise of fact-checking coincides with the decline in the number of news outlets. Consider the analogy of the EMH: it's defensible provided there are many buyers and sellers and with none having the ability to dominate the market. It's when too much power is concentrated in too few that the EMH breaks down. So too with news. I would make the analogy to the "adaptive market hypothesis": it holds that markets are sufficiently close to the ideal that is works well enough. It works because the irrational behavior of a few buyers and sellers is overwhelmed by the rational behavior of many buyers and sellers. But it works only if there are many buyers and sellers to "correct" the irrational behavior of a few buyers and sellers. Similarly, when there are many "sellers" of news the accurate purveyors will overwhelm the inaccurate purveyors and, thus, come close enough to the ideal of accuracy. It's when a few purveyors dominate that accuracy no longer holds; it's a market failure in news. In the past there were many purveyors of the news, even small to medium-sized communities with many outlets. Today that's no longer true. Indeed, only a few outlets dominate the national news (the NYT and WP in print and Fox News in television). What's risen as the anti-dote isn't a return to many news outlets but intentionally biased outlets on the internet, each catering to a particular mythology. In this environment, fact-checkers attempt to provide the accuracy that is lost due to a market failure. It's won't work in the market for news any more than it will work in the market for investments.

Juvenal's Sixth Satire updated: "Who'll fact-check the fact-checkers?"

The purpose of "fact-checkers" is to enable leftists to argue from authority. They are all operated by leftists and they are all extremely biased.

New subtitle for the Andrew Hall book: "You get what you pay for."

The real problem is selection bias.

All statements cannot possibly be checked. They cherry pick bad ones from conservatives and good ones from liberals so they can show that conservative X lies more than liberal Y.

Then they draw only favorable inferences from liberals and draw only negative inferences from conservative statements. More manipulation.

This is a reasonable point, but are there any fact checkers who are more 'fair' in your opinion?

What evidence do you have that they are cherry picking? It is certainly possible on the part of one or more fact-checking outlets but that is a claim that needs to be backed up with analysis and not just assertion. Aren't conservatives supposed to be skeptical of claims of discrimination or disparate impact?

Not news to me. Those sites are just one more portion of the lying media. For instance, When Trump claimed that Clinton wanted "open borders" Politifact said that was "mostly false" because while she did use those exact words, she was talking about "energy," being the first person to ever in history use that phrase to refer to energy. And even if she did mean immigration, it was really "support" for open borders because "her statement sounded aspirational and contained no timeline or explanation as to how she would make it happen politically." By that logic, no politician is ever in favor of anything.

Well no, because no one outside the hardest core anarcho-libertarians actually want "open" borders. Politifact was correct to argue Trump's accusation was false. If he had said she favors increased immigration, that would be different.

Thankfully we will never find out precisely how bad Hillary's immigration policies would have been.

"because no one outside the hardest core anarcho-libertarians actually want “open” borders."

Pants on fire lie. Many on the far Left openly want it, the Wall Street Journal has endorsed it on multiple occasions, Vox has published articles in support of it. You know this, just as you know that Hillary Clinton stated she wanted open borders when she thought the public couldn't hear her, just as politifact knows this.(Though at least they're getting paid to shill.)

This is another one of those cases where if you actually read the full article instead of focusing on the graphic at the top, it is a pretty balanced discussion. We don't have text of the full speech so we cannot actually evaluate the full context of her remarks but Politifact includes quotes from people who are a bit skeptical, as you and I are both are, that she was talking strictly about "energy." Two of the three outside analysts they consult think it is plausible she is hoping for something analogous to the Schengen Agreement or the European Union Common Market that makes it easier for people to move across borders to work or do business. Given her use of the term "dream," though, she almost certainly would consider this to be something that would take decades to eventually come to fruition and is clearly not an actionable policy proposal. Ronald Reagan once dreamed of complete abolition of nuclear weapons -- nobody interpreted his lofty speech as indicating he was planning unilateral disarmament for the U.S. during his term, though.

In cases like this, it seems almost better to avoid a "ruling" and just lay the facts out.

Anyone fact-check this fact-checker-checker's facts?

The statement that "14 out of 70 statements (20 percent) received two completely opposite ratings from the fact-checkers" is based on the author assign scores of 3 into the binary "True" camp. 7 of those 14 "disagreements" are when one Fact Checker is calling the statement "Half True" and the other is calling it "Mostly False". How much of a difference is there between half true and mostly false? In my estimation, not very much. 5 more were comparing 3s and 5s. This is especially inappropriate given FactChecker's statement: "The line between Two and Three can be bit fuzzy and we do not award half-Pinocchios. So we strive to explain the factors that tipped us toward a Three." (Source:

The author provides Hillary's statement about Bernie and negative ads as an example, but it is the only example where the two fact checkers truly disagree with one another. All of the others are basically just disagreements on shades of gray.

Personally, I'd rate this study "Mostly False".

My reaction to the study as well. It's the work of a PhD student elevated beyond its merits.

Ironically, when a researcher who rubs elbows with the fact-checking community performed a superficially similar study (using a measure of coder consistency to look at fact-checking, the fact checkers welcomed it.

Lim's study, if anything, improves on the earlier study by Michelle A. Amazeen, but isn't likely to receive anything but criticism from fact-checkers (most likely they'll ignore it). And that's how selection bias and ideological bias work.

The main problem with both studies? Fact checker rating systems were not designed as instructions for coders. Good coding instructions need to come with objective markers to guide the coders. Fact-checker rating systems have almost nothing of the kind, and the fact checkers display little reluctance in ignoring what little clear guidance their rating definitions offer. It's a fact that PolitiFact's creator, Bill Adair, called the decision on the ratings "entirely subjective."

It's not reasonable to expect the fact-checkers to rate consistently when their decisions are guided by subjectivity, is it?

Some might be interested in this as a corollary: In the current issue of the US Naval Institute's journal Proceedings, there is a commentary on the problems with pedantic obsessions with literal truth in some statements

When something like Wikipedia becomes a widely used source, trouble is afoot.

Comments for this post are closed