Fact-gathering and fact-checking are foundational to journalism and politics [16, 17]. Journalists often advocate for and value fact-checking because it coincides with their professional values and offers a possible solution to public distrust in news and institutions [18]. Fact-checking is no longer optional for major, well-intentioned newsrooms and news organizations; it has become a central need and ethos for them in the wake of misinformation and propaganda, requiring journalists to make fast and accurate decisions about content veracity [19]. Citizens also believe fact-checking is crucial, especially for politicians [20]. Identifying the truth, and communicating it in a way that creates an informed public, are core parts of the journalism profession and are instrumental to a democracy [21]. Fact-checking is part of this process because it promotes accountability and attempts to ensure that people receive high-quality information from their leaders [22]. Additionally, to the extent that valid fact-checking is a time-consuming, effortful, and difficult task that benefits from professional training and experience, fact-checkers aid the public by making veracity assessments more accessible.

The critical issues evaluated in this work are how much two high profile fact-checkers observing U.S. politics—The Washington Post’s Fact Checker and PolitiFact—agree on what to fact-check and, when they do check the same facts, the degree of veracity agreement. Should two (or more) fact-checkers overwhelmingly agree on what is worth fact-checking and agree in their assessments of the factual accuracy, news consumers should have trust and confidence in a consistent set of facts from news providers to stay informed. Should there be considerable disagreement between fact-checkers on what to check, news consumers might question what they read, distrust news media, and consider fact-checkers to be partisan. They may also be left with doubts about what constitutes ground truth.

We note that prior work has taken a similar approach by evaluating statement selection from one presidency [7]. Nevertheless, there are outstanding questions to resolve. First, Lim [7] only evaluated President Trump’s statements as a presidential candidate and those fact-checked by The Washington Post and PolitiFact between September 2013 and November 2016. Statements during the Trump presidency were unevaluated, and it is therefore unclear how fact-checkers might have selected statements to check as President Trump took office and his deception rate increased [23]. Second, the severity or grade of a false statement (e.g., 1 Pinocchio vs. 4 Pinocchios) likely matters for how people evaluate the truth. If fact-checkers show small-to-middling agreement on fact-checking deceptiveness, this might suggest scale sensitivity discrepancies or differences in how deception is conceptualized by sources. Finally, we evaluate if fact-checking selection disagreements are moderated by the political party in office. Our work considers fact-checked statements for President Trump while in office and compared the rate of selection disagreements to the first 100 days of President Biden’s term. This helps ensure that our analysis is non-partisan and can generalize beyond a single administration.

Honesty matters to an electorate [24, 25], and fact-checking is a key way to provide a report card for deceptive politicians. The interest in fact-checking presidents has increased over time, not only because it is seen as an innovation to journalism but also as an obligation of journalists [26]. Online access to technology and resources at scale, which began accelerating during the time of the Obama presidency, have amplified and accelerated the ability to record and verify information [6] and consequently, one of the most fact-checked public figure has been former President Trump [27, 28]. Given the sheer number of Trump’s falsehoods [23], because his approval ratings remained relatively flat as his falsehoods increased over time [29], and because his statements were consequential for the U.S. population, we evaluated how much fact-checkers agreed on their assessment of his statements. Empirically, President Trump’s statements yielded a sample size larger than any other public figure in the U.S. or elsewhere, providing a unique opportunity to evaluate fact-checkers.

Finally, we believe it is critical to evaluate statements in this manner because citizens should rely on fact-checkers to “get it right,” and report correctly and consistently on facts. If the news is not vetted properly or in a reliable manner, this can lead to the unchecked spread of fake news and disinformation. Most people are poor “fake news” detectors [30], suggesting Trump’s repeated falsehoods may have subtly undermined the public’s ability to discriminate truths from lies [31], or the public’s trust that news organizations will present facts honestly. The public expects fact-checkers and news media to keep them informed about politics, which begins with a consistent set of facts.

Fact-checking and deception detection are not synonymous, but the overlap is considerable. According to most definitions of deception, false statements are not lies if the sender lacks deceptive intent [32]. A politician, for example, may believe what they say, even if it is patently false. In such cases, the statement is not a lie, but it could be fact-checked as false or misleading. Further, public figures may lie about non-factual matters (e.g., opinions that are not rooted in fact). Not all false and misleading statements can be rectified by fact-checking and the intersection between what can be documented as false and what is deceptive is imperfect. Nevertheless, the overlap between deception detection and fact-checking is substantial and consequently, theories of deception detection may inform fact-checking.

Most deception detection research has investigated lie detection based on senders’ specific and observable behaviors referred to as deception cues [33, 34]. Decades of research on cue-based lie detection suggest that people are poor lie detectors (54% accuracy; [35]). In a finding called the “veracity effect,” people are consistently worse at identifying lies than truths [36]. Thus, the 54% accuracy result is correct truth-lie discrimination; but the probability of correctly identifying a lie, per se, is below 50% [35].

Truth-default theory [32, 37] provides important insights that are relevant to the current discussion, and we draw on TDT for much of our theoretical grounding. TDT is unique among theories of deception detection in its focus on communication content rather than cues, and in addition to fact-checking as a method of deception detection. Further, TDT’s concept of triggers may also be useful in understanding journalistic fact-checking. Simply stated, a trigger is any stimulus that either prompts suspicion or skepticism, or that shifts a veracity assessment from under suspicion to considered deceptive.

The core idea of TDT is that of people defaulting to the truth. This proposition is rooted in the social need for efficient and effective communication. People tend to uncritically believe communication content unless skepticism is actively triggered [37]. Consequently, the public’s vulnerability to deception is likely greater than deception detection research suggests because deception detection experiments prompt conscious truth-lie assessments that might not otherwise come to mind. Truth-default research shows, for example, that a communicator’s identity as a well-known politician, for example, is not sufficient to trigger critical considerations of veracity [38]. Outside of the laboratory, the idea that some content might be false or misleading often does not register. Fact-checking can thus serve as a trigger for the public to assess veracity and honesty [32]. Interestingly, the trigger construct may also apply to fact checker’s selections of what to fact check. Differential trigger sensitivity may offer a useful theoretical frame for understanding differences in selection criteria and choices.

Truth-default theory prioritizes checking facts over behavioral cues and demeanor [32, 37]. The theory values journalistic fact-checking because (a) checking facts is theoretically specified as the most preferred and efficacious method of deception detection, and (b) due to the truth-default, the public is unlikely to critically assess content in the absence of dedicated fact-checkers. People tend to believe that others are honest, most of the time. Evidence of the efficacy in fact-checking by identifying false and misleading statements would be consistent with the theory; robust inaccuracies would undercut a major assumption of the theory.

Presently, most fact-checking is a human activity. Therefore, issues of how a fact-check is triggered, the ways in which it yields information to confirm or disconfirm the veracity of a statement, and its ability to be shared with many people rather than informing the judgment of a single individual are important consequences to applying the TDT propositions. Importantly, TDT states (regardless of who is doing the detecting), deception is most accurately detected, “by comparison of the contextualized communication content to some external evidence or preexisting knowledge” [37]. As this seems to be the essence of fact-checking, we therefore raise the third research question:

RQ₃: To what degree does truth-default theory apply to journalistic fact-checking?

In the U.S., two of the most prominent media-based political fact-checkers are The Washington Post Fact Checker and PolitiFact. The present work draws on the perspective of truth-default theory to evaluate how much these fact-checkers agree in their assessment of facts. Here, we focus on how much overlap exists among statements that are checked by leading fact-checkers (i.e., the influence of human abilities). We also evaluate the strength of the association between scaling across The Washington Post Fact Checker and PolitiFact to assess if they frequently give the same or similar deceptiveness ratings. Finally, to broaden the scope of our investigation, we evaluated how often two non-journalistic sources (i.e., Snopes, FactCheck.org) fact-checked the same statements as PolitiFact.

Our cross-checking revealed 22.6% of statements fact-checked by PolitiFact did not appear in The Washington Post database (116/513) (An exploratory thematic analysis using the Meaning Extraction Method [39, 40] for the statements PolitiFact checked but The Washington Post did not is available in the S1 File). As the top panel of Table 1 suggests, a range of 20–35% of the statements across the four deceptiveness rating categories used by PolitiFact were not checked by The Washington Post or were not included in the database of 2–4 Pinocchio statements. The verbatim check for true and mostly true statements found that only 1 out of 64 statements (1.6%) was given a false and misleading rating and included in The Washington Post selection, when PolitiFact tagged it as mostly true. This discrepancy is investigated further in the S1 File.

Our general pattern of results replicated prior work well [7]. We therefore attempted to confirm that the fact-checkers are consistent in their identification of false statements in other ways. Specifically, to provide an independent assessment of these data and further understand the difficulties involved with fact-checking, all four authors independently fact-checked a random selection of 10 statements previously fact-checked by both organizations. Each coder searched online sources for a claim’s veracity without using The Washington Post Fact Checker, PolitiFact, or other published fact-checkers as sources. Coders made true, false, or uncertain ratings based on such evidence and coders also indicated the sources they used for veracity determinations. On 8 of the 10 statements, coders reached 100% agreement. The remaining 2 statements reached majority agreement (75%). In both cases, the fourth coder believed the statement was false, but felt the evidence was sufficiently ambiguous that a “false” rating could not be assigned. These data demonstrate that fact-checking is not easy, even for trained coders using clear guidelines. Poor grammar, conflating of ideas, obfuscation (intentional or not), conflicting source data for claims, and subjective interpretation of source data all contribute to the difficulty of ascertaining the veracity of a claim. Despite these challenges, our independent fact-check of the previously fact-checked statements validated the published results and therefore serves as an important robustness check.

We collected a random sample of 100 statements fact-checked by PolitiFact and identified if they were also checked by Snopes and FactCheck.org. This analysis helped to indicate if a subtle, yet unknown fact-checking bias existed for The Washington Post or if a similar rate of selection agreement/disagreement existed for other sources as well.

Only eight PolitiFact statements were also checked by Snopes and 21 PolitiFact statements were also checked by FactCheck.org. No disagreements in veracity were observed and all statements fact-checked by Snopes were also checked by FactCheck.org. The bottom panel of Table 1 reveals most statements checked by PolitiFact were not checked by Snopes or FactCheck.org. At most, 40% of statements from Snopes or FactCheck.org were also checked by PolitiFact. Taken together, the evidence collectively suggests more correspondence in statement selection between The Washington Post and PolitiFact compared to other sources and PolitiFact (All 8 PolitiFact statements checked by Snopes were also checked by The Washington Post and two-thirds of the statements (14/21) also checked by FactCheck.org were checked by The Washington Post). The data further suggest that discrepancies are evident in what is fact-checked, but not in the bottom-line determinations of veracity (i.e., was the statement true or false?). When different fact-checkers check the same statement, they invariably agreed on the substance of the check.

We calculated a bivariate correlation between deceptiveness ratings for any statement with a Pinocchio rating from The Washington Post’s Fact Checker and a Truth-O-Meter rating from PolitiFact. The relationship between these deceptiveness ratings was positive and statistically significant, r(107) = .451, p < .001. A non-parametric bivariate correlation was also consistent in magnitude and operated in the same direction, ρ = .419, p < .001.

It is important to note that PolitiFact defines false as “The statement is not accurate” and pants on fire as “The statement is not accurate and makes a ridiculous claim.” Therefore, pants on fire is the scale equivalent of a false statement; the ridiculous nature of a claim does not alter the “falseness” of it, but instead adds a qualifier about how the claim is made. Converting values of the pants on fire category to false produced a similar result, r(107) = .447, p < .001.

Deceptiveness agreements systematically decrease from pants on fire distinctions to half truths (Fig 1). Half truths have a seemingly random distribution of Pinocchios and there is near perfect agreement for pants on fire statements. This suggests that there is moderate deceptiveness agreement across The Washington Post Fact Checker’s Pinocchios and the Truth-O-Meter from PolitiFact. Correlations show general agreement but blurring of adjacent categories is also common. Most discrepancies were only off by one category, and there were only 6 instances of being off by two category discrepancies. Major off-diagonal disagreements were largely exaggerations by President Trump that The Washington Post considered egregious and PolitiFact believed they had some validity. For example, three examples in the top left corner of Fig 1 related to exaggerations by President Trump (e.g., “I’ve won awards on environmental protection;” 4 Pinocchio’s), but were half-truths via PolitiFact (he received a personal award for donating land in New York State after not building a golf course there).

The current results have several implications for political fact-checkers, media scholars, and news consumers. Although understandable, differences in statement selection are substantial.

Journalists and political fact-checkers should be aware of uniquely high-frequency deceivers (prolific liars, or those who lie prolifically; [43–47]) and when they might be more likely to lie (e.g., as suggested by TDT, when topics are consequential or when the truth is a problem; [32]). For example, during his final year as president and as the 2020 U.S. presidential election approached, President Trump’s lying rate increased dramatically compared to other years [23]. Thus, fact-check frequency should increase proportionally. Yet, fact-checkers have finite resources and they must be selective. Newsrooms and political fact-checkers should create processes to be systematic and comprehensive with their cataloguing of deception, while adjusting to variable base-rates of lying. Relatedly, the fact-checkers have a responsibility to be unbiased and balanced in their approach. Politicians and political candidates may be unequal in their frequency of deceptive messaging, but the fact-checking procedures should reveal that imbalance rather than focusing on politicians who create an a priori expectation of dishonesty. Finally, while selection variation may be clear to researchers, the implications may not be so clear to those using these fact-checking resources. Fact-checkers need to communicate these differences with greater precision and clarity to the public.

This investigation provides an understanding of fact-checkers’ and political fact-checking blind spots. When fact-checkers disagree on what to fact-check, and their disagreements are prevalent, news consumers might doubt what they read and how much they trust a source. The implications of fact-checking disagreements are exacerbated when they become aggregated in news stories, read by a wide circulation on social media, or when a concerned citizen cannot find the same statement fact-checked by another source. News consumers should not have to doubt or fact-check the fact-checkers. Perhaps having a consistent set of standards in journalistic fact-checking (e.g., definitions of deception, deceptiveness rating systems that are consistent across sources) and removing ad-hoc policies at the source level will alleviate public concerns about fact assessments. More widely promoting and creating public awareness of fact-checking standards might also be beneficial (https://www.poynter.org/ifcn/).

Finally, in response to RQ₃ about the degree to which TDT applies to journalistic fact-checking, it is also important to describe how our results inform truth-default theory [32, 37]. Our evidence suggests bottom-line judgments of statement veracity were almost perfectly aligned for professional fact-checkers. Among those checked by PolitiFact, there were broad similarities and modest differences across scaling compared to those checked by The Washington Post. Therefore, fact-checkers agree on veracity when they both check a statement, but their selection and scaling can be misaligned. These results inform truth-default theory because the theory champions fact-checking (labeled in the theory as correspondence information) as one of the most preferred methods of lie detection. Our results show that fact-checking appears valid, and the theory can be applied to non-interactive (e.g., not interpersonal) settings of consequence (e.g., political and journalistic fact-checking). The process of detecting deception is not limited to one individual confirming the veracity of another. The triggers for suspicion can arise from many sources, and while the fact-checking organization functions in ways that are analogous to an individual’s efforts to confirm or disconfirm a message, the results may be widely circulated to a mass audience of news consumers.

According to truth-default theory, most people presume a message is honest independent of its actual veracity [32, 37]. Therefore, if false or misleading statements are not fact-checked (or not agreed upon to be fact-checked by multiple sources), their ability to mislead the public goes unchallenged. The truth-default remains until a trigger exists. Fact-checking can serve as a trigger or be the response to a trigger. Nevertheless, absent public confidence in fact-checkers, the utility of the triggering function is undercut.

Our test of TDT in the domain of fact-checking has implications beyond U.S. fact-checking sources like The Washington Post Fact Checker and PolitiFact because the theory is pan-cultural [32, 37]. As prior work suggests, the theory’s “core presumptions concern human nature and are not tied to any particular culture, religion, government, or social structure” [44], suggesting that its principles, propositions, and foundations can extend to fact-checking in other regions of the world. Fact-checking efforts are global due to the pervasive and consequential nature of disinformation [2–5] and therefore, it may be helpful for future work to compare how fact-checking approaches can inform each other in a pursuit of best practices, identifying elusive perpetrators of false information, and understanding cultural differences related to the dissemination and detection of such false information.

We learned that political fact-checking is difficult, and fact-checkers face nontrivial challenges as they try to evaluate the truth. In some cases, as our analysis involving Snopes and FactCheck.org revealed, only one organization will fact-check a claim. People often want information fast, though fact-checking requires time and thorough investigation. Even then, fact-checkers are susceptible to human error and human biases that might affect their judgment. We advocate for reliability estimates in fact-checking and cross-checking sources to ensure consistent and credible news. Two of the coders in the PolitiFact, Snopes, and FactCheck.org cross-check also noticed that FactCheck.org’s search algorithm was enhanced by Google. Therefore, if sorted by “Relevance,” coders or fact-checkers might receive slightly different search results on FactCheck.org because of a person’s unique Google history and what Google assumes the searcher would be interested in. Algorithmic bias in fact-checking and citizens’ experiences should be acknowledged.

Our investigations purposefully chose premier fact-checkers across analyses, but there are others that would be worthy of evaluation in future work. We selected The Washington Post’s Fact Checker, PolitiFact, and others because of their popularity and deceptiveness scaling, but more sources, including those outside the U.S. could be assessed as well (e.g., CNN, USA Today, Agence France-Presse, and other members of the International Fact-Checking Network). A cross-cultural analysis of fact-checking might be helpful to illuminate cultural or norms-based differences in what people decide to fact-check and how veracity might be scored differently. However, this may require new methods as the opportunity for direct comparison of checked facts is limited. We also evaluated two presidents in this paper. Therefore, it will be important to evaluate how checkers might continue to disagree on facts from other political figures.

Although the results of this work described disagreement between two main fact-checkers, it is unclear why these patterns emerged. We offered two possible explanations based on methodological and theory-based evidence, though future work would benefit from interviewing fact-checkers to understand their process and decisions on what to check in difficult cases [48]. Pairing qualitative evidence about editorial decision-making and fact-checking principles with quantitative work would make for a more holistic evaluation of how much fact-checking disagreements exist and why they might exist.

In addition, we only evaluated false statements when comparing PolitiFact and The Washington Post; comparing true and false statements required additional fact-checking with Snopes and FactCheck.org. Therefore, a limitation of some analyses is that we cannot evaluate potential disagreements on truths, which are also critical to fact-checking. Nonetheless, the data suggest that fact-check results are mostly accurate. It is also important to note that while coders made independent cross-checks and were reliable in the paper, future work may consider additional approaches to confirm objectivity. For example, it is unclear if artificial intelligence (e.g., ChatGPT) can perform cross-checks, but this may be an important step to consider.

Finally, our research focused on the actions of the fact-checkers. When assessing the outcomes of fact-checking, it would be useful to obtain feedback from the consumers of fact-checking news. If a news consumer observed that a fact-checker had an incorrect fact-check, does this impact the source’s credibility and if so, what can be done to repair it? While our professional judgments resulted in a high level of agreement, news consumers—who may be untrained and whose judgments may be constrained by past exposure to the candidates, biased by repetitive exposure to the messages being checked, or influenced by the values of others with whom they interact—may not interpret fact-checking results with the same clarity or be universally forgiving if a mistakenly incorrect fact-check occurs. This suggests that the results not only need to be accurate, but they need to be credible.