Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?

by   Mohammed Saeed, et al.

Fact-checking is one of the effective solutions in fighting online misinformation. However, traditional fact-checking is a process requiring scarce expert human resources, and thus does not scale well on social media because of the continuous flow of new content to be checked. Methods based on crowdsourcing have been proposed to tackle this challenge, as they can scale with a smaller cost, but, while they have shown to be feasible, have always been studied in controlled environments. In this work, we study the first large-scale effort of crowdsourced fact-checking deployed in practice, started by Twitter with the Birdwatch program. Our analysis shows that crowdsourcing may be an effective fact-checking strategy in some settings, even comparable to results obtained by human experts, but does not lead to consistent, actionable results in others. We processed 11.9k tweets verified by the Birdwatch program and report empirical evidence of i) differences in how the crowd and experts select content to be fact-checked, ii) how the crowd and the experts retrieve different resources to fact-check, and iii) the edge the crowd shows in fact-checking scalability and efficiency as compared to expert checkers.


page 1

page 2

page 3

page 4


The Role of the Crowd in Countering Misinformation: A Case Study of the COVID-19 Infodemic

Fact checking by professionals is viewed as a vital defense in the fight...

Designing and Evaluating Presentation Strategies for Fact-Checked Content

With the rapid growth of online misinformation, it is crucial to have re...

Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria

Misinformation about critical issues such as climate change and vaccine ...

CoVERT: A Corpus of Fact-checked Biomedical COVID-19 Tweets

Over the course of the COVID-19 pandemic, large volumes of biomedical in...

Human and technological infrastructures of fact-checking

Increasing demands for fact-checking has led to a growing interest in de...

Does AI-Assisted Fact-Checking Disproportionately Benefit Majority Groups Online?

In recent years, algorithms have been incorporated into fact-checking pi...

Fact-Checking at Scale with DimensionRank

The most important problem that has emerged after twenty years of popula...

Please sign up or login with your details

Forgot password? Click here to reset