Crowd-based Multi-Predicate Screening of Papers in Literature Reviews

by   Evgeny Krivosheev, et al.

Systematic literature reviews (SLRs) are one of the most common and useful form of scientific research and publication. Tens of thousands of SLRs are published each year, and this rate is growing across all fields of science. Performing an accurate, complete and unbiased SLR is however a difficult and expensive endeavor. This is true in general for all phases of a literature review, and in particular for the paper screening phase, where authors lter a set of potentially in-scope papers based on a number of exclusion criteria. To address the problem, in recent years the research community has began to explore the use of the crowd to allow for a faster, accurate, cheaper and unbiased screening of papers. Initial results show that crowdsourcing can be effective, even for relatively complex reviews. In this paper we derive and analyze a set of strategies for crowd-based screening, and show that an adaptive strategy, that continuously re-assesses the statistical properties of the problem to minimize the number of votes needed to take decisions for each paper, significantly outperforms a number of non-adaptive approaches in terms of cost and accuracy. We validate both applicability and results of the approach through a set of crowdsourcing experiments, and discuss properties of the problem and algorithms that we believe to be generally of interest for classification problems where items are classified via a series of successive tests (as it often happens in medicine).


page 1

page 2

page 3

page 4


CrowdRev: A platform for Crowd-based Screening of Literature Reviews

In this paper and demo we present a crowd and crowd+AI based system, cal...

Crowdsourcing Paper Screening in Systematic Literature Reviews

Literature reviews allow scientists to stand on the shoulders of giants,...

Crowd-Machine Collaboration for Item Screening

In this paper we describe how crowd and machine classifier can be effici...

CRUISE-Screening: Living Literature Reviews Toolbox

Keeping up with research and finding related work is still a time-consum...

A Review on the Applications of Crowdsourcing in Human Pathology

The advent of the digital pathology has introduced new avenues of diagno...

Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study

In the process of Systematic Literature Review, citation screening is es...

Automated Paper Screening for Clinical Reviews Using Large Language Models

Objective: To assess the performance of the OpenAI GPT API in accurately...

Please sign up or login with your details

Forgot password? Click here to reset