Deep forecasting of translational impact in medical research

by   Amy PK Nelson, et al.

The value of biomedical research–a 1.7 trillion annual investment–is ultimately determined by its downstream, real-world impact. Current objective predictors of impact rest on proxy, reductive metrics of dissemination, such as paper citation rates, whose relation to real-world translation remains unquantified. Here we sought to determine the comparative predictability of future real-world translation–as indexed by inclusion in patents, guidelines or policy documents–from complex models of the abstract-level content of biomedical publications versus citations and publication meta-data alone. We develop a suite of representational and discriminative mathematical models of multi-scale publication data, quantifying predictive performance out-of-sample, ahead-of-time, across major biomedical domains, using the entire corpus of biomedical research captured by Microsoft Academic Graph from 1990 to 2019, encompassing 43.3 million papers across all domains. We show that citations are only moderately predictive of translational impact as judged by inclusion in patents, guidelines, or policy documents. By contrast, high-dimensional models of publication titles, abstracts and metadata exhibit high fidelity (AUROC > 0.9), generalise across time and thematic domain, and transfer to the task of recognising papers of Nobel Laureates. The translational impact of a paper indexed by inclusion in patents, guidelines, or policy documents can be predicted–out-of-sample and ahead-of-time–with substantially higher fidelity from complex models of its abstract-level content than from models of publication meta-data or citation metrics. We argue that content-based models of impact are superior in performance to conventional, citation-based measures, and sustain a stronger evidence-based claim to the objective measurement of translational potential.


page 22

page 25


Direct Citations between Citing Publications

This paper defines and explores the direct citations between citing publ...

Citation histories of papers: sometimes the rich get richer, sometimes they don't

We describe a simple model of how a publication's citations change over ...

The citation disadvantage of clinical research

Biomedical research encompasses diverse types of activities, from basic ...

Interdisciplinary research and technological impact

Interdisciplinary research has been considered as a solution to today's ...

Predicting the Citations of Scholarly Paper

Citation prediction of scholarly papers is of great significance in guid...

Machine Identification of High Impact Research through Text and Image Analysis

The volume of academic paper submissions and publications is growing at ...

Please sign up or login with your details

Forgot password? Click here to reset