Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges

by   Hiba Ahsan, et al.

Unstructured Electronic Health Record (EHR) data often contains critical information complementary to imaging data that would inform radiologists' diagnoses. However, time constraints and the large volume of notes frequently associated with individual patients renders manual perusal of such data to identify relevant evidence infeasible in practice. Modern Large Language Models (LLMs) provide a flexible means of interacting with unstructured EHR data, and may provide a mechanism to efficiently retrieve and summarize unstructured evidence relevant to a given query. In this work, we propose and evaluate an LLM (Flan-T5 XXL) for this purpose. Specifically, in a zero-shot setting we task the LLM to infer whether a patient has or is at risk of a particular condition; if so, we prompt the model to summarize the supporting evidence. Enlisting radiologists for manual evaluation, we find that this LLM-based approach provides outputs consistently preferred to a standard information retrieval baseline, but we also highlight the key outstanding challenge: LLMs are prone to hallucinating evidence. However, we provide results indicating that model confidence in outputs might indicate when LLMs are hallucinating, potentially providing a means to address this.


page 1

page 2

page 3

page 4


Conceptualizing Machine Learning for Dynamic Information Retrieval of Electronic Health Record Notes

The large amount of time clinicians spend sifting through patient notes ...

Leveraging Contextual Relatedness to Identify Suicide Documentation in Clinical Notes through Zero Shot Learning

Identifying suicidality including suicidal ideation, attempts, and risk ...

Large Language Models are Zero-Shot Clinical Information Extractors

We show that large language models, such as GPT-3, perform well at zero-...

Extending Cross-Modal Retrieval with Interactive Learning to Improve Image Retrieval Performance in Forensics

Nowadays, one of the critical challenges in forensics is analyzing the e...

Prediction Using Note Text: Synthetic Feature Creation with word2vec

word2vec affords a simple yet powerful approach of extracting quantitati...

RELIC: Retrieving Evidence for Literary Claims

Humanities scholars commonly provide evidence for claims that they make ...

Please sign up or login with your details

Forgot password? Click here to reset