SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data

05/04/2023
by   Mael Jullien, et al.
17

This paper describes the results of SemEval 2023 task 7 – Multi-Evidence Natural Language Inference for Clinical Trial Data (NLI4CT) – consisting of 2 tasks, a Natural Language Inference (NLI) task, and an evidence selection task on clinical trial data. The proposed challenges require multi-hop biomedical and numerical reasoning, which are of significant importance to the development of systems capable of large-scale interpretation and retrieval of medical evidence, to provide personalized evidence-based care. Task 1, the entailment task, received 643 submissions from 40 participants, and Task 2, the evidence selection task, received 364 submissions from 23 participants. The tasks are challenging, with the majority of submitted systems failing to significantly outperform the majority class baseline on the entailment task, and we observe significantly better performance on the evidence selection task than on the entailment task. Increasing the number of model parameters leads to a direct increase in performance, far more significant than the effect of biomedical pre-training. Future works could explore the limitations of large models for generalization and numerical inference, and investigate methods to augment clinical datasets to allow for more rigorous testing and to facilitate fine-tuning. We envisage that the dataset, models, and results of this task will be useful to the biomedical NLI and evidence retrieval communities. The dataset, competition leaderboard, and website are publicly available.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/25/2023

Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports

With the increasing number of clinical trial reports generated every day...
research
05/05/2023

NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports

How can we interpret and retrieve medical evidence to support clinical d...
research
06/02/2023

THiFLY Research at SemEval-2023 Task 7: A Multi-granularity System for CTR-based Textual Entailment and Evidence Retrieval

The NLI4CT task aims to entail hypotheses based on Clinical Trial Report...
research
05/08/2020

Evidence Inference 2.0: More Data, Better Models

How do we most effectively treat a disease or condition? Ideally, we cou...
research
05/02/2018

Hypothesis Only Baselines in Natural Language Inference

We propose a hypothesis only baseline for diagnosing Natural Language In...
research
10/12/2020

Predicting Clinical Trial Results by Implicit Evidence Integration

Clinical trials provide essential guidance for practicing Evidence-Based...
research
03/13/2019

Adversarial attacks against Fact Extraction and VERification

This paper describes a baseline for the second iteration of the Fact Ext...

Please sign up or login with your details

Forgot password? Click here to reset