Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies

by   Gabriele Pergola, et al.

Biomedical question-answering (QA) has gained increased attention for its capability to provide users with high-quality information from a vast scientific literature. Although an increasing number of biomedical QA datasets has been recently made available, those resources are still rather limited and expensive to produce. Transfer learning via pre-trained language models (LMs) has been shown as a promising approach to leverage existing general-purpose knowledge. However, finetuning these large models can be costly and time consuming, often yielding limited benefits when adapting to specific themes of specialised domains, such as the COVID-19 literature. To bootstrap further their domain adaptation, we propose a simple yet unexplored approach, which we call biomedical entity-aware masking (BEM). We encourage masked language models to learn entity-centric knowledge based on the pivotal entities characterizing the domain at hand, and employ those entities to drive the LM fine-tuning. The resulting strategy is a downstream process applicable to a wide variety of masked LMs, not requiring additional memory or components in the neural architectures. Experimental results show performance on par with state-of-the-art models on several biomedical QA datasets.


page 1

page 2

page 3

page 4


Neural Domain Adaptation for Biomedical Question Answering

Factoid question answering (QA) has recently benefited from the developm...

Transferability of Natural Language Inference to Biomedical Question Answering

Biomedical question answering (QA) is a challenging problem due to the s...

Contextual embedding and model weighting by fusing domain knowledge on Biomedical Question Answering

Biomedical Question Answering aims to obtain an answer to the given ques...

Large Language Models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery

Inferring over and extracting information from Large Language Models (LL...

LED down the rabbit hole: exploring the potential of global attention for biomedical multi-document summarisation

In this paper we report on our submission to the Multidocument Summarisa...

A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models

Pre-trained language models (PLMs) cannot well recall rich factual knowl...

Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification

Recent advances in large language models (LLMs) have shown impressive ab...

Please sign up or login with your details

Forgot password? Click here to reset