Pre-training transformer-based framework on large-scale pediatric claims data for downstream population-specific tasks

by   Xianlong Zeng, et al.

The adoption of electronic health records (EHR) has become universal during the past decade, which has afforded in-depth data-based research. By learning from the large amount of healthcare data, various data-driven models have been built to predict future events for different medical tasks, such as auto diagnosis and heart-attack prediction. Although EHR is abundant, the population that satisfies specific criteria for learning population-specific tasks is scarce, making it challenging to train data-hungry deep learning models. This study presents the Claim Pre-Training (Claim-PT) framework, a generic pre-training model that first trains on the entire pediatric claims dataset, followed by a discriminative fine-tuning on each population-specific task. The semantic meaning of medical events can be captured in the pre-training stage, and the effective knowledge transfer is completed through the task-aware fine-tuning stage. The fine-tuning process requires minimal parameter modification without changing the model architecture, which mitigates the data scarcity issue and helps train the deep learning model adequately on small patient cohorts. We conducted experiments on a real-world claims dataset with more than one million patient records. Experimental results on two downstream tasks demonstrated the effectiveness of our method: our general task-agnostic pre-training framework outperformed tailored task-specific models, achieving more than 10% higher in model performance as compared to baselines. In addition, our framework showed a great generalizability potential to transfer learned knowledge from one institution to another, paving the way for future healthcare model pre-training across institutions.


page 1

page 4

page 7

page 8


Unsupervised Pre-Training on Patient Population Graphs for Patient-Level Predictions

Pre-training has shown success in different areas of machine learning, s...

Medical SANSformers: Training self-supervised transformers without attention for Electronic Medical Records

We leverage deep sequential models to tackle the problem of predicting h...

Train No Evil: Selective Masking for Task-guided Pre-training

Recently, pre-trained language models mostly follow the pre-training-the...

Deep Claim: Payer Response Prediction from Claims Data with Deep Learning

Each year, almost 10 plans). With the cost to recover these denials and ...

Predicting User-specific Future Activities using LSTM-based Multi-label Classification

User-specific future activity prediction in the healthcare domain based ...

Data Transfer Approaches to Improve Seq-to-Seq Retrosynthesis

Retrosynthesis is a problem to infer reactant compounds to synthesize a ...

ProtoKD: Learning from Extremely Scarce Data for Parasite Ova Recognition

Developing reliable computational frameworks for early parasite detectio...

Please sign up or login with your details

Forgot password? Click here to reset