Multimodal Machine Learning in Precision Health

by   Adrienne Kline, et al.

As machine learning and artificial intelligence are more frequently being leveraged to tackle problems in the health sector, there has been increased interest in utilizing them in clinical decision-support. This has historically been the case in single modal data such as electronic health record data. Attempts to improve prediction and resemble the multimodal nature of clinical expert decision-making this has been met in the computational field of machine learning by a fusion of disparate data. This review was conducted to summarize this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) extension for Scoping Reviews to characterize multi-modal data fusion in health. We used a combination of content analysis and literature searches to establish search strings and databases of PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 125 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. However, there exist a wide breadth of current applications. The most common form of information fusion was early fusion. Notably, there was an improvement in predictive performance performing heterogeneous data fusion. Lacking from the papers were clear clinical deployment strategies and pursuit of FDA-approved tools. These findings provide a map of the current literature on multimodal data fusion as applied to health diagnosis/prognosis problems. Multi-modal machine learning, while more robust in its estimations over unimodal methods, has drawbacks in its scalability and the time-consuming nature of information concatenation.


page 6

page 8

page 11

page 16


Artificial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data

Healthcare data are inherently multimodal, including electronic health r...

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

The rapid development of diagnostic technologies in healthcare is leadin...

Multimodal Learning with Channel-Mixing and Masked Autoencoder on Facial Action Unit Detection

Recent studies utilizing multi-modal data aimed at building a robust mod...

A Scoping Review of Publicly Available Language Tasks in Clinical Natural Language Processing

Objective: to provide a scoping review of papers on clinical natural lan...

Predicting Depression Severity by Multi-Modal Feature Engineering and Fusion

We present our preliminary work to determine if patient's vocal acoustic...

A Machine learning approach for rapid disaster response based on multi-modal data. The case of housing shelter needs

Along with climate change, more frequent extreme events, such as floodin...

Network Module Detection from Multi-Modal Node Features with a Greedy Decision Forest for Actionable Explainable AI

Network-based algorithms are used in most domains of research and indust...

Please sign up or login with your details

Forgot password? Click here to reset