Multimodal Deep Learning System for Early Detection of Chronic Diseases using Medical Images + EHR Data

Abstract

Early detection of chronic diseases is essential for reducing long-term health complications and improving patient survival outcomes. Traditional diagnostic systems rely heavily on single-modality data, such as medical imaging or clinical records, which often fail to capture the multidimensional nature of chronic disease progression. This research presents a multimodal deep learning framework that integrates medical images with Electronic Health Records (EHR) to enhance early disease prediction. The proposed system utilizes a Convolutional Neural Network (CNN) for extracting structural and morphological patterns from imaging modalities such as MRI, CT, X-ray, and retinal fundus images. In parallel, an LSTM/Transformer-based encoder processes EHR variables, including laboratory values, comorbidities, vitals, and demographic information. The latent representations from both modalities are fused using an intermediate multimodal fusion strategy to generate a unified patient-level diagnostic prediction. Experimental results show that the proposed multimodal model significantly outperforms image-only and EHR-only models, achieving an overall accuracy of 92.8%, an F1-score of 91.0%, and an AUC of 0.96. Per-class analysis demonstrates substantial improvement in detecting early-stage conditions such as diabetic retinopathy, chronic kidney disease, cardiovascular diseases, and COPD. The inclusion of Grad-CAM and SHAP-based interpretability analyses further enhances the clinical reliability of the model. Overall, the findings confirm that integrating imaging and EHR data through multimodal deep learning provides a more comprehensive and accurate approach for early chronic disease detection and has strong potential for real clinical implementation.