Deep Multimodal Integration of Whole Slide Images and Pathology Reports for Precise Estrogen Receptor Classification in Breast Cancer

Author First name, Last name, Institution

• Mohamed T. Azam, J.B. Speed School of Engineering
• Walid Mohamed, J.B. Speed School of Engineering
• Khadiga Ali, Faculty of Medicine
• Ahmed Aboudessouki, J.B. Speed School of Engineering
• Hossam Magdy Balaha, J.B. Speed School of Engineering
• Moumen El-Melegy, J.B. Speed School of Engineering
• Asem Ali, J.B. Speed School of Engineering
• Ali Mahmoud, J.B. Speed School of Engineering
• Fatma Taher, Zayed University
• Mohammed A. Ghazal, Abu Dhabi University
• Dibson Gondim, University of Louisville School of Medicine
• Ayman El-Baz, J.B. Speed School of Engineering

Document Type

Conference Proceeding

Source of Publication

Proceedings International Symposium on Biomedical Imaging

Publication Date

4-8-2026

Abstract

Accurate determination of estrogen receptor (ER) status is critical for breast cancer prognosis; however, conventional assessments remain semi-quantitative and prone to inter-observer variability. This work proposes a dual-modality deep learning framework that integrates pathology reports and immunohistochemical whole slide images (WSIs) for automated ER classification. The framework consists of two parallel branches: an attention-based Multiple Instance Learning (MIL) module that captures morphological patterns and staining distributions from WSIs, and a transformer-based textual encoder that extracts semantic embeddings from pathology reports. These diagnostic and staining information are complementary visual and textual features that are fused into a unified multimodal feature space, enabling the model to exploit cross-modal relationships between tissue morphology and textual diagnostic knowledge. Evaluation using stratified 5 -fold cross-validation shows that multimodal fusion consistently outperforms unimodal baselines across all utilized classifiers. CatBoost achieved the highest performance, with 92.5% accuracy, 92.0% precision, 95.0% recall, and an F1-score of 93.5%, and substantially outperformed single-modality results. The proposed fusion approach improved performance across all models, with ensemble methods such as Random Forest and XGBoost reaching 90.0% accuracy, while simpler classifiers, including Decision Tree and Naive Bayes, improved by 2.5%-5.0%. These results validate the strength of the proposed multimodal integration approach and underscore the synergistic benefits of visual and textual information for precise and dependable ER classification in clinical practice.

DOI Link

10.1109/isbi61048.2026.11515817

ISBN

[9798331577636]

ISSN

1945-7928

Publisher

IEEE

Volume

2026-April

Disciplines

Computer Sciences | Medicine and Health Sciences

Keywords

Breast cancer, Estrogen receptor, Multimodal deep learning, Pathology reports, Whole slide images

Scopus ID

105041608061

Recommended Citation

Azam, Mohamed T.; Mohamed, Walid; Ali, Khadiga; Aboudessouki, Ahmed; Balaha, Hossam Magdy; El-Melegy, Moumen; Ali, Asem; Mahmoud, Ali; Taher, Fatma; Ghazal, Mohammed A.; Gondim, Dibson; and El-Baz, Ayman, "Deep Multimodal Integration of Whole Slide Images and Pathology Reports for Precise Estrogen Receptor Classification in Breast Cancer" (2026). All Works. 8199.
https://zuscholars.zu.ac.ae/works/8199

Indexed in Scopus

yes

Open Access

no