Bidirectional Cross-Modal Attention Gating for Multimodal Estrogen Receptor Status Classification in Breast Cancer

Author First name, Last name, Institution

• Mohamed T. Azam, J.B. Speed School of Engineering
• Walid Mohamed, J.B. Speed School of Engineering
• Khadiga Ali, Faculty of Medicine
• Ahmed Aboudessouki, J.B. Speed School of Engineering
• Hossam Magdy Balaha, J.B. Speed School of Engineering
• Moumen El-Melegy, J.B. Speed School of Engineering
• Asem Ali, J.B. Speed School of Engineering
• Mohammed Ghazal, Abu Dhabi University
• Ashraf Khalil, Zayed University, Abu Dhabi CampusFollow
• Dibson Gondim, University of Louisville School of Medicine
• Ayman El-Baz, J.B. Speed School of Engineering

Document Type

Conference Proceeding

Source of Publication

Lecture Notes in Computer Science

Publication Date

8-2-2026

Abstract

Precise estrogen receptor (ER) status assessment is fundamental to breast cancer treatment; however, current computational pathology methods rely solely on whole slide images (WSIs), overlooking valuable diagnostic information contained in pathology reports. In this work, we propose a bidirectional cross-modal attention gating framework that combines visual and textual modalities for robust automated ER classification. The framework employs attention-based Multiple Instance Learning (MIL) to capture morphological patterns and ER expression heterogeneity from WSIs, while a biomedical transformer encoder extracts contextualized semantic knowledge from pathology reports. A bidirectional attention module enables each modality to selectively query the other, providing diagnostic context to visual features and morphological grounding to textual features. An adaptive gating network then learns to dynamically balance these complementary information streams. Evaluated using 5-fold stratified cross-validation, the method achieves 92.11% balanced accuracy, outperforming unimodal baselines by 5.54% (p<0.01) and attaining consistent gains of 1.25–6.25% across diverse classifiers. The system reaches a mean precision of 94.17% and a recall of 91.79%, exceeding naive fusion strategies by 6.41% (p<0.01), indicating that learned cross-modal interactions capture genuine complementary information. These findings indicate that multimodal integration can improve automated ER biomarker assessment and move toward reliable clinical deployment in digital pathology.

DOI Link

10.1007/978-3-032-31930-2_11

ISBN

[9783032319296]

ISSN

0302-9743

Publisher

Springer Nature Switzerland

Volume

16825 LNCS

First Page

158

Last Page

171

Disciplines

Computer Sciences

Keywords

Classification, Cross-Attention, ER Biomarker, Histopathology, Multimodal Fusion, Multiple Instance Learning (MIL)

Scopus ID

105047052418

Recommended Citation

Azam, Mohamed T.; Mohamed, Walid; Ali, Khadiga; Aboudessouki, Ahmed; Balaha, Hossam Magdy; El-Melegy, Moumen; Ali, Asem; Ghazal, Mohammed; Khalil, Ashraf; Gondim, Dibson; and El-Baz, Ayman, "Bidirectional Cross-Modal Attention Gating for Multimodal Estrogen Receptor Status Classification in Breast Cancer" (2026). All Works. 8038.
https://zuscholars.zu.ac.ae/works/8038

Indexed in Scopus

yes

Open Access

no