Deep multimodal state-space fusion of endoscopic-radiomic and clinical data for survival prediction in colorectal cancer

Sep 28, 2026

Colorectal cancer requires accurate preoperative assessment and prognosis, yet endoscopic imagery and pelvic CT represent anatomy at different scales and are often analyzed separately. The study introduces HydraMamba, a multimodal selective state space framework that fuses endoscopy and CT for joint lesion segmentation, detection, and survival prediction. Using a shared state space backbone with lightweight modules, HydraMamba achieves strong lesion analysis performance (endoscopy Dice 0.856, F1 0.918; CT Dice 0.812, F1 0.888) and calibrated survival modeling on CT data (Harrell’s C 0.832, Uno’s C@1y 0.853, Brier 0.161, calibration slope ≈ 1.01).

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Motivation and clinical need for multimodal assessment

Background: advances and remaining challenges

Method: HydraMamba framework

Multimodal selective state space fusion

Joint lesion analysis and survival prediction

Results and evaluation

Lesion segmentation/detection metrics

Survival modeling and calibration

npj | digital medicine Article | Published in partnership with Seoul National University Bundang Hospital | | | | --- | --- | --- | | [https://doi.org/10.1038/s41746-025-02236-3](https://doi.org/10.1038/s41746-025-02236-3) | | | | Deep multimodal state-space fusion of endoscopic-radiomic and clinical data for survival prediction in colorectal cancer
Check for updates | | | | Ning Wang1,5, Jiajing Lin2,5, Wujin Li2,5, Yahui Lyu1,5, Yiqing Jiang3, Zhizhan Ni4, Qi Huang4, Hong Chen2,4 , Qiang Yan1 & Chenshen Huang2,4 | | | | Integrating complementary surface and cross sectional cues is central to preoperative assessment of colorectal cancer, but technically challenging because endoscopic images and pelvic CT encode anatomy at different scales. Here we present HydraMamba, a multimodal selective state space framework that fuses endoscopy and CT for joint lesion segmentation, lesion detection, and survival prediction. The model couples a shared state space backbone with two lightweight modules. Across the endoscopic dataset and the CT dataset, HydraMamba achieved state-of-the-art lesion analysis (endoscopy: Dice 0.856, F1 0.918; CT: Dice 0.812, F1 0.888) and delivered calibrated survival modeling on the CT dataset (Harrell’s C index 0.832, Uno ’s C@1y 0.853, integrated Brier score 0.161, calibration slope ≈ 1.01) . By unifying endoscopic and CT information in a single coherent architecture, HydraMamba provides an accurate and well-calibrated foundation for lesion analysis and prognostication in colorectal cancer. | | | | Colorectal cancer remains a signiﬁcant health burden, with rectal cancer as a major contributor to morbidity and mortality1. Effective management of colorectal tumors relies on accurate lesion characterization and staging, which increasingly beneﬁt from advanced medical imaging andartiﬁcial intelligence. In clinical practice, endoscopy (colonoscopy) provides high resolution visualization of mucosal lesions, whereas crosssectional imaging such as computed tomography (CT) delineates deeper anatomical context and distant spread. These modalities offer complementary information; however, traditionally they have been analyzed in isolation. There is a growing consensus that multimodal learning can unlock synergistic value from heterogeneous data in oncology2. For instance, integrating endoscopic and radiologic data has been shown to improve diagnostic accuracy and prognostic modeling. In gastric cancer, a deep learning model that fuses CT scans with endoscopy images achieved an area under the ROC curve (AUC) of 0.93 in predicting pathological stage, signiﬁcantly outperforming models using either modality alone3. Similarly, combining radiologic features with digitized pathology has yielded robust predictors of patient outcomes; a recent multimodal model integrating CT-based and histopathology-based deep features (along with clinical factors) attained high concordance indices (0 . 74) in survival prediction for head and neck cancer4–6. | At the same time, deep learning has revolutionized image-based detection and segmentation of lesions in gastroenterology and oncology. Convolutional neural networksand theirvariantsnow approach expert level performance in localizing tumors or precancerous polyps on medical images. In colonoscopy, numerous studies have demonstrated automated polyp detection and delineation systems powered by deep learning, which can assist endoscopists by reducing miss rates and providing real-time decision support7. On radiologic scans, AI algorithms excel at highlighting tumors and metastases that might be subtle to human observers8. These advances are not merely academic—they translate into practical tools that improve diagnostic precision and workﬂow efﬁciency. Nevertheless, challenges remain in generalizing these models across diverse patient populations and imaging devices7. Robust performance in lesion detection/ segmentation requires large, diverse datasets and architectures capable

FAQs

Why is combining endoscopy and pelvic CT important for colorectal cancer assessment?

Endoscopy provides high-resolution visualization of mucosal lesions, while CT supplies deeper anatomical context and distant spread. Their complementary information can improve diagnosis and prognostic modeling.

What is HydraMamba and what does it fuse?

HydraMamba is a multimodal selective state space framework that fuses endoscopic data and CT to support joint lesion segmentation, lesion detection, and survival prediction.

How does HydraMamba perform for lesion analysis and survival prediction?

It achieves state-of-the-art lesion analysis on both modalities (endoscopy Dice 0.856, F1 0.918; CT Dice 0.812, F1 0.888) and delivers calibrated survival modeling on CT data (Harrell’s C 0.832 and calibration slope about 1.01).

Sep 28, 2026

Uploaded by WPS_1786070896