Case-Specific Priors-Guided Multimodal Fusion for Prognosis Prediction in Head and Neck Cancer

Abstract

Accurate prognosis prediction for head and neck cancer typically necessitates integrating highly heterogeneous clinical and pathological evidence by sophisticated multimodal fusion. However, existing fusion techniques are usually grounded in case-generic clinical rationales about complementary synergies across modalities, underestimating case-specific variance. In this study, we propose a prognosis prediction framework that explicitly incorporates Case-specific Priors to guide Multimodal fusion (named CPM). It leverages broad medical knowledge of LLMs to characterize each case's (case-specific) intra-modal contexts and inter-modal relations, as well as case-generic multimodal rationale, as priors, enabling three-stage priors-guided multimodal fusion: (i) Intra-modal Context-guided Calibration (IntraCC) that leverages case-specific intra-modal priors to guide feature calibration within each modality; (ii) Inter-modal Relation-guided Interaction (InterRI) that leverages case-specific inter-modal priors to guide pair-wise feature interaction in modality pairs; and (iii) Multimodal Rationale-guided Aggregation (MultiRA) that leverages case-generic multimodal priors to guide final feature aggregation over all modalities. On the well-established HANCOCK challenge dataset involving four distinct medical modalities, our CPM achieved a mean AUC of 83.98% for predicting 5-year overall survival and 2-year recurrence-free survival, improving upon the baseline by 7.58%, pioneeringly demonstrating the effectiveness of LLM-derived medical priors in guiding multimodal fusion for prognosis prediction. The code is available at https://github.com/yuelin421/CPM.