SHAP-Weighted Cross-Modal Fusion Matches Early Fusion on Emotion Recognition, Exceeds Late Fusion

A new arXiv study revisits XAI-guided adaptive fusion (XGAF), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights come from TreeSHAP attribution magnitudes. The key finding: when experts have unequal feature dimensionalities, mean-abs and median-abs SHAP reductions suppress high-dimensional cross-modal experts, while sum-abs reduction preserves total attribution mass. On MELD 7-class emotion recognition, sum-abs XGAF nearly matches early fusion across three face-sequence aggregators; the Transformer variant reaches 0.5983 weighted F1 versus 0.6018 for early fusion and 0.4598 for probability-average late fusion. McNemar testing shows no significant difference between sum-abs XGAF and early fusion on MELD (p=1.000), while XGAF remains significantly better than late fusion (p<0.0001). On CMU-MOSEI 3-class sentiment recognition, sum-abs XGAF reaches 0.6519 weighted F1, slightly exceeding early fusion (0.6485) and late fusion (0.5696). Ablation studies show the main gain comes from adding cross-modal experts, especially the trimodal expert, rather than from complex per-sample routing. Diagnostics confirm mean-abs and median-abs weights are nearly uniform, while sum-abs weights concentrate on the trimodal expert.