Robust Multimodal Sentiment Analysis Based on Dynamic Semantic Routing and Symmetric Bidirectional Enhancement
-
Abstract
Multimodal sentiment analysis aims to recognize complex human emotions by effectively synthesizing heterogeneous information sources, including textual, acoustic, and visual modalities. However, when processing real-world data characterized by ubiquitous non-alignment and high ambiguity, existing studies still grapple with two fundamental challenges: cross-modal semantic level misalignment and unimodal dependency bias toward the textual modality. Traditional fusion methodologies often implicitly assume that distinct modalities share semantic correspondence at the exact same physical depth within neural networks, thereby neglecting the inherent heterogeneity between deep textual intents and shallow audio-visual cues, such as micro-expressions. Furthermore, prevalent models predominantly adopt a text-centric unidirectional interaction paradigm. Consequently, when the textual modality itself is defective or deceptive—as seen in sarcasm—these models lack the capability to utilize non-verbal cues to inversely correct the misleading semantics. To address these limitations, this paper proposes a robust framework named dynamic semantic routing and symmetric-enhanced modality-consistent embedding network (DySym-MCEN). Drawing inspiration from the dual cognitive mechanism of “prediction and error-correction”, our model constructs a robust feature interaction manifold through two key innovations. First, we introduce a dynamic semantic routing (DSR) mechanism. By calculating a global cross-modal affinity matrix, DSR effectively breaks the rigid assumption of fixed physical layer alignment, adaptively achieving the elastic alignment of heterogeneous modalities at a logical semantic level. Second, we propose a symmetric bidirectional enhancement (SBE) module, which constructs a mathematically symmetric dual-stream interaction architecture. Coupled with an adaptive complementary gating unit and a cycle consistency loss constraint, this approach endows the model with bidirectional error-correction capabilities in scenarios involving modal conflict. Finally, extensive experiments conducted on three mainstream benchmark datasets—MOSI, MOSEI, and CH-SIMS—demonstrated that DySym-MCEN achieved excellent performance across all evaluation metrics. Notably, the model exhibited superior robustness and generalization performance, particularly in complex scenarios characterized by a large number of unaligned samples.
-
-