Methods
Cross-attentive multimodal emotion recognition
Encoding audio, text and video, learning cross-modal dependencies and evaluating emotion predictions.
Different modalities provide different observations of the same interaction. The Canadian AI 2025 study investigates how cross-attention can refine the fusion of audio, text and video features for emotion recognition.
The method was evaluated on IEMOCAP. Its evidence concerns that experimental setting; reliability under missing, degraded or conflicting modalities remains an important broader research question.
Walk through the method
Open each step to inspect the inputs, process and outputs described by its authors.
01 Encode each modality
- Input
- Audio, text and video
- Output
- Modality-specific representations
Use modality-specific encoders to represent speech audio, text and video. Each encoder prepares a feature representation appropriate to the information carried by its input.
02 Model dependencies through cross-attention
- Input
- Encoded features
- Output
- Fused multimodal representation
Cross-attention lets the model learn relationships between modalities when fusing their features. The resulting representation combines complementary information for the emotion-recognition task.
03 Account for class imbalance
- Input
- Fused features and labels
- Output
- Emotion predictions and per-class evaluation
The study uses focal loss to address class imbalance. Its evaluation reports accuracy, precision, recall and F1, so that performance can be examined beyond a single aggregate score.
Projects
- 01
Projects
Cross-attentive emotion fusion
Studying the relationship between audio, text and video through cross-attention and feature fusion.
Research directions
- 01
Research programs
Interpretable & multimodal learning
Understanding acoustic features and examining how audio, language and visual cues interact in emotion recognition.