CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition
Merged summary
TL;DR — CF-Net is a deep multimodal network for recognizing ambivalence and hesitancy (AH) in unconstrained video, fusing visual, audio, and text streams to detect subtle cross-modal incongruence. It matters as a challenge entry (ABAW 11th, ECCV 2026) tackling inherently ambiguous affective signals.
- Encodes three modalities with frozen backbones (SigLIP2 for vision, HuBERT for audio, DistilBERT for transcripts), plus per-speaker feature normalization to reduce identity leakage.
- Introduces a ConflictFusion module that explicitly computes pairwise cross-modal incongruence, targeting the signal that distinguishes AH from prototypical affect.
- Training combines certainty-weighted focal loss, manifold mixup, and modality dropout, with an auxiliary certainty-regression head using ambiguity annotations to stabilize learning on borderline samples.
- Reports Macro F1 of 0.7155 on the BAH validation set and 0.7364 (AP = 0.7492) on the private challenge test set.
Sources (1)
CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition
TL;DR — CF-Net is a deep multimodal network for recognizing ambivalence and hesitancy (AH) in unconstrained video, fusing visual, audio, and text streams to detect subtle cross-modal incongruence. It matters as a challenge entry (ABAW 11th, ECCV 2026) tackling inherently ambiguous affective signals.
- Encodes three modalities with frozen backbones (SigLIP2 for vision, HuBERT for audio, DistilBERT for transcripts), plus per-speaker feature normalization to reduce identity leakage.
- Introduces a ConflictFusion module that explicitly computes pairwise cross-modal incongruence, targeting the signal that distinguishes AH from prototypical affect.
- Training combines certainty-weighted focal loss, manifold mixup, and modality dropout, with an auxiliary certainty-regression head using ambiguity annotations to stabilize learning on borderline samples.
- Reports Macro F1 of 0.7155 on the BAH validation set and 0.7364 (AP = 0.7492) on the private challenge test set.