Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution

TL;DR - Standard vision-language embeddings fail to encode a separable, generalizable signal for cross-modal negation. A new cross-modal attention architecture improves negation detection by up to 7.03% F1 over unimodal baselines.

• Analysis covers 3,222 political video-text pairs automatically annotated with Qwen2.5-VL.
• Pretrained embeddings mainly capture modality-specific features rather than shared negation semantics.
• Visual negation depends more heavily on linguistic context, while textual negation often occurs independently.
• The approach combines cross-modal attention with JEPA2 video representations to model temporal negation.