On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
Merged summary
TL;DR - This arXiv paper introduces SimVLA, a simplified transferable adversarial attack pipeline against Vision-Language Pre-training Models (VLPMs) that outperforms more complex prior methods while using far less time and memory. It matters because it challenges the assumption that sophisticated multi-stage attacks are necessary, showing domain knowledge beats brute-force complexity.
- Identifies three overlooked issues in existing transferable VLPM attacks stemming from inappropriate cross-modal interactions and excessive operations.
- Proposes SimVLA, replacing complex loss functions and multi-stage designs with a simpler pipeline centered on cross-modal word identification.
- On Flickr30k text-image retrieval, SimVLA beats the SOTA baseline in R@1 transferability by 8.01%–14.71%, using ~35.73% of the time and ~46.26% of max VRAM.
- Validated across four datasets and three downstream tasks; argues intricate operations can be harmful, and offers SimVLA as a backbone for future work (code released).
Sources (1)
On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
TL;DR - This arXiv paper introduces SimVLA, a simplified transferable adversarial attack pipeline against Vision-Language Pre-training Models (VLPMs) that outperforms more complex prior methods while using far less time and memory. It matters because it challenges the assumption that sophisticated multi-stage attacks are necessary, showing domain knowledge beats brute-force complexity.
- Identifies three overlooked issues in existing transferable VLPM attacks stemming from inappropriate cross-modal interactions and excessive operations.
- Proposes SimVLA, replacing complex loss functions and multi-stage designs with a simpler pipeline centered on cross-modal word identification.
- On Flickr30k text-image retrieval, SimVLA beats the SOTA baseline in R@1 transferability by 8.01%–14.71%, using ~35.73% of the time and ~46.26% of max VRAM.
- Validated across four datasets and three downstream tasks; argues intricate operations can be harmful, and offers SimVLA as a backbone for future work (code released).