The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluation

TL;DR - This paper proposes the “convention gap,” a metric for quantifying how much cooperative success depends on implicit communication beyond literal messages. Across roughly 101,000 Hanabi actions, it better distinguished human-like convention use than game score and may help predict AI effectiveness with human partners.

• The convention gap compares failure probability predicted from literal communication with observed failure rates; Hanabi’s deterministic hint rules make it exactly computable.
• The gap was +26.2 percentage points for human-human pairs, −0.7 for AI-AI pairs, and +16.4 for human-AI pairs.
• Human implicit coordination was strongest for cards receiving no hints, where the human-pair gap reached approximately +46 points.
• In human-AI games, partners eliciting larger convention gaps produced fewer human failures despite providing similar literal information.