Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv study that measures whether tool-using agents take the same actions (not just reach the same answers) when a task is posed in different languages, across 8 models, 6 parallel benchmarks, 41 languages, and 2.38M rollouts. It matters because action traces determine cost, latency, failure modes, and auditability — and they diverge structurally across languages.
- Naive trace-similarity measurement is broken by five confounds: short traces score higher, empty traces score perfectly, unrelated traces agree by chance >50% of the time, the gap is capped by model reproducibility, and models are not self-consistent even within one language. Correcting all five makes the measured divergence larger, not smaller.
- Divergence is structural, not sampling noise: it persists under greedy decoding in every cell and stays flat as temperature rises. Normalized by their own reproducibility, four frontier models converge at 71–73% action-policy retention across languages, with model identity explaining only 5.7% of variance.
- Below roughly 10B parameters retention breaks down, and rankings among smaller models are largely an artifact of a chance floor the authors measure by permutation rather than assume.
- Agents pivot non-English tasks through English; this is causally load-bearing (confirmed via a pre-registered prediction across four models) and models will not abandon it when instructed to. Separately, a single trace-extraction regex — not the model — manufactured an apparent multilingual failure: two worked examples raised one model's measured accuracy 26x while accuracy on readable outputs barely changed.
Sources (1)
Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
TL;DR - An arXiv study that measures whether tool-using agents take the same actions (not just reach the same answers) when a task is posed in different languages, across 8 models, 6 parallel benchmarks, 41 languages, and 2.38M rollouts. It matters because action traces determine cost, latency, failure modes, and auditability — and they diverge structurally across languages.
- Naive trace-similarity measurement is broken by five confounds: short traces score higher, empty traces score perfectly, unrelated traces agree by chance >50% of the time, the gap is capped by model reproducibility, and models are not self-consistent even within one language. Correcting all five makes the measured divergence larger, not smaller.
- Divergence is structural, not sampling noise: it persists under greedy decoding in every cell and stays flat as temperature rises. Normalized by their own reproducibility, four frontier models converge at 71–73% action-policy retention across languages, with model identity explaining only 5.7% of variance.
- Below roughly 10B parameters retention breaks down, and rankings among smaller models are largely an artifact of a chance floor the authors measure by permutation rather than assume.
- Agents pivot non-English tasks through English; this is causally load-bearing (confirmed via a pre-registered prediction across four models) and models will not abandon it when instructed to. Separately, a single trace-extraction regex — not the model — manufactured an apparent multilingual failure: two worked examples raised one model's measured accuracy 26x while accuracy on readable outputs barely changed.