Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

TL;DR - Prior scores embedded as metadata can anchor LLM-as-a-Judge systems, undermining the assumed independence of successive evaluations. The effect spans numerical scoring and categorical decisions, and common prompting mitigations do not eliminate it.

• Seven of eight evaluated models showed a statistically significant overall anchoring effect across 185,271 successful evaluations, with absolute Cohen’s (d) reaching 0.71.
• Token-level probes suggest a threshold-like response: adding anchored metadata sharply shifts score probabilities, while varying below-threshold anchor values has less impact.
• On human-labeled industry data, anchoring prevented 48% of error corrections and flipped 10.18% of correct judgments to an assigned wrong label.
• Chain-of-Thought and metadata-disregard warnings did not reduce the total effect, highlighting the need for model- and task-specific mitigation validation.