Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
Ranking
Overall
79
Content
95
Popularity
42
Observed public metrics from 1 member.
Merged summary
TL;DR - Prior scores embedded as metadata can anchor LLM-as-a-Judge systems, undermining the assumed independence of successive evaluations. The effect spans numerical scoring and categorical decisions, and common prompting mitigations do not eliminate it.
- Seven of eight evaluated models showed a statistically significant overall anchoring effect across 185,271 successful evaluations, with absolute Cohen’s (d) reaching 0.71.
- Token-level probes suggest a threshold-like response: adding anchored metadata sharply shifts score probabilities, while varying below-threshold anchor values has less impact.
- On human-labeled industry data, anchoring prevented 48% of error corrections and flipped 10.18% of correct judgments to an assigned wrong label.
- Chain-of-Thought and metadata-disregard warnings did not reduce the total effect, highlighting the need for model- and task-specific mitigation validation.
Sources (1)
Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - Prior scores embedded as metadata can anchor LLM-as-a-Judge systems, undermining the assumed independence of successive evaluations. The effect spans numerical scoring and categorical decisions, and common prompting mitigations do not eliminate it.
- Seven of eight evaluated models showed a statistically significant overall anchoring effect across 185,271 successful evaluations, with absolute Cohen’s (d) reaching 0.71.
- Token-level probes suggest a threshold-like response: adding anchored metadata sharply shifts score probabilities, while varying below-threshold anchor values has less impact.
- On human-labeled industry data, anchoring prevented 48% of error corrections and flipped 10.18% of correct judgments to an assigned wrong label.
- Chain-of-Thought and metadata-disregard warnings did not reduce the total effect, highlighting the need for model- and task-specific mitigation validation.