LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - LangChoiceBench is a project-level code-generation benchmark that measures how strongly LLMs default to Python, and finds the bias is pervasive, poorly justified, and often inconsistent with the models' own recommendations. It matters because language choice shapes real project outcomes, and an unexamined Python default silently degrades code quality in domains where Python is a bad fit.
- Benchmark spans 28 projects across seven software areas where Python is often a poor default, scoring Python preference, recommendation-implementation consistency, and language diversity.
- Across 25 diverse LLMs, Python is heavily over-selected and recommendation-implementation consistency is low; smaller open-weight models show stronger Python bias and less language diversity.
- Analysis of 9,826 reasoning traces shows most Python choices are automatic or ease-driven rather than grounded in project requirements.
- Identifies "phantom evidence" — models fabricating contextual justification for Python — plus cases where emitted code contradicts the language chosen in the model's own reasoning.
Sources (1)
LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
TL;DR - LangChoiceBench is a project-level code-generation benchmark that measures how strongly LLMs default to Python, and finds the bias is pervasive, poorly justified, and often inconsistent with the models' own recommendations. It matters because language choice shapes real project outcomes, and an unexamined Python default silently degrades code quality in domains where Python is a bad fit.
- Benchmark spans 28 projects across seven software areas where Python is often a poor default, scoring Python preference, recommendation-implementation consistency, and language diversity.
- Across 25 diverse LLMs, Python is heavily over-selected and recommendation-implementation consistency is low; smaller open-weight models show stronger Python bias and less language diversity.
- Analysis of 9,826 reasoning traces shows most Python choices are automatic or ease-driven rather than grounded in project requirements.
- Identifies "phantom evidence" — models fabricating contextual justification for Python — plus cases where emitted code contradicts the language chosen in the model's own reasoning.