🛰️ Daily AI Frontier
‹ back to 2026-09-21

GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

Research LLM Agents

Ranking

Overall 82
Content 95
Popularity N/A

No observed public metrics; popularity remains neutral/archived.

Representative image for GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

Merged summary

TL;DR - GameLogicBench evaluates coding agents on 72 Godot gameplay-logic tasks using reproducible, tick-level state assertions across varied scenarios. The best model-scaffold combination solved only 52.78% of tasks, highlighting persistent difficulty with runtime correctness and repository-scale changes.

  • The benchmark includes 403 hand-designed scenarios expanded through seeded variations into 1,451 test cases.
  • Evaluators accept different valid implementations while using capability-removed mutants to verify that incorrect behavior is rejected.
  • Performance declines as tasks progress from isolated mechanics to multi-system interactions and repository-scale features.
  • Most failed submissions run successfully but implement required behavior incorrectly; open network access also creates a risk of agents copying public code.

Sources (1)

GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

arXiv cs.SE Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Yanghai Wang, Zixuan Dong, Yifan Yao, Qianqian Xie, Letian Zhu, Jiaheng Liu 2026-09-18 arXiv:2609.21562
Public signals N/A
Providers: Hugging Face · N/A OpenAlex · N/A Publisher · N/A Semantic Scholar · N/A X · N/A Fetched 2026-09-26 14:14:53.383325 UTC

TL;DR - GameLogicBench evaluates coding agents on 72 Godot gameplay-logic tasks using reproducible, tick-level state assertions across varied scenarios. The best model-scaffold combination solved only 52.78% of tasks, highlighting persistent difficulty with runtime correctness and repository-scale changes.

  • The benchmark includes 403 hand-designed scenarios expanded through seeded variations into 1,451 test cases.
  • Evaluators accept different valid implementations while using capability-removed mutants to verify that incorrect behavior is rejected.
  • Performance declines as tasks progress from isolated mechanics to multi-system interactions and repository-scale features.
  • Most failed submissions run successfully but implement required behavior incorrectly; open network access also creates a risk of agents copying public code.
item →