GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

TL;DR - GameLogicBench evaluates coding agents on 72 Godot gameplay-logic tasks using reproducible, tick-level state assertions across varied scenarios. The best model-scaffold combination solved only 52.78% of tasks, highlighting persistent difficulty with runtime correctness and repository-scale changes.

• The benchmark includes 403 hand-designed scenarios expanded through seeded variations into 1,451 test cases.
• Evaluators accept different valid implementations while using capability-removed mutants to verify that incorrect behavior is rejected.
• Performance declines as tasks progress from isolated mechanics to multi-system interactions and repository-scale features.
• Most failed submissions run successfully but implement required behavior incorrectly; open network access also creates a risk of agents copying public code.