FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration

TL;DR - FDE-Bench evaluates LLM agents on 136 reproducible Docker, Compose, and Kubernetes deployment tasks using programmatic checks rather than LLM judges. Leading models solve 52.9–75.0% of tasks, with service readiness emerging as the largest failure stage.

• The benchmark covers greenfield deployment and diagnosis-and-repair, grading build success, readiness, behavior, and specification conformance in pristine environments.
• Seven models from four providers use the same four-tool scaffold; repair tasks average 30.7 percentage points higher resolution than disjoint greenfield tasks.
• Shortcut-resistant release gates reject do-nothing, specification-copying, and generic-stub solutions; three additional adversarial strategies solve none of the 135 applicable tasks.
• In a 25-task case study, an engineer directing Claude-Sonnet-5 achieves 92% resolution versus 72% for the autonomous baseline.