ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

TL;DR - ProgramDistill is a benchmark for testing whether coding agents can reconstruct web-app features by interacting with working reference applications rather than following explicit specifications. It offers scalable, automatically generated tasks with controlled difficulty and replay-based verification.

• The mine-craft-patch pipeline discovered 1,975 replay-verified behaviors across 26 applications and generated 4,063 tasks without human intervention.
• Tasks factor applications into features of varying granularity, each paired with replayable behavior executable through a gold patch.
• In full-application reconstruction, GPT-6 Astra and Claude Opus 5 achieved 49.2% and 28.8% success, respectively, on cumulative workflows.
• Partial reconstruction performance declined sharply as restoration depth increased from 1 to 8, indicating substantial difficulty in recovering longer feature sequences.