ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
Ranking
Overall
90
Content
100
Popularity
66
Observed public metrics from 1 member.
Merged summary
TL;DR - ProgramDistill is a benchmark for testing whether coding agents can reconstruct web-app features by interacting with working reference applications rather than following explicit specifications. It offers scalable, automatically generated tasks with controlled difficulty and replay-based verification.
- The mine-craft-patch pipeline discovered 1,975 replay-verified behaviors across 26 applications and generated 4,063 tasks without human intervention.
- Tasks factor applications into features of varying granularity, each paired with replayable behavior executable through a gold patch.
- In full-application reconstruction, GPT-6 Astra and Claude Opus 5 achieved 49.2% and 28.8% success, respectively, on cumulative workflows.
- Partial reconstruction performance declined sharply as restoration depth increased from 1 to 8, indicating substantial difficulty in recovering longer feature sequences.
Sources (1)
ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
Public signals
Hugging Face upvotes 64
TL;DR - ProgramDistill is a benchmark for testing whether coding agents can reconstruct web-app features by interacting with working reference applications rather than following explicit specifications. It offers scalable, automatically generated tasks with controlled difficulty and replay-based verification.
- The mine-craft-patch pipeline discovered 1,975 replay-verified behaviors across 26 applications and generated 4,063 tasks without human intervention.
- Tasks factor applications into features of varying granularity, each paired with replayable behavior executable through a gold patch.
- In full-application reconstruction, GPT-6 Astra and Claude Opus 5 achieved 49.2% and 28.8% success, respectively, on cumulative workflows.
- Partial reconstruction performance declined sharply as restoration depth increased from 1 to 8, indicating substantial difficulty in recovering longer feature sequences.