SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
Ranking
Overall
72
Content
75
Popularity
66
Observed public metrics from 1 member.
Merged summary
TL;DR - SWE-Touch is a benchmark framework that stress-tests coding agents in shared workspaces by injecting plausible, task-conflicting "Counter-Edits" mid-task, revealing that strong autonomous agents lack awareness of a workspace being changed by a human collaborator.
- Counter-Edits are built by mining task-critical code regions from multiple repair trajectories, generating conflicting edits via a separate User Patch Generator, and injecting them with contextual user messages when the agent reaches the relevant code.
- Across nine coding models, Counter-Edits cut average resolve rate on SWE-bench Verified by 7.7 percentage points, with degradation persisting on longer-horizon SWE-Bench Pro and DeepSWE tasks.
- Trajectory analysis attributes failures to poor state awareness: agents keep conflicting code or overwrite it without re-inspecting the repository or validating changes with targeted tests.
- The authors frame change detection, edit-task reconciliation, and verification of affected behavior as the key missing capabilities for human-agent collaborative development.
Sources (1)
SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
Public signals
Hugging Face upvotes 24
TL;DR - SWE-Touch is a benchmark framework that stress-tests coding agents in shared workspaces by injecting plausible, task-conflicting "Counter-Edits" mid-task, revealing that strong autonomous agents lack awareness of a workspace being changed by a human collaborator.
- Counter-Edits are built by mining task-critical code regions from multiple repair trajectories, generating conflicting edits via a separate User Patch Generator, and injecting them with contextual user messages when the agent reaches the relevant code.
- Across nine coding models, Counter-Edits cut average resolve rate on SWE-bench Verified by 7.7 percentage points, with degradation persisting on longer-horizon SWE-Bench Pro and DeepSWE tasks.
- Trajectory analysis attributes failures to poor state awareness: agents keep conflicting code or overwrite it without re-inspecting the repository or validating changes with targeted tests.
- The authors frame change detection, edit-task reconciliation, and verification of affected behavior as the key missing capabilities for human-agent collaborative development.