ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step

TL;DR - ScrambleToolBench is an interactive terminal benchmark that strips semantic tool schemas away, forcing agents to discover hidden tool behavior purely through trial-and-error interaction. It matters because it shows frontier LLM agents can make initial discoveries but fail to adapt deductively when the environment shifts.

• Removes semantic cues and uses a continuous task curriculum so agents cannot lean on prior knowledge; behavior must be inferred from interaction alone.
• Adds dynamic stressors — mapping drift, stochastic action failures, and temporal execution windows — to test hypothesis revision as the environment changes.
• Under structural change, agents show belief inertia or revert to exhaustive search instead of deductive strategies like cycle tracing.
• Scaling test-time reasoning amplifies brute-force search rather than enabling deduction; persistent memory cuts compounding errors but still doesn't yield efficient structural inference.