Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents

TL;DR - An empirical study of six browser-observation modes (text, pixels, both) for web agents across eight site-model cells on VisualWebArena and WebArena, showing that per-task routing between modes yields little robust benefit today because routing labels are only produced where agents already succeed. It matters because it reframes "adaptive observation" from a free win into a problem gated by base agent capability.

• Observation modes are genuinely complementary — each solves tasks the others miss, they fail in structurally different ways, and the best mode reverses between task sets — but the apparent oracle gain is inflated by noise: rerunning the same mode changes 12–14% of outcomes, so a second run of an existing mode gains about as much as adding a new mode.
• The one durable win is cost, not accuracy: routing only the tasks no mode solves to the cheapest mode cuts cost 9.5–30.6% in 8 of 8 cells at unchanged success.
• Five routing policies (mode picking, when-to-spend on the strong mode, a zero-cost rule from task text, a confidence cascade, pooled cost tiers) fail to robustly beat simply fixing one well-chosen mode; the sole exception is a fragile result in the sparsest cell.
• Core obstruction: routing supervision is generated at the agent's own success rate, so weaker agents get fewer labels exactly where routing would help most; label supply and routing opportunity correlate at 0.95 across cells, implying stronger future agents could overturn the negative result.