🛰️ Daily AI Frontier
‹ back to 2026-08-10

贵57.1倍的Claude Opus 4.8五项全输,赢它的不是模型,是Harness

WeChat: 机器之心 LLM Agents 2026-08-08
Representative image for 贵57.1倍的Claude Opus 4.8五项全输,赢它的不是模型,是Harness

TL;DR - AOE Tech Labs published third-party benchmark results for its Floatboat Harness, claiming that running the cheap DeepSeek-V4-Flash inside its own agent harness beats Claude Opus 4.8 on all five benchmarks at ~1/57 the blended token cost. The pitch is that the non-model half of an agent system (runtime, loop, tools, infra) is a measurable, high-leverage performance lever.

  • Single-variable setup: same DeepSeek-V4-Flash 0731 base, compared against DeepSeek's own official harness (not a bare API) in isolated sandboxes. Official harness scores 54.4/73.2/82.7/25.1/70.7; Floatboat lifts DeepSWE 54.4 → 67.25 and hits 87.80 on OpenAI's BrowseComp, above GPT-5.6 Terra (87.5) and Opus 4.8 (84.3).
  • Gains scale monotonically with task horizon: 1.9% → 9.6% → 12.6% → 19.9% → 23.6%, attributed to long-horizon loop convergence rather than prompt tricks (typically 1–3 points).
  • Proposes HLR (Harness Leverage Ratio) = harness gain ÷ model-upgrade gain; values 0.78× → 3.57×, e.g. DeepSWE's 12.85-point harness gain vs. the 3.6-point Opus 4.8 spread. Self-admittedly a new, non-standard metric sensitive to reference-model choice.
  • Architectural claim: self-built Runtime, Agent Loop, Tools, Infra plus FloatSail evolution system are required, since SDK-wrapper products can't patch a loop that drops context at step 20. Note this is vendor-published data, not independent replication.

view merged work →