How Much Does a Reasoning Summary Reveal? An Observability Ladder for Large Language Models

TL;DR - An arXiv study introduces an "observability ladder" that fixes a completed LLM run and varies only what a reader sees — response, self-written reasoning summary, full trace, or internal signals, with and without the prompt — to test how well correctness can be judged. It matters because it shows reasoning summaries are far weaker monitoring surfaces than full traces once the prompt is known, undercutting monitorability claims that don't specify the reader.

• Across three benchmarks and five open-weight Qwen3 / gpt-oss models, matched linear correctness predictors gave summaries mean AUROC 0.774 vs 0.813 for traces without the prompt (+0.156 over response alone); with the prompt visible, the summary's gain collapsed to +0.019 while the trace still added +0.041.
• At equal length, the trace's last words predicted correctness as well as or slightly better than summaries, and carried denser, more discriminative uncertainty and self-correction cues.
• On MMLU-Pro items with both correct and incorrect runs, linear summary readers were near chance and trace readers only modestly informative (prompt-withheld AUROC 0.503–0.545 vs 0.544–0.590); a stronger GPT-5-mini reader recovered more signal on gpt-oss-20b, with the trace retaining a +0.034 edge.
• Much of the linear readers' trace signal was associated with length; the authors conclude monitorability is a joint property of display and reader, so claims — including about faithfulness — must specify both.