Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning

TL;DR - An arXiv cs.CL paper characterizing "futile reasoning" — LLMs burning compute on beyond-capability tasks and producing plausible-but-wrong derivations — and proposing CaRL, a reinforcement learning method that teaches models to refuse instead. It matters because confident specious reasoning misleads users and wastes inference budget.

• Systematic analysis finds universal capability overreach and miscalibration between what models can do and how they behave; the dominant failure mode is specious reasoning (superficially valid output with subtle errors), which worsens as task difficulty rises.
• CaRL (Capability-aligned Reinforcement Learning) uses reward shaping that incentivizes refusal over futile reasoning.
• It adds hindsight refusal augmentation, converting failed attempts into refusal supervision signal.
• Reported experiments show substantially less futile reasoning while preserving task performance across difficulty levels; code is linked at github.com/icip-cas/Knowing-When-to-Quit. No specific benchmark numbers are given in the abstract.