Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents

TL;DR - InvestLogicBench is a process-native benchmark of 201,247 documented decisions from 151 real-world investors that evaluates how financial LLM agents reason, not just whether they turn a profit. It matters because it shows leading LLMs produce fluent investment logic that is largely ungrounded in actual market evidence — a failure mode invisible to outcome-only evaluation.

• Each episode follows a P→E→R→D→O trace (investor Profile, market Events, Reasoning, executable Decision, delayed Outcome), supporting comprehension, profile-conditioned generation, and end-to-end replay.
• Across four leading LLMs, logical plausibility scored near 4/5 while event grounding scored only 0.8–2.8/5; return and process-quality metrics also disagreed with each other.
• The authors argue existing evaluation uses the "wrong ruler": static QA omits agency, and terminal P&L cannot distinguish grounded, profile-consistent action from luck.
• They propose P→E→R→D→O as a data-system interface requiring versioned profiles, temporal provenance (point-in-time event binding), inspectable retrieval, decision ledgers, and replayable outcomes — with finance as a stress test for personalized, consequential agents generally.