Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?

TL;DR - CreditCardQA is a 1,800-question benchmark testing numerical reasoning over real credit card agreements. It shows that models struggle more with interpreting contractual rules and conditions than with arithmetic, with potentially greater consequences for financially vulnerable users.

• Program-of-Thought prompting consistently outperforms Chain-of-Thought, especially for weaker models.
• PoT narrows performance gaps between open- and closed-source systems.
• Comparisons, conditional logic, and monetary constraints are particularly challenging.
• Common failures include missed conditions and misapplied rules around edge cases such as late fees and small balances.