Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?
Ranking
Overall
62
Content
70
Popularity
44
Observed public metrics from 1 member.
Merged summary
TL;DR - CreditCardQA is a 1,800-question benchmark testing numerical reasoning over real credit card agreements. It shows that models struggle more with interpreting contractual rules and conditions than with arithmetic, with potentially greater consequences for financially vulnerable users.
- Program-of-Thought prompting consistently outperforms Chain-of-Thought, especially for weaker models.
- PoT narrows performance gaps between open- and closed-source systems.
- Comparisons, conditional logic, and monetary constraints are particularly challenging.
- Common failures include missed conditions and misapplied rules around edge cases such as late fees and small balances.
Sources (1)
Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - CreditCardQA is a 1,800-question benchmark testing numerical reasoning over real credit card agreements. It shows that models struggle more with interpreting contractual rules and conditions than with arithmetic, with potentially greater consequences for financially vulnerable users.
- Program-of-Thought prompting consistently outperforms Chain-of-Thought, especially for weaker models.
- PoT narrows performance gaps between open- and closed-source systems.
- Comparisons, conditional logic, and monetary constraints are particularly challenging.
- Common failures include missed conditions and misapplied rules around edge cases such as late fees and small balances.