From Test-Time Scaling to Reusable Memory: Measuring Crystallization in Text-to-SQL
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv paper defining the "crystallization problem": how to measure whether test-time-scaling compute, when saved as reusable memory, actually helps on future unseen text-to-SQL questions rather than just replaying past ones. It matters because current end-to-end scores conflate replay with genuine transfer and hide which memory design choice is doing the work.
- Proposes a controlled evaluation that fixes the single-shot solver and varies one memory choice at a time, separately reporting replay, cross-question retention, and held-out same-database transfer.
- On BIRD, storing verified corrected queries raises held-out first-attempt accuracy by 4.34 percentage points, capturing 44.4% of the headroom that on-demand repair provides on the same questions.
- Interventions point to database-specific content as the main active ingredient; reliable verification and broader retrieval coverage help, while richer memory formats and more elaborate retrievers do not.
- Code, evaluation artifacts, and reproduction instructions are released openly (github.com/ai-jiaqian/text-to-sql-memory-crystallization).
Sources (1)
From Test-Time Scaling to Reusable Memory: Measuring Crystallization in Text-to-SQL
TL;DR - An arXiv paper defining the "crystallization problem": how to measure whether test-time-scaling compute, when saved as reusable memory, actually helps on future unseen text-to-SQL questions rather than just replaying past ones. It matters because current end-to-end scores conflate replay with genuine transfer and hide which memory design choice is doing the work.
- Proposes a controlled evaluation that fixes the single-shot solver and varies one memory choice at a time, separately reporting replay, cross-question retention, and held-out same-database transfer.
- On BIRD, storing verified corrected queries raises held-out first-attempt accuracy by 4.34 percentage points, capturing 44.4% of the headroom that on-demand repair provides on the same questions.
- Interventions point to database-specific content as the main active ingredient; reliable verification and broader retrieval coverage help, while richer memory formats and more elaborate retrievers do not.
- Code, evaluation artifacts, and reproduction instructions are released openly (github.com/ai-jiaqian/text-to-sql-memory-crystallization).