CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

TL;DR - CRAFT diagnoses model-specific capability gaps by clustering rubric criteria into a hierarchical capability tree, then uses weak nodes to generate targeted supervised fine-tuning data. This criterion-level approach generally outperformed prompt-level clustering and random data generation across finance and legal evaluations.

• Converts prompt-rubric pairs into capability descriptions and clusters them hierarchically.
• Dynamically selects low-performing capabilities at the tree level where each failure is clearest.
• Achieved the best finance-domain average across all four tested open-source models.
• Led on legal benchmarks for three models and remained within decoding variance of the best baseline on the fourth.