PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
Merged summary
TL;DR - PyroDash trains a small language model to trigger a one-time, token-level handoff to a frozen LLM when assistance is needed. This reduces serving costs while maintaining—or in one configuration improving—mathematical reasoning accuracy.
- The SLM learns an internal control-token policy, avoiding a separate router, LLM retraining, or access to LLM logits.
- Training combines control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware GRPO alignment.
- At λ=0.05, PyroDash averaged 64.04% accuracy across five benchmarks, 6.36 points above the LLM-only baseline, while cutting cost by 20.4%.
- At λ=0.6, it achieved 54.55% accuracy while reducing total cost from $49.36 to $1.78, using a 1.90% LLM-token ratio and 0.012 LLM calls per example.
Sources (1)
PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
TL;DR - PyroDash trains a small language model to trigger a one-time, token-level handoff to a frozen LLM when assistance is needed. This reduces serving costs while maintaining—or in one configuration improving—mathematical reasoning accuracy.
- The SLM learns an internal control-token policy, avoiding a separate router, LLM retraining, or access to LLM logits.
- Training combines control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware GRPO alignment.
- At λ=0.05, PyroDash averaged 64.04% accuracy across five benchmarks, 6.36 points above the LLM-only baseline, while cutting cost by 20.4%.
- At λ=0.6, it achieved 54.55% accuracy while reducing total cost from $49.36 to $1.78, using a 1.90% LLM-token ratio and 0.012 LLM calls per example.