Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

TL;DR - Power-Law Entropy Search (PLES) uses cost-aware, multi-fidelity Bayesian optimization to estimate optimal LLM training hyperparameter scaling laws efficiently. It achieves accurate estimates with less than one-tenth the computational budget of grid search and other baselines.

• PLES selects experiments based on expected reduction in overall scaling-law uncertainty per unit of compute.
• Its adaptive strategy naturally favors informative, lower-cost small-scale training runs.
• Evaluations cover synthetic benchmarks, surrogates fitted to real LLM training data, and actual LLM pre-training runs.
• The method targets scaling-law estimation rather than optimization of a single objective function.