AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

TL;DR - AI4AI-Bench evaluates whether LLM agents can improve AI training algorithms across 10 frozen research repositories. Current agents show limited recursive self-improvement capability, with the best system reaching 0.250 on a scale where 0.1 is the original algorithm and 1.0 is optimal.

• The benchmark separates genuine learning-algorithm changes from data collection and hyperparameter tuning.
• It evaluates 29 configurations of six systems, giving each agent four hours on one B300 GPU before independently rerunning and scoring its code.
• The mean score was 0.166; most submissions did not alter how the model learns.
• Submissions that changed the learning algorithm averaged 0.226 versus 0.126 for others, and increased reasoning effort raised their frequency from 8% to 64%.