Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents

TL;DR - An arXiv preprint proposing a zeroth-order optimization framework that lets self-evolving LLM agents learn from difficult examples they cannot solve by sampling, pushing past their inherent capability boundary without trajectory annotations.

• Perturbs LoRA parameters, runs the agent under perturbed vs. original weights, and uses the loss difference to estimate gradients and update LoRA — no backprop or labeled trajectories required.
• The improved model then samples trajectories used for supervised fine-tuning, closing a self-evolution loop that breaks the sampling-based capability ceiling.
• Efficiency and stability additions: parallel perturbation inference, an adaptive lookup mechanism to cut zeroth-order overhead, and an answer-perplexity loss for smooth, stable loss signals.
• Reported gains on multiple deep research benchmarks: more successful trajectories and consistent improvement over strong baselines, most notably on hard examples; code released at github.com/hidk1911/ZOForLLMAgents.