Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements

Summary

Reinforcement learning (RL) has shown promise in single-turn LLM fine-tuning, but long-horizon agentic reasoning introduces increasingly branching interactions and sparse rewards, exposing limitations: heavyweight backpropagation-based training stacks and harder credit assignment. This paper argues that evolution strategies (ES) are a better choice, offering model scalability (full-parameter optimization with inference-level GPU memory), flexibility (composable with prompt-space evolution), and long-horizon scalability (trajectory-level attribution). They propose Agentic ESOpt, a full-parameter fine-tuning framework with online reward-weighted updates and a cosine decay schedule for perturbation scale. On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B improves the No Skill baseline by 6.69%. In test-time automatic heuristic design, it improves its matched baseline in 28 of 36 settings.