SPADE: Self-Play in Adaptive Synthetic Executable Environments

Summary

SPADE is a self-play RL framework where a single LLM acts as both an Environment Designer, which writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. The designer optimizes the agent's regret signal (the gap between reward with and without privileged hints) to generate environments at the edge of capability while keeping them feasible. Key components include grounding the designer on documents from a large pretraining corpus and giving it accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out benchmarks, and lifts tool-use by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent. This makes environment design a learnable component, advancing open-ended self-improvement.