Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses

Summary

This work introduces Hierarchical Self-Improvement (HSI), a framework where a single frozen LLM operates across three hierarchical scopes: a task harness, an evolver that rewrites the harness, and a meta-evolver that rewrites the evolver's strategy. The harness is task-specific and continuously evolvable via a fixed injection seam, using environment feedback. A thinking-on/off design isolates the contribution of harness evolution. Experiments on BALROG with DeepSeek-V4-Flash-Preview as the frozen backbone show consistent gains on moderate-difficulty tasks (BabyAI +39.3, Crafter +33.0, TextWorld +25.0, MiniHack +15.0 in raw % Progress) and strong held-out generalization on BabaIsAI sub-suites (0.98 on BreakStop, 1.00 on GoTo from a 20% unseen split). However, tasks beyond the backbone's capability (NLE) showed no improvement, indicating limits due to feedback fidelity and backbone capability. Code is available.