How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

Summary

AutoResearchEval introduces 100 tasks spanning 7 scientific domains and the full research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Evaluating 8 harness-model combinations yields 800 trajectories with process-level annotation, leading to the AutoResearch Failure Taxonomy (ARFT), a framework of 45 empirically-grounded failure patterns. A human-calibrated agent-as-a-judge pipeline enables scalable fine-grained attribution. The failures converge on a single overarching limitation: current agents lack a metacognitive loop—the ability to check what they produced against what they found, revise when it doesn't hold up, and question the soundness of their path. This pattern recurs across all combinations, including the strongest models, locating the deficit at the model level rather than in any particular scaffold. The work publicly releases AutoResearchEval and ARFT to facilitate further research.