Is Pre-training Bitter Enough? Task Construction as the Control Surface in Continued Pretraining

Summary

This paper examines task construction as a control surface in continued pretraining. Standard pretraining fixes the self-supervised objective before training, with downstream feedback only available after the entire run, making the control loop coarse. The authors propose value-based pretraining (V-pretraining), which separates the learner from a lightweight task designer. The learner updates only via self-supervised loss on unlabeled data, while the designer learns to construct self-supervised tasks—such as target distributions in language modeling or view selection in vision—using a small set of verifiable downstream examples, thereby making the control loop more direct and tighter.