AINews: 10% worse, 100x cheaper, 10000x faster: Why Simulation is taking over

Summary

According to Latent Space, since 2022, components of the AI pipeline have been shifting from human-made to model-made, each flip starting with a 'patient zero' at a frontier lab. This trend is characterized as simulation: slightly worse (10%), but 100x cheaper and 10,000x faster. Stage 1 (2022): Reward signal went synthetic with InstructGPT's reward model, followed by Constitutional AI and RLAIF, where AI critiques itself. Stage 2 (2023): Training data became synthetic; Microsoft's Phi series showed 'Textbooks Are All You Need', Apple's WRAP rephrased the web with an LLM, making pretraining about 3x more efficient, and NVIDIA's Nemotron-4 340B shipped with a permissively licensed synthetic data generation pipeline.