Why Models Are AI’s Next Training Dataset with Damian Borth - #772

Summary

As high-quality training data becomes scarce and pretraining costs rise, Damian Borth, professor of AI at the University of St. Gallen, argues that we've overlooked a key knowledge source: already-trained models. His group's work on weight space learning treats trained neural networks as data, learning from the distilled results of millions of GPU hours of optimization rather than starting from raw data. This episode explores what it means to build foundation models of neural networks, how knowledge can transfer across architectures and domains, how this approach could cut the cost of specialized model development, and whether future AI systems might be trained on collections of models rather than ever-growing datasets.