NAPE: Next Patch Embedding Prediction Enables Scalable Audio Learners

Summary

Self-supervised learning has driven audio representation learning, but existing methods often rely on complex pre-training recipes. The authors introduce NAPE, a framework where a causal Transformer predicts the next patch embedding of a log-mel spectrogram from previous ones, using only causal masking and stop-gradient as training signals, avoiding reconstruction decoders, acoustic tokenizers, and other components. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently with encoder size, and yields strong linear-probing results, while producing structured attention patterns without explicit supervision.