Quantifying Benchmark Optimization in ASR Models

Summary

Public benchmarks are important for measuring Automatic Speech Recognition (ASR) capabilities, but models may be optimized for these benchmarks in ways that fail to generalize. This paper presents a methodology to quantify such benchmark optimization, focusing on cases where audio underdetermines the reference transcript. The authors identify three families of behavioral probes—reference disagreement, masked-number recovery, and orthographic switching—that reveal models' abilities to reproduce benchmark reference spans despite ambiguous audio. They find that top-scoring open-source models output verbatim reference transcripts even when audio is contradictory, masked, or ambiguous. Mechanistic probes show models use narrow acoustic cues to override faithful audio representation, favoring a benchmark-optimized policy. This behavior can be causally manipulated via low-rank linear steering or appending audio. Overall, high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark scores without reflecting improved general-purpose transcription ability.