Dynamic Multi-Byte Prediction with Hierarchical Language Models

Summary

Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations: a variable-length prediction window aligned with the latent tokens (segments) of a hierarchical LM, and a novel attention-masking scheme enabling parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across generative tasks—instruction following, question answering, summarization, and machine translation—achieving the best balance between performance and inference throughput.