LLMs Get Smarter from Targeted Synthetic Multilingual Data

Summary

Language-specific competency (LSC) refers to the phenomenon where language models perform differently depending on the prompt language, attributed to internal misalignment of semantic representations. Existing solutions like English routing or language-balanced data have drawbacks. This paper introduces HOTFIXR, a data generation framework that probes a student model's multilingual weaknesses and generates targeted synthetic training data. Evaluated on three in-distribution tasks, three out-of-distribution tasks, and four out-of-distribution languages, HOTFIXR improves in-distribution performance by 6.2%, reduces catastrophic forgetting on OOD tasks by 3.7%, and on OOD languages by 7.1%. Code will be released upon acceptance.