When a model reads a drug's class from its name—not its knowledge

Summary

When a model reads a drug's class from its name—not its knowledge August 18, 2026 Ai2 People increasingly bring health and medication questions to AI models, often naming specific drugs. But a model can sound knowledgeable without knowing much about the drug itself. Many drug names encode their class through affixes – often suffixes like -pril, -olol, and -azoline – giving a model a shortcut: infer the answer from the name rather than from learned facts about the medication. In a new study of Olmo 3 accepted to EMNLP, researchers found that 51–59% of the drugs they tested elicited little sign of drug-specific knowledge, while another 12–18% appeared affix-driven. By tracing those behaviors back to the model’s open training data, the team showed not only that the shortcut exists, but where it comes from—and why that matters when models answer health questions. Kaijie Mo at the University of Texas at Austin, working with collaborators at Northeastern University and MD Anderson Cancer Center, led the study. The team set out to determine when a model was relying on the morphology of a drug’s name rather than drug-specific knowledge—and to trace that behavior back to the data the model was trained on. They chose Olmo 3 because the analysis required a fully open model with public weights and training corpora, documentation spanning pretraining through post-training, and released intermediate checkpoints. "Olmo 3 was particularly valuable because of its openness and because its training corpus includes substantial medical content," says Mo, "allowing us to connect behavior, mechanisms, and training data. Open models make it possible to move from observing a behavior to understanding its cause." On Olmo 3 7B Instruct, the team built a diagnostic that isolates three things a model might draw on when it answers drug-related health questions: the affix of a drug’s name (the shared ending, like -pril), the stem (the part of the name unique to a drug), and real knowledge of a specific drug. They swapped the stem, the affix, or both for invented, meaningless ("nonce") strings and measured how the model's answers shifted. The diagnostic found that Olmo 3 knows less about drugs than it lets on. Depending on the evaluation format, about 12–18% of drugs appeared affix-driven. For a much larger 51–59%, the model showed no real sign of knowing a particular drug—its answer to a…