Characterizing interference weights in a tiny language model

Summary

Mechanistic interpretability aims to read neural networks like programs, but superposition forces features into low-dimensional projections. This study trains a one-layer transformer and decomposes its virtual weights between tokens, positions, features, and logits, scoring weights by effectiveness and helpfulness. They find an interference weight: the path from 'IN' to logits via residual stream votes for 'utions', a token that never follows 'IN' in training, so every time it affects output, it worsens loss. This demonstrates, for the first time, an interference weight inside a trained transformer by measuring its effect on training loss.