Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

Summary

Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, making them unreliable measures of model knowledge. This paper tests whether preventing the model from seeing option labels while committing to an answer removes positional influence and improves performance. Two strategies are evaluated: generation-then-matching and scoring options in isolation (the latter positionally unbiased by construction). Neither reliably improves accuracy. A complete decomposition shows the bottleneck is withholding options, not the matching step. The only configuration consistently matching the baseline shows all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, both an aggregate recall-imbalance measure and a direct per-question order-sensitivity measure fail to show reliable debiasing.