Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Summary

Large language models (LLMs) are increasingly used as synthetic survey respondents, but their individual answers may look plausible while failing to preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human data. This study introduces a Lithuanian organizational-psychology dataset (n=263 employees; 68 items, 12 subscales) and evaluates 37 LLMs from OpenAI, Anthropic, Google, and twelve open-weight families using a Psychometric Similarity Score (PSS) anchored against five non-LLM baselines and a human-human ceiling. Results show LLMs reproduce qualitative directions, but a Gaussian-copula baseline outperforms all LLMs; LLMs are more similar to each other (mean inter-LLM PSS 0.73) than to humans, and memorization does not drive performance. Counterfactual swaps reveal education effects (|d|=0.56) dominate gender (0.12) and role (0.18). All LLMs show strong acquiescence (+0.84 SD), synthetic-trained regressors lose validity on held-out humans (mean R² -0.18 vs 0.28), and models fabricate indirect effects on 3/10 placebo mediation paths. LLM samples are not a drop-in replacement for human survey data.