Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization

Summary

Large language models often fail to answer questions about a bounded document collection without retrieval at inference. This work frames the problem as document knowledge internalization, converting a fixed corpus into usable parametric knowledge. The proposed IAR (Inject, Align, Recover) framework separates structured knowledge injection, QA alignment, and general ability recovery. Inject uses continuation, rewrite, and instruction-conditioned reconstruction objectives; Align adapts with answer-only QA supervision; Recover merges the domain-adapted model with the base instruction model. Across Common Corpus and CCI, and Llama, Phi, Qwen, and SmolLM families, IAR improves the domain-general frontier. In the main comparison, IAR outperforms Vanilla SFT in 7 of 8 dataset-model settings on all four metrics, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 points in mean general performance (IFEval, MMLU, MSBench). While LoRA and FAPM can win individual general metrics, IAR retains one of the strongest general profiles among methods with leading domain internalization.