New research reveals that open-book assessments primarily measure reading behavior rather than knowledge retention.
An arXiv paper published on October 24, 2026 (2610.31017v1) points out that, in document-level LoRA adapter evaluations, open-book scoring relies heavily on the model’s ability to read on the fly, rather than on whether the adapter has truly encoded knowledge specific to the document; random adapters can reproduce 82% of the performance advantage of expert adapters on open-book tasks, but on cloze tasks involving unstructured text, they only replicate 2%—statistically indistinguishable from zero.
Why it matters：This study, through systematic controlled experiments—comprising 150 Wikipedia articles, the Llama-3.1-8B-Instruct model, and a two-task comparison—reveals the fundamental limitations of the current open-book evaluation paradigm: it fails to effectively distinguish between “reading” and “remembering,” posing methodological challenges to the practical utility of knowledge-editing and retrieval‑oriented adapters. The paper also reports four silent errors that lead to “clean yet incorrect results,” highlighting risks to reproducibility.