An arXiv paper published on October 26, 2026 (ID 2610.31678v1) warns that, in interactive evaluations of large language models (LLMs), relying solely on overall accuracy or F1 scores to validate an automated scorer can seriously mislead assessments of prompt‑strategy effectiveness. For example, when comparing model responses elicited by two types of prompts—“neutral requests to reconsider” and “assertions that fictional movies are real”—if the scorer exhibits a significant increase in error rate under one prompt, the measured “dialogue pressure effect” could be exaggerated, underestimated, or even reversed in direction.
The study validated this issue through two audits: The first involved a balanced sample of 288 cases, with the scorer achieving an overall F1 score of 0.916; however, its false-positive rate varied dramatically across different prompts—ranging from 2.6% to 33.3%—and the estimates of prompt specificity themselves were uncertain. The second audit had the same set of 300 model–question pairs evaluated jointly by the scorer and two independent LLM reviewers, revealing that what initially appeared to be a 9.3-percentage-point effect difference narrowed to 2.3 percentage points under dual review, with the lead reviewer’s confidence interval including zero, thus failing to establish statistical significance.
Furthermore, the comparison results of the other two prompts remained consistent in direction under dual review, whereas a single exploratory attempt at specificity correction, due to excessive uncertainty, also failed to yield a definitive conclusion. The paper ultimately calls for researchers to directly validate the specific prompt differences they intend to report, rather than merely assessing the overall performance of the scorer—meaning that both prompts should be audited concurrently on matched cases, and the uncertainties of both the scorer and human reviewers should be explicitly presented in the final results.