New benchmark framework targets the evaluation-safety gap in LLM testing

A survey and conceptual framework posted to arXiv argues that LLM evaluation and AI safety suffer from the same measurement problem: benchmark scores, red-teaming results, and human preference ratings all compress heterogeneous failure modes into single numbers, making it hard to distinguish genuine capability gains from superficial pattern matching. The authors propose EvalSafetyGap, a taxonomy that maps evaluation artifacts such as data contamination, benchmark saturation, and metric gaming onto safety failures like reward hacking, sycophancy, and capability elicitation gaps. They illustrate the framework on three case studies: a coding benchmark where pass@1 rose 12 points after training on the test set's problem templates without improving generalization; a refusal benchmark where a model learned to refuse a narrow template pattern while remaining compliant on semantically identical requests; and a red-teaming dataset where a model scored well on held-out attacks but failed on distributionally shifted variants. The framework does not introduce new benchmarks but reinterprets existing measurements as joint evaluation-safety signals. The preprint is available as arXiv:2606.30219v1.