COFFEQA preprint reports gains in small-model question answering

Researchers report that COFFEQA, a method for training smaller language models with limited examples, improves question answering over training with factual examples alone. In a preprint, they report gains on TriviaQA of 8.77 average exact-match points and 9.98 average F1 points using T5 models and 32 training pairs. The method pairs factual contexts with minimally modified versions that cause a larger teacher model to give a different answer. It checks that the alternative answer depends on the changed context before using the pairs to train the smaller model. Tests covered five in-domain and three out-of-domain benchmarks using T5 and Qwen2.5 families; the reported advantage is over factual-only distillation at the same few-shot budget.