Preprint releases bibliographic benchmark with 62,899 synthetic documents

A preprint introduced SHELF, a synthetic benchmarking system for bibliographic work, and released its source code and data under permissive licenses. Its first release contains 62,899 model-written documents derived from Library of Congress vocabularies and supports classification, clustering, retrieval, pair classification and instruction retrieval tasks. The authors compared sparse methods, encoders and zero-shot decoders where applicable. They report that SHELF scores do not estimate accuracy on production catalogue data.