Preprint demonstrates generation of language-model weights for held-out data mixtures

Researchers report generating complete pretrained language-model weights for training-data mixtures withheld from their generator's training, in a new arXiv preprint. They found minimal changes in downstream test-set perplexity and general capability benchmark scores compared with models in their reference collection. The team trained 100 GPT-2 models at each of three sizes: 51M, 124M and 355M parameters. Training mixtures varied across web text, books, mathematics, coding and French. A flow-matching model generated weights in a compressed representation, conditioned on those mixtures. The researchers provide source code and artifacts. Their demonstrated generalization concerns held-out mixtures within that data-mixture space.