Preprint reports 42-fold sample advantage for aligning text and image representations

Researchers report that a method for training text encoders reached the same alignment with image representations using 42 times fewer training samples than next-token prediction on the same corpus. Their arXiv preprint studies joint-embedding predictive architectures, which learn by relating different observations of the same scene. A 109M-parameter text encoder aligned with DINOv2 and CLIP image encoders better than every language model evaluated, including models up to 72B parameters. The researchers also demonstrated image captioning by applying a rotation between separately trained encoders. The sample advantage concerns representation alignment, rather than a demonstrated reduction in the cost of training a general-purpose language model.