Dust researchers report competitive transformer training without backpropagation

Researchers at qlabs.sh report that Dust, a method for training transformer language models without backpropagation, matched and sometimes exceeded that standard training method in experiments using large populations of trials. Achieving those results required substantially more computation. Dust adds independent noise to internal model outputs at each token, then rewards changes that reduce prediction errors. This lets a single forward pass evaluate many trials in parallel, rather than evaluating a separate altered set of model weights for each trial. The researchers tested alignment with backpropagation's training signals up to 1B tokens. They explicitly do not aim to make Dust efficient enough to replace backpropagation today.