SimForcing researchers report gains in robot video prediction and policy learning

Researchers report in a preprint that SimForcing, a system for predicting robot videos from robot actions, outperformed the compared methods on four video-quality measures on the Bridge dataset without external embodied pretraining. Using the trained model to initialize a model that maps visual and language inputs to actions also improved LIBERO task success. SimForcing transfers motion patterns from a simulation teacher and uses simulated trajectories to guide video prediction. Its training reduces dependence on inaccurate simulation predictions and limits interference from differences in visual appearance. The trained system generates both simulation conditions and real-domain videos without an additional world model at inference; the reported gains concern dataset evaluations.