Researchers report gains from jointly training hand reconstruction and action prediction

Researchers report that P³-WAM, a model trained jointly to reconstruct hand movements, predict video from actions and plan manipulation sequences, outperformed specialized baselines on hand reconstruction and action-conditioned video prediction across four datasets in a preprint. The model combines a structured representation of 3D actions with a pretrained video-generation transformer. It can use video and actions as either inputs or prediction targets. Unified training improved all three capabilities over training each mode separately. Planning gains were assessed through a vision-language model, while generated manipulation videos showed stronger instruction adherence and action progression than video-generation baselines.