AWS shares training checks to detect silent GPU errors on HyperPod

AWS published code for detecting silent hardware errors during distributed AI training on SageMaker HyperPod and tracing suspect GPUs to nodes for replacement. The checks compare repeated calculations and model weights to catch incorrect results that hardware monitoring can miss. AWS tested the approach with software-injected faults on 32 NVIDIA H100 GPUs and repeated every test on three instances, the minimum required. Detection runs inside the training code; node replacement follows review. The approach requires replicated model weights, and a finite error that occurs only once on a step without an audit can escape detection.