# AWS shares training checks to detect silent GPU errors on HyperPod

_Published Wednesday, November 11, 2026 at 8:06 PM EST · AI, Infrastructure · Latest · Tier 2 — Notable_

![AWS shares training checks to detect silent GPU errors on HyperPod — Primary](https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/03/ML-21841-featured-image.png)

AWS published code for detecting silent hardware errors during distributed AI training on SageMaker HyperPod and tracing suspect GPUs to nodes for replacement. The checks compare repeated calculations and model weights to catch incorrect results that hardware monitoring can miss.

AWS tested the approach with software-injected faults on 32 NVIDIA H100 GPUs and repeated every test on three instances, the minimum required. Detection runs inside the training code; node replacement follows review. The approach requires replicated model weights, and a finite error that occurs only once on a step without an audit can escape detection.

## Sources

- [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/detecting-and-recovering-from-silent-data-corruption-on-amazon-sagemaker-hyperpod/)

---
Canonical: https://techandbusiness.org/newswire/vw0w72h5t7HNJXYMExEqQN
Published: 2026-11-12T01:06:02.154Z
Story chronology: 2026-11-11T23:30:03.000Z
Retrieved: 2026-11-12T02:44:42.790Z
Publisher: Tech & Business (techandbusiness.org)