# Nvidia details open-source service for automated training recovery

_Published Monday, November 2, 2026 at 6:02 PM EST · AI, Infrastructure · Latest · Tier 2 — Notable_

![Nvidia details open-source service for automated training recovery — Primary](https://developer-blogs.nvidia.com/wp-content/uploads/2026/11/image8.webp)

Nvidia says its open-source NVIDIA Recovery service can detect failed or stalled AI training jobs, exclude implicated nodes and coordinate restarts from saved checkpoints across a cluster. The Apache 2.0 software supports Slurm, the system that allocates cluster resources, and runs without changes to application code.

IBM Research has deployed the service and enabled autonomous action after comparing its recommendations with manual recovery decisions. New deployments initially monitor workloads without taking scheduler actions. Native Kubernetes recovery remains on the roadmap, and the time needed to schedule resources and restore checkpoints still determines when productive training resumes.

## Sources

- [NVIDIA Technical Blog](https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery/)

---
Canonical: https://techandbusiness.org/newswire/5EeKvRJts7X4x02NyDPanM
Published: 2026-11-02T23:02:16.923Z
Story chronology: 2026-11-02T22:43:05.000Z
Retrieved: 2026-11-03T01:42:46.311Z
Publisher: Tech & Business (techandbusiness.org)