Nvidia details open-source service for automated training recovery

Image: Primary

Published Monday, November 2, 2026 · 6:02 PM EST

Nvidia says its open-source NVIDIA Recovery service can detect failed or stalled AI training jobs, exclude implicated nodes and coordinate restarts from saved checkpoints across a cluster. The Apache 2.0 software supports Slurm, the system that allocates cluster resources, and runs without changes to application code. IBM Research has deployed the service and enabled autonomous action after comparing its recommendations with manual recovery decisions. New deployments initially monitor workloads without taking scheduler actions. Native Kubernetes recovery remains on the roadmap, and the time needed to schedule resources and restore checkpoints still determines when productive training resumes.

Sources

NVIDIA Technical Blog

In this story

Nvidia IBM

Published by Tech & Business, a media brand covering technology and business. This story was sourced from NVIDIA Technical Blog and reviewed by the T&B editorial agent team.

Back to Newswire

Keep reading

Full wire

Products Infrastructure

Marvell announces $115 billion Lumentum acquisition

Marvell announced plans to acquire optical-component maker Lumentum for $115 billion, SiliconANGLE reported. Lumentum shareholders would receive 4.5 Marvell shares for each share they own. Marvell expects the transaction to close in the first half of 2028.

AI Capital

Trajectory raises $45 million and opens model post-training platform

Trajectory has raised $45 million in Series A funding at a $300 million valuation and is making its model post-training platform publicly available, TechCrunch reported. Sequoia led the round, with participation from Nvidia, Bessemer Venture Partners and Bain Capital Ventures.

Capital AI

42AI emerges from stealth with $62.5 million raised over three years

Israeli startup 42AI emerged from stealth with $62.5 million in seed funding accumulated over three years from its founders and Infinity Group, TechCrunch reported.

Capital

Larridin raises $30 million to measure enterprise AI spending

Larridin has raised $30 million in Series A funding led by Pelion Ventures, bringing its total financing to $47 million, Unite.AI reported. The company will use the funding for product development and team expansion.

AI Capital

Clark raises $11M and launches AI agents for audit work

Clark said Monday it raised an $11M seed round led by Touring Capital and launched Clerks, AI agents that carry out audit tasks for accounting firms. Marathon Management Partners, Rerail and Sofeon Ventures also invested, bringing the company's total funding to $16M.

Robotics AI

Tutor raises $75M Series B led by a16z

Tutor raised a $75M Series B led by a16z, bringing its total funding to $117M, Forbes reported. The company builds AI warehouse robots and is training a foundation model for robotics, putting the financing alongside its work on both robots and their underlying AI.