Plumbline preprint reports 61 training problems found across five AI models

Researchers report that Plumbline, a system used in production at Amazon to validate large language model training, found 61 problems across three frameworks and five models. Of those problems, 32 were silent failures. The system checks the training lifecycle from input data through the model used for serving, covering individual layers and whole runs. It reuses those checks to trace failures to their root causes and supports validating newly released models before teams rely on their training. The preprint says 42 problems have been fixed or reported upstream, including eight first raised by the researchers.