AWS says security agent exploited intended flaws in 33 of 37 tests

AWS said its autonomous penetration-testing agent exploited the intended vulnerabilities in 33 of 37 tests involving recently disclosed software flaws. The agent started with an endpoint and minimal information, without an exploit or proof of concept, and had to retrieve a hidden flag from each live target. The company ran the benchmark on Claude Opus 5 with a 10-hour limit per target and disabled runtime vulnerability lookup. An integrity audit rejected two additional flag captures because they bypassed the intended flaws. In two other tests, the agent exploited the bug but disabled the target before retrieving the flag. AWS describes the benchmark as reducing contamination from prior training data, rather than eliminating it.