Preprint finds open-weight security agents match closed models at lower cost

An open-weight language model can match the strongest closed models at less than a tenth of their cost on a security benchmark when given enough turns, researchers report in a preprint. They evaluated twenty-six models on CTI-REALM, where agents turn threat-intelligence reports into working detections by querying a live telemetry store. The researchers found that the benchmark's message limit assigns zero to unfinished tasks, including checkpoints already passed. Relaxing that limit let truncated models recover, with open-weight models earning the highest rewards. Separate tests that executed detection queries and checked outputs found different models leading different workflow stages, limiting the usefulness of choosing one model by its overall score.