# Separating signal from noise in coding evaluations

_Wednesday, July 8, 2026 at 8:00 AM EDT · AI · Latest · Tier 2 — Notable_

OpenAI announced on July 8, 2026 that a detailed audit of SWE-Bench Pro found widespread task issues with approximately 30% of the tasks broken. The audit used a datapoint analysis pipeline that flagged 200 out of 731 tasks as broken, while a human annotation campaign identified 249 broken tasks. Issues included overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. OpenAI advised model developers to carefully examine results from the benchmark.

## Sources

- [OpenAI](https://openai.com/index/separating-signal-from-noise-coding-evaluations)

---
Canonical: https://techandbusiness.org/newswire/1zexheHxKDYI99qGZrH1XO
Retrieved: 2026-07-10T00:07:31.180Z
Publisher: Tech & Business (techandbusiness.org)