Separating signal from noise in coding evaluations

Published Thursday, July 9, 2026 · 4:53 PM EDT

OpenAI announced on July 8, 2026 that a detailed audit of SWE-Bench Pro found widespread task issues with approximately 30% of the tasks broken. The audit used a datapoint analysis pipeline that flagged 200 out of 731 tasks as broken, while a human annotation campaign identified 249 broken tasks. Issues included overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. OpenAI advised model developers to carefully examine results from the benchmark.

Sources

In this story

Published by Tech & Business, a media brand covering technology and business. This story was sourced from OpenAI and reviewed by the T&B editorial agent team.

Back to Newswire

Keep reading

Capital

Capital

Endeavor Catalyst closes $320 million fund for its founder network

Endeavor Catalyst has closed its fifth fund with $320 million in capital commitments, bringing assets under management to more than $850 million. The venture arm of nonprofit Endeavor plans to invest in up to 150 companies through...

Capital AI

Capital AI

Nous Research raises $90 million for enterprise agent push

Nous Research raised a $90 million Series B at a $1.5 billion valuation, TechCrunch reported, confirming its earlier reporting. Robot Ventures led the round, with backers including Nvidia, Union Square Ventures, Menlo Ventures, Sa...

AI Infrastructure

AI Infrastructure

Broadcom seeks more than $50B in financing for OpenAI chip

Broadcom has been working to arrange more than $50B in financing for OpenAI's custom AI chip, the Wall Street Journal reported, citing sources. Oracle is also discussing financing for a large chip purchase. Apollo, Blackstone and ...

Capital

Capital

Navra raises $19 million for access to blockchain financial markets

Navra, a startup led by SoFi and Figure co-founder Mike Cagney, raised $19 million in Series A funding, Bitcoinist reported. Ribbit led the oversubscribed round, with Baseline, DCM, Jump Crypto and Figure Technology Solutions part...

Capital AI

Capital AI

Stuut announces $52.5 million Series B for finance automation

Stuut announced a $52.5 million Series B led by Insight Partners, with Andreessen Horowitz and Microsoft's venture fund M12 participating. The round brings total funding to $93 million, The Next Web reports. The company automates...

Robotics Capital

Robotics Capital

Mecka raises $60M Series B led by Sequoia

Mecka raised a $60M Series B led by Sequoia, with participation from Nvidia, M12 and other investors, TechCrunch reported. The startup collects and analyzes human motion data used to train humanoid robots and other kinds of robots...