Researchers report 66 faults in six software security testing tools

Researchers report that LLM4SAST, a framework for testing software security scanners, identified 66 unique faults across six tools. Developers confirmed or fixed 32 after receiving reports, according to the arXiv preprint. The framework uses large language models to generate code variants in multiple programming languages, then compares scanners' reports on the original programs and their variants to identify faults. Feedback from running programs helps select variants, while comparisons of code changes help group faults. The researchers report detecting 3.4 times more faults using only 39.8% of the variants required by existing approaches. Their filtering heuristics eliminate 47.8% of false positives; the findings concern faults in scanning tools, rather than vulnerabilities discovered in customer applications.