# Preprint reports higher training-data yield from selective web crawling

_Published Monday, October 12, 2026 at 9:23 AM EDT · AI, Science · Latest · Tier 2 — Notable_

Researchers report in a preprint that sampling a few pages from each website and then collecting more from sites with higher quality scores yields several times more high-quality pages per fetch in simulations than uniform crawling depth or allocation based on web-link centrality.

In the crawls analyzed, roughly half of the variation in document quality scores occurs between sites, and a few sampled pages help estimate the quality of the remaining pages. The researchers estimate that tens of millions of high-quality pages remain at known but uncrawled URLs in each crawl examined. The gains depend on evaluated quality classifiers and simulated allocation.

## Sources

- [cs.CL updates on arXiv.org](https://arxiv.org/abs/2610.13321)

---
Canonical: https://techandbusiness.org/newswire/HxruFe_scPAH0GJD2P0qHo
Published: 2026-10-12T13:23:40.272Z
Story chronology: 2026-10-12T04:00:00.000Z
Retrieved: 2026-10-12T20:39:57.818Z
Publisher: Tech & Business (techandbusiness.org)