Preprint reports higher training-data yield from selective web crawling

Researchers report in a preprint that sampling a few pages from each website and then collecting more from sites with higher quality scores yields several times more high-quality pages per fetch in simulations than uniform crawling depth or allocation based on web-link centrality. In the crawls analyzed, roughly half of the variation in document quality scores occurs between sites, and a few sampled pages help estimate the quality of the remaining pages. The researchers estimate that tens of millions of high-quality pages remain at known but uncrawled URLs in each crawl examined. The gains depend on evaluated quality classifiers and simulated allocation.