An update on residential proxies and the scraper situation

LWN.net said its article on AI scraper bots published in early 2025 described a problem that has continued to grow over the past year. The site said scraper attacks come from millions of unique IP addresses, predominantly residential and mobile networks directed by command-and-control nodes. LWN said software is installed on ordinary systems, often without the owner's knowledge or consent, to fetch pages and forward data. The site said Google took down a bot network called IPIDEA at the beginning of the year, which correlated with a temporary reduction in scraper traffic. LWN said media-streaming devices have recently been identified as a major carrier of malicious scraping software. The site said companies such as Bright Data operate residential-proxy networks offering "ethically sourced" IP addresses and advertise the ability to bypass access controls. LWN said these operators can run code accessing resources on networks where millions of devices are connected. The site said high-profile model developers do their own scraping but are not the biggest problem. LWN said it is unclear who is paying residential proxies to attack sites. The site said defenses like Anubis proof-of-work challenges and commercial services are widespread. LWN said it was recently subjected to its heaviest scraper attack yet but defenses held.