Bright data, another company that goes dark with site-scraping practices.
Recently I was noticing a new wave of AI scrapers on one of my websites. This time, they were using sneaky tactics.
Once they couldn’t get the page they wanted (anti-bot measures), they started fetching those from various residential IP addresses, eventually using a full-fledged browser to get through the page challenge.
However, due to the nature of the site — using random parameters in the URLs — I’ve had a way to identify these bots by the URLs requested.
I don’t know if they are using VPN-like connections or the residential IPs are actual servers of theirs (ipapi.is didn’t see them as proxies or VPNs), but they were american IP addresses. They used the following tactics:
- First they tried to fetch the URL from their “official” IP ranges, disclosed on their page: https://brightdata.com/brightbot#toc_1_1
. The request arrived with the
Brightbot 1.0user agent. - When the URL wasn’t served with an
HTTP 200, within 5 seconds they rerequested the same URL, but with legit-looking user agent strings, from residential IPs. - This re-requesting repeated 2-4 times from different IPs.
- When even those bots couldn’t fetch the page because of the challenge it served, they’ve changed tactics and used another bot that probably was an actual headless browser. It loaded and solved the challenge, and then scraped the page.
This whole process repeated about minutely with various URLs, all of those were trackable because of the same randomized parameter in the URL, used in each of their scraping sessions.
One such example:
82.97.199.180 - - [04/Sep/2025:17:04:35 +0200] "GET /url?parameter=random-number&other-param=xxx HTTP/2.0" 418 2060 "-" "Brightbot 1.0" "-"
136.227.180.208 - - [04/Sep/2025:17:04:38 +0200] "GET /url?parameter=random-number&other-param=xxx HTTP/2.0" 418 2060 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:141.0) Gecko/20100101 Firefox/141.0" "-"
172.98.191.147 - - [04/Sep/2025:17:04:41 +0200] "GET /url?parameter=random-number&other-param=xxx HTTP/2.0" 418 2060 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:141.0) Gecko/20100101 Firefox/141.0" "-"
167.224.230.231 - - [04/Sep/2025:17:04:53 +0200] "GET /url?parameter=random-number&other-param=xxx HTTP/2.0" 418 2060 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36" "-"
Of these IPs, the last one was the full-fledged, probably headless browser, that loaded and solved the challenge.
These residential IPs are reported nowhere. AbuseIPDB doesn’t contain them, and other places I’ve looked them up on, didn’t categorize them as doing malicious activity.
I’ve made measures to mitigate the scraping, the details of which I can’t disclose here (you can email me on the email address in the contact-imprint page for tips and tricks), There is enough evidence that bright data, a company that boasts about “99.99% protection from abuse” , are abusers themselves.
Without going into presumptions here, this “protection” they claim, seems more like holding sites at ransom.
Recent posts
- I'm glad and relieved that „AI is taking my job away“ June 19, 2026
- Fact-Checking the Cryptographic Issues Report of Matrix's Vodozemac May 6, 2026
- Rewriting my IceGStreamer project in 4 days with Claude Code March 23, 2026
- Postmortem: taming Synapse's runaway database growth March 5, 2026
- Tarpitting bots is a reality October 29, 2025