Why Anti-AI Scraping

The War Nobody Is Winning

There is a war being fought on the internet. Website owners are building walls. AI crawlers are walking around them. The website owners build a bigger wall. The crawlers change their behaviour. Someone invents a poison. Someone else builds a detector. Someone builds a tarpit. The crawler learns to avoid it. Repeat until everyone is exhausted. Then someone sends an invoice.

The question nobody is asking clearly enough is whether any of this is worth it. Most of the people spending money on anti-scraping infrastructure have not done that calculation. They have simply decided that crawlers are the enemy and that stopping them is self-evidently good. That is not analysis. That is a reaction.

The defences are real. The costs are real. The outcomes, for most sites, were decided before the first line of configuration was written.

robots.txt: A Polite Sign on an Unlocked Door

The first line of defence is robots.txt. It is also the weakest. The file is a convention. It tells crawlers which parts of a site they are permitted to access. A legitimate crawler may respect it. A crawler operated by someone who has decided your content is worth taking does not have to.

There is no enforcement mechanism. No technical barrier. No connection refusal. The file is read, or it is not. The bytes on the other side of it are still served to anyone who asks. Placing your hopes in robots.txt against a determined scraper is the equivalent of putting a sign on your front door that says PLEASE DO NOT ENTER and then leaving it unlocked. Polite. Not effective.

Where robots.txt retains value is in signalling intent to crawlers that already operate within the rules. That is a narrower use case than most people imagine, and it requires trusting that the crawler in question has decided to behave. That trust has been misplaced enough times to be worth examining before extending it.

Bot Detection and the Evasion Problem

The logical next step is active detection. Block the bots. Identify them by user-agent string. By IP address. By request rate. By the absence of browser behaviour. By TLS fingerprint. By header patterns. By the timing between requests. Build a model of what a crawler looks like and refuse access when the model fires.

The problem is not that this is technically difficult. It is that the crawler has a straightforward response to every signal you check. Change the user-agent. Rotate the IP. Slow down. Distribute requests across dozens of exit nodes. Use residential proxies that are indistinguishable from real users. Render JavaScript. Pause between pages. Come back next week from a different address.

The more accurate your detection becomes, the more sophisticated the evasion becomes. That is not a problem with a solution. It is an arms race with no terminal state. Every improvement in detection is also a specification for the next generation of evasion. You are, in effect, publishing your own vulnerability research every time you tighten your defences.

The nasty part is the collateral damage. Legitimate users get caught. CDNs, VPNs, corporate proxies, Tor exit nodes, shared hosting — all of these produce traffic that a naive detection system misidentifies as hostile. The harder you push the detection threshold, the more real visitors you block. The scraper does not care. The person who just wanted to read an article does.

Poisoning: An Elegant Idea With a Ceiling

Data poisoning takes a different approach. Rather than preventing the crawler from accessing your content, you corrupt what it finds. Feed it misleading information. Subtly wrong facts. Plausible but false associations. Contradictory claims that look legitimate. If enough poisoned material enters a training dataset, the theory goes, the resulting model becomes less reliable.

The theory is sound in principle. The practice runs into a significant obstacle: AI companies already know the internet contains rubbish.

They have strong economic incentives to filter it. They deduplicate. They score sources by domain authority and historical reliability. They detect anomalies by comparing claims across multiple independent sources. They weight information from established references more heavily than information from unknown domains. They actively research adversarial content injection and build pipelines to resist it.

For poisoning to cause serious damage, it would need to be sufficiently widespread and convincing to evade all of that. At which point the obvious response is to build better filters. Then better poison. Then better filters. Another arms race, with the additional problem that you have no reliable way to measure whether your poison is having any effect at all. You could spend significant effort producing content that is simply ignored.

Tarpits: Economically Interesting, Temporarily

A tarpit is more mechanically interesting than either of the above. The principle is to waste the crawler's resources rather than block it or mislead it. Generate thousands of pages. Link them together endlessly. Make every page appear legitimate. Make crawling the site expensive in time, bandwidth and compute.

This has economic logic behind it. If harvesting a website costs less than the value of the data it produces, the crawler will harvest it. If the cost exceeds the value, it may not. Making crawling expensive is a legitimate lever.

The crawler can adapt. It can learn to measure the information density of pages before committing resources to them. It can detect repetitive link structures. It can identify domains that consistently produce low-value content and deprioritise them. It can set a time budget per domain and abandon sites that exceed it. A sufficiently sophisticated tarpit teaches the crawler which patterns to avoid, which is useful information for the crawler's operator and a cost you have paid to provide it.

The Fundamental Problem

Underneath all of the specific techniques is a problem that none of them resolve.

If a human can access your content without authentication, a machine can potentially access it too. The machine does not need to understand the content in the way a human does. It only needs to retrieve the bytes. And if your server is configured to send those bytes to an unauthenticated request, the fundamental question of access has already been answered.

You can raise the cost. You can increase the difficulty. You can introduce friction that filters out low-effort scrapers. But if the requirement is that humans may read this information while machines must never obtain it — and the information is publicly accessible — you are working against the design of the network rather than with it. The network was built to move bytes. It does that very effectively regardless of who is asking.

Authentication and subscription access are the only technically meaningful controls. If content is behind a login that a machine cannot obtain, the machine cannot access it. Everything else is friction, not a barrier.

What Is Actually Worth Doing

None of this means anti-scraping measures are universally useless. The argument needs to be made more carefully than that.

Rate limiting is legitimate infrastructure management. If a crawler is hammering your server with ten thousand requests per minute, you are not having a philosophical debate about data rights. You are having a performance incident. Rate limiting solves that problem irrespective of whether the traffic is a scraper, a misconfigured client or a DDoS.

Bot detection that blocks crawlers which voluntarily identify themselves is sensible and low-cost. If a crawler announces itself in its user-agent and you have decided you do not want it on your site, blocking it is straightforward. The crawlers that are genuinely problematic are precisely the ones that will not announce themselves, but blocking the cooperative ones is still reasonable housekeeping.

Authentication is the only reliable technical control for genuinely sensitive content. If the data has real commercial value, putting it behind a login is not optional. robots.txt is not a substitute.

The questionable part is the belief that you can build an impregnable wall around publicly accessible information. You almost certainly cannot. And attempting to do so at scale — WAFs, CAPTCHA systems, fingerprinting infrastructure, engineering time — has costs that need to be weighed honestly against the damage the crawler is actually causing.

The Question That Matters

The internet has always had this argument. Search engines indexed pages without asking. Aggregators republished headlines. Scrapers harvested prices. Social platforms extracted links. Each time, the owner said: that is my data. Each time, the other side said: it was publicly available. Each time, engineers built walls and other engineers built ladders.

AI training data is the current version of that argument. The technology is new. The structure of the dispute is not.

The useful question is not how to stop AI scraping your website. The useful question is what it is worth spending to try. If the engineering time, the infrastructure cost, the false positives and the lost legitimate visitors add up to more than the damage the crawler is causing, then something has gone wrong with the priorities.

There is also a harder question underneath that one. If an AI can reproduce everything on your website accurately, the problem may not ultimately be that it scraped the content. The problem may be that the content contained nothing that could not be reproduced. Making something genuinely irreplaceable is a more durable defence than any technical measure. It is also considerably more difficult.

Blocking a crawler will not fix that.