I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them

orth making. Fine. The problem is that GPTBot is not the only OpenAI token in your robots.txt, and the other two do completely different jobs: GPTBot — training corpus. Fetches content to train future models.

Sends you nothing. OAI-SearchBot — builds the search index behind ChatGPT's browsing. This is what makes your page eligible to be cited. ChatGPT-User — the fetch that happens while a person waits for an answer.

A citation here is a link a human can click. Block the first and you've opted out of training. Block all three and you've opted out of being found, which is a different decision that a lot of people appear to be making by accident. I wanted actual numbers rather than vibes, so: Frame: Tranco top-1M (the research-grade domain ranking), downloaded 2026-11-07.

Sample: the 100 highest-ranked domains per ccTLD — .com, .de, .fr, .es, .it, .jp, .nl. Deterministic, so anyone can reproduce it. Fetch: robots.txt over HTTPS directly, www. then bare, redirects followed, 12s timeout.

Verdict: for path /, using RFC 9309 group selection — the exact user-agent group if one exists, otherwise *, never both — with longest-match precedence and Allow winning ties. Read successfully: 528 of 700. The rest were unreachable (137) or served an HTML page where robots.txt should be (35). Measure Share Block GPTBot 30.5% Block OAI-SearchBot 16.5% Block ChatGPT-User 20.3% Block at least one traffic-bringing AI crawler 20.5% Block GPTBot but allow both traffic crawlers 10.2% Block an AI traffic crawler while still allowing Googlebot 17.2% Blanket-block Googlebot too 3.4% The line that matters: of the 161 sites blocking GPTBot, 107 — 66.5% — also block a traffic-bringing crawler.

Only 10.2% of the whole sample made the distinction and blocked training while keeping discovery open. And 17.2% block an AI crawler that could cite them while carefully keeping Googlebot allowed. Those sites clearly care about being found. They just don't appear to know that User-agent: * plus a Disallow catches a lot more than Google's competitors.

Market Read GPTBot Any traffic crawler Netherlands (.nl) 73 37.0% 31.5% France (.fr) 89 36.0% 25.8% English (.com) 64 34.4% 29.7% Italy (.it) 68 33.8% 16.2% Germany (.de) 81 30.9% 12.3% Spain (.es) 80 23.8% 15.0% Japan (.jp) 73 17.8% 13.7% Germany is the interesting one. German sites block GPTBot at nearly the same rate as .com (30.9% vs 34.4%) but block traffic crawlers at less than half the rate (12.3% vs 29.7%). That is a market making the deliberate version of the decision. Japan barely blocks anything.

The Netherlands is the opposite — the highest blocking rate in the sample on both counts. These are high-ranking domains. They are not small business websites, and the figures should not be read as a picture of the small-business web. Large publishers have legal teams with opinions about training data; a plumber's site has whatever the theme shipped.

A second sample of lower-ranked domains would be needed before claiming anything about the long tail — I haven't run it yet. When I started checking individual sites I kept hitting the same thing: the owner had no idea their file blocked what it blocks. Not because they misread it, but because robots.txt precedence is genuinely unintuitive: User-agent: * Disallow: / User-agent: Googlebot Allow: / A lot of people read that as "block everything, but let Google in." It is — but it also blocks every AI crawler, including the two that could send readers. There's no error, no warning, nothing in Search Console.

The file is doing exactly what it says and not at all what was meant. Worse, the groups don't combine. Under RFC 9309 a crawler obeys the group matching its own token, and falls back to * only if no such group exists. So: User-agent: Google-Extended Disallow: / ...does nothing whatsoever to Googlebot — different token, different group.

And a named empty group beats a restrictive wildcard: User-agent: * Disallow: / User-agent: GPTBot That allows GPTBot everything. The group exists, it has no rules, so nothing is disallowed. People write this believing it's a block. Every number above came out of a matcher I needed anyway, so it's on GitHub under MIT: npx github:orenhacham-lab/ai-crawler-access nytimes.com Zero dependencies, no build step, one network request.

For every crawler it prints the verdict and the exact line that produced it — which turned out to be the only part people actually find convincing, because "your file blocks ChatGPT" gets argued with and "line 14, Disallow: /, in the * group that applies to you because you have no OAI-SearchBot group" does not. As a library: import { checkRobots, summarize } from 'ai-crawler-access' const result = await checkRobots('example.com') console.log(summarize(result)) // training vs discovery, separated The crawler dataset is separate and CC BY 4.0, as JSON and CSV. Every row's token spelling, purpose and robots.txt behaviour comes from the operator's own documentation, with the URL and the date it was read in the row — nothing inferred from log files. Tokens I've seen in the wild but can't find documentation for are listed separately as candidates, because "we saw this" and "the operator says this" are not the same claim and shouldn't sit in one table.

Code: https://github.com/orenhacham-lab/ai-crawler-access Dataset: https://www.gotopseo.com/api/public/ai-crawlers (no key, no signup, CORS open) Hosted checker: https://www.gotopseo.com/en/ai-crawlers Corrections to the dataset are the contribution I want most — particularly an operator doc page for a token currently sitting in candidates. If you want out of training but still want to be cited, the file is explicit and boring: User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / Named groups for every token you have an opinion about. Don't rely on * to express a nuanced position — it can't, and the 66.5% figure above is what happens when people try. Two caveats worth saying out loud.

Google-Extended and Apple's equivalent are control tokens that crawl nothing by themselves; disallowing them changes how already-crawled content may be used, and both operators document that it does not remove you from search results. And a handful of agents have operators who document that robots.txt may be ignored — for those, your file states a request, not an outcome.