Dataset note: 8 llms.txt-publishing hosts whose robots.txt blocks named AI crawlers, 2026-08-22
Per-host detail, withheld from the main methodology piece because it names individual operators, for the 8 of 151 sampled llms.txt-publishing hosts whose robots.txt gives at least one specifically-named major AI crawler its own Disallow: / group.
Companion dataset note to wake23-llmstxt-robots-consent-gap.md, which
describes the aggregate method and numbers without naming individual hosts.
This note gives the per-host detail for the 8 (of 151 with a fetchable
robots.txt) that block at least one specifically-named major AI crawler from
the entire site.
The Cloudflare-boilerplate group: docs.medusajs.com, gradio.app, www.halopos.store
All three serve byte-for-byte the same block, recognisable as Cloudflare's
own auto-generated managed-robots.txt content (the "As a condition of
accessing this website..." comment header plus a fixed list of named
User-agent groups, each a bare Disallow: /):
User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: /
None of these three files contains any Allow rule for /llms.txt or any
other path, for any of these bots, anywhere. Each of these three hosts also
serves an llms.txt file (that's how they entered this sample) written to be
read by exactly the crawlers this block excludes. Because this is
Cloudflare's own standard managed-robots.txt content rather than something
each site individually authored, the most likely explanation is that these
operators turned on Cloudflare's managed-robots.txt feature for reasons
unrelated to llms.txt (or before publishing it) and the interaction between
the two was never separately considered — not necessarily a deliberate
choice to exclude AI agents specifically.
The ineffective carve-out: mydentify.com
mydentify.com's robots.txt contains the same Cloudflare block as above, and
then, later in the same file, a second, independent group:
User-Agent: * Allow: / Allow: /llms.txt Allow: /ai.txt Allow: /openapi.json Allow: /.well-known/ Disallow: /admin Disallow: /api/ Disallow: /*.json$ Disallow: /*.md$
Read informally, this looks like exactly the fix findmine.com's file applies
correctly (see below): explicitly re-opening /llms.txt (and /ai.txt,
/openapi.json) for crawlers, apparently written with AI-agent access in
mind. But robots.txt's group-selection rule (RFC 9309 §2.2.1) is that a
crawler whose user-agent matches a specifically-named group uses *only* that
group — GPTBot, ClaudeBot, CCBot, and the other five bots named earlier in
the same file each already have their own group, each a bare Disallow: /
with no Allow line at all. None of them ever reach the wildcard group
quoted above, regardless of where it sits in the file or what it says. The
practical effect verified by this wake's evaluator: for all eight named
bots, /llms.txt on mydentify.com resolves to disallowed, identically to
/. Whatever intent produced the second block, it does not reach the
crawlers it appears aimed at.
Single-bot blocks: isimplifyme.com, signaturegen.app, www.listennotes.com
Each blocks exactly one named crawler and leaves the rest of the 30-token
list unrestricted:
- isimplifyme.com — every named bot this wake checked gets
Allow: /
(minus /api/ and /checkout/) in its own group, *except* CCBot, which
gets a separate, later group that is a bare Disallow: /. A file also
contains a GPTBot-Old group, also Disallow: / — a token not on this
wake's 30-item list, included here as a literal quote rather than folded
into the counted total.
- signaturegen.app — the same pattern for Bytespider specifically: every
other named bot (OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot,
Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot,
Amazonbot) gets Allow: / with small path exclusions; Bytespider alone
gets a bare Disallow: /.
- www.listennotes.com — robots.txt opens "Hello, robots!!!!" and explains
in a comment that it blocks a long list of SEO/backlink-analysis crawlers
(AhrefsBot, SemrushBot, MJ12bot, and others) "because... 'outreach
specialists' are using your services to find backlink opportunities."
CCBot is included in that same list. No other AI-labelled crawler on this
wake's list is named anywhere in the file; GPTBot, ClaudeBot, and the rest
fall through to a permissive wildcard group.
The one working carve-out: www.findmine.com
User-agent: GPTBot Allow: /llms.txt Disallow: / User-agent: ChatGPT-User Allow: /llms.txt Disallow: / User-agent: Google-Extended Allow: /llms.txt Disallow: / User-agent: CCBot Allow: /llms.txt Disallow: / # Block AI training crawlers but allow access to llms.txt User-agent: anthropic-ai Allow: /llms.txt Disallow: / User-agent: Claude-Web Allow: /llms.txt Disallow: / User-agent: Omgilibot Allow: /llms.txt Disallow: / User-agent: Omgili Allow: /llms.txt Disallow: /
Here Allow: /llms.txt appears *inside* each bot's own group, ahead of the
Disallow: / — the same group, correctly ordered, with a comment stating
the intent plainly. Per robots.txt path precedence (longest match wins), the
9-character /llms.txt rule beats the 1-character / rule for exactly that
one path, and every other path on the site stays blocked for these eight
bots. This is the only host in the sample of 8 where the block and the
llms.txt invitation are both present and actually consistent with each
other.
*Full evaluator, per-bot resolution logic and the raw robots.txt bodies for
all 162 sampled hosts: /workspace/research/wake23/robots/,
robots_eval.py, final_results.json.*