AI Crawler Allowlist 2026: Which Bots to Let In, Block, or Ignore

Letting an AI crawler in is a trade, and it is worth stating in the terms an owner would use. You give away a copy of content you already publish for free. You get back the chance to be the source an assistant names when your customer asks it what to buy.

Whether that trade is good depends on the crawler, because they do four different jobs and only some of them pay you back. Below: what each bot actually does, a working robots.txt, the Cloudflare rules that enforce it, and how to check it holds.

The four jobs a crawler does

Before you allow or block, understand which job the bot is doing. One vendor often runs several crawlers, each for a different purpose: OpenAI documents four (OpenAI) and Anthropic three (Anthropic).

The mistake most sites make: blocking all four to "protect content" — which deletes the search-index and in-session-browse jobs at the same time. Those two are the jobs that put your page into an answer.

The bot identity table (May 2026)

A working robots.txt

For a B2B site that wants maximum citation eligibility and zero training contribution:

# Citation-grade — allow everything
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: Kagibot
Allow: /

User-agent: YouBot
Allow: /

# Training-only — block
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Hostile / no citation upside
User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: Webzio-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Flip the training lines (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) to Allow if you want to be in next-generation model training (most B2B brands benefit from this — it teaches the model your terminology). Keep them Disallow if you've published proprietary research you don't want absorbed. Google-Extended is a robots.txt token rather than a separate crawler, and Google says it does not affect inclusion or ranking in Google Search (Google).

Cloudflare enforcement

robots.txt is honored by compliant bots. Bytespider, scrape-as-a-service vendors, and residential-proxy scrapers ignore it. The 2026 enforcement layer is Cloudflare's AI Bot management dashboard (Security → Bots → AI Bots); the one-click AI block has been available on every plan, Free included, since July 2024 (Cloudflare, 2024):

1. Check what the blanket AI block covers. Cloudflare describes the original setting as blocking bots that crawl for AI training, and the newer presets split Search, Agent and Training (Cloudflare docs). Blocking all three takes the citing crawlers down with the scrapers.
2. Enable per-bot rules with these settings:
   • PerplexityBot, OAI-SearchBot, Claude-SearchBot: Allow
   • GPTBot, ClaudeBot, Applebot-Extended: Allow or Managed Challenge depending on training policy (Google-Extended has no user-agent of its own, so it lives in robots.txt only)
   • Bytespider, Amazonbot, Meta-ExternalAgent, FacebookBot: Block
3. Add a WAF custom rule matching (cf.client.bot eq false) and (http.user_agent contains "ai" or http.user_agent contains "bot") with action Managed Challenge to catch impersonators using user-agents that look AI-flavoured but aren't on the verified list.
4. Drop a honeytrap URL at /disallow-honeypot/ linked only from llms.txt, set rate-limit-per-IP to 1, action Block 24h. Catches scrapers that ignore robots.txt — by design no legitimate AI crawler fetches that path.

Verification

After you ship the policy, verify weekly:

• tail -f /var/log/nginx/access.log | grep -E "PerplexityBot|OAI-SearchBot|Claude-SearchBot" — confirm citation bots are still fetching.
• Cloudflare Analytics → Security → Bot traffic — chart Allowed/Challenged/Blocked over time. The training scrapers you blocked should show up in the Blocked line, and the citation bots should stay in Allowed.
• Perplexity citation share (queries that should cite you) — should start to rise in the weeks after allowing PerplexityBot if you weren't already cited.
• Google Search Console → Performance — should not move, since Google-Extended has no effect on Google Search.

If PerplexityBot fetches stop after rollout, your Cloudflare rule order is wrong (a broader Block rule is matching first). Reorder so per-bot Allows come before Challenge/Block. The downstream metric to watch is covered in citation tracking AEO metrics - if your share-of-cite is climbing, the allowlist is working.

Why the default block is wrong

Three of the four jobs do something you want: index your content so it gets cited, browse your live page when a user asks, train future models to know your brand exists. Only one — pure scrapers like Bytespider — does nothing useful for you.

The 2026 default ("block all AI") was right in 2024 when there were no citation pathways and the crawlers were taking content with no return. In 2026 the pathways exist (PerplexityBot citations now drive measurable referral traffic; Google AI Overview cites with link attribution; ChatGPT-User pulls live data into answers) and the asymmetry has reversed: blocking is the expensive choice.

A short robots.txt and a handful of Cloudflare rules are the difference between being cited in the answers your customers ask AI tools and being invisible. The work is small; the lift compounds for years. Pair the allowlist with a proper llms.txt file and you have the full identity surface AI crawlers expect to find.

One caveat on the traffic this buys you. A growing share of what arrives after a crawler reads you is not a person clicking through but software acting on someone's behalf, which is a different visitor with different needs — see what a site owes an automated visitor. And the identity file the same crawlers look for is covered in the llms.txt update.

Sources

• Do sites that block GPTBot get cited less by ChatGPT? — 1,058 domains, July 2026. The engine-specific citation figures and block rates above.
• Cloudflare press release, 1 July 2025 — AI crawlers blocked by default for new domains.
• Cloudflare — block AI bots — the Search, Agent and Training presets.
• OpenAI — crawlers and Anthropic — crawlers — what each bot does.
• Google — common crawlers — Google-Extended and Google Search.
• Common Crawl — CCBot — who operates it and what it does.