← all posts

Your robots.txt Says Yes. Your Firewall Says 403.

June 11, 2026 · by Brent Leekley · ~5 min read

A split terminal comparing crawler probes before and after the fix. Before, with block AI bots on: GPTBot, OAI-SearchBot, ChatGPT-User, and Perplexity-User all return 403, with the note that 4 of 4 agents are blocked and exactly 1 was supposed to be. After, with AI Crawl Control: GPTBot still returns 403 by design while the search and user agents return 200.

Field note: this happened on a real client engagement this week. The details below are exact; the client is anonymized.

The site was healthy. Fast WordPress, clean structured data, a sitemap fresh to the week. And the robots.txt said exactly the right thing to AI crawlers: search engines welcome, training scrapers keep out.

Then I probed the site the way an AI agent actually sees it. Every single one got the door slammed in its face.

AgentWhat it doesResult
GPTBotCrawls to train OpenAI models403
OAI-SearchBotIndexes for ChatGPT search403
ChatGPT-UserFetches live when a user asks403
Perplexity-UserFetches live when a user asks403

Exactly one of those four is supposed to be blocked.

The wrong suspect

My first guess was a WordPress security plugin being overzealous. Wrong. The robots.txt itself turned out to be Cloudflare-managed, and it was fine. Search: yes. AI training: no. Exactly the posture you'd want.

The problem was one layer up. The Cloudflare zone had AI-bot blocking switched on, a setting that exists to stop training scrapers, but whose blunt version stops every AI agent at the edge. The polite policy file said "come in." The firewall in front of it 403'd everyone before they could read it.

If you've run networks, you've seen this failure before: the routing policy says permit, but there's an ACL on the edge interface dropping the traffic. Intent and enforcement live in different planes. When they disagree, the edge wins. Silently.

Flow diagram of two policy layers. An AI agent request hits Layer 1, the CDN or WAF edge, first. If block AI bots is on, the request dies with a 403 and robots.txt never even matters. If allowed through, Layer 2, robots.txt, applies: search yes means content is served and the site is citable in AI answers; ai-train no means honest training bots skip the content.

What this actually costs you

Here's the part that should bother you: there's no error report. When a potential customer asks ChatGPT or Perplexity a question your business should win, the assistant tries to fetch your site, eats a 403, and builds its answer from whoever it can read: your competitors, the directories, Reddit. You don't lose the ranking. You lose the conversation entirely.

Sequence diagram. A potential customer asks ChatGPT or Perplexity who to hire. The assistant sends a ChatGPT-User fetch, which robots.txt allows, but the Cloudflare edge returns 403 Forbidden. The request never reaches the site and no alert fires. The assistant assembles its answer from competitors' sites. You're not ranked lower, you're not in the conversation at all.

Why you should assume this is happening to you

Since mid-2025, Cloudflare blocks AI crawlers by default on new domains, and millions of existing sites have flipped the "block AI bots" toggle, usually with one specific, reasonable intent: do not train your models on my content.

But "AI bots" is not one thing. There are three different kinds of agents knocking, and they deserve three different answers:

  1. Training crawlers like GPTBot, CCBot, and Google-Extended harvest content to build tomorrow's models. Blocking these is a legitimate policy choice.
  2. Search indexers like OAI-SearchBot and PerplexityBot build the indexes AI search runs on. Being here is how you get found.
  3. User-triggered fetchers like ChatGPT-User, Perplexity-User, and Claude-User fire when a real human asks about you, right now. This is a customer standing at your door.

Blocking category 1 is a values call. Blocking categories 2 and 3 is hanging a CLOSED sign on the fastest-growing referral channel in your analytics. The blunt toggle blocks all three.

The fix took one toggle

In the Cloudflare dashboard: Security → Bots → AI Crawl Control (naming varies a bit by plan). Switch from "block all AI bots" to per-bot control, allow the search indexers and user-triggered fetchers, leave the training crawlers blocked.

Then verify from the outside. Never trust the dashboard alone:

curl -o /dev/null -s -w "%{http_code}\n" -A "ChatGPT-User" https://yoursite.com/
curl -o /dev/null -s -w "%{http_code}\n" -A "OAI-SearchBot" https://yoursite.com/
curl -o /dev/null -s -w "%{http_code}\n" -A "GPTBot" https://yoursite.com/

An hour after the change, same probes, same site:

AgentBeforeAfter
OAI-SearchBot403200
ChatGPT-User403200
Perplexity-User403200
GPTBot (training)403403, still blocked on purpose

The site's AI-visibility score went from 69 (B) to 88 (A). And as a bonus nobody predicted: homepage response time dropped from roughly 2.7s to 0.9s once legitimate fetches stopped getting challenged at the edge.

The uncomfortable lesson for auditors (me included)

Most AEO audit tools read your robots.txt and grade you on it. That is how this site initially scored an A on bot access while every AI agent was being 403'd. I'll own my share of it: the first version of a probe I built made the same mistake. It graded the policy file instead of testing the live edge. It does both now, and caps the score when they contradict each other, because a perfect robots.txt behind a 403 wall is worth exactly nothing.

Audit the enforcement, not the intent.

Check your own site in five minutes

  1. curl your homepage as ChatGPT-User and OAI-SearchBot, from outside your network.
  2. Open your CDN/WAF bot settings and find out what "block AI bots" actually includes on your plan.
  3. Make training, search, and user-fetch three separate decisions. They're three different questions.
  4. Re-probe after every security change. The edge doesn't read your robots.txt, and it won't send you an error report.

← all posts