Hi everyone, this is Neo.

Over the past year, dozens of site owners have asked me the same question: “Neo, AI crawlers hit my site every day — no traffic back, just bandwidth burned. Should I block them all?”

My answer has always been: don’t rush to block everything. First, figure out who you’re blocking, at which layer, and what it costs you.

Right on cue, Search Engine Journal’s “Ask An SEO” column just published a systematic piece titled Should I Block AI Crawlers At Robots.txt Or Server Level? — it lays out the full math. Combined with Cloudflare’s new AI crawler classification policy that landed in 2026, now is the perfect moment to get this right.

In this post I’ll cover the whole thing: the fundamental difference between the two blocking approaches, Cloudflare’s three-category system, and a copy-paste-ready decision checklist with configuration code.

First, Understand the Basics: Robots.txt Is a “Gentleman’s Agreement.” Server-Level Blocking Is a Physical Lock.

There are only two technical routes for blocking AI crawlers, and understanding the difference is the foundation of every decision.

Route A: robots.txt — a polite “no trespassing” sign.

The concept is dead simple: in the robots.txt file at your site’s root, add a Disallow rule per crawler name (User-agent). For example, to stop OpenAI’s GPTBot from crawling your site:

User-agent: GPTBot
Disallow: /

Or block only certain directories:

User-agent: GPTBot
Disallow: /products/

The major AI companies — OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot, Google’s Google-Extended, Perplexity’s PerplexityBot — all officially state they respect the robots.txt protocol.

Route B: server/CDN/WAF — actually locking the gate.

At the server, CDN, or WAF layer, you evaluate incoming requests (by IP, User-Agent, behavior) and deny, allow, or redirect specific crawlers:

• Server level: reads the request and applies your rules (e.g., deny GPTBot).
• CDN level: intercepts the bot before it ever reaches your server — best for saving bandwidth, since the bot never touches your origin. Cloudflare and others offer preset AI bot blocking toggles.
• WAF level: the strictest. It doesn’t just read request headers — it analyzes request behavior, so it can catch sophisticated crawlers that spoof other User-Agents to slip under the radar. Cloudflare WAF and AWS WAF fall in this category.

Here’s the one-line summary of the fundamental difference, and I love SEJ’s analogy:

Robots.txt is a “no trespassing” sign on the fence; server/CDN/WAF blocking is a padlock on the gate. The sign relies on crawlers choosing to comply; the lock doesn’t need their cooperation — it detects and stops them, whether they like it or not.

Full Pros and Cons of Both Approaches

Robots.txt pros:

1. Most accessible. Any SEO-savvy person can edit it without waiting on developers.
2. Officially supported by major AI companies. GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc., all say they honor it.
3. Fine-grained. You can target specific crawlers, specific directories, and adjust anytime.

Robots.txt cons:

1. Compliance is voluntary. Robots.txt is a request, not a block — OpenAI itself has acknowledged its fetch bots may not strictly follow it. If a crawler ignores the rules, you have no recourse.
2. Easy to break by accident. CMS robots.txt controls are foolproofed for non-technical users, but one slip can produce a catastrophic “block everything” rule that also locks out Google and other legitimate search engines:

User-agent: *
Disallow: /

3. Doesn’t update itself. With 170+ AI crawlers now in existence, every new bot requires a manual rule — the maintenance burden keeps growing.

Server/CDN/WAF pros:

1. It’s a real block. No crawler “cooperation” needed — detected means stopped, physically.
2. Saves bandwidth. CDN-level interception happens before requests hit your origin, so crawlers consume none of your server resources.
3. Logs and data. Every blocked attempt is recorded — you can see exactly who tried to crawl your site and when. Those records are evidence if you ever need to take the issue up with the bot’s operator (including legally).

Server/CDN/WAF cons:

1. High maintenance threshold. Changes generally require a developer — especially if a third party manages your server. Every adjustment costs time, resources, and money.
2. No perfect defense. Even the best WAF can be bypassed by advanced bots — it’s a probability game. “Completely block all AI crawlers” doesn’t exist in practice; “block the vast majority” does.

The Biggest Change of 2026: Cloudflare Splits AI Bots Into Three Categories

The theory above is old news. The biggest variable in 2026 is Cloudflare’s new classification system. In July, Cloudflare split AI-related traffic into three categories, each independently configurable to allow or block:

Category	What it is	Typical crawlers	What it’s worth
Search	Crawls your content to “answer questions later,” and sends referral traffic back	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Usually allow — this is what gets you cited in AI search
Agent	Automated actions taken in real time on a person’s behalf (chat assistants fetching live pages, browser-use agents)	ChatGPT-User, Gemini/Claude driving a browser	Most controversial — could be “a prospect browsing you” or “freeloading”
Training	Crawls your content to train or fine-tune models; content is absorbed permanently	GPTBot, ClaudeBot, Google-Extended	The top candidate for blocking — takes without giving back

Here’s the trap within the trap — read this twice: Cloudflare explicitly states that multi-purpose crawlers like Googlebot, Applebot, and BingBot (which both index for search and may be used for training) are judged by all of their behaviors. That means if your site blocks the Training category, Googlebot gets blocked along with it — unless you separately configure Search to allow. This is the #1 reason sellers mysteriously lose Google indexing after “blocking AI crawlers.”

The default rules starting September 15, 2026 (confirmed on Cloudflare’s official blog):

From September 15, all domains newly onboarded to Cloudflare will have Training and Agent blocked by default on pages displaying ads, while Search remains allowed by default. Cloudflare’s logic: an ad is a signal that the site owner wants humans to land there and generate revenue — so it allows the category that sends traffic (Search) and blocks the categories that only consume (Training and Agent).

Cloudflare’s background data is sobering: as of June 2026, AI training crawlers made up 52% of all crawler requests (up from 22% in spring 2025), and automated bot traffic passed human traffic long ago. Training crawlers have an abysmal give-to-take ratio — one analysis shows ClaudeBot crawls over 11,000 pages for every referral it sends back, while Googlebot’s ratio is roughly 5:1. That’s the evidence behind “training crawlers deserve blocking most.”

One side note: Cloudflare has been pushing its “Pay Per Crawl” marketplace since 2025 — the idea being to turn “AI using your content” into “AI paying for your content.” For content-driven independent sites, this may be a smarter long-term option than blanket blocking, but it’s still early days.

Don’t Rush to Block: The Three AI Traffic Types Have Completely Different Value

Enough mechanics — back to the decision. Most people’s mistake is treating all AI crawlers as one monolith. Using the three-category framework, their business value is completely different:

Category one: AI search crawlers (Search) — this is your future Googlebot. Don’t block it.

OAI-SearchBot, Claude-SearchBot, and PerplexityBot crawl your pages so you appear in the citations of ChatGPT, Perplexity, and other AI search engines. This is already a traffic channel independent sites can’t ignore. And here’s the data: 88.2% of sites that block GPTBot still appear in AI answers (because AI search uses a separate fetching pipeline). But if you block the Search category along with everything else, you’re personally deleting yourself from AI search results.

One-liner: you can hate “being trained on,” but don’t refuse “being searched.” Training is freeloading off your past; search is your road to the future.

Category two: training crawlers (Training) — the purest freeloaders. Block them first.

GPTBot, ClaudeBot, Google-Extended… they permanently absorb your content into model weights, returning neither traffic nor money. For content-driven sites (blogs, tutorials, resource hubs), these crawlers are pure loss. Unless you deliberately want your content in a model’s training corpus (some site owners treat it as a brand exposure strategy — your call), Training is the category to block.

Category three: agents (Agent) — the most agonizing choice. Observe first.

Real-time agent fetches (like ChatGPT-User) may happen because a human user is talking to ChatGPT and it’s checking your page live on their behalf. In that moment, it may be your prospective customer. A blanket block is closing the door before the customer walks in. But if malicious agents are hammering you (scraping APIs, racing inventory, fake orders), that’s an attack. My advice: watch your logs for a while first. If your agent traffic is mostly real conversational fetches, allow it; if it’s mostly abnormal high-frequency calls, add a targeted WAF rule.

So which blocking method should you choose? I largely agree with SEJ’s recommendation — here’s the priority:

1. Blocking just one or two mainstream AI crawlers (e.g., only GPTBot training) → start with robots.txt. Cheapest.
2. Clear content-protection needs → go as high up the stack as you can: WAF first, then CDN, then server — the earlier you block, the more you save.
3. On Cloudflare → use Security → Bots → AI Crawl Control in the dashboard and configure the three categories separately. Far less work than hand-writing rules, and it’s on the free plan.
4. Whatever method you choose, keep logging on — periodically check whether bots your robots.txt supposedly blocked are still actually visiting. If they are, they’re ignoring the rules, and it’s time to escalate to server-level blocking.

Copy-Paste Configuration: Block Training, Allow Search

On Cloudflare, just configure the three categories in the dashboard. If you manage robots.txt and your server yourself, here’s a reference config that blocks training crawlers while allowing AI search crawlers:

# Allow all regular crawlers (keep defaults)
User-agent: *
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /

# Allow AI search/citation crawlers (so AI search can cite you)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

Watch the Google-Extended detail above: Google keeps “training use” (Google-Extended) and “search indexing” (Googlebot) as separate crawlers, so blocking Google-Extended does not affect your Google rankings. The real danger is Cloudflare’s multi-purpose classification (the trap in section three), where you need to confirm Search stays allowed.

If you need Nginx server-level blocking (for crawlers that won’t behave), the core idea is matching User-Agents in a server block and returning 403. I’ll skip the full config here — comment below if you want it and I’ll write a dedicated Nginx walkthrough.

Neo’s Take: How Should Independent Site Sellers Decide?

Here’s my verdict, broken down by site type — find yours:

Content/blog sites (living off ads, indexing, and AI citations): You can block training crawlers, but always allow Search — AI search citations are becoming these sites’ second growth curve. Also keep a serious eye on Cloudflare’s Pay Per Crawl model; it may be worth more than “blocking” in the long run.

E-commerce/product sites (living off real buyers): Be careful with Agent traffic — it contains genuine purchase intent. Block only Training; allow Search and Agent initially, then adjust after a month of log analysis. Your core goal is “keep my product copy out of training sets,” not “keep everyone out of my store.”

Sites whose moat is proprietary content (tutorials, data, exclusive resources): Max out your defenses: robots.txt for training + WAF-level blocking + close monitoring. Once content like this gets absorbed at scale, the moat is gone — physical-layer protection is worth the cost.

Whatever your site type, remember three principles:

First, no blanket blocking. “Block all AI” sounds cathartic, but the price is disappearing from AI search and AI recommendations simultaneously. Separate Search / Agent / Training before you act.

Second, soft first, hard second. Start with robots.txt (zero cost, reversible), then escalate misbehaving crawlers to the server layer. Don’t overcomplicate your server config on day one — robots.txt handles 90% of what you need.

Third, blocking should leave evidence. Server-level logs are your arsenal — if a crawler ignores the rules and barges in anyway, those logs are your leverage for negotiation or even legal action.

One honest thought to close: in 2026, “AI reading your content” isn’t the scary part. The scary part is “AI freeloading your content while your brand has zero presence in AI.” Block training crawlers to protect your content’s value; allow search crawlers so AI mentions you in its answers. One defends, one attacks — you need both hands.

Figure out that you’re blocking “training,” not “search,” and your blocking strategy is half done.

I’m Neo, and I write about independent site SEO. If you’ve set this up, come back and tell me what monsters you found in your block logs.