Warning: You Might Be Losing 55% of Your AI Search Traffic — Neo's Deep Dive on the New AI Crawler Landscape

Hi everyone, this is Neo.

Lately, the e-commerce community has been buzzing about one question: “AI crawlers are going crazy and crashing my server — should I just block them all?”

My usual advice is: calm down, don’t throw out the goose that lays golden eggs along with the thieves.

The latest data is proving me right. Hostinger just published a major research report that reveals a striking trend: site owners are mass-blocking OpenAI’s training bots, while quietly opening their doors to AI search bots.

What’s the logic behind this? As a seller going global from China, how do we win this “AI data battle” — protecting our intellectual property without missing out on the AI-era traffic dividend?

Today, Neo is going to walk you through this report and give you some actionable advice.

1. What the Data Says: A Cliff Dive from 84% to 12%

Hostinger analyzed over 66 billion requests across more than 5 million websites and found a dramatic split.

Training Bots (The Takers) Are Being Mass-Blocked

OpenAI’s GPTBot (which mainly crawls data to train ChatGPT models) saw its coverage plummet from 84% to just 12%.

That means the vast majority of site owners have realized: giving my data away for free to train AI models does nothing for me — I even foot the server bandwidth bill. So everyone has been busy writing block rules in their robots.txt.

Search Bots (The Givers) Are on the Rise

Meanwhile, OpenAI’s OAI-SearchBot (which powers ChatGPT’s real-time search) has climbed to 55.67% coverage.

Neo’s take: This contrast is fascinating. It shows web admins around the world are getting savvier. People aren’t blindly afraid of AI anymore — they’re deciding based on value exchange:

  • You want my data to train your model (GPTBot)? No way — that’s freeloading.
  • You want my data to answer users’ questions and credit my links (OAI-SearchBot)? Sure — that’s a partnership.

2. Why You Must Tell These Two Bots Apart

A lot of sellers habitually write User-agent: * Disallow: / in their robots.txt, or block every User-Agent that contains “GPT.”

In the LLM era, that could be an expensive mistake.

Training Bots

  • Who they are: GPTBot, ClaudeBot, CCBot (Common Crawl).
  • What they do: Devour your site’s content in huge gulps and store it in their databases.
  • The consequence: Your content becomes part of their model. Users ask ChatGPT a question, the AI answers using your knowledge — but it usually doesn’t hand out a link back to your site.
  • Neo’s advice: Block them firmly. Unless you’re Wikipedia doing public service, there’s no reason to donate your exclusive content.

Assistant/Search Bots

  • Who they are: OAI-SearchBot, Applebot, Bingbot.
  • What they do: When a user searches a related topic on SearchGPT or ChatGPT, they crawl your pages in real time.
  • The consequence: The AI summarizes your content and cites your links in the answer.
  • Neo’s advice: Let them in. This is the future of SEO (what we call AEO — AI Engine Optimization). Block them, and you’re invisible in the AI search world.

3. A “Smart Gatekeeping” Strategy for E-commerce Sellers

Hostinger’s data tells us the middle path is the best strategy right now: block training, allow search.

Now that we know the theory, how do we actually do it? Neo has prepared a robots.txt template you can copy and adapt to your own situation.

# 1. Block OpenAI's model training crawler (no freeloading)
User-agent: GPTBot
Disallow: /

# 2. Block Anthropic's (Claude) crawler
User-agent: ClaudeBot
Disallow: /

# 3. Block Common Crawl (many AI models train on its data)
User-agent: CCBot
Disallow: /

# 4. Allow OpenAI's search crawler (get SearchGPT traffic)
User-agent: OAI-SearchBot
Allow: /

# 5. Allow Google's crawler (keep traditional SEO working)
User-agent: Googlebot
Allow: /

Neo’s take: The core logic here is Block Training, Allow Search.

  • For OAI-SearchBot, OpenAI’s official docs state it controls whether your content appears in ChatGPT search results.
  • For GPTBot, it’s only used for training. Blocking it won’t affect your rankings in search results.

4. Summary: Embrace Change, But Hold Your Ground

In the AI era, the traffic gateway is shifting from the Google search box to the ChatGPT dialog.

Hostinger’s report isn’t just a pile of cold numbers — it represents a new contract for the internet: content creators are no longer donating their blood for free; they’re demanding fair traffic in return.

As e-commerce site operators, we don’t need to panic, and we don’t need to fight. What we need to do is:

  1. Protect your assets: Block training crawlers like GPTBot to safeguard your original images, descriptions, and blog content.
  2. Embrace the traffic: Open up to OAI-SearchBot, optimize your structured data, and make it easy for AI to understand you — so you carve out a place in the future of AI search.

The future is already here. Don’t let your site become an “information island” in the AI age.

References: