Hi everyone, this is Neo.

Lately, in a few cross-border e-commerce groups, everyone’s been discussing one technical move: how to block OpenAI’s crawler in robots.txt?

The common line of thinking is: “My content is hard-earned original work — why should AI get to train on it for free? Block it and protect my copyright!”

I completely understand the feeling. But a heavyweight report Hostinger just published, covering 66.7 billion crawler interactions, delivers the opposite warning: blindly blocking AI training crawlers could be one of the most dangerous decisions an independent site seller makes in the AI era.

Why? Because you might be wiping your own brand out of AI’s “brain” entirely.

Today, let’s dig into this report together, and figure out how to build a more forward-looking SEO strategy in the age of AI search (SearchGPT, Perplexity).

01 The Great Blockade: What the Data Really Shows

Hostinger analyzed backend logs from 5 million websites and found a strikingly divided picture:

1. Training bots are getting slaughtered:

   • OpenAI’s GPTBot (used to train models) dropped from 84% to 12% in just three months.
   • Meta’s ExternalAgent also fell from 60% to 41%.
   • Everyone is busy “closing the door” so AI can’t “learn” from their data.

2. Search bots, meanwhile, are taking off:

   • OpenAI’s SearchBot (used for real-time search answers) saw coverage rise from 52% to 68%.
   • Apple’s Applebot (used for Siri and Spotlight) more than doubled, reaching 34%.

Neo’s take: This reveals a clear trend: AI is shifting from “training mode” to “application mode.” On one hand, site owners refuse to be “trained”; on the other, they have no choice but to let AI assistants (like ChatGPT Search) crawl real-time information to answer users. Hidden in all this is a huge misconception — people are confusing “parametric knowledge” with “retrieval augmentation.”

02 The Fatal Mistake: Giving Up “Parametric Knowledge”

A lot of sellers think: “If I let SearchBot in, ChatGPT can find my site when it searches — why would I also let GPTBot in?”

Here we need a core concept: parametric knowledge.

• Parametric knowledge (long-term memory): knowledge the AI “memorized” during training, stored in its model weights. Mention “Nike,” for example, and the AI knows without searching that it’s “a sports brand, Just Do It, an American company.” It’s etched into its brain.
• RAG (retrieval-augmented generation, short-term memory): information the AI temporarily “searches” online when answering a question.

What happens if you block the training crawler (GPTBot)?

It means you’re voluntarily giving up your shot at AI’s “long-term memory.” When a user asks, “recommend a few tent brands that are good for outdoor camping,” if your brand isn’t in the AI’s parametric knowledge, it won’t even remember you.

The AI will only fetch your pages via RAG (search crawlers) when a user searches your brand name very precisely. But here’s the thing — in the AI era, users’ search habits are getting vaguer and vaguer (like “what are the best pet vacuums?”). If you’re not in the AI’s subconscious (parametric knowledge), you never even get a seat at the table.

As Hostinger’s analysis puts it: “By blocking training crawlers, companies are essentially opting out of the LLM’s parametric knowledge base — and giving up the ability to tell their own brand story.”

03 Losing Control of Your Brand Narrative

Imagine this: ChatGPT doesn’t know who you are (because you blocked training), but users still ask about your product (through third-party reviews or news). How will the AI answer?

It will scrape third-party reviews, Reddit complaints, or competitor comparison articles to stitch together an answer. The result:

1. You lose the official narrative: AI knows nothing about your carefully crafted “About Us,” brand vision, or core selling points.
2. Pricing and value get distorted: AI may quote outdated discounts or wrong competitor comparisons.
3. Trust drops: to the AI, you’re a “stranger,” not an entity it “knows.”

Tomas Rasymas, Hostinger’s head of AI, put it perfectly:

“The real risk isn’t AI access itself — it’s losing control over how your pricing, positioning, and value are presented when users make decisions.”

04 Neo’s Advice: From Total Blockade to Smart Openness

In 2026 and beyond, AEO (Answer Engine Optimization) will replace traditional SEO as the core.

We can’t throw out the baby with the bathwater. For different crawlers, my recommendation is a tiered management strategy:

1. Core brand assets: keep them open

• Page types: homepage, About Us, core product detail pages, brand story pages, FAQ.
• Strategy: allow all crawlers (including GPTBot).
• Goal: let AI “learn” your brand as much as possible, etching your brand name, core business, and unique selling points into its parametric knowledge. Make AI your “brand ambassador.”

2. High-value original/paid content: you can block it

• Page types: exclusive data reports, members-only articles, core algorithm logic, unique teaching content.
• Strategy: block training crawlers (GPTBot); allow search crawlers (SearchBot) as appropriate.
• Goal: protect core IP from being learned by AI for free and turned into direct substitutes (like that Reddit case — if you sell courses, once AI learns them, nobody buys the courses anymore).

3. Dynamic info/e-commerce pages: optimize them to the max

• Page types: product prices, inventory, promotions, news.
• Strategy: allow search crawlers (OAI-SearchBot, Applebot) and set up proper Schema structured data.
• Goal: this information changes fast — AI doesn’t need to memorize it (it goes stale quickly), it needs to fetch it in real time (RAG) and answer users accurately.

05 Summary

In an era where the traffic gateway is shifting from “search engines” to “AI assistants,” staying invisible is no longer protection — it’s suicide.

Don’t treat AI as a “thief” stealing your content. Think of it as a “super salesperson” with hundreds of millions of users. If you’re willing to spend money advertising on Google and Facebook so more people see you, why wouldn’t you let this “super salesperson” get to know your brand for free?

Action checklist:

1. Check your robots.txt — don’t blindly Disallow: /.
2. Treat GPTBot (training) and OAI-SearchBot (search) differently.
3. Write a better “About Us” — that’s your “self-introduction” written for AI.

I’m Neo — follow me to run independent sites with technical know-how in the AI era.