SEO

Why can't ChatGPT, Claude or Perplexity see your website? Check your hosting

A server with a shield that lets Googlebot through and stops AI assistant crawlers

In short

Often the problem isn't your content but the front door: the hosting's bot protection, a security plugin or the CDN answers ChatGPT, Claude or Perplexity crawlers with a 403 error, while Googlebot gets in without trouble. Check it by simulating those crawlers, look at the robots.txt your site actually serves, and allow at least the search crawlers in your security settings. Whether to let training crawlers in is a separate decision.

You ask ChatGPT about your own business and it replies with old details, or details from another site. You ask Claude or Perplexity to open your prices page and it says it can't access it. Your site shows up on Google, so you assume the content is the problem. Often it isn't: it's the front door.

Small business websites usually sit behind some kind of bot protection (from the hosting company, a plugin or a CDN). That protection lets Googlebot through but can turn away the crawlers used by AI assistants. From the outside you notice nothing: the site works for you and for Google.

It happened to us: a real case

On 8 November 2026 we checked our own site, web-sensei.com. The robots.txt file allowed everything. Even so, the per-site bot rules on our hosting returned a 403 (forbidden) error to GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. At the same time, Googlebot got a 200 (page served normally).

Worse, they couldn't read robots.txt or llms.txt either. They never even got to see that we had given them permission.

The fix had nothing to do with code: we allowed those crawlers in the bots section of the hosting control panel. After that change, all of them got a 200.

The lesson: robots.txt is a request; a firewall is a locked door. If the server answers 403, whatever robots.txt says doesn't matter.

Which crawlers each assistant uses

Not all AI bots do the same job. Each company separates the ones that search and answer from the ones that collect content to train models. This is what each company's official documentation says:

Company Crawler What it's for If you block it
OpenAI OAI-SearchBot Showing websites in ChatGPT search You don't appear in ChatGPT search answers (only as a navigational link)
OpenAI ChatGPT-User Visits ChatGPT makes when a user asks it to ChatGPT can't open your page for that user
OpenAI GPTBot Content that may be used to train models Your content shouldn't be used for training
Anthropic Claude-SearchBot Indexing content to improve search results Less visibility in Claude's search
Anthropic Claude-User Reading pages when a user asks Claude Claude can't retrieve your content to answer
Anthropic ClaudeBot Content that may contribute to training Your future content is excluded from training
Perplexity PerplexityBot Showing and linking websites in Perplexity results (not used for training) You don't appear in its results
Perplexity Perplexity-User Visits made at a user's request Perplexity can't open your page to answer
Google Google-Extended Use of your content to train Gemini and to ground its answers No effect on your inclusion or ranking in Google Search

Two useful details:

  • OpenAI says each setting is independent: you can block GPTBot and still allow OAI-SearchBot. It also notes that a robots.txt change can take about 24 hours to be reflected in its search.
  • ChatGPT-User and Perplexity-User act on a person's request, and their companies say robots.txt may not apply to them. A 403 from the server still stops them, though.

How to check it yourself in 10 minutes

Step 1. Look at the robots.txt your site really serves

Open https://yourwebsite.es/robots.txt in your browser. Look at what the public sees, not the file you think you uploaded: some hosting companies, content management systems and plugins generate or change this file themselves.

Look for blocks such as User-agent: GPTBot or User-agent: PerplexityBot followed by Disallow: /. Bear in mind that, according to Google, a crawler follows only the most specific group that names it: if there is a block for OAI-SearchBot, it overrides the general User-agent: * block.

Step 2. Simulate the crawlers

If you have a computer with a terminal (Mac, Linux, or Windows with curl), run:

for ua in Googlebot GPTBot OAI-SearchBot \
  ChatGPT-User ClaudeBot Claude-SearchBot \
  Claude-User PerplexityBot Perplexity-User
do
  printf "%-18s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" \
    -A "$ua" https://yourwebsite.es/
done

How to read the result:

  • 200: the page is served.
  • 403: forbidden. The clearest sign of a block.
  • 429 or 503: too many requests or a challenge page. Find out which protection produces it.
  • 301/302: a redirect; repeat the test with the final URL.

Repeat the test with /robots.txt and with an inner page, such as your prices page. If Googlebot gets 200 and the others get 403, the block is on the server or the CDN, not in your content.

An honest caveat: this test only imitates the bot's name. Some protections also check the IP address (Perplexity, for instance, recommends that site administrators verify both). So the result may not match exactly what the real bot receives. Treat it as a strong signal and confirm it in your control panel or logs.

Step 3. Ask an assistant to open your page

In ChatGPT, Claude or Perplexity, type: "Open https://yourwebsite.es/prices/ and tell me what prices it shows." If it says it can't access the page, or makes the details up, that's another signal. It isn't conclusive, because the assistant may rely on an old copy or another source.

Step 4. Check the server logs

In your hosting's statistics or access logs, search for these crawler names and the status code they receive. If you see GPTBot or PerplexityBot with 403, you have your confirmation.

Where the block usually is

Every hosting company names its settings differently, so search by concept:

Where to look What to look for
Hosting control panel Bot protection, firewall or suspicious IP settings, with per-site rules
Security plugin (WordPress or another CMS) Blocking of bots, user agents or countries
CDN (for example Cloudflare) AI crawler policies, bot fight mode, firewall (WAF) rules
.htaccess file or server configuration Rules that deny by user agent

In Cloudflare, the official documentation puts the AI crawler and agent policies under Security > Settings, filtered by bot traffic. You choose an action for three behaviours: search, agent and training. According to that documentation, all three are set to "allow" on new domains, but someone may have changed them. Perplexity also warns that Cloudflare's Bot Fight Mode can't be skipped with custom firewall rules.

If you don't have access to the panel, send whoever manages your site this sentence: "I need GPTBot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User to get a 200, not a 403. Which rule is blocking them?"

What your robots.txt should say

For a local business that wants to appear in search engines and assistants, the simplest option is to allow everything and list your sitemap:

User-agent: *
Allow: /

Sitemap: https://yourwebsite.es/sitemap.xml

If you'd rather your content wasn't used to train models but still want to appear in assistant search, add blocks just for the training crawlers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourwebsite.es/sitemap.xml

Note that, according to Google, Google-Extended covers both Gemini training and the use of your content to ground its answers. Blocking it doesn't remove you from Google Search, but it does opt you out of that second use.

Block training or not? Your decision

There's no single right answer. A practical way to decide:

If your situation is… A sensible option
A local business that wants to be found (garage, clinic, builder) Allow search and agents; training is up to you
Your value is original content (paid guides, writing you sell) Allow search and agents; block training
You don't want any AI on your site Block everything, accepting you won't appear in their answers

What you want to avoid is the most common case: blocking everything without knowing it, because of a default setting or a rule someone switched on years ago.

FAQ

If I block GPTBot, do I disappear from ChatGPT? Not necessarily. According to OpenAI, ChatGPT search depends on OAI-SearchBot, which is controlled separately.

Does allowing AI bots affect my Google ranking? Google says Google-Extended has no effect on inclusion or ranking in Search. What you do need to watch is not blocking Googlebot by mistake.

Is a correct robots.txt enough? No. If the server or CDN answers 403, the bot never even reads the file. Anthropic also advises against blocking its IP addresses instead of using robots.txt, because then it can't read your preferences.

How long until the change shows? OpenAI mentions about 24 hours for its search. The other companies don't publish a timeframe; repeat the tests the next day and check the logs over the week.

Checklist

  • I've opened yourwebsite.es/robots.txt and know what the served file says.
  • I've simulated the crawlers and no search crawler gets a 403.
  • robots.txt and llms.txt (if I have one) also return 200 to those crawlers.
  • I've checked the hosting's bot protection, the security plugin and the CDN.
  • I've made a conscious decision about GPTBot, ClaudeBot and Google-Extended.
  • Googlebot still gets a 200.
  • I've repeated the test the next day and checked the logs.

Unlocking the door is only the first step: what the assistants find then has to be clear and consistent. We explain that in how to get ChatGPT and other AI tools to recommend your business. For a local business, the other half is your Google profile: see our local visibility work.

If you'd rather we checked it for you, our AI visibility audit includes a review of crawler access (from €350 excl. VAT; details on our pricing page). Or start with the free short review.

Sources

You may also like