Why can't ChatGPT, Claude or Perplexity see your website? Check your hosting

In short
Often the problem isn't your content but the front door: the hosting's bot protection, a security plugin or the CDN answers ChatGPT, Claude or Perplexity crawlers with a 403 error, while Googlebot gets in without trouble. Check it by simulating those crawlers, look at the robots.txt your site actually serves, and allow at least the search crawlers in your security settings. Whether to let training crawlers in is a separate decision.
You ask ChatGPT about your own business and it replies with old details, or details from another site. You ask Claude or Perplexity to open your prices page and it says it can't access it. Your site shows up on Google, so you assume the content is the problem. Often it isn't: it's the front door.
Small business websites usually sit behind some kind of bot protection (from the hosting company, a plugin or a CDN). That protection lets Googlebot through but can turn away the crawlers used by AI assistants. From the outside you notice nothing: the site works for you and for Google.
It happened to us: a real case
On 8 November 2026 we checked our own site, web-sensei.com. The robots.txt file allowed everything. Even so, the per-site bot rules on our hosting returned a 403 (forbidden) error to GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. At the same time, Googlebot got a 200 (page served normally).
Worse, they couldn't read robots.txt or llms.txt either. They never even got to see that we had given them permission.
The fix had nothing to do with code: we allowed those crawlers in the bots section of the hosting control panel. After that change, all of them got a 200.
The lesson: robots.txt is a request; a firewall is a locked door. If the server answers 403, whatever robots.txt says doesn't matter.
Which crawlers each assistant uses
Not all AI bots do the same job. Each company separates the ones that search and answer from the ones that collect content to train models. This is what each company's official documentation says:
| Company | Crawler | What it's for | If you block it |
|---|---|---|---|
| OpenAI | OAI-SearchBot |
Showing websites in ChatGPT search | You don't appear in ChatGPT search answers (only as a navigational link) |
| OpenAI | ChatGPT-User |
Visits ChatGPT makes when a user asks it to | ChatGPT can't open your page for that user |
| OpenAI | GPTBot |
Content that may be used to train models | Your content shouldn't be used for training |
| Anthropic | Claude-SearchBot |
Indexing content to improve search results | Less visibility in Claude's search |
| Anthropic | Claude-User |
Reading pages when a user asks Claude | Claude can't retrieve your content to answer |
| Anthropic | ClaudeBot |
Content that may contribute to training | Your future content is excluded from training |
| Perplexity | PerplexityBot |
Showing and linking websites in Perplexity results (not used for training) | You don't appear in its results |
| Perplexity | Perplexity-User |
Visits made at a user's request | Perplexity can't open your page to answer |
Google-Extended |
Use of your content to train Gemini and to ground its answers | No effect on your inclusion or ranking in Google Search |
Two useful details:
- OpenAI says each setting is independent: you can block
GPTBotand still allowOAI-SearchBot. It also notes that arobots.txtchange can take about 24 hours to be reflected in its search. ChatGPT-UserandPerplexity-Useract on a person's request, and their companies sayrobots.txtmay not apply to them. A 403 from the server still stops them, though.
How to check it yourself in 10 minutes
Step 1. Look at the robots.txt your site really serves
Open https://yourwebsite.es/robots.txt in your browser. Look at what the public sees, not the file you think you uploaded: some hosting companies, content management systems and plugins generate or change this file themselves.
Look for blocks such as User-agent: GPTBot or User-agent: PerplexityBot followed by Disallow: /. Bear in mind that, according to Google, a crawler follows only the most specific group that names it: if there is a block for OAI-SearchBot, it overrides the general User-agent: * block.
Step 2. Simulate the crawlers
If you have a computer with a terminal (Mac, Linux, or Windows with curl), run:
for ua in Googlebot GPTBot OAI-SearchBot \
ChatGPT-User ClaudeBot Claude-SearchBot \
Claude-User PerplexityBot Perplexity-User
do
printf "%-18s " "$ua"
curl -s -o /dev/null -w "%{http_code}\n" \
-A "$ua" https://yourwebsite.es/
done
How to read the result:
- 200: the page is served.
- 403: forbidden. The clearest sign of a block.
- 429 or 503: too many requests or a challenge page. Find out which protection produces it.
- 301/302: a redirect; repeat the test with the final URL.
Repeat the test with /robots.txt and with an inner page, such as your prices page. If Googlebot gets 200 and the others get 403, the block is on the server or the CDN, not in your content.
An honest caveat: this test only imitates the bot's name. Some protections also check the IP address (Perplexity, for instance, recommends that site administrators verify both). So the result may not match exactly what the real bot receives. Treat it as a strong signal and confirm it in your control panel or logs.
Step 3. Ask an assistant to open your page
In ChatGPT, Claude or Perplexity, type: "Open https://yourwebsite.es/prices/ and tell me what prices it shows." If it says it can't access the page, or makes the details up, that's another signal. It isn't conclusive, because the assistant may rely on an old copy or another source.
Step 4. Check the server logs
In your hosting's statistics or access logs, search for these crawler names and the status code they receive. If you see GPTBot or PerplexityBot with 403, you have your confirmation.
Where the block usually is
Every hosting company names its settings differently, so search by concept:
| Where to look | What to look for |
|---|---|
| Hosting control panel | Bot protection, firewall or suspicious IP settings, with per-site rules |
| Security plugin (WordPress or another CMS) | Blocking of bots, user agents or countries |
| CDN (for example Cloudflare) | AI crawler policies, bot fight mode, firewall (WAF) rules |
.htaccess file or server configuration |
Rules that deny by user agent |
In Cloudflare, the official documentation puts the AI crawler and agent policies under Security > Settings, filtered by bot traffic. You choose an action for three behaviours: search, agent and training. According to that documentation, all three are set to "allow" on new domains, but someone may have changed them. Perplexity also warns that Cloudflare's Bot Fight Mode can't be skipped with custom firewall rules.
If you don't have access to the panel, send whoever manages your site this sentence: "I need GPTBot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User to get a 200, not a 403. Which rule is blocking them?"
What your robots.txt should say
For a local business that wants to appear in search engines and assistants, the simplest option is to allow everything and list your sitemap:
User-agent: *
Allow: /
Sitemap: https://yourwebsite.es/sitemap.xml
If you'd rather your content wasn't used to train models but still want to appear in assistant search, add blocks just for the training crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourwebsite.es/sitemap.xml
Note that, according to Google, Google-Extended covers both Gemini training and the use of your content to ground its answers. Blocking it doesn't remove you from Google Search, but it does opt you out of that second use.
Block training or not? Your decision
There's no single right answer. A practical way to decide:
| If your situation is… | A sensible option |
|---|---|
| A local business that wants to be found (garage, clinic, builder) | Allow search and agents; training is up to you |
| Your value is original content (paid guides, writing you sell) | Allow search and agents; block training |
| You don't want any AI on your site | Block everything, accepting you won't appear in their answers |
What you want to avoid is the most common case: blocking everything without knowing it, because of a default setting or a rule someone switched on years ago.
FAQ
If I block GPTBot, do I disappear from ChatGPT? Not necessarily. According to OpenAI, ChatGPT search depends on OAI-SearchBot, which is controlled separately.
Does allowing AI bots affect my Google ranking? Google says Google-Extended has no effect on inclusion or ranking in Search. What you do need to watch is not blocking Googlebot by mistake.
Is a correct robots.txt enough? No. If the server or CDN answers 403, the bot never even reads the file. Anthropic also advises against blocking its IP addresses instead of using robots.txt, because then it can't read your preferences.
How long until the change shows? OpenAI mentions about 24 hours for its search. The other companies don't publish a timeframe; repeat the tests the next day and check the logs over the week.
Checklist
- I've opened
yourwebsite.es/robots.txtand know what the served file says. - I've simulated the crawlers and no search crawler gets a 403.
robots.txtandllms.txt(if I have one) also return 200 to those crawlers.- I've checked the hosting's bot protection, the security plugin and the CDN.
- I've made a conscious decision about
GPTBot,ClaudeBotandGoogle-Extended. - Googlebot still gets a 200.
- I've repeated the test the next day and checked the logs.
Unlocking the door is only the first step: what the assistants find then has to be clear and consistent. We explain that in how to get ChatGPT and other AI tools to recommend your business. For a local business, the other half is your Google profile: see our local visibility work.
If you'd rather we checked it for you, our AI visibility audit includes a review of crawler access (from €350 excl. VAT; details on our pricing page). Or start with the free short review.
