Skip to content
arduralab
·11 min

403 Forbidden and AI Crawlers: robots.txt Is Not Access

technical SEOAIAI searchcrawlability
MG
Marcin Godula

Współzałożyciel & Head of SEO/Tech

Specjalista SEO, GEO i web development z ponad 15-letnim doświadczeniem. Pomaga firmom B2B budować widoczność w wyszukiwarkach klasycznych i AI.

A 403 Forbidden error means the server understood the request but refuses to fulfill it. For AI crawlers, that has a concrete consequence: a robots.txt file that allows GPTBot, ClaudeBot or PerplexityBot is only a statement of intent, while the actual response comes from the server or from a layer in front of it, such as a web application firewall or bot protection on a content delivery network. That layer can answer with a 403 or a challenge page before the request ever reaches the site. That's why OpenAI and Perplexity recommend in their crawler documentation not just allowing their bots in robots.txt but also allowing requests from their published IP ranges, and why the proof of access is in the logs for those addresses, not in the robots.txt file.

What a 403 Forbidden Error Means

A 403 is one of the HTTP status codes in the 4xx class, which covers client errors. The HTTP specification, RFC 9110, sums it up in one sentence: the server understood the request but refuses to fulfill it. A server that wants to explain why it refused can describe the reason in the response content. If the request included credentials, a 403 means the server considered them insufficient, but the specification points out that the refusal can have reasons unrelated to credentials. A server that doesn't want to reveal that a resource exists can respond with a 404 instead.

In practice, a 403 shows up for a handful of typical reasons. On the server itself, those are file and directory permissions, access rules for IP addresses or countries, and a directory with no index file while directory listing is turned off. In front of the server, they are a web application firewall (WAF), bot protection and rate limits on a content delivery network (CDN), and security plugins in a content management system. For a human using a browser, the result looks the same: a page that says "Forbidden" or "Access denied." For a bot crawling the site on behalf of a search engine or an AI assistant, the difference between those sources matters, because each one gets fixed in a different place.

The 403 Status Code in Google: A Quick Recap

According to Google's documentation on HTTP status codes, all 4xx errors except 429 are treated the same way. Google doesn't use the content of such URLs, doesn't index them, and over time removes the ones it had already indexed. The same page warns against using 401 and 403 to limit the crawl rate, because those codes have no effect on it.

There's one exception that's easy to forget: the robots.txt file itself. The robots.txt specification in Google's documentation says Google's crawlers treat any 4xx status other than 429 when fetching that file as if no valid robots.txt existed, which means they assume there are no restrictions. A robots.txt that returns a 403 therefore doesn't block Googlebot. It just takes away the site owner's control over what the crawler fetches. How AI crawlers behave in the same situation is something their operators don't publicly describe in the documentation we read.

robots.txt Is a Statement, a 403 Is the Server's Decision

A robots.txt file is a request addressed to crawlers: it says which parts of the site they may fetch. It isn't an access control mechanism. A crawler's request first hits the content delivery network, the firewall and the bot protection, and only then the server with the page. Each of those layers can respond on its own without consulting the site. If it decides the request is unwanted, the crawler gets a 403, a challenge page or an empty response, and the allow in robots.txt has nothing to do with it.

This happens most often when bot protection is turned on with its default settings. Rules meant to stop automated scrapers don't necessarily tell an AI search crawler apart from a script copying content. robots.txt and edge rules are also usually configured by different people in different dashboards, so a site owner who deliberately allowed OAI-SearchBot in robots.txt can be blocking it at the edge at the same time without knowing it.

A challenge page isn't a way in for a crawler, either. Anthropic writes in its description of its crawlers that they respect anti-circumvention technologies and won't attempt to bypass CAPTCHAs. For that kind of crawler, a challenge works like a block, even if nobody meant to set one up.

AI Crawlers and What Each One Is For

Operators of large language models run several crawlers, each with a different role, which matters when you write the rules. OpenAI's crawler documentation lists three. OAI-SearchBot is used to surface websites in ChatGPT's search results. GPTBot collects content that may be used to train models. ChatGPT-User visits pages when a user asks it to, doesn't crawl the web automatically, and OpenAI notes that robots.txt rules may not apply to it. According to the same page, it can take about 24 hours after a robots.txt update for ChatGPT search to reflect it. The page doesn't mention Bing at all: the documented gate to ChatGPT search is OAI-SearchBot.

Anthropic describes three crawlers: ClaudeBot collects content that could contribute to model training, Claude-User fetches pages when users ask Claude questions, and Claude-SearchBot analyzes content to improve the quality of search results. In its crawler documentation, Perplexity says PerplexityBot surfaces and links websites in search results and isn't used to crawl content for training models, while Perplexity-User visits pages in response to user questions and generally ignores robots.txt rules.

Google-Extended is a separate case. According to Google's overview of its common crawlers, it's only a product token in robots.txt, with no User-Agent string of its own: the crawling is done by Google's regular crawlers, and the token decides whether the fetched content can be used to train Gemini models and to ground their answers. Google notes that Google-Extended doesn't affect a site's inclusion in Search and isn't a ranking signal. Blocking it in a firewall makes no sense, because there's nothing there to block.

What the Crawler Operators Recommend

Two companies address access directly. For a site to appear in ChatGPT's search results, OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, listed in a separate JSON file for each crawler. Perplexity recommends the same for PerplexityBot. Both sets of documentation treat robots.txt and network access as two separate conditions.

Perplexity goes further, with a section of its documentation on web application firewalls. It says a site protected by a WAF may need to explicitly allow Perplexity's crawlers, and it gives example rules for two popular firewalls. The principle behind them is the same: match the crawler's User-Agent header against its own IP ranges instead of letting the header through on its own. Perplexity also notes that one of those vendors' bot-fighting mode may still challenge crawlers that match an allow rule, because the firewall rule doesn't turn it off.

Anthropic approaches it from the other side. To opt out of crawling, it recommends robots.txt and warns that blocking its crawlers' IP addresses may not work correctly, because it prevents them from reading the robots.txt file. It also publishes the list of addresses its crawlers operate from. The takeaway for a site owner is simple: the decision about AI crawlers is made in robots.txt, and the firewall shouldn't quietly reverse it.

Checking Access: A Spoofed User-Agent Is Half a Test

The simplest test is a request sent with the crawler's User-Agent header, for example with curl -A and the header string copied from the operator's documentation. It shows what the server returns to a request carrying that header. It doesn't show what the real crawler gets. A firewall that verifies crawlers by IP address may block that test as impersonation while letting the real crawler through. A firewall that only looks at the header will let the test through and may still stop the real crawler with a different rule, such as a rate limit.

That's why the evidence lives in your server or CDN logs: which status codes went to requests from the addresses the crawler's operator publishes as its own. Reading them is ordinary log file analysis. For Googlebot, Google describes verifying its crawlers with a reverse and a forward DNS lookup or by matching against its published IP ranges, and you can check a page with the live test in the URL Inspection tool in Search Console. Bing offers a Verify Bingbot tool that checks whether a given IP address belongs to bingbot.

For AI crawlers, the operators don't describe a tool like that in their crawler documentation. OpenAI and Perplexity publish their IP ranges in JSON files, and Anthropic publishes a list of addresses. So the test goes like this: pull the requests carrying a given crawler's header from the logs, check whether the source address falls within the published ranges, and see which status code they got. A request with a GPTBot header from outside OpenAI's ranges is a fake, and blocking it is fine. A 403 for a request from a published range is a problem to fix.

On This Site: robots.txt and What the Edge Actually Serves

This site's robots.txt is generated from code at build time, and besides the general rule for all crawlers, it contains a separate group that explicitly allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended and OAI-SearchBot. That's a statement of our decision, but it isn't proof that each of those crawlers gets the page. We learned the difference through a different example.

Until September 2026, a feature on the network edge that hid email addresses from bots was rewriting the finished HTML after the site was built and stripping out the mailto: links. The files coming out of the build were correct, and the page served to readers wasn't. That's why our post-deployment checks test what the edge serves, not what we built. Each one starts with a positive control, such as confirming that the homepage responds with a 200. If that control fails, the result is flagged as a blind measurement, not reported as "zero errors."

Crawler access works the same way. An allow in robots.txt says what we want, and whether the crawler actually gets through the edge is a separate measurement. The same mental shortcut, treating configuration as proof, leads to the belief that a site is accessible to AI crawlers because "robots.txt lets them in, after all."

What We Look for in a Crawler Access Audit

In audits, we regularly come across sites that allow AI crawlers in robots.txt while running bot protection on their content delivery network that sends those same crawlers a 403 or a challenge page. Usually nobody made that decision on purpose: the protection was turned on with default settings or as a side effect of another change. That's why we check access at three levels.

The robots.txt file and its status code. We read the rules for each crawler separately and check that the file itself responds with a 200. A file that returns a 403 or a 5xx changes how crawlers behave no matter what it says.

Edge rules. We go through the firewall rules, bot protection modes and rate limits: whether any of them catch crawlers that the site allows in robots.txt, and whether the exceptions rely on the operator's IP ranges rather than on the header alone.

Logs for verified addresses. We match crawler requests against the published IP ranges and check the status codes. Only this part tells you what the crawler actually received.

Summary

A 403 Forbidden error is the server's refusal, and it can come from several layers, from file permissions to bot protection on a content delivery network. For Google, a page that returns a 403 drops out of the index, and a robots.txt file that returns a 403 is treated as if it didn't exist. For AI crawlers, the key point is that an allow in robots.txt doesn't open the way through a firewall. OpenAI and Perplexity recommend allowing their crawlers both in robots.txt and for their published IP ranges, and Anthropic advises against blocking addresses instead of using robots.txt rules.

You check a crawler's access in the logs, for the addresses its operator publishes as its own, not in a configuration file. We review robots.txt, edge rules and logs as part of our audits of visibility in search engines and AI answers. If you want to find out whether AI crawlers can see your site at all, request a quote.

Need help with this topic?

Get a free audit and find out how we can help your business grow online.

Get a free quote