To check whether search systems can access your important pages, test five things separately for each page: what your robots.txt file says to each named crawler, whether the page carries a noindex or snippet rule, what your server or CDN returns to that crawler, whether the important text is in the HTML response, and what your logs show actually happened. A problem at any of those layers can keep a page out of an answer. The fix depends on which layer it is, and on whether someone blocked that crawler on purpose.
Most access audits stop at the first layer. Someone opens robots.txt, sees no obvious block and moves on, while a firewall rule refuses the same crawler a level below. Or the reverse: a team sees an AI crawler blocked and allows everything, undoing a decision someone made deliberately about model training.
Start by separating what each crawler is for
Crawlers from the same company can have different jobs, and blocking one does not necessarily block the others. The vendors describe their own crawlers this way:
| Robots.txt token | Purpose, as the vendor describes it | What a block means, per the vendor |
|---|---|---|
| OAI-SearchBot | Surfaces websites in ChatGPT’s search features | Opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links |
| GPTBot | Crawls content that may be used to train OpenAI’s foundation models | Signals the content should not be used for that training |
| ChatGPT-User | Visits pages for certain user actions in ChatGPT | Not automatic crawling; because the visits are user-initiated, robots.txt rules may not apply |
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | Googlebot’s robots.txt rules are the control for how a site is crawled for Search |
| Google-Extended | A control token with no user agent of its own | Manages use in training future Gemini models and in grounding for Gemini Apps and Vertex AI; Google says it does not affect inclusion or ranking in Search |
| ClaudeBot | Collects content that could contribute to Anthropic’s model training | Signals future content should be excluded from training data |
| Claude-SearchBot | Improves Claude’s search results | May reduce the site’s visibility in Claude’s search results |
| Claude-User | Visits pages when a person asks Claude a question | Prevents retrieval in response to user queries |
Sources: OpenAI’s crawler overview, Google’s common crawlers list, Google’s AI features documentation and Anthropic’s crawler article.
OpenAI states that its settings are independent: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to opt out of training. It also notes that search systems can take about 24 hours to adjust after a robots.txt change. So “we block AI” is not a single setting. It is a set of choices that can be made one crawler at a time, and the audit should record each one.
Check the five layers that decide access
1. Robots.txt. Find the group that applies to each token you care about, and check the rule for the exact path, not just the homepage. Look in particular for rules left over from a staging site and broad wildcard disallows. Robots.txt is a stated preference, not a lock: the standard itself, RFC 9309, says its rules are not a form of access authorization, and that listing paths in the file makes them discoverable. Private areas need authentication, not a disallow line.
2. Indexing and snippet rules. A page can be crawlable and still be excluded. For Google, a page must be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode, and noindex, nosnippet, data-nosnippet and max-snippet limit what is shown. Note the trap in combining the two layers: if robots.txt blocks a page, Google never sees its noindex rule, and the URL can still appear in results if other pages link to it.
3. Server and CDN response. A page that returns 200 in your browser can return 403, a challenge page or a timeout to a crawler. Hosting platforms, security plugins and CDNs make these decisions below robots.txt, often without the content team knowing. Cloudflare is a current example. Its AI bot policies sort crawlers into Search, Agent and Training, and it classifies mixed-purpose crawlers, used for both search and training, as Training. Its documentation says that from September 15, 2026, all configurations that block AI training also block those mixed-purpose crawlers, and describes new-domain defaults from that date that block Training and Agent bots on pages showing ads while allowing Search. A choice labelled “training” can therefore reach a crawler that also feeds search.
4. The text in the HTML response. If your price table, specifications or answer only appear after scripts run, test what a crawler receives. Google documents that Googlebot queues pages for rendering in a headless browser. OpenAI’s and Anthropic’s crawler pages cited above do not say whether their crawlers run JavaScript, so treat text that exists only after rendering as a risk to remove from important pages rather than a confirmed failure.
5. Log evidence. Your server or CDN logs show which crawlers requested which pages, and what they received. OpenAI publishes IP ranges for its crawlers, so a request can be checked against them; a user-agent string alone can be imitated. A page with no crawler visits in your logs and no errors is a different finding from a page that is visited and refused.
A crawler-access diagnostic table, with fictional data
The following example is fictional. Larchfield Joinery is an invented kitchen-cabinet maker; its domain, pages, rules and log entries are made up and are not a Seenvero customer result.
| Page (fictional) | Crawler and purpose | Robots rule | HTTP response | Text in HTML response | Log evidence | Next action |
|---|---|---|---|---|---|---|
| larchfield.example/kitchens/oak-shaker | OAI-SearchBot, search | Allowed | 403 from the host’s firewall | Full product copy | Requests from OpenAI’s published search ranges refused | Ask the host whether the rule is deliberate; if not, allow the verified crawler on public pages |
| larchfield.example/guides/lead-times | GPTBot, training | Disallow: / | Not requested | Full guide | No requests | None: a recorded decision to opt out of training. Confirm OAI-SearchBot is still allowed |
| larchfield.example/pricing | Googlebot, Search | Allowed | 200 | Price table missing; loaded by a script | Regular crawls | Put the price table in the HTML; recheck after the change |
| larchfield.example/showroom-booking | Claude-SearchBot, search | Disallow: /showroom left from a 2023 staging site | Not requested | Full page | No requests | Remove the stale rule if the page should be found |
| larchfield.example/trade-portal | All crawlers | Disallow: /trade-portal | 302 to a login page | Login form only | Occasional refused requests | Keep the login; the robots line adds no protection and makes the path easier to find |
Read the rows across, not down. The first row looks fine in robots.txt and fails at the firewall. The second is blocked, and that is correct, because someone chose it. The third is fully accessible and still hides the information a buyer asked about. Only the fourth is the classic robots.txt mistake.
A sixth column worth adding in your own sheet is owner: who decided the rule and when. If nobody can say, the rule is a candidate for review, not automatically for removal.
Do not overcorrect
The tempting fix after finding one bad block is to allow every crawler everywhere and switch off bot protection. Neither follows from the audit. Some sites have real reasons to opt out of training, to limit crawl load or to keep private sections closed, and those choices can coexist with being visible in search. Change the specific rule that contradicts a documented decision, record the change, and leave deliberate rules alone.
A single manual check is also weaker evidence than it feels. Asking an assistant to read your page is one user-initiated fetch, from one product, at one moment. ChatGPT-User, for example, is not the crawler OpenAI uses to decide search inclusion. A success says that fetch worked; it does not prove every search crawler can reach the page, and a failure may come from a temporary error rather than a block.
Run the check on your own pages with Seenvero
Seenvero’s free page check covers the first layers for one page without an account. It fetches the page you name, following up to five redirects, and reports the status, response time and the URL that answered. It reads the title, meta description, canonical, meta robots, X-Robots-Tag and whether structured data is present, then evaluates the site’s robots.txt for that exact path for OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider and bingbot. It does not render JavaScript, and “allowed” means robots.txt permits the crawler, not that the crawler came or that any answer used the page.
Inside the product, Page Audit runs the same kind of check on URLs you submit and can compare robots.txt across your domain and your competitors’ in one table, and Crawl Insights reads AI crawler traffic from a connected CDN account. Once access is settled, the rest of Seenvero shows whether answers to your buyers’ questions actually name or cite you. If you are fixing the markup on the same pages, our article on reviewing titles, alt text and structured data covers that layer.
Check one of your important pages for free, then add the result as the first row of your own diagnostic table.
Common questions
If I block GPTBot, will my site disappear from ChatGPT search?
Not according to OpenAI. It describes GPTBot and OAI-SearchBot as independent settings, and says sites that allow OAI-SearchBot can appear in ChatGPT search results while disallowing GPTBot for training.
Does blocking Google-Extended remove my pages from AI Overviews?
Google says Google-Extended does not affect inclusion in Google Search or act as a ranking signal. Access for Search, including AI Overviews and AI Mode, is controlled through Googlebot and the usual indexing and snippet rules.
Is robots.txt enough to keep a page private?
No. Robots.txt is a request that well-behaved crawlers follow, and listing a path makes it easier to find. Use authentication for private pages, and noindex on pages that should be reachable but not shown in search.