Is Your Website Blocking ChatGPT? How to Check robots.txt for AI Bots

By The Check My AI Search Team, AI search visibility for local businesses · October 2026

To check whether your website is blocking ChatGPT or other AI crawlers, go to yourdomain.com/robots.txt in your browser. Look for lines that say User-agent: GPTBot or User-agent: * followed by Disallow: /. Either one can lock AI crawlers out of your entire site — and if they can't crawl it, they can't cite it.

What is robots.txt and why does it matter for AI search?

robots.txt is a plain-text file that sits at the root of your website and tells crawlers — search engines, AI bots, and other automated visitors — which pages they're allowed to read. Every major AI engine (ChatGPT, Perplexity, Google's AI Overviews, Copilot) operates its own crawler. If your robots.txt tells those crawlers to stay out, the AI has no way to read your site, verify your services, or cite you as a source in its answers.

For a local business, this matters more than most owners realize. When someone asks ChatGPT "best HVAC company in Columbus," the AI draws on what it has crawled and indexed. A site it was never allowed to read is a site it effectively doesn't know. Your Google Business Profile and third-party reviews can still get you named, but your own website — your service pages, your city pages, your schema markup — contributes nothing if the crawler was turned away at the door.

Which AI crawlers does your robots.txt need to allow?

Each major AI platform uses its own user-agent string. As of late 2026, the ones that matter most for local business visibility are:

AI Platform	Crawler User-Agent	What it feeds
ChatGPT (OpenAI)	`GPTBot`	ChatGPT browsing & training
OpenAI search	`OAI-SearchBot`	ChatGPT real-time search answers
Google AI Overviews	`Google-Extended`	Gemini & AI Overviews training
Perplexity	`PerplexityBot`	Perplexity answers & citations
Microsoft Copilot	`Bingbot` (shared)	Copilot answers via Bing index
Anthropic (Claude)	`ClaudeBot`	Claude training & answers

You don't need to explicitly list every one of these in your robots.txt. The key is to make sure none of them are blocked — either by a specific rule targeting their user-agent or by a catch-all rule that blocks everything.

How to find and read your robots.txt file in 60 seconds

Finding your robots.txt takes about ten seconds. Reading it takes another fifty. Here's the process:

Open your browser and type your domain followed by /robots.txt. For example: https://www.acmeplumbing.com/robots.txt. Hit enter.
If you see a plain-text file, you have a robots.txt. Read on. If you see a 404 error, your site has no robots.txt at all — which means all crawlers are allowed by default. You're fine, but you may want to add one for control.
Scan the file top to bottom. Look for any User-agent: line followed by a Disallow: line. The Disallow: line is what blocks access. A Disallow: / (with a forward slash and nothing after it) blocks the entire site for that user-agent.
Check for wildcard rules. A User-agent: * block applies to every crawler that isn't named elsewhere in the file. If it has a broad Disallow:, all unnamed bots — including newer AI crawlers — are blocked.

The whole audit takes under a minute. No tools required, no login needed.

What a blocking rule looks like — and how to spot it

Here are the three most common patterns that lock out AI crawlers. Any of these in your robots.txt is a problem:

Pattern 1 — Blocking everything for all bots:

User-agent: *
Disallow: /

This is the most damaging. It tells every crawler — including GPTBot, PerplexityBot, and Google-Extended — to leave your entire site alone. It's often added by mistake during a site build and never removed.

Pattern 2 — Blocking GPTBot specifically:

User-agent: GPTBot
Disallow: /

This was commonly recommended in 2023 when some site owners wanted to opt out of OpenAI training data. It blocks ChatGPT's crawler from your entire site.

Pattern 3 — Blocking Google-Extended:

User-agent: Google-Extended
Disallow: /

This blocks Google's AI training crawler, which affects Gemini and Google AI Overviews. Many site owners added this without realizing the downstream effect on AI search visibility.

A rule that only blocks a specific subfolder — like Disallow: /wp-admin/ — is harmless and normal. The problem is any rule that disallows the root path (/) or major content directories.

How to fix a robots.txt that's locking out AI bots

The fix depends on what's causing the block. Here are the three scenarios and what to do in each:

Scenario A — The catch-all block (User-agent: * / Disallow: /): Delete the Disallow: / line entirely, or replace it with Disallow: (blank, which means "block nothing"). Keep any specific subfolder blocks you actually want, like /wp-admin/.

Scenario B — A named AI crawler is blocked: Delete the entire block for that crawler. For example, remove both the User-agent: GPTBot line and the Disallow: / line beneath it. Omitting a user-agent from robots.txt means it's allowed — you don't need to explicitly whitelist anyone.

Scenario C — You want to allow AI crawlers but keep other bots out: List the bots you want to block explicitly, and leave AI crawlers unmentioned (which means allowed). For example:

User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

This approach gives you control without accidentally sweeping in AI crawlers.

After editing, save the file and re-upload it to your server root (or update it through your CMS — WordPress users can find it under Settings → Reading, or use an SEO plugin like Yoast or Rank Math). Then re-visit yourdomain.com/robots.txt to confirm the change is live.

If you're not sure where your robots.txt lives or how to edit it, your web host's support team can usually help in under ten minutes. This is a one-file fix, not a development project.

Does blocking AI crawlers actually hurt your recommendations?

Based on what we see running thousands of real AI queries for local businesses — questions like "best roofer in Tampa" or "emergency dentist in Austin" — your own website is one of several inputs AI uses, but it's not the only one. Third-party sources (Google Business Profile, Yelp, Angi, BBB, Healthgrades) carry significant weight and don't depend on your robots.txt at all.

That said, your website is the one source you control completely. It's where you can publish service pages, city-specific content, and structured data (like LocalBusiness schema) that third-party directories can't replicate. Blocking AI crawlers from reading that content means you're competing with one hand tied behind your back.

We've also seen cases where a business appears in AI answers despite having a blocked site — because the AI pulled from a Yelp listing or a local news mention. But those businesses are relying on sources they don't control, and their citations tend to be thinner. Fixing your robots.txt doesn't guarantee a recommendation, but it removes a self-inflicted obstacle.

For more on what actually drives AI recommendations for local businesses, see our AI visibility guide and the help center, which covers the full list of technical fixes including schema markup and citation sources.

What to check next after you fix robots.txt

robots.txt is the first thing to fix because it's a hard gate — if it's closed, nothing else matters. Once it's open, the next layer of checks is:

LocalBusiness schema: Does your homepage or contact page include structured data that tells AI your business name, address, phone, hours, and service area? Schema is machine-readable context that AI engines use directly.
Service pages: Does each service you offer have its own page with clear, specific language? "We replace water heaters in Phoenix" is more useful to an AI than a generic "plumbing services" paragraph.
Third-party profiles: Are your Google Business Profile, Yelp, and trade-specific directories (Angi, BBB, Healthgrades, Avvo) complete and consistent? These are the sources AI quotes most often for local recommendations.
Reviews: Recent, specific reviews on Google and Yelp are cited heavily in AI answers. Volume and recency both matter.
Crawlability beyond robots.txt: Check that your pages aren't set to noindex in their meta tags, which is a separate instruction that prevents indexing even if the crawler can access the page.

The fastest way to see where you actually stand — whether ChatGPT and Gemini are naming your business, which sources they're citing, and what's missing — is to run a visibility check. Our free AI visibility check asks the real questions customers ask in your city and trade, and shows you the results in about two minutes.

Want to know whether ChatGPT and Gemini name your business today? Run the free AI visibility check: no account needed, results in about two minutes. Weekly tracking starts at $29 a month, with a full refund within 7 days.

Frequently asked questions

Does ChatGPT respect robots.txt?

Yes. OpenAI's GPTBot crawler honors robots.txt directives. If your file includes User-agent: GPTBot with Disallow: /, OpenAI's crawler will not read your site. The same applies to OAI-SearchBot, which powers ChatGPT's real-time search answers. Removing the block allows both crawlers in.

If I remove the block, will ChatGPT start recommending me right away?

Not instantly. AI crawlers re-visit sites on their own schedule, which can range from days to weeks. Fixing robots.txt removes the obstacle, but it doesn't trigger an immediate re-crawl. Think of it as opening the door — the crawler still has to walk through it on its next pass.

Is it ever okay to block AI crawlers?

Yes, if you have a specific reason — for example, blocking AI training data for proprietary content, or limiting bandwidth on a low-resource server. The trade-off is reduced AI search visibility. For most local businesses whose goal is to be found and recommended, blocking AI crawlers works against that goal.

What if my robots.txt looks fine but I'm still not being recommended?

robots.txt is one factor among many. If your file is clean, the next things to check are your LocalBusiness schema, the completeness of your Google Business Profile, the volume and recency of your reviews, and whether your site has specific service and city pages. Our help center covers each of these fixes step by step.

Can a WordPress plugin accidentally block AI crawlers?

Yes. Some security plugins, SEO plugins, or site-builder tools generate or modify robots.txt automatically. A common culprit is a plugin that adds a broad Disallow: / under User-agent: * during setup. Always check your live robots.txt file directly in the browser rather than relying on what a plugin's settings screen shows — the live file is what crawlers actually see.

Last updated: October 2026