Is Your Website Blocking ChatGPT? How to Check in 60 Seconds

What Is robots.txt and Why Does It Matter for AI Search?

Every website has a file called robots.txt that lives at yourdomain.com/robots.txt. It is a plain text file that tells web crawlers — both traditional search engine bots and AI crawlers — which parts of your site they are allowed to visit.

When robots.txt is configured correctly, crawlers can access every page on your site, read your content, and use it to answer user queries. When it is misconfigured, crawlers hit a wall and leave. Your content becomes invisible to them no matter how good it is.

In 2026, the crawlers that matter most are not just Googlebot and Bingbot. They are GPTBot (ChatGPT), ClaudeBot (Anthropic), PerplexityBot, and a growing list of AI-powered tools that are sending real customers to service businesses every day.

How to Check Your robots.txt in 60 Seconds

Open a browser tab and go to: yourdomain.com/robots.txt (replace yourdomain.com with your actual website URL).

You will see a plain text file. Here is what you are looking for:

Safe configuration (allows everyone):

User-agent: *
Allow: /

Warning signs to look for: Any line that says "Disallow: /" under a User-agent entry. Specific disallow rules for GPTBot, ClaudeBot, or PerplexityBot. A completely empty file — which some security plugins create — that defaults to blocking everything not explicitly allowed.

If you see "Disallow: /" anywhere in that file, there is a strong chance you are blocking AI crawlers. The impact: any AI engine that respects robots.txt — including ChatGPT — cannot crawl your site or cite your business.

The Most Common Bad Configurations

After auditing hundreds of service business websites across the United States, we have found four configurations that cause the most AI crawler blocking.

The WordPress Default: Many WordPress installations, especially those using security plugins like Wordfence or iThemes Security, add aggressive disallow rules as a "security measure." These often block all non-Googlebot crawlers by default.

The Elementor Export: Websites built with Elementor page builder sometimes generate robots.txt files that block access to core directories that AI crawlers need to read your full page content.

The Hosting Provider Default: Some shared hosting providers — especially budget hosts — enable a default robots.txt that blocks crawlers while a site is being built. They sometimes forget to remove it after launch.

The Security Plugin Overreach: Plugins like Sucuri, Cloudflare's bot fight mode, or overly aggressive firewall rules can return a 403 or 429 response to AI crawlers before they even reach your robots.txt — which is even worse than a Disallow rule.

What a Correct robots.txt Looks Like for a Service Business

Here is the exact robots.txt configuration we install on every website we build or audit:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: cohere-ai
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

This configuration explicitly invites every major AI crawler in by name. It is not enough to have a generic "Allow: *" — some AI crawlers require an explicit User-agent entry to confirm they are welcome.

Why the Competitive Window Is Open Right Now

Here is the business reality: most of your competitors have not checked their robots.txt. Most of the agencies they work with have not checked it either, because most agencies are still optimizing for traditional search engines.

In every market we audit — roofing, HVAC, plumbing, law, medical, cleaning — we find that between 60 and 80 percent of competitor websites have some form of AI crawler restriction. That means a properly configured robots.txt alone puts you ahead of the majority of your market in AI search visibility.

This window will not stay open. As awareness of AEO grows, your competitors will eventually fix their configurations. The businesses that get indexed first build citation history that is very difficult to displace once established. Read our full guide on what AEO is or get a free robots.txt audit from our team.

Frequently Asked Questions