Robots.txt Checker: Test Googlebot, GPTBot and Every AI Crawler

A robots.txt checker reads your robots.txt file and tells you which crawlers it lets in, which it blocks, and which lines are broken. This free robots.txt checker goes further: it tests 45 crawlers (Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and more), detects whether your site runs on WordPress, Shopify, Next.js or another platform, checks the exclusions that platform needs, and hands you a corrected robots.txt you can paste in. It needs no signup and has no limit.

How to use the robots.txt checker

1. Enter your homepage or any page URL in the box above. The checker finds the robots.txt file at the root of that domain.
2. Read the top opportunities first. Each one shows what we found, such as the exact robots.txt line, the blocked file or the crawler that was refused, so you can confirm it yourself.
3. Check the rules for your platform. Each recommended rule was tested against your live file, with the URL we tried and the line that matched.
4. Copy the corrected robots.txt, review it, and publish it. Then run the check again to confirm.

What this robots.txt checker tests

• Every crawler, rule by rule. We apply your robots.txt the way Google's specification says crawlers read it: a crawler follows only the most specific User-agent group that names it, and within that group the longest matching rule wins.
• A live firewall test. CDNs and security plugins can block crawlers before robots.txt is ever read. We request your page with each crawler's user agent and compare the response with a normal browser visit.
• Your platform's exclusions. WordPress, WooCommerce, Shopify, Next.js, Wix, Squarespace, Webflow, Drupal, Joomla, Magento, BigCommerce and Ghost each create URLs that waste crawl budget: admin areas, internal search, carts, sorting and filter parameters, feeds.
• The files Google needs. We take the CSS, JavaScript and images your page loads and test each one against your robots.txt. Blocking them can make pages look broken to Google.
• A robots.txt validator. Typos such as "Disalow", rules before any User-agent line, two directives on one line, relative Sitemap URLs, unsupported noindex lines and files over Google's 500 KiB limit.
• AI Overviews eligibility. noindex, nosnippet and max-snippet:0 on the page or in the X-Robots-Tag header keep it out of Google AI Overviews, whatever robots.txt says.

What is a robots.txt file?

Robots.txt is a plain text file at the root of a domain, such as example.com/robots.txt, that tells crawlers which paths they may request. It controls crawling, not indexing. A page blocked in robots.txt can still appear in Google if other sites link to it, and Google then can't see a noindex tag on it. To keep a page out of search results, allow crawling and use noindex.

Three details trip most people up:

• Groups don't combine. If your file has a User-agent: Googlebot group, Googlebot ignores everything under User-agent: *. Rules you add to * don't reach crawlers that have their own group.
• The longest rule wins. Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ because it is longer and more specific.
• Google reads 500 KiB. Anything after that is ignored.

How to check robots.txt without a tool

Open yourdomain.com/robots.txt in a browser to read the file. In Google Search Console, the robots.txt report (under Settings) shows the version Google last fetched and any parsing errors. Neither shows how AI crawlers or your firewall treat each bot, which is what the checker above adds.

Robots.txt for WordPress

WordPress serves a virtual robots.txt until you add a real one, usually through Yoast, Rank Math or AIOSEO. A sensible WordPress robots.txt blocks the admin area but keeps admin-ajax.php open, and blocks internal search results:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=

Sitemap: https://example.com/sitemap_index.xml

Don't block /wp-content/ or /wp-includes/. They hold the CSS and JavaScript your theme needs. Leave your main /feed/ open too, because Google uses it to find new posts. WooCommerce stores should also block /cart/, /checkout/, /my-account/ and ?add-to-cart= links, plus sorting (?orderby=) and filter parameters.

Robots.txt for Shopify

Shopify generates robots.txt with good defaults: it blocks the cart, checkout, account pages, internal search, and sorted or tag-filtered collection URLs. You can change it only through the robots.txt.liquid theme template. The most common Shopify mistake is a custom template that drops those defaults. The checker tests each one against your live file.

Robots.txt for Next.js

In Next.js, robots.txt comes from app/robots.ts or public/robots.txt. Never block /_next/: /_next/static/ holds your CSS and JavaScript and /_next/image serves your images. On the App Router, Googlebot sometimes crawls the ?_rsc= payloads Next.js fetches during in-app navigation, and Search Console then lists them. A Disallow: /*?_rsc= rule keeps those out without touching your pages.

Robots.txt for Wix and Squarespace

Wix generates robots.txt and lets you edit it under Settings, SEO, Robots.txt Editor. Squarespace generates it and doesn't allow edits, but has a setting to block known AI crawlers. On both, the checker is most useful for the AI crawler and firewall results.

Robots.txt and AI crawlers: which to allow

AI companies run separate crawlers for search and for training, so you can block one and keep the other.

For most business sites, allow the search and assistant crawlers, because those are the ones that cite you and send visitors, and decide on training crawlers separately. Per OpenAI's crawler documentation, a robots.txt change "can take ~24 hours" to reach its systems.

Older guides list anthropic-ai and Claude-Web. Anthropic no longer uses those names, so a rule that only names them doesn't block ClaudeBot. The checker flags this.

Common robots.txt mistakes

• Blocking CSS and JavaScript. Rules like Disallow: /_next/, Disallow: /wp-includes/ or, on old Drupal and Joomla sites, /misc/ and /media/ hide the files Google renders pages with.
• Leaving Disallow: / on after launch. Staging sites block everything. If the file goes live unchanged, the site drops out of search.
• Using robots.txt to remove a page. Blocking a URL stops Google seeing its noindex tag. Use noindex instead.
• noindex or Crawl-delay lines. Google stopped supporting noindex in robots.txt in 2019 and ignores Crawl-delay.
• Two rules on one line. A missing line break turns two rules into one long path that matches nothing.
• Allowing bots in robots.txt but blocking them at the firewall. Cloudflare's "Block AI bots" setting, Wordfence and Sucuri can refuse crawlers that your file welcomes.

Worked example

Take a WordPress site whose robots.txt disallows GPTBot and says nothing about OAI-SearchBot.

1. Enter the home page URL.
2. The checker detects WordPress from the /wp-content/ assets and the generator tag.
3. GPTBot, a training crawler, shows as blocked. OAI-SearchBot, an AI search crawler, shows as allowed, because no rule matches it. ChatGPT search can still cite the site.
4. The platform rules show /wp-admin/ blocked and admin-ajax.php allowed, but internal search (/?s=test) crawlable. That's one rule to add.
5. The corrected robots.txt keeps your existing rules, adds Disallow: /?s=, and keeps GPTBot blocked because that was your choice.
6. You publish it on Monday at 09:00 and re-run the check on Tuesday, after OpenAI's roughly 24-hour delay.

Related free tools

Crawler access is step one. The AI SEO audit checks 100+ technical and content issues across your site, and the AI Visibility Checker shows whether ChatGPT and other AI search engines actually recommend your business.