Do I need to name AI bots in robots.txt?

No. A bot with no group of its own follows the User-agent: * rules. Name a bot only when you want it to behave differently from everyone else.

Does blocking GPTBot remove me from ChatGPT?

It stops that crawler from fetching your pages. Vendors can run separate bots for training and for search, so read each vendor's bot documentation before you decide.

Is robots.txt a security tool?

No. It is a request that well-behaved crawlers follow. Anything private needs real access control.

Why are these results not in my score?

They are Site checks. They need the -site flag or the site option in the API, and they are reported apart from the SEO, AEO and GEO score.

robots.txt and AI crawlers: GPTBot, ClaudeBot, PerplexityBot

By findable team. Published 2026-10-15. Updated 2026-10-15.

Contents

What does findable test?
How does findable read the rules?
What does a clear policy look like?
How do you block an AI bot completely?
What are the common mistakes?
Key takeaways

Whether an AI tool can read your page starts with robots.txt. findable tests this with its Site checks. They run only when you ask for them, and they do not change your score.

What does findable test?

Check id	What it tests
`robots-txt`	`/robots.txt` exists and returns 200
`robots-page`	The `*` group lets crawlers fetch the page you scanned
`robots-gptbot`	GPTBot may fetch the page
`robots-claudebot`	ClaudeBot may fetch the page
`robots-perplexitybot`	PerplexityBot may fetch the page
`robots-google-extended`	Google-Extended may fetch the page

Run them with findable -site https://example.com, or send "site": true to /api/scan. The check tests the exact path you scanned, so a rule that blocks /blog/ fails for a blog URL and passes for the home page.

How does findable read the rules?

It follows the usual matching rules:

A bot uses the group that names it. If no group names it, it uses the * group.
A named group replaces the * group completely. Rules are not merged.
The longest matching pattern wins. On a tie, Allow wins.
* matches any characters and a trailing $ anchors the end of the path.
An empty Disallow: blocks nothing.

Agent names are matched without case.

What does a clear policy look like?

This allows everyone, and keeps GPTBot out of one folder:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /private/

Note the trap: GPTBot now follows only its own group, so /admin/ is no longer blocked for it. If you want both, repeat the line inside the GPTBot group.

How do you block an AI bot completely?

User-agent: ClaudeBot
Disallow: /

Do this on purpose. If you want to be cited by AI answers, a block works against you. See why AI answer engines skip your site.

What are the common mistakes?

A blanket Disallow: / under User-agent: * left over from a staging site.
Adding a named group and forgetting it drops your * rules.
Serving a 404, or a redirect that lands on an HTML page. Crawlers follow redirects, so the file counts as present, but an HTML page holds no rules and nothing is restricted.
Treating robots.txt as a way to hide pages. A blocked URL can still be listed if other sites link to it.

Key takeaways

Run the checks with -site. They are unscored.
A named bot group overrides * entirely.
Longest pattern wins, and Allow wins ties.
Decide each AI bot on purpose, then confirm with a scan.
Next, see llms.txt.

FAQ

Do I need to name AI bots in robots.txt?: No. A bot with no group of its own follows the User-agent: * rules. Name a bot only when you want it to behave differently from everyone else.
Does blocking GPTBot remove me from ChatGPT?: It stops that crawler from fetching your pages. Vendors can run separate bots for training and for search, so read each vendor's bot documentation before you decide.
Is robots.txt a security tool?: No. It is a request that well-behaved crawlers follow. Anything private needs real access control.
Why are these results not in my score?: They are Site checks. They need the -site flag or the site option in the API, and they are reported apart from the SEO, AEO and GEO score.

How does findable score a page?

Family	Weight	Question it asks
SEO	50%	Can crawlers read and index this page?
AEO	25%	Is it easy to lift an answer from this page?
GEO	25%	Does the page give a generative model a reason to cite it?

The hosted audit waits up to 10 s for a page and fetches at most 5 MB. It reads robots.txt and llms.txt up to 512 KB each.

More guides

Every findable AEO Check ID and How to Fix It: All 35 AEO checks in findable explained, from aeo_faq to aeo_summary. See what each id tests, how loose the test is and how to fix it. AEO is 25% of the score.
Canonical Tags and Duplicate Content: How to Get Them Right: Use one canonical link per page that points to its own absolute URL. Learn the checks canonical, canonical_self and canonical_absolute and fix them.
FAQ Schema for Answer Engines: Markup and Checks: How to add FAQPage JSON-LD that answer engines can lift, and how findable tests aeo_faq, aeo_faq_count and faq_valid. Includes a working example.
Get Findable in 15 Minutes: A Step by Step Checklist: Install findable, audit a page and your site, fix the top failures, then add MCP, the web UI, the API and a CI gate. Every step has a copy-paste command.
Every findable GEO Check ID, Plus AI Crawler Checks: All 27 GEO checks in findable explained, plus the unscored Site checks for GPTBot, ClaudeBot and llms.txt. GEO is 25% of the score. Fixes included.
Image Alt Text, Dimensions and Lazy Loading Explained: Fix img_alt, img_dimensions, img_alt_quality, img_lazy_loading and img_modern_format. See what each image check in findable tests and how to pass.
JSON-LD Schema Basics: What findable Checks and Why: Learn what JSON-LD is, which schema.org types findable validates, and how to fix jsonld_valid, jsonld_context and empty or relative values.
llms.txt: What It Is and How findable Checks It: llms.txt is a plain Markdown file at your site root that points AI tools to your key pages. See the llms-txt check and how to write the file.
How to Write a Meta Description That Passes findable: Write a meta description of 120 to 160 characters, one per page, that matches the page. See the checks desc_exists, desc_length and desc_single.
Open Graph and Twitter Cards: Fix Your Share Previews: Add og:title, og:description, og:image, og:url and twitter:card so links unfurl well. See every social check findable runs and how to pass it.
Every findable SEO Check ID and How to Fix It: A plain guide to all 98 SEO checks in findable, from title_length to csp. See what each id tests, when it passes and the fix. SEO is 50% of your score.
SEO vs AEO vs GEO: The Difference in Plain Words: SEO, AEO and GEO each target a different reader: a search crawler, an answer box, or a generative AI model. See what wins in each and how to check them.
Why AI Answer Engines Skip Your Site and How to Fix It: ChatGPT, Perplexity and Claude may ignore your pages for five fixable reasons, from robots.txt blocks to missing answers. Learn each cause and its check id.