Why AI Answer Engines Skip Your Site and How to Fix It
Contents
You wrote the page. It ranks fine. Then you ask ChatGPT, Perplexity or Claude about your topic and your site never shows up. This post lists the five most common causes and the findable check that catches each one.
Why does robots.txt keep AI bots out?
It is easy to block AI crawlers without noticing. A broad Disallow rule, or a rule copied from a template, tells bots like GPTBot, ClaudeBot and PerplexityBot to stay away. If a bot cannot fetch your page, it cannot use it.
Check your file at /robots.txt. The Google robots.txt intro explains the format. Each bot has its own docs: OpenAI crawlers, ClaudeBot, PerplexityBot and Google-Extended.
This snippet allows all four bots:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Allow: /If you do want to block one of them, that is your call. Just make it a choice. With -site, findable reports robots-txt, robots-page, robots-gptbot, robots-claudebot, robots-perplexitybot and robots-google-extended.
Why does a missing llms.txt matter?
An llms.txt file is a plain Markdown file at your site root that points language models to your best pages. It is a proposal, so no engine promises to read it. It costs little to add and gives tools a clean map of your site.
Read the llms.txt proposal for the format. findable checks for it with the llms-txt site check. It is unscored, so a miss will not drop your grade. The check only tests that the file exists; it does not parse the Markdown or follow the links in it.
Why can't bots read JavaScript-only content?
If your text only appears after a script runs, some crawlers will see an empty shell. Google can render JavaScript, but you should not assume every AI bot does. Plain HTML in the first server response is the safe path for every reader.
Read Google's JavaScript SEO basics for how rendering works. In findable, a thin shell shows up as a failing word_count (300+ words), a low text_html_ratio (15% or more) and a missing h1_exists. If those fail on a page that looks full in your browser, you are probably rendering on the client. Move the key text into server-rendered HTML.
Why does missing structured data hurt?
Structured data tells a machine what a block of your page is: an article, an FAQ, a person, an organization. Without it, a model has to guess what you meant. With it, your meaning is explicit, and the machine does not need to infer it.
The structured data intro shows how to add JSON-LD, and schema.org lists the types. findable checks jsonld, aeo_structured_data, aeo_faq, geo_org_schema, geo_author_schema, geo_knowledge_graph and geo_entity_markup. Start with one JSON-LD block, then add FAQPage and Organization.
Why don't engines find a direct answer on your page?
A short passage that stands alone is the easiest thing to quote. If your page buries the point under a long intro, there is nothing clean to lift. Put the answer first, right under a question heading, and add detail after it.
Write a paragraph of 30 to 80 words for aeo_direct_answers, and a question heading for aeo_question_headings. Add aeo_lists, aeo_tables and aeo_numbered_steps where they fit. For citability, geo_passage_citability wants a paragraph of 40 to 120 words, and geo_citations wants at least one external link.
How do you find all five problems at once?
Run one command against your home page with -site. You get the page checks plus the robots.txt and llms.txt results in one report. Each bot gets its own row, so you can see that GPTBot is allowed while ClaudeBot is blocked.
findable -site https://example.comFix the robots.txt rules first, since a blocked bot ignores everything else. Then server-render your text, add JSON-LD, and rewrite your top sections as question plus answer.
Key takeaways
- Bots skip sites for five common reasons, and all five are fixable.
- Check robots.txt first. Allow GPTBot, ClaudeBot, PerplexityBot and Google-Extended if you want to be found.
- Add
/llms.txt, put your text in server-rendered HTML, and add JSON-LD. - Lead each section with a direct answer.
- Run
findable -site <url>to see every one of these as a check id.
FAQ
- Does blocking GPTBot hide my site from ChatGPT?
- It stops the GPTBot crawler from fetching your pages. OpenAI documents separate bots for training and for search, so read their bot page and decide each one on purpose.
- Is llms.txt required?
- No. It is a proposal, not a standard that every engine reads. It is cheap to add, and findable reports it as the llms-txt site check.
- How do I see which bots my robots.txt blocks?
- Run findable with -site. It reports robots-gptbot, robots-claudebot, robots-perplexitybot and robots-google-extended as separate results.
How does findable score a page?
| Family | Weight | Question it asks |
|---|---|---|
| SEO | 50% | Can crawlers read and index this page? |
| AEO | 25% | Is it easy to lift an answer from this page? |
| GEO | 25% | Does the page give a generative model a reason to cite it? |
The hosted audit waits up to 10 s for a page and fetches at most 5 MB. It reads robots.txt and llms.txt up to 512 KB each.
More guides
- Every findable AEO Check ID and How to Fix It
- All 35 AEO checks in findable explained, from aeo_faq to aeo_summary. See what each id tests, how loose the test is and how to fix it. AEO is 25% of the score.
- Canonical Tags and Duplicate Content: How to Get Them Right
- Use one canonical link per page that points to its own absolute URL. Learn the checks canonical, canonical_self and canonical_absolute and fix them.
- FAQ Schema for Answer Engines: Markup and Checks
- How to add FAQPage JSON-LD that answer engines can lift, and how findable tests aeo_faq, aeo_faq_count and faq_valid. Includes a working example.
- Get Findable in 15 Minutes: A Step by Step Checklist
- Install findable, audit a page and your site, fix the top failures, then add MCP, the web UI, the API and a CI gate. Every step has a copy-paste command.
- Every findable GEO Check ID, Plus AI Crawler Checks
- All 27 GEO checks in findable explained, plus the unscored Site checks for GPTBot, ClaudeBot and llms.txt. GEO is 25% of the score. Fixes included.
- Image Alt Text, Dimensions and Lazy Loading Explained
- Fix img_alt, img_dimensions, img_alt_quality, img_lazy_loading and img_modern_format. See what each image check in findable tests and how to pass.
- JSON-LD Schema Basics: What findable Checks and Why
- Learn what JSON-LD is, which schema.org types findable validates, and how to fix jsonld_valid, jsonld_context and empty or relative values.
- llms.txt: What It Is and How findable Checks It
- llms.txt is a plain Markdown file at your site root that points AI tools to your key pages. See the llms-txt check and how to write the file.
- How to Write a Meta Description That Passes findable
- Write a meta description of 120 to 160 characters, one per page, that matches the page. See the checks desc_exists, desc_length and desc_single.
- Open Graph and Twitter Cards: Fix Your Share Previews
- Add og:title, og:description, og:image, og:url and twitter:card so links unfurl well. See every social check findable runs and how to pass it.
- robots.txt and AI Crawlers: GPTBot, ClaudeBot, PerplexityBot
- How robots.txt rules decide whether GPTBot, ClaudeBot, PerplexityBot and Google-Extended can fetch your page, and how findable tests each one.
- Every findable SEO Check ID and How to Fix It
- A plain guide to all 98 SEO checks in findable, from title_length to csp. See what each id tests, when it passes and the fix. SEO is 50% of your score.
- SEO vs AEO vs GEO: The Difference in Plain Words
- SEO, AEO and GEO each target a different reader: a search crawler, an answer box, or a generative AI model. See what wins in each and how to check them.