Reference
The five gates
Every search and AI-answer failure is a failure at one of five gates: crawl, index, retrieve, cite, influence. Knowing the gates lets you place any finding anyone shows you, including findings you have never seen before. Classic SEO lives at gates one and two; AI-answer visibility lives at three, four and five.
Gate 1 — Crawl. Can a bot fetch the URL?
Fails on robots.txt rules, server errors, and — most often in practice — CDN or WAF behaviour nobody on the marketing side knows about. We verify crawler identity by published IP range rather than by user-agent string, because a user-agent is a claim anyone can make.
The most common technical error in the industry belongs here.
robots.txt controls crawling; noindex controls
indexing. Disallow a URL and the bot never fetches it, therefore
never sees the noindex, and the URL can still appear in results.
To remove something you must let it be crawled.
Gate 2 — Index. Did the engine keep it?
Fails on thin or duplicate content, canonicalisation, or an explicit
noindex. Submitted is not crawled and crawled is not indexed —
three separate stages, each a filter. Rendering is a deferred queue, and
pages can sit in it far longer than the few seconds people assume.
Two things worth knowing because they are widely sold and largely
irrelevant: rel=canonical is a strong signal, not a directive,
and Google overrides it routinely. And almost no site has a crawl-budget
problem — Google's rough thresholds are a million-plus pages changing
weekly, or ten thousand-plus daily. Below that, claiming otherwise is a tell.
Gate 3 — Retrieve. Was it pulled for this query?
Fails when the page does not answer the sub-query the fan-out actually issued. The model rewrites one question into many concurrent sub-queries and each retrieves separately, so the competition is across a set of decomposed questions rather than a keyword.
This is the first gate where classic ranking stops predicting the outcome. Ahrefs tested 15,000 long-tail queries across six engines in August 2025 and found only 12% of links cited by ChatGPT, Gemini and Copilot appeared in Google's top ten for the same prompt, with 80% not ranking anywhere for it. Perplexity is the outlier at 28.6%.
Gate 4 — Cite. Did it get named in the answer?
Being inside the page an engine read does not get you named. This is the gate almost nobody sells honestly, and it is demonstrable live in under a minute.
Run a category query and watch what happens. An AI Overview will cite a directory's "top ten" listing and then name firms that were not in that ten — pulling names instead from businesses that had their own citable page in the source set. The engine read the directory; it named someone else.
Engines also read different sources from one another. In a single live test, Google's AI Overview drew on Yelp, Perplexity drew on Clutch and Thumbtack, and ChatGPT drew on Expertise.com — for the same question. "Getting into AI answers" is not one job, and anyone offering it as one has not looked.
Source overlap between engines is low wherever it has been measured, but the figure depends on the pair. Grossman et al. (SIGIR 2026, 11,500 queries, December 2025) report URL-level Jaccard similarity of 0.11 between AI Overviews and Gemini, 0.16 between Gemini and the standard result page, and 0.18 between AI Overviews and that page — all three inside the Google family.
Gate 5 — Influence. Did it shape what was said?
Fails when you are cited navigationally and contribute nothing to the substance of the answer. A citation that appears in a footnote list while the answer's actual claims came from elsewhere is presence without influence. It is the least discussed gate and the hardest to move.
What Google says does not work
Google's guidance on AI features says it directly: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." And specifically on the files and markup an entire vendor category is built on: "You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add."
Google's documentation has also clarified that its spam policies apply to generative AI responses in Search. There is no separate rulebook for AI Overviews and AI Mode, which means every AI-content shortcut carries the same policy exposure it always did.
On llms.txt specifically, the measurement is unambiguous: Ahrefs found 97% of llms.txt files received zero requests in a month across 137,000 domains. That panel is its own Web Analytics customers rather than the open web — a population biased toward people who would notice, which makes the result stronger rather than weaker.
Two more dated deliverables worth checking your current agency against. FAQ rich results are retired entirely — they stopped appearing on 7 May 2026, documentation was removed on 15 June, and Search Console API support went in August. And structured data is not a ranking factor; it makes pages eligible for rich results, which is a different claim.
How AI visibility should be measured
Ask an assistant the same question twice and you can get different brands. A panel of ten or twenty-five prompts sampled once a day produces a number whose margin of error is wider than the range it is meant to move within. It will show a change every month, and none of those changes will mean anything.
The noise is not a rounding detail. It is the dominant term:
- 65% of AI-cited sources change from one day to the next
- ±44 points of error on a single prompt asked once
- 57.8% of ChatGPT runs produce zero citations at all
What we do instead:
| Design choice | Why |
|---|---|
| At least 120 prompts, frozen and approved in writing | Precision comes from the number of distinct prompts, not from re-asking one. Freezing the set is what makes month two comparable to month one. |
| Five repetitions per prompt per engine | Enough to characterise the noise. Beyond roughly ten, repetition buys almost nothing. |
| Reported per engine, never blended | Engines retrieve differently and move at different times. A blended figure hides the only actionable information. |
| Every figure with a confidence interval | So you can tell real movement from sampling noise. Most months the honest answer is that nothing measurable changed, and we will say so. |
| The measurement layer declared on every number | API sampling, consumer interface and search-result capture are three different measurements. We label which produced the figure. |
We also measure your own noise floor at onboarding, on your category and an approved competitor set, before reporting any movement — because the floor differs by market.
A note on the founding academic work. The 2023 GEO paper is frequently quoted as evidence of a "40% visibility lift". That figure came from a GPT-3.5-era system, with the source already inside a fixed five-document context, judged by a model from the same family. The mechanisms it identified are worth knowing; the percentage does not transfer to today's engines, and we will not quote it at you as though it does.
Which gate are you failing?
The diagnostic audit answers that in ten business days, for $1,500 — credited against month one if you continue, refunded if it finds nothing material.
Sources
- Google Search Central, AI features and your website — both passages quoted verbatim; and the documentation changelog on spam policies applying to generative AI responses.
- Ahrefs, citation overlap across six engines, 15,000 long-tail queries, August 2025.
- Ahrefs, llms.txt request measurement, 137,000 domains — its own Web Analytics customers, not the open web.
- Grossman et al., SIGIR 2026, 11,500 queries, December 2025 — URL-level Jaccard similarity between engines.
- Aggarwal et al., GEO: Generative Engine Optimization, IIT Delhi and Princeton, KDD 2024. Cited here for its mechanisms and for the limits of its headline figure.