Auspex.

Reference

The five gates

Every search and AI-answer failure is a failure at one of five gates: crawl, index, retrieve, cite, influence. Knowing the gates lets you place any finding anyone shows you, including findings you have never seen before. Classic SEO lives at gates one and two. AI-answer visibility lives at three, four and five.

Fig. 1 The five gates
  1. Gate 1 Crawl Can a bot fetch the URL? robots.txt · server errors · CDN and WAF rules · budget spent elsewhere
  2. Gate 2 Index Did the engine keep it? thin · duplicate · canonicalised away · noindex
  3. Gate 3 Retrieve Was it pulled for this query? does not answer the sub-query the fan-out issued
  4. Gate 4 Cite Did it get named in the answer? present in the source, not lifted out
  5. Gate 5 Influence Did it shape what was said? cited navigationally, contributing nothing
Gates 1–2 · Classic SEOWhere your current agency works.
Gates 3–5 · AI-answer visibilityWhere we work. Gates 4 and 5 are the ones nobody sells honestly.
Sequence only — no quantities implied Gate four, in detail ↓

Gate 1: Crawl. Can a bot fetch the URL?

Fails on robots.txt rules, server errors, and, most often, CDN or WAF behaviour nobody on the marketing side knows about. We verify crawler identity by published IP range rather than by user-agent string, because a user-agent is a claim anyone can make.

Fig. 3 The crawl path
The crawl path A bot's request for a URL meets robots.txt, then the CDN or WAF, then the server. Any one of them can stop it — disallowed, blocked, or a server error — and the URL is fetched only if it survives all three. THE CRAWLER EVERY ONE CAN STOP THE FETCH GATE 1 PASSED A bot robots.txt CDN or WAF The server Fetched A URL ALLOWED LET THROUGH SERVES IT VERIFIED BY IP RANGE, NOT BY USER-AGENT DISALLOW · NEVER FETCHED BLOCKED · NOT FETCHED MOST OFTEN, THIS ONE SERVER ERROR · NOT FETCHED
Gate 1 fails on robots.txt rules, server errors, and, most often, CDN or WAF behaviour nobody on the marketing side knows about.

The most common technical error in the industry belongs here. robots.txt controls crawling, and noindex controls indexing. Disallow a URL and the bot never fetches it, therefore never sees the noindex, and the URL can still appear in results. To remove something you must let it be crawled.

Two free checks read this gate from outside. The crawl check reads what your site returns to each named crawler, and the identity check reads whether a crawler in your logs was the crawler at all.

Gate 2: Index. Did the engine keep it?

Fails on thin or duplicate content, canonicalisation, or an explicit noindex. Submitted, crawled and indexed are three separate stages, and each one is a filter. Rendering is a deferred queue, and pages can sit in it far longer than the few seconds people assume.

Fig. 4 The three index stages
What the index keeps A page that was fetched at gate 1 still has to survive three separate decisions before the engine keeps it: thin or duplicate content, canonicalisation, and an explicit noindex. Any one of them drops it, and only what survives all three is in the index. FETCHED AT GATE 1 EVERY ONE CAN DROP THE PAGE GATE 2 PASSED Fetched Thin or duplicate rel=canonical noindex Kept in the index NOT YET KEPT NEITHER THE ONE KEPT NO NOINDEX SUBMITTED, CRAWLED AND INDEXED ARE SEPARATE NOT KEPT CANONICALISED AWAY A SIGNAL, NOT A DIRECTIVE NOT KEPT
Submitted, crawled and indexed are three separate stages, and each one is a filter.

Two things worth knowing because they are widely sold and largely irrelevant: rel=canonical is a strong signal, not a directive, and Google overrides it often. And almost no site has a crawl-budget problem. Google's rough thresholds are a million-plus pages changing weekly, or ten thousand-plus daily. Below that, it is very unlikely to be the problem.

Two free checks read this gate from outside. The index check reads what the index would keep from one URL, and the sitemap check reads whether your sitemap offers what it then takes back.

Gate 3: Retrieve. Was it pulled for this query?

Fails when the page does not answer the sub-query the fan-out issued. The model rewrites one question into many concurrent sub-queries and each retrieves separately, so the competition is across a set of decomposed questions rather than a keyword.

This is the first gate where classic ranking stops predicting the outcome. Ahrefs tested 15,000 long-tail queries across six engines in August 2025 and found only 12% of links cited by ChatGPT, Gemini and Copilot appeared in Google's top ten for the same prompt, with 80% not ranking anywhere for it. Perplexity is the outlier at 28.6%.

Fig. 5 Query fan-out
Query fan-out One question is rewritten into several concurrent sub-queries. Each sub-query retrieves separately; that retrieval is gate 3. Your page is pulled only for the sub-queries it answers, another site's page is pulled for the ones it does not, and the answer is synthesised from whatever was retrieved. QUESTION SUB-QUERIES ANSWER RETRIEVAL · GATE 3 One question AS THE BUYER TYPED IT REWRITTEN sub-query 1 sub-query 2 sub-query 3 … AND N MORE, CONCURRENTLY RETRIEVES Your page A directory that lists you Someone else's page about you YOUR PAGE DID NOT ANSWER THESE SYNTHESISED Answer
A page that answers the question as asked can still be absent from the sub-queries the engine actually issued. Each one it misses is a retrieval that pulls someone else's page.

Gate 4: Cite. Did it get named in the answer?

Being inside the page an engine read does not get you named. This is the gate almost nobody sells, and you can see it happen in under a minute.

Run a category query and watch what happens. An AI Overview will cite a directory's "top ten" listing and then name firms that were not in that ten, pulling names instead from businesses that had their own citable page in the source set. The engine read the directory and named someone else.

Engines also read different sources from one another. In a single live test of the same question, Google's AI Overview drew on Yelp, Perplexity drew on Clutch and Thumbtack, and ChatGPT drew on Expertise.com. "Getting into AI answers" is not one job.

Source overlap between engines is low wherever it has been measured, but the figure depends on the pair. Grossman et al. (SIGIR 2026, 11,500 queries, December 2025) report URL-level Jaccard similarity of 0.11 between AI Overviews and Gemini, 0.16 between Gemini and the standard result page, and 0.18 between AI Overviews and that page, all three inside the Google family.

Gate 5: Influence. Did it shape what was said?

Fails when you are cited navigationally and contribute nothing to the substance of the answer. A citation that appears in a footnote list while the answer's actual claims came from elsewhere is presence without influence. It is the least discussed gate and the hardest to move.

Fig. 6 The four outcomes at gates 4 and 5
Named, cited, or neither Gate 4 asks whether the page was named as a linked source in the answer; gate 5 asks whether that citation shaped what the answer said. Only one of the four outcomes clears both — cited, and the claims in the answer came from your page. THE SOURCE SET FOUR OUTCOMES GATE 4 · CITE GATE 5 · INFLUENCE WITH YOUR PAGE AMONG THEM Retrieved passages CITED Cited as a source SHAPED IT DID NOT NOT CITED NOT CITED EITHER It shaped the answer Cited navigationally Mentioned, not cited Absent from the answer GATE 5 PASSED THE CLAIMS ARE YOURS GATE 5 FAILED PRESENCE WITHOUT INFLUENCE GATE 4 FAILED A DIRECTORY IS CITED GATE 4 FAILED NOTHING WAS LIFTED OUT
A mention is being named in an answer and a citation is being named as a linked source for it — and a citation sitting in a footnote list, while the answer's actual claims came from elsewhere, is presence without influence. Citation vs mention →

What Google says does not work

Google's guidance on AI features says it directly: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." And specifically on the files and markup an entire vendor category is built on: "You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add."

Google's documentation has also clarified that its spam policies apply to generative AI responses in Search. There is no separate rulebook for AI Overviews and AI Mode, which means every AI-content shortcut carries the same policy exposure it always did.

On llms.txt specifically, the measurement is unambiguous: Ahrefs found 97% of llms.txt files received zero requests in a month across 137,000 domains. That panel is its own Web Analytics customers rather than the open web, a population biased toward people who would notice. That strengthens the result.

Two more dated deliverables worth checking your current agency against. FAQ rich results are retired entirely. They stopped appearing on 7 May 2026, documentation was removed on 15 June, and Search Console API support went in August. And structured data is not a ranking factor. It makes pages eligible for rich results, which is a different claim.

How AI visibility should be measured

Ask an assistant the same question twice and you can get different brands. A panel of ten or twenty-five prompts sampled once a day produces a number whose margin of error exceeds the movement it is supposed to detect. It will show a change every month, and none of those changes will mean anything.

Noise is the dominant term in these measurements:

  • 65% of AI-cited sources change from one day to the next
  • ±44 points of error on a single prompt asked once
  • 57.8% of ChatGPT runs produce zero citations at all

What we do instead, set out in full on the methodology page:

Design choiceWhy
At least 120 prompts, frozen and approved in writing Precision comes from the number of distinct prompts, not from re-asking one. Freezing the set is what makes month two comparable to month one.
Five repetitions per prompt per engine Enough to characterise the noise. Beyond roughly ten, repetition buys almost nothing.
Reported per engine, never blended Engines retrieve differently and move at different times. A blended figure hides the only actionable information.
Every figure with a confidence interval So you can tell real movement from sampling noise. Most months there is no movement we can separate from noise, and we will say so.
The measurement layer declared on every number API sampling, consumer interface and search-result capture are three different measurements. We label which produced the figure.

We also measure your own noise floor at onboarding, on your category and an approved competitor set, before reporting any movement, because the floor differs by market.

A note on the founding academic work. The 2023 GEO paper is frequently quoted as evidence of a "40% visibility lift". That figure came from a GPT-3.5-era system, with the source already inside a fixed five-document context, judged by a model from the same family. The mechanisms it identified are worth knowing. The percentage does not transfer to today's engines, and we will not quote it at you as though it does.

Which gate are you failing?

The diagnostic audit answers that in ten business days. Prices and terms are published on the pricing page.

Request an audit

Sources

  • Google Search Central, AI features and your website — both passages quoted verbatim; and the documentation changelog on spam policies applying to generative AI responses.
  • Ahrefs, citation overlap across six engines, 15,000 long-tail queries, August 2025.
  • Ahrefs, llms.txt request measurement, 137,000 domains — its own Web Analytics customers, not the open web.
  • Grossman et al., SIGIR 2026, 11,500 queries, December 2025 — URL-level Jaccard similarity between engines.
  • Aggarwal et al., GEO: Generative Engine Optimization, IIT Delhi and Princeton, KDD 2024. Cited here for its mechanisms and for the limits of its headline figure.