Auspex.

Tools / Crawl check

Does your site let the crawlers in?

Gate 1 is crawl access, and it is the gate most sites fail without knowing. This check requests one address as each of nine named search and AI crawlers, reads the site's robots.txt for each, and reports what came back. It is free, it stores nothing, and it says what it cannot see.

Sends ten requests to the address: one for robots.txt, then one each as Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, GPTBot, Applebot and Amazonbot. Bodies are not read. Six checks a minute per visitor, four a minute per site.

What it reports

For each agent, three things. Whether the site's robots.txt allows the path for that agent, under the same longest-match rule the crawlers use. The status the address returned to a request carrying that agent's name, with any redirects followed. And the headers that decide the rest of gate 1 and part of gate 2: X-Robots-Tag, the server, and any Cloudflare mitigation header, which is where a challenge shows up.

The interesting result is disagreement. A robots.txt that allows an agent the edge then blocks. A rule written for a name the crawler no longer uses. A retrieval agent treated as a training crawler. Each is a gate-1 failure that a robots.txt audit on its own would pass.

What it does not report

Whether the page is indexed, whether it is retrieved for the sub-queries a buyer's question turns into, whether it is cited, or whether the citation shaped anything. Those are gates 2 to 5. The audit run on this site shows the form the full reading takes, and the services page says what the client version adds.