Auspex.

Tools / Index check

What would the index keep?

Gate 2 is indexation. A page every crawler can fetch can still be canonicalised to another URL, excluded by a directive, or served with no text until scripts run. This check fetches one address as Googlebot and reads what the HTML says before anything renders. It is free, stores nothing, and does not claim to know what the engine decided.

Fetches the page once as Googlebot, the site's robots.txt, and its sitemap. The first megabyte of HTML is read and discarded. Six checks a minute per visitor, four a minute per site.

What it reports

The status and any redirects. The canonical, and whether it names this URL or another. Every robots directive, in the page and in the X-Robots-Tag header. The title, description, h1 and language. How many words exist in the HTML before any script runs, and whether the page's root element is empty, which is the shape of a site that renders everything client-side. The structured-data types declared. And whether the URL appears in the sitemap the site publishes.

The verdict is one of three. Not kept, when a directive excludes the URL. Canonicalised away, when the page names another URL as the one to keep. Eligible, when nothing in the HTML excludes it. Eligible is not indexed. The last row of every result says so.

The gate before this one

A page has to be fetchable before any of this matters. The crawl check reads gate 1 per named crawler: what the site returns to each, and what its robots.txt says to each.