Website not indexed: why Google isn't showing your pages

In Search Console, under Indexing > Pages, Google splits your site into two groups: pages that are indexed and pages that aren't, with the reason each one was left out. The second number is often bigger than the first, and on its own it tells you nothing. To the question "Is it OK if a page isn't indexed?", the report's help page answers "Absolutely". The same page also says that all your important pages should be indexed, and that's where to start.

First, look at which pages are missing

In its guide to how Search works Google says it doesn't guarantee it will crawl or index a page, even one that follows all the rules. The report spells out the goal: get the canonical version of every important page indexed.

For each excluded address, ask whether a customer should be able to find it on Google. A service page or an article, yes. An address with filter parameters or an old URL that now redirects, no, and it's right that they stay out.

On a large site the list won't show you everything: the same help page warns that the example URLs are capped at 1,000. It also suggests submitting a sitemap with only your most important pages and filtering the report by it. Google offers that as a way to speed up fix validation, but in the meantime the list shows only the addresses that matter. To check one page at a time there's the URL Inspection tool, which tells you whether Google knows the page, which canonical it picked and whether it found a noindex.

The exclusion reasons that matter

The names below are the ones in Google's documentation; in Search Console you'll find them in the "Why pages aren't indexed" table.

Crawled - currently not indexed

Google read the page and decided, for now, not to put it in the index. The help page says there's no need to resubmit it for crawling, because Google has already seen it.

There's no technical error to fix. Usually Google found nothing in the page that it didn't already have somewhere else: a product page with the description copied from the manufacturer, or the same page rebuilt once for every city. The guidance on helpful content asks whether you are "mainly summarizing what others have to say without adding much value", and the spam policies treat large numbers of pages made to rank as abuse, no matter how they were created. So either the page becomes useful, or you leave it out and link better to the ones that matter, starting from pages Google already considers important.

Discovered - currently not indexed

Google knows the address but hasn't visited it yet. According to the help page, it typically rescheduled the crawl because crawling was expected to overload the site. If this status sits on many pages for weeks, either the server is slow to respond or there are lots of pages poorly linked to each other.

Duplicate, Google chose a different canonical

The full name is "Duplicate, Google chose different canonical than user". You marked a page as the main one with the canonical tag and Google preferred another. It's allowed to: the canonicalization guide says a canonical preference is a hint, not a rule.

The typical case is one page served from several addresses: with and without www, over http and https, with and without a trailing slash, with and without a language prefix. The fix is to keep one address and permanently redirect the others: for Google 301 and 308 are permanent redirects and signal that the destination should be the canonical.

URLs with noindex or blocked by robots.txt

"URL marked 'noindex'" means the page itself is asking to stay out. If you put it there, fine. If not, take it off. It happens after a rebuild, when the staging site had noindex and the new version goes live on the same template.

A noindex can live in two places: a meta tag inside the page, or the X-Robots-Tag header the server sends along with it. You won't see the second one by looking at the page source. Our free visibility tool checks both on the home page.

"URL blocked by robots.txt" is a different thing, and it's the wrong tool for hiding a page. The report's help page warns that the block doesn't guarantee the page stays out of the index. Google can't read the page, so it can't see a noindex either: to keep the page out, remove the block and use noindex.

Redirects and 404 pages

An address that redirects doesn't go into the index; Google evaluates the destination instead. A 404 is fine if the page was removed and has no replacement. If it moved, it needs a 301 to the new address.

A soft 404 is a page that says "not found" but answers with a success code. Google discards it, and when there are many of them Googlebot wastes its visits on empty pages. Internal searches with no results and e-commerce categories with no products left are the usual sources.

The /it URLs Google had found on our site

Our site has Italian at the root and English under /en. In September, going through Search Console, we found some pages indexed with /it in front as well: /it/servizi/seo next to /servizi/seo, for example. A configuration flaw made the site answer at those addresses too, so we had two URLs for the same page and Google could pick the wrong one.

On 19 September we closed it with a permanent redirect (308) from /it and everything under it to the address without the prefix. Every page declares itself as canonical and points to the version in the other language with hreflang, as the guide to localized versions asks. On 22 October the sitemap listed 83 addresses across Italian and English.

We'll read the report again on 7 December and 4 January, and the number we watch is pages crawled but not indexed. If it grows while we publish articles, Google finds them not very useful, and the sensible move is to publish fewer and improve the ones we have.

Timing and indexing requests

There's no fixed time for a new page to get indexed. Google writes that crawling can take anywhere from a few days to a few weeks and that requesting it doesn't guarantee inclusion in the results.

The "Request indexing" button in URL Inspection is for a page that's new or has changed a lot. There's a quota for individual submissions, and asking several times for the same address won't get it crawled any faster. For many pages you use a sitemap, which helps Google find addresses but doesn't guarantee they'll be crawled and indexed; on a site of about 500 pages or fewer, well linked to each other, Google says you might not need one. If you do submit it, use real modification dates: Google uses the lastmod field only if it's consistently accurate, and ignores priority and change frequency.

For a first check from the outside there's our free visibility tool: give it the address and it tells you whether the page asks to be kept out of the index, whether the text arrives without JavaScript and whether the other pages can be reached from the first one through internal links and the sitemap.

If the report shows important pages left out of the index and you can't see why, write to us: we'll go through it with you and check addresses, canonicals and redirects in the site's code.