noindex on production: the silent launch killer

A client’s new site went live on a Tuesday. Three weeks later they called: not one page was in Google, and the old site had already been redirected away. The cause was four words in the <head> that nobody looked at, carried over from the staging build.

This is the most expensive pre-launch defect there is, because it is silent. Nothing is broken. Every page loads, every form submits, the client clicks around and signs off. The damage only surfaces weeks later, when the traffic that should have arrived never does.

noindex hides in four places, and only one of them is in your page source

Most launch checklists say “check for noindex” and stop there, which quietly means “view-source and search for the word.” That catches the easiest case and misses three harder ones.

The header case is the one that ruins launches. Vercel sets X-Robots-Tag: noindex on preview deployments automatically, and plenty of agencies add it to an nginx or Cloudflare rule while a site is in build. Promote that config to production and the page source is spotlessly clean while every response tells Google to stay away.

The thirty-second check

Headers first, because they are the invisible case:

$ curl -sI https://clientsite.com/ | grep -i robots
x-robots-tag: noindex, nofollow

No output means no header, which is what you want. Then the page source, the robots file, and the canonical value:

$ curl -s https://clientsite.com/ | grep -iE 'robots|canonical'
$ curl -s https://clientsite.com/robots.txt

Run all four on the production hostname over HTTPS, not on staging and not on the www variant you happen to have open. Which brings us to the trap.

The trap: Disallow and noindex together do the opposite of what you want

This one catches experienced developers. The instinct, on discovering an accidentally indexed page, is to belt-and-braces it — add noindex to the page and Disallow it in robots.txt. That combination is self-defeating.

“Disallow stops the crawl. If Google can’t crawl the page, it never sees the noindex — so the URL stays in the index indefinitely.”

Blocked-but-indexed URLs are exactly the ones that show up in results with no snippet and the line “No information is available for this page.” The fix is to pick one: noindex and let it be crawled until it drops out, or Disallow for pages you simply never want fetched. Never both on the same URL.

If you have already shipped it

Removing the tag is step one, but Google will not re-crawl on your schedule. The recovery has a shape worth setting expectations around:

1. Remove every instance — meta tag, header, robots rule, and any canonical pointing off-host. Verify with the curl commands above, on production.
2. Request indexing in Search Console via URL Inspection for the homepage and your three or four most important pages. This is the only lever that meaningfully speeds things up.
3. Submit the sitemap again so the remaining URLs get queued.
4. Expect days, not hours, for the pages you requested, and longer for the rest of the site. The deeper the page and the newer the domain, the slower it goes.

There is no penalty to recover from — you were never punished, you were obeyed. But the weeks the site spent invisible are not refundable, which is the entire argument for catching it before the handoff rather than after.

Why this check belongs on the staging build

The honest problem with every checklist item above is when you run it. Checking production after launch tells you about a mistake you have already made. The check is only worth anything on the staging build, before the DNS moves — and staging is usually behind HTTP basic auth or a platform password, which is precisely where a paste-a-URL scanner returns the login wall instead of your site.