Robots.txt vs Meta Noindex: Which One to Use
Robots.txt vs meta noindex explained simply: robots.txt controls crawling, noindex controls indexing. Learn which to use, and the order mistake that keeps pages stuck.
Use meta noindex when you want a page kept out of search results, and use robots.txt when you want to stop crawlers from fetching a page at all. They are not interchangeable. Robots.txt controls crawling. Meta noindex controls indexing. The common mistake is blocking a page in robots.txt and expecting it to disappear from Google. That does the opposite. If Google cannot crawl the page, it cannot read the noindex tag, so the page can stay indexed with no description.
What does robots.txt actually do?
Robots.txt is a file at the root of your domain that tells crawlers which paths they may request. A Disallow rule means "do not fetch this." It is a crawling instruction, not an indexing one. A page blocked in robots.txt can still show up in search if other sites link to it. Google just shows the URL with no snippet because it never read the content.
Robots.txt is good for saving crawl budget on large sites, blocking admin or faceted URLs, and keeping bots out of heavy endpoints. It is bad at hiding pages from results.
What does meta noindex do?
Meta noindex is a tag in the page head or an X-Robots-Tag HTTP header. It says "you may crawl this, but do not put it in the index." Google fetches the page, reads the tag, and drops it from results. This is the correct tool when you want a page gone from search.
- HTML tag:
<meta name="robots" content="noindex"> - Header version:
X-Robots-Tag: noindex(use this for PDFs and non-HTML files) - Add
nofollowonly if you also want to stop link equity flowing from that page
Which one to use, and when?
- Want a page out of search results: use meta noindex, and make sure the page is NOT blocked in robots.txt so the tag can be read.
- Want to save crawl budget on junk URLs that were never indexed: use robots.txt Disallow.
- Want to hide a private page: use real authentication, not either of these. Both are public and advisory.
- Want a non-HTML file (PDF, image) out of results: use the X-Robots-Tag header.
The trap that keeps pages stuck in Google
If a page is already indexed and you want it removed, do not add a robots.txt block. Leave crawling open, add noindex, and wait for Google to recrawl. Once it drops out, you can block it in robots.txt if you want. Do it in the wrong order and the page lingers for months. This is the single most common indexing bug on small sites.
Why this matters for AI answers too
Search engines are not the only readers now. ChatGPT, Perplexity, and Google AI Overviews crawl and cite pages. If your best pages are accidentally blocked or noindexed, an AI assistant reading the web may skip you and recommend a competitor instead. Directive mistakes quietly cost you both rankings and citations. If you are thinking about how models see your site, start with what AI visibility is.
A quick checklist before you ship
- One directive per goal. Do not stack robots.txt Disallow and noindex on the same page.
- Removing a page from search: noindex first, crawlable, then block later if needed.
- Saving crawl budget on never-indexed URLs: robots.txt is fine.
- Confirm your sitemap does not list noindexed or disallowed URLs.
- Recheck after deploy. Templates and CMS plugins love to add stray noindex tags.
Get the order right and your important pages stay visible to search engines and to the AI tools people now ask for recommendations. When you are ready to go deeper on the technical side, a full free SEO audit covers the rest of the crawl and index picture.