robots.txt vs meta robots vs X-Robots-Tag: Which Controls What
July 7, 2026 · DevTools
There are three ways to tell search engines what to do with your pages, and confusing them causes real damage. The key distinction: crawling (can the bot fetch the page?) versus indexing (can the page appear in results?).
robots.txt — controls crawling
A root-level file that tells bots which URL paths they may request. It's about crawl access, not indexing.
User-agent: *
Disallow: /admin/
Crucially, a Disallowed URL can still appear in search results (as a bare link with no snippet) if other sites link to it — because the bot was told not to fetch it, so it never saw a noindex.
meta robots — controls indexing (per page)
An HTML tag in the <head> that controls whether this page is indexed:
<meta name="robots" content="noindex, follow">
noindex keeps the page out of results while still letting the bot crawl links on it. This is the correct tool for "don't show this page."
X-Robots-Tag — indexing for non-HTML
The same directives as meta robots, but sent as an HTTP header — so it works for PDFs, images, and other non-HTML resources that can't carry a <meta> tag:
X-Robots-Tag: noindex
The classic mistake
Blocking a page in robots.txt and adding noindex to it. The bot obeys robots.txt, never fetches the page, and therefore never sees the noindex — so the URL can linger in the index. To reliably remove a page, allow crawling and use noindex.
Generate them right
Use the Robots.txt Generator for crawl rules and the Meta Tag Generator for per-page <meta> directives — and remember which layer each one operates on.