How to Block AI Crawlers with robots.txt (GPTBot, ClaudeBot & More)

Over the last couple of years, most large AI labs have started publishing named crawlers so site owners can control whether their content is used for training. The mechanism is the same one search engines have used for decades: robots.txt.

What robots.txt can (and can't) do

robots.txt is a request, not an enforcement mechanism. Well-behaved crawlers read it and obey it; a malicious scraper can ignore it entirely. The major AI companies have committed to honoring it, so for opting out of training corpora it's the right tool — but don't treat it as access control. Anything you truly need private belongs behind authentication.

The crawlers worth knowing

Each company uses one or more user-agents:

• OpenAI — GPTBot (training), ChatGPT-User (live browsing on a user's behalf)
• Anthropic — ClaudeBot, anthropic-ai
• Google — Google-Extended (controls Gemini/Vertex training without affecting Search)
• Common Crawl — CCBot (a dataset many models are trained on)
• Perplexity — PerplexityBot

Note Google-Extended: blocking it opts you out of AI training while keeping normal Google Search indexing intact. That's usually what people actually want.

A ready-to-use block

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: *
Disallow:

The final User-agent: * group with an empty Disallow: keeps your site fully open to everything else — including Googlebot and Bingbot.

Do it without the typos

Grouping and slashes in robots.txt are easy to get subtly wrong, and a stray Disallow: / in the wrong group can deindex your whole site. The Robots.txt Generator has a one-click "Block AI bots" preset that emits exactly the groups above, and lets you add your own allow/disallow paths and a sitemap directive on top. Pair it with an XML sitemap so the search engines you do welcome can find every page.