Robots.txt Generator
Build a valid robots.txt for search engines and AI crawlers — configured entirely in your browser
Examples
Protect private paths and publish a sitemap
User-agent: *
Allow: /public/
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlUser-agent: *
Allow: /public/
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
Allow and Disallow values are emitted in their input order, and the sitemap is appended after a blank line.
Block every path for every crawler
Preset: Block allUser-agent: *
Disallow: /
The Block all preset creates one wildcard group with a root disallow rule.
Add a crawler-specific delay
User-agent: Googlebot
Allow: /
Disallow: /private/
Crawl-delay: 5User-agent: Googlebot
Allow: /
Disallow: /private/
Crawl-delay: 5
A trimmed non-empty crawl-delay is included after the path rules for that user-agent group.
About this tool
A robots.txt file gives automated crawlers path rules at the root of a site. This generator builds one or more User-agent groups, adds Allow and Disallow paths one per line, optionally emits Crawl-delay, and appends a Sitemap directive when a sitemap URL is provided.
The output preserves the order of your groups and paths, trims each value, defaults a blank user-agent to *, and separates groups with blank lines. Three presets cover common starting points: Allow all, Block all and Block AI bots, the last of which creates separate disallow-all groups for the AI bot names included in the component.
Generation is entirely local and performs string assembly rather than fetching or validating your site. Save the result as robots.txt at the site's root, then verify the rules against the crawler behavior and deployment path you actually use.
How to use
Choose a starting preset
Use Allow all, Block all or Block AI bots when one of those policies is close to your target, or edit the initial group directly.
Configure a user-agent group
Enter a crawler name, put allowed paths in the Allow box, and put blocked paths in the Disallow box with one path per line.
Add optional controls
Add more groups when different crawlers need different rules, and enter a Crawl-delay only when the target crawler supports the directive you need.
Add the sitemap and copy
Enter an optional sitemap URL, copy the generated file, and save it as /robots.txt at the site root.
Use cases
Keeping administration paths out of crawler queues
Create a wildcard group that disallows admin, private or internal paths while leaving public content crawlable.
Creating a crawler-specific policy
Add separate user-agent groups when a search crawler, partner bot or internal crawler needs rules different from the wildcard group.
Publishing a sitemap location
Append the canonical sitemap URL to the root file so crawlers can discover the site's URL inventory from the same entry point.
Starting an AI crawler policy
Use the Block AI bots preset as a starting list, then review and maintain the named bot groups according to the policy your site actually wants.
Common mistakes
Mistake:Saving robots.txt somewhere other than the site root.
Fix:Place it at the origin root, such as https://example.com/robots.txt. Rules in a subdirectory do not control the whole site.
Mistake:Assuming Disallow is an access-control mechanism.
Fix:Robots rules are crawler instructions, not authentication. Protect private data with authorization and server-side access controls as well.
Mistake:Expecting every crawler to honor Crawl-delay.
Fix:Crawl-delay is not uniformly supported. Use it only as a crawler-specific hint and rely on rate limiting or server controls when traffic must be constrained.
Mistake:Assuming the Block AI bots preset covers every automated client.
Fix:The preset only creates groups for the AI bot names listed in the component. Review the generated names and maintain the file as crawler identifiers change.
Frequently asked questions
Related guides
XML Sitemaps Explained: lastmod, changefreq, and priority
What an XML sitemap is, what the optional tags actually do, how search engines treat them, and how to generate a valid one from a list of URLs.
How to Block AI Crawlers with robots.txt (GPTBot, ClaudeBot & More)
A practical guide to opting your site out of AI training crawlers using robots.txt — which user-agents to block, what it does and doesn't do, and a ready-to-copy example.
Four Files Every Production Website Should Have
robots.txt, sitemap.xml, security.txt, and the web app manifest — the small root-level files that improve crawling, security, and installability.
robots.txt vs meta robots vs X-Robots-Tag: Which Controls What
Three different mechanisms control crawling and indexing, and they're not interchangeable. Here's what each one does and the classic mistake that hides pages from Google.
References & standards
Related tools
Address Generator
Generate realistic addresses in the correct format for 21 countries — single-line, multi-line, JSON or HTML — plus a postcode validator. Test data only, runs in your browser.
ASCII Text Art Generator
Turn text into large ASCII art banners with multiple figlet fonts
Credit Card Generator
Generate Luhn-valid test credit card numbers for development and QA (not real cards)
Driver's License Generator
Generate driver's license numbers in the correct format for 21 countries and all 50 US states, plus a format and Luhn validator — test data only, runs in your browser.
Email Signature Generator
Build a clean, cross-client HTML email signature — name, title, company, and contact links with an accent color and a live preview
Favicon Generator
Generate favicons (16x16, 32x32) from any image.