Four Files Every Production Website Should Have

Some of the highest-leverage improvements you can make to a website aren't in the app at all — they're small static files at the root that crawlers, browsers, and researchers look for by convention. Here are four worth having on every production site.

1. robots.txt

Lives at /robots.txt. Tells crawlers which paths they may request, points them at your sitemap, and — increasingly relevant — lets you opt out of AI training crawlers like GPTBot and ClaudeBot. Keep it permissive by default; only disallow what genuinely shouldn't be indexed. → Robots.txt Generator

2. sitemap.xml

A list of the URLs you want discovered, in the sitemaps.org XML format. It won't force indexing, but it helps search engines find pages that aren't well linked, and honest <lastmod> dates help them re-crawl efficiently. Reference it from robots.txt and submit it in Search Console. → Sitemap.xml Generator

3. .well-known/security.txt

An RFC 9116 file at /.well-known/security.txt that tells security researchers how to report a vulnerability. Two required fields (Contact and Expires) and a handful of optional ones. It shortens the path from "found a bug" to "reported it responsibly." → Security.txt Generator

4. manifest.json

The web app manifest makes your site installable as a Progressive Web App — controlling the home-screen name, icons, splash screen, and display mode. Link it with <link rel="manifest">. → PWA Manifest Generator

The theme

None of these require a backend or a build step — they're static files you generate once and drop at a known path. Together they make your site easier to crawl, safer to disclose to, and installable. Each of the tools linked above builds the file for you, entirely in the browser.