What Is robots.txt?
robots.txt is a plain-text file in the root directory of your site that tells search engine bots which sections they may crawl and which they must stay out of. Every reputable bot, Googlebot included, reads this file before crawling a site and follows its rules. Without the file, bots may crawl everything; with a badly written one, your entire site can drop out of search — few files can do that much damage with a single wrong character. This tool generates a standards-compliant robots.txt without the risk of typos: enter the paths to block and your sitemap address, and get a ready-to-copy file.
How to Generate One
- Choose general access: "allow all bots" is for normal sites; "block everything" belongs only on test sites that are not meant to be public.
- Enter the paths to block: up to 5 paths bots should not enter, such as /admin, /cart or /account. Forget the leading slash and the tool adds it.
- Define an exception if needed: open a single path inside a blocked folder with an Allow line.
- Add your sitemap address: the full URL of your sitemap, so bots discover it while reading the file.
- Copy it to your root directory: save the output as robots.txt; it must be reachable at yoursite.com/robots.txt.
What the Lines Mean
- User-agent: * — the rules apply to all bots. This tool produces a single-group file with the same rules for every bot; extend it by hand if you need per-bot rules.
- Disallow: /admin — blocks crawling of every address starting with /admin. An empty Disallow line means "everything is open"; the tool uses this explicit form recommended by the standard when you enter no paths.
- Allow: /admin/help — opens an exception inside a blocked area; Google and modern bots support this directive.
- Sitemap: https://yoursite.com/sitemap.xml — announces where your sitemap lives; duplicate the line if you have several.
What to Block — and What Not To
Typical candidates for blocking: admin panels, cart and checkout steps, account pages, internal search results and filter parameters that generate endless combinations — pages that have no business in search results and waste crawl budget. On the other hand, never block CSS and JavaScript files: Google evaluates pages by rendering them visually, and a bot that cannot reach your stylesheets sees a broken page, which can hurt your mobile-friendliness assessment. The most critical warning: robots.txt is not a security tool. The file is public; writing "Disallow: /secret-panel" does not hide that address — it advertises it to anyone malicious. Pages that need privacy need password protection.
robots.txt and AI Bots
In recent years the file has gained a new dimension: sites are now crawled not only by search engines but by AI companies' bots such as GPTBot and ClaudeBot, and that content feeds the answers AI assistants give users. robots.txt is where you declare your stance towards these bots: leave the door open and your content gets a chance to appear in AI answers; close it and you forgo visibility on that surface — for a business website, staying open usually makes sense. This tool produces a single-group file applying the same rule to all bots; to treat a specific AI bot differently, add an extra User-agent group with that bot's name to the generated file.
Common Mistakes
- Forgetting the block at launch: a "block everything" file carried over from staging takes your live site out of search entirely; put it on your launch checklist.
- Confusing blocking with noindex: a page blocked from crawling can still appear in the index if other sites link to it. Removing a page from search takes a noindex tag — and for the tag to be seen, the page must remain crawlable.
- Wrong location: the file does not work in a subfolder (/blog/robots.txt); only the root directory counts, and the filename must be all lowercase.
- Not testing: after uploading, verify your rules behave as expected with the robots.txt report in Google Search Console.
Why It Matters
robots.txt is the doorbell of technical SEO: it is the first point of contact between bots and your site, and a mistake here can zero out your visibility no matter how good the content is. A well-configured file steers crawl budget towards your valuable pages and announces your sitemap. Nothing you enter is stored on the server; the file is generated on the fly and returned only to you. To help bots not just crawl but truly understand your pages, add structured data with our Schema.org Generator — or leave technical SEO to the experts via our SEO optimisation service.