Free technical SEO tool · no sign-up
Free Robots.txt Generator
Build a valid robots.txt in seconds. Choose a preset, add your sitemap and any paths to block, then copy or download the file to your site root.
What is a robots.txt file?
robots.txt is a plain-text file at the root of your site (example.com/robots.txt) that tells search-engine crawlers which parts of your site they may or may not crawl. It is the first file most bots request when they visit.
It is a set of polite instructions, not a security barrier — well-behaved crawlers like Googlebot obey it, but it does not stop a page from being indexed if other sites link to it. To truly keep a page out of search, use a noindex tag or password protection.
Why robots.txt matters for SEO
- Crawl budget. Blocking low-value paths (faceted search, internal cart pages) lets crawlers spend their time on your important pages.
- Sitemap discovery. Listing your sitemap helps search engines find every URL you want indexed.
- Avoid wasted crawling of staging files, admin areas, or duplicate parameters.
robots.txt syntax basics
User-agent: *— applies the rules to all crawlers.Disallow: /admin/— blocks a folder.Disallow:(empty) — allows everything.Allow: /folder/page.html— re-allows one URL inside a blocked folder.Sitemap: https://…/sitemap.xml— points crawlers to your sitemap.
Common mistakes to avoid
- Disallowing your whole site by accident (
Disallow: /) — this de-indexes everything. - Blocking CSS or JS Google needs to render the page — keep those crawlable.
- Using robots.txt to hide private pages — it does not stop indexing; use noindex or auth.
- Forgetting to upload the file to the root (it must be at
/robots.txt).
Frequently asked questions
Where do I put the robots.txt file?
In the root directory of your domain, so it is reachable at https://yourdomain.com/robots.txt. It will not work in a subfolder.
Does robots.txt stop a page from being indexed?
No. It only requests that crawlers do not crawl a path. A blocked page can still be indexed if other pages link to it. To prevent indexing, use a noindex meta tag or password protection.
Should I block CSS and JavaScript?
No. Google renders pages like a browser and needs your CSS and JS to understand layout and content. Blocking them can hurt rankings.
Do I need a robots.txt at all?
It is optional, but recommended. Even a simple file that allows everything and points to your sitemap helps crawlers, and avoids 404 noise in your logs.
Is this generator free?
Yes — free, no sign-up, generated entirely in your browser.