Robots.txt generator without the mistakes
Choose what your site runs on, add the paths robots should stay out of and the address of your sitemap. You copy or download the finished robots.txt, and the tool warns you about the most common mistakes. Nothing is sent anywhere: the file is built in your browser.
robots.txt
User-agent, Disallow, Sitemap
What goes into a robots.txt
Robots.txt is a plain text file in the root of your site (yoursite.com/robots.txt) in which you tell search engines and other robots which parts of the site they may visit and which they should skip. The format is defined by RFC 9309. The file should be UTF-8, and Google reads the first 500 KiB of it, which is far more than an ordinary site needs.
For most business sites a few lines are enough:
- The paths that waste your server's time or do not belong in search: the admin area, the cart, the login, the internal search.
- A Sitemap line with the address of your sitemap, so robots find all your pages faster.
- On WordPress, a block on the /wp-admin/ folder with an exception for admin-ajax.php, which some features of the public pages need.
The mistakes it catches
A mistake in this file is usually noticed late, sometimes only when the site drops out of search. So the generator checks for:
- a path with no leading slash, or with a space inside it,
- a block on the whole site (Disallow: /), which belongs only on a site that is still in development,
- a block on CSS or JavaScript files, which search engines need to render your pages,
- a missing sitemap address.
What robots.txt cannot do
It is not a lock and it hides nothing. A page that other sites link to can still show up in search, just without a description. What must stay private belongs behind a password, and what should only stay out of search needs a noindex tag, which a robot has to be allowed to read, so do not block that page in robots.txt.
Google ignores the Crawl-delay line, so the generator does not write it. Do not block the folder with your site icon either: Google must be able to fetch it to show it in search. And remember that the rules are a request that well-behaved robots follow, not a way to keep out bad ones.
Frequently asked questions
How do I create a robots.txt file?
Choose what your site runs on, adjust the paths and enter your sitemap address. Download the file and upload it to the root of your site so that it opens at yoursite.com/robots.txt. Then open that address in your browser to check it.
Where does robots.txt go?
In the root of the domain, not in a subfolder. Robots look for it at the address with the file name straight after the domain. Every subdomain needs a file of its own.
What does a robots.txt for WordPress look like?
Most often it blocks the /wp-admin/ folder, allows /wp-admin/admin-ajax.php and adds a Sitemap line. Pick WordPress at the top and the generator fills in exactly that.
How do I add my sitemap to robots.txt?
On a line of its own write Sitemap: followed by the full address of the sitemap, for example https://www.example.com/sitemap.xml. The address must be complete, with https. You can build the sitemap with our sitemap.xml generator.
Does robots.txt remove a page from Google?
No. It stops robots from fetching the page, but the address can still appear in results. To keep a page out of Google, use a noindex tag and let robots read the page.
How do I test my robots.txt?
Open yoursite.com/robots.txt in a browser and check that you see plain text with no errors. The robots.txt report in Google Search Console shows which file Google fetched and any problems it found with it.
Does this tool block AI crawlers?
It writes the rules for all robots (User-agent: *). Rules for a specific AI crawler are a separate group that names its user agent, and they are only followed by crawlers that choose to respect robots.txt.
Need a website as well?
Search settings such as robots.txt and the sitemap are set up by us on the sites we build. You see a free demo of your site before you pay anything.

