Free Robots.txt Generator

What goes in a robots.txt file?

A robots.txt file asks crawlers (the programs search engines and other services use to read websites) to stay out of some parts of your site. It lives at yourbusiness.com/robots.txt. Most small-business sites need only a few lines:

User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

• User-agent names the crawler the next rules are for. * means every crawler that has no group of its own.
• Disallow asks it not to read addresses that start with the path. Disallow: /cart covers /cart, /cart/ and /cart?item=2.
• Allow makes an exception inside a blocked folder. When two rules match, Google follows the longer one; if they are the same length, Allow wins.
• Sitemap gives the full address of your sitemap, so search engines find all your pages.
• * in a path matches anything, and $ at the end means “the address ends here”: Disallow: /*.pdf$ covers every PDF.

You don't have to block anything. If you have nothing to keep out, the “Everything” choice above is the right one.

Good things to block (and things not to)

• Worth blocking: shopping cart and checkout pages, internal search results (/search, /?s=), and admin areas. They aren't useful in search results.
• Don't block your pages, images, style sheets or scripts. Google needs them to see your site the way visitors do.
• Never use Disallow: / for all crawlers on a live site unless you want it out of search engines: it asks Google to stop reading every page. That's why this generator doesn't offer it.

robots.txt is a request, not a lock

• It isn't security. Anyone can open your robots.txt and read it, and badly behaved crawlers ignore it. Listing a private folder there even tells people where to look. Pages that must stay private need a password or a login.
• Blocking Googlebot hurts your Google results: it can't read the pages, and they fade from search.
• Disallow doesn't remove a page from Google. A blocked page can still be listed (without a description) if other sites link to it. To keep a page out of search results, let Google read it and add a noindex tag to the page.
• AI crawlers: the AI lines only work for crawlers that follow robots.txt and use those names. Some AI tools read a page only when a person asks about it, under a different name. Blocking AI crawlers doesn't affect Google or Bing search.

How to add it to your website

Wix

Wix makes a robots.txt file for you, so you don't upload one. To add your rules, open your site's dashboard, go to the SEO settings (under Marketing & SEO) and look for the Robots.txt Editor in the tools and settings. Paste in the lines you need, then save. Wix's help center has the current steps if the menu has moved.

Squarespace

Squarespace makes robots.txt for you and doesn't let you edit it. If you want to ask AI crawlers to stay out, use its crawler setting (search the settings for “crawlers”). Your Squarespace sitemap is at yourdomain.com/sitemap.xml.

Shopify

Shopify makes a robots.txt file for every store. To change it, edit your theme's code and add a robots.txt.liquid template; Shopify's help center explains how to add rules there. Do this only if you are comfortable editing theme code: a mistake can affect your whole store's search listing.

WordPress

If there is no real file, WordPress shows a basic robots.txt for you. To use yours: SEO plugins such as Yoast SEO and Rank Math have a robots.txt editor in their tools or settings, where you can paste it. Or upload the downloaded robots.txt to your site's main folder (the one with wp-config.php) with your host's file manager or FTP.

Other hosts

Upload the file to the main folder of your website (often called public_html or www), so it opens at yourbusiness.com/robots.txt. Then check it with the Robots.txt Checker.

What this generator does, and its limits

• Runs in your browser. Nothing you type is sent anywhere.
• Checks the file with the same rules our Robots.txt Checker uses, which follow Google's documented reading of robots.txt. Other crawlers may read some details differently.
• Doesn't check that your sitemap address works; the checker does, once the file is on your site.
• Also free: the Broken Link Checker tests every link on your pages.
• Also free: the Sitemap Generator makes an XML sitemap of your pages and checks the one you have.
• Also free: the Google Maps Embed Generator makes the code to show your business on a Google map on your website.
• Also free: the WhatsApp Link Generator makes WhatsApp, click-to-call and text links, website buttons and a QR code.
• Also free: the Google Review Link Generator turns your Google review link into a QR code, a printable sign and ready-to-send messages.
• Also free: the Website Down Checker checks from our server whether a website answers right now, and why not.
• Also free: the Redirect Checker shows every redirect a web address goes through, step by step.
• Also free: the SSL Checker checks that your site opens securely on https and shows its certificate details.
• Also free: the Robots.txt Checker reads your robots.txt and shows whether it blocks Google, AI crawlers or your sitemap.
• Also free: the Website Accessibility Checker scans one page for common accessibility problems, like images without alt text and form fields without labels.
• Also free: the Meta Tag Checker shows the title, description and other tags Google reads on a page, with a search preview.
• Also free: the Open Graph Checker shows how a page will look when shared on Facebook, LinkedIn and X, and checks its og:image.

Questions