Free Robots.txt Checker

What is a robots.txt file?

A robots.txt file is a short text file at the top of your website, at yourbusiness.com/robots.txt. It tells crawlers (the programs search engines and other services use to read websites) which parts of your site they may read. A simple one looks like this:

User-agent: *
Disallow: /admin/

Sitemap: https://yourbusiness.com/sitemap.xml

User-agent says which crawler the rules below are for (* means all of them). Disallow asks it not to read addresses that start with that path, and Allow makes an exception. Sitemap points to the list of your pages.

You don't have to have one. With no robots.txt file, search engines simply read everything they can find, which is fine for most small-business websites.

How to read your results

• Found / not found: what your site answered when we asked for /robots.txt. “Not found” (a 404) means crawlers may read every page. A server error is different: Google pauses crawling a site while its robots.txt gives server errors, so that one matters.
• What each crawler may do: for Google, Bing, the best-known AI crawlers and our own reader, whether the file lets them read everything, blocks some paths, or blocks the whole site.
• Sitemaps: each Sitemap line, and whether that address loads. A sitemap that doesn't load is no help to search engines.
• Problems and notes: lines crawlers ignore or may misread, with the line number, such as a misspelled rule, a missing colon, or a rule before any User-agent line.
• Test a web address: type a page and pick a crawler to see if it is allowed, and which line decides it.

The most common mistake: blocking your whole site

These two lines ask every crawler, Google included, to stay out of every page:

User-agent: *
Disallow: /

They are often added while a site is being built, then forgotten at launch, or copied over from a test copy of the site. Google then stops reading your pages, and over time they can drop out of search results or show up with no description. If the checker shows Blocked from the whole site for Googlebot and you didn't mean it, delete the Disallow: / line (or change it to Disallow: with nothing after it), then check again.

On WordPress, also look at Settings → Reading → Search engine visibility. If “Discourage search engines from indexing this site” is ticked, current WordPress versions add a noindex tag to your pages instead of a robots.txt rule, which has much the same effect. Untick it once the site is live.

Blocking AI crawlers

Some AI companies read websites to train their models or to answer questions. You can ask them to stay out, crawler by crawler, for example:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Whether to do this is up to you: some owners want their content kept out of AI training; others want AI assistants to know about their business. A few things to know:

• It only works for crawlers that follow robots.txt and use the name you block. The big companies say theirs do; not every crawler on the internet does.
• Blocking an AI crawler doesn't remove your pages from Google or Bing search, as long as Googlebot and Bingbot are still allowed.
• Some AI tools fetch a page only when a person asks about it, under a different name than their training crawler, so one block may not cover everything a company does.

Our Robots.txt Generator can add these lines for you.

robots.txt is a request, not a lock

• It isn't security. Anyone can open your robots.txt and read it, and badly behaved crawlers ignore it. Listing a private folder in it even tells people where to look. Pages that must stay private need a password or a login.
• Blocking Googlebot hurts your Google results: it can't read the pages, and they fade from search.
• Disallow doesn't remove a page from Google. A blocked page can still be listed (without a description) if other sites link to it. To keep a page out of search results, let Google read it and add a noindex tag to the page instead.

How to change robots.txt on your website

Wix

Wix makes a robots.txt file for you. To add your own rules, open your site's dashboard, go to the SEO settings (under Marketing & SEO) and look for the Robots.txt Editor in the tools and settings. Wix's help center has the current steps if the menu has moved.

Squarespace

Squarespace makes the file for you and doesn't let you edit it. Its settings include a switch to ask AI crawlers to stay out (search the settings for “crawlers”).

Shopify

Shopify makes a robots.txt file for every store. To change it, edit your theme's code and add a robots.txt.liquid template. Do this only if you are comfortable editing theme code; Shopify's help center explains the steps.

WordPress

If there is no real file, WordPress shows a basic robots.txt for you. SEO plugins such as Yoast SEO and Rank Math have a robots.txt editor in their tools or settings. You can also upload a file named robots.txt to your site's main folder with your host's file manager.

After any change, run the checker again. Search engines read robots.txt again on their own, usually within a day.

What this checker does, and its limits

• Asks your site for /robots.txt from our server, following up to 5 redirects (each one checked), and reads up to 1 MB. Google reads the first 500 KiB, so we use the same part.
• Reads the file the way Google documents: groups of rules per crawler, the longest matching rule wins, Allow wins a tie, and * and $ work as wildcards. Other search engines and crawlers may read some details differently.
• Checks up to 4 sitemap addresses from the file, by asking for each one without downloading it. It doesn't check what is inside a sitemap; the Sitemap Generator does that.
• Robots.txt covers one address. www.yourbusiness.com and yourbusiness.com (and http:// and https://) each have their own file; we read the one for the address you enter.
• Introduces itself as FlatRateChatBot. Some sites block automated checks or show them a different answer than search engines get; then the result says so instead of guessing.
• Up to 60 checks a day from one connection, and 30 a day of the same website. No report is saved: we keep only the counters for these daily limits and anonymous usage counts.
• Also free: the Broken Link Checker tests every link on your pages.
• Also free: the Sitemap Generator makes an XML sitemap of your pages and checks the one you have.
• Also free: the Google Maps Embed Generator makes the code to show your business on a Google map on your website.
• Also free: the WhatsApp Link Generator makes WhatsApp, click-to-call and text links, website buttons and a QR code.
• Also free: the Google Review Link Generator turns your Google review link into a QR code, a printable sign and ready-to-send messages.
• Also free: the Website Down Checker checks from our server whether a website answers right now, and why not.
• Also free: the Redirect Checker shows every redirect a web address goes through, step by step.
• Also free: the SSL Checker checks that your site opens securely on https and shows its certificate details.
• Also free: the Robots.txt Generator makes a robots.txt file for your website and checks it as you type.
• Also free: the Website Accessibility Checker scans one page for common accessibility problems, like images without alt text and form fields without labels.
• Also free: the Meta Tag Checker shows the title, description and other tags Google reads on a page, with a search preview.
• Also free: the Open Graph Checker shows how a page will look when shared on Facebook, LinkedIn and X, and checks its og:image.

Questions