Home / Free tools / Robots.txt tester
Free tool

Robots.txt tester. Free, no signup.

Read a site's robots.txt and test whether a path is allowed for Googlebot, Bingbot or AI crawlers such as GPTBot.

How to use it

  1. Type the domain and the path you want to test.
  2. Pick the crawler.
  3. See whether the path is allowed and which rule decides it.
FAQ

Questions about the robots.txt tester.

Which rule wins when two rules match?

The longest matching rule wins; when an Allow and a Disallow are the same length, Allow wins.

Does Disallow remove a page from Google?

No. It stops crawling, not indexing. Use a noindex tag on a crawlable page to remove it.

Can I block AI crawlers?

Yes, GPTBot, ClaudeBot and others follow their own user-agent groups in robots.txt.

What if there is no robots.txt?

Then everything may be crawled. That is fine for most sites.

More free SEO tools

Website traffic checkerSee how much organic search traffic a website gets, how many keywords it ranks for, and the searches and pages that bring it the most visits.FreeBacklink checkerCount the links pointing to any website and the sites they come from, and see the strongest linking domains with their anchor text.FreeSpam score checkerCheck how spammy a website's backlink profile looks and see the links with the highest spam score.FreeCompetitor keyword finderSee the keywords any website ranks for in Google, with search volume, position and the page that ranks.FreeKeyword generatorTurn one word or phrase into keyword ideas with monthly search volume, difficulty and search intent.FreeCompetitor analysisSee a competitor's top keywords and top pages, and, with your own domain, the keywords they rank for that you do not.Free

Want every result, not just the top 10? Create a free account: 1,000 credits every month, full results on every free tool, no card.

Create a free accountSee pricing
14free SEO tools