Free tool · Technical SEO
Robots.txt and sitemap checker in one pass
I fetch robots.txt, the sitemaps it lists (or /sitemap.xml) and a sample of the URLs inside, from my server. Takes up to 20 seconds. Nothing is saved.
The guide
Two small files that decide what Google crawls
When a site loses traffic, robots.txt and the sitemap are two of the first files I open. They're tiny and nobody looks at them after launch, which is exactly why they go wrong. A staging rule that blocks everything gets copied to the live site. A plugin update starts listing redirected URLs in the sitemap. A developer blocks a folder that turns out to hold the site's JavaScript.
The fix is rarely hard once you see it. The hard part is noticing, because nothing on the site looks broken to a visitor. Pages simply stop being crawled, or new ones never get found, and the traffic graph drifts down for weeks before anyone asks why. Checking these files is part of every audit on my technical SEO services page, and the first thing I look at after a site migration or URL change.
Most robots.txt testers check one thing: whether a URL is allowed. That matters, but the expensive mistakes are elsewhere: a server error on robots.txt, a sitemap full of redirects, or a sitemap that lists pages your own robots.txt blocks. This checker looks at all of it in one pass.
What each result means
Status code of robots.txt
200: Google reads the rules. 404 or 410: no file, so everything is crawlable. Other 4xx: treated like no file. 5xx or timeout: the dangerous one. Google stops crawling the site for a while and then falls back on its last cached copy.
Lines Google ignores
Crawl-delay, Host, Noindex and typos such as "Dissallow" do nothing for Googlebot. Noindex in robots.txt stopped working in 2019, yet I still find it on live sites.
Site-wide and asset blocks
Disallow: / under User-agent: * or Googlebot blocks the whole site. Blocking folders like /_next/static/ or /wp-includes/ can stop Google rendering the page properly.
Sitemap health
A sitemap should list only final, indexable URLs on the same host: no redirects, no 404s, nothing blocked. The sample table shows what a crawler gets for a spread of URLs from the file.
Reading a robots.txt file, line by line
A typical small-business robots.txt is four or five lines. User-agent: * opens a group that applies to every crawler without a group of its own. Disallow: /wp-admin/ keeps crawlers out of the admin area. Allow: /wp-admin/admin-ajax.php makes one exception, and because it's the longer, more specific rule, it wins for that file. Sitemap: https://www.example.com/sitemap.xml points every search engine to the sitemap and can go anywhere in the file.
The traps are in the details. A group for User-agent: Googlebot replaces the * group for Google entirely; it doesn't add to it, so rules you wrote for everyone stop applying to Googlebot. Paths are case-sensitive, so /Blog/ doesn't block /blog/. And an empty Disallow: means allow everything, while Disallow: / means block everything. One character apart. The path test above applies all of these rules, so you can check a URL instead of reasoning it out.
How I use it in an audit
Check the home domain first
Enter the bare domain. The tool follows the redirect to the version the site actually serves, so you're checking the right robots.txt.
Fix anything red
Server errors, site-wide blocks and sitemaps that don't load come before everything else.
Test your key paths
Run your money pages, a blog post and a category page through the path test. If any is blocked, find the rule it names.
Read the sample table
Redirects and 404s in the sitemap are a sign it isn't generated from the live site. Fix the source, not the file.
Compare with Search Console
The robots.txt report and Sitemaps report show what Google last fetched. If they disagree with this tool, something changed recently.
Check one URL in depth
For a page that still won't index, the indexing checker adds the page-level checks: noindex, canonical and redirects.
Common robots.txt and sitemap mistakes
- Blocking a page to remove it from Google. Use noindex and let it be crawled. A blocked page can stay indexed.
- Staging rules on the live site.
Disallow: /survives more launches than you'd think. - Sitemaps generated once and never again. New pages are missing and deleted ones return 404.
- Listing both www and non-www, or http and https. List the version your canonical tags point to.
- A lastmod that is always today. Use the date the content changed, or leave it out.
- Blocking AI crawlers and expecting it to affect Search. Blocking GPTBot or CCBot doesn't change Google rankings. Google-Extended only controls use in Google's AI models, not Search crawling.
When the files are clean but pages still don't rank
Crawlable isn't the same as indexed, and indexed isn't the same as ranking. Once these files are right, run important pages through the free SEO audit, check structured data with the JSON-LD schema generator, and if traffic dropped on a particular date, see whether it lines up with a confirmed change in the Google algorithm update tracker.
I wrote about what usually causes the bigger drops in my post on core updates. If you're dealing with one now, I'm happy to take a look.
FAQ
Robots.txt and sitemap questions
What does this robots.txt checker test?
It fetches your robots.txt the way a crawler would and reads it line by line: directives Google doesn’t support, typos, rules outside a user-agent group, paths that can never match, rules that block CSS or JavaScript, and anything that blocks the whole site. It also reports the status code, because a 5xx on robots.txt can stop Google crawling everything.
How do I test whether a URL is blocked by robots.txt?
Enter the path in the second box, for example /blog/ or a full URL. The tool applies Google’s matching rules: the most specific user-agent group wins, then the longest matching rule, with Allow winning a tie. It shows the result for Googlebot and for all crawlers, and which rule decided it.
What does the sitemap check look at?
Whether the file is a valid urlset or sitemap index, the namespace, the 50,000 URL and 50 MB limits, relative URLs, URLs on another domain or the other www version, http URLs on an https site, lastmod format and future dates, duplicates and tracking parameters. Then it opens up to eight URLs from the sitemap to check they return 200 and aren’t blocked by your own robots.txt.
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt controls crawling, not indexing. A blocked URL can still be indexed if other pages link to it, just without a description. To keep a page out of results, let Google crawl it and add a noindex tag. Blocking it in robots.txt stops Google seeing the noindex.
Do I need a robots.txt file?
No. Without one, Google crawls everything it can find. I still add one to every site, mainly to point crawlers to the sitemap and to keep them out of things like internal search results.
Does Google use priority and changefreq in sitemaps?
No. Google has said it ignores both. It uses lastmod, but only when the dates are consistently accurate. If every URL shows the same date, it learns to ignore that too.
More free tools
Pages dropping out of Google?
I'll check crawling, indexing and redirects across your site and tell you what's holding pages back. The review is free.