What are sitemap.xml and robots.txt, and are mine right?
Worth a look
Two small files at the root of your website. robots.txt tells search engines what they may read; sitemap.xml gives them a list of your pages so they don't have to find everything by following links. Neither will make you rank, but both can quietly stop you ranking when they're wrong.
What each one is for
robots.txt is a note left at your front door for automated visitors. It says which parts of the site they should leave alone — typically admin areas, search result pages, or a checkout. It's advisory: well-behaved crawlers respect it, and it is not a security measure, since anything listed in it is public and readable by anyone curious.
sitemap.xml is an index of the pages you'd like listed, with a hint at how often each changes. It matters most for large sites, new sites, and sites where some pages aren't well linked from elsewhere.
Convention is that robots.txt points at the sitemap, so a crawler finding one finds the other.
The mistakes Gus sees most
A sitemap pointing at a different domain entirely. This is nearly always a website template that was never tidied up after launch — the sitemap line still names the developer's example address or the previous owner's site. It's harmless-looking and genuinely confusing to search engines.
A sitemap listing pages that don't load, redirect elsewhere, or don't exist any more. A sitemap is a set of claims about your site; when a third of them are wrong, it stops being useful evidence.
robots.txt served as a web page rather than plain text, usually because a platform's 404 handler is answering for it. Crawlers get HTML where they expected instructions, and behaviour becomes unpredictable.
And the expensive one: Disallow: / left over from the build, blocking the entire site. That's covered separately under “why isn't my site showing up on Google”.
What good looks like
robots.txt as plain text, blocking only what genuinely shouldn't be crawled, and naming your sitemap on your own domain.
A sitemap generated automatically from the live site rather than maintained by hand, so it can't drift. Every platform worth using does this, and on WordPress it's what the SEO plugins are largely for.
Every URL in it returning a page directly — no redirects, no errors, all on the single address you've chosen as canonical.
How to check this yourself
- Visit yourbusiness.co.uk/robots.txt. You should see a few lines of plain text, not your website's design and not a 404 page.
- Read it for a line saying Disallow: / on its own — that blocks everything.
- Check the Sitemap: line at the bottom points at your own domain, not somebody else's.
- Visit that sitemap address. Click three or four of the URLs at random and confirm each opens a real page without redirecting.
What to ask your web person
Copy this. It's phrased so you don't have to know the jargon to ask for the right thing.
Could you check our robots.txt is served as plain text and only blocks what it should, and that our sitemap is generated from the live site, lists only working URLs on our canonical domain, and is declared in robots.txt and submitted to Google Search Console?
Want to know if this applies to your site?
Gus checks this and everything else in the free check-up, then explains what he found in plain English. It's free, there's no account, and it takes about a minute.