Your Sitemap Might Be Publishing Pages You Have Never Seen

Two files decide what search engines think your website is. Almost nobody who owns a website has ever opened either one.

robots.txt tells crawlers where they may go and where your sitemap lives. The sitemap tells them what pages exist. Both are plain text, both sit at predictable addresses, and both are readable by anyone in about four seconds. They are also where the strangest problems on the internet hide, because nothing on your website surfaces their contents to you.

We read these files on every site we look at. Here is what has turned up recently.

The photography studio publishing 2,124 casino pages

A photography business. Small, regional, the kind of site you would expect to hold maybe forty pages.

Their sitemap listed 2,124 URLs, and they were effectively all Australian online casino pages. Not links out to a casino site. Pages on their own domain.

We spot-checked them. Each one returned a 200 status. Each one rendered inside the studio’s own template, with their own navigation and their own branding wrapped around gambling content. Each carried index, follow, so every one of them was explicitly asking to be indexed.

And every single lastmod value was stamped with that day’s date. It was not a leftover from an old compromise somebody had cleaned up and forgotten. It was still generating pages.

We did not do a forensic investigation, and we are not going to assert a mechanism we did not verify. What we can say is that this pattern, thousands of unrelated commercial pages rendering inside a legitimate site’s template, is a well-documented outcome of a compromised content management system, a vulnerable plugin, or an unsecured upload path. Somebody is borrowing the domain’s credibility to rank content the domain owner has never seen.

The owner almost certainly has no idea. Nothing in the admin dashboard lists these pages. They do not appear in the navigation. The homepage looks exactly the way it always has. The only places this is visible are the sitemap, a site: search, and Search Console’s indexing report, and it is entirely normal for a small business to never open any of the three.

The cost is not hypothetical. Google has a specific enforcement category for hacked content, and it applies to the whole domain, not just the injected pages. A business can lose its own rankings for its own name because of pages it did not create and cannot see.

The site with 48,007 URLs in one file

A jewelry retailer shipping a single sitemap file weighing 11.4 MB and containing 48,007 URLs.

The sitemap protocol sets a hard limit of 50,000 URLs and 50 MB uncompressed per file. Past that a crawler is entitled to stop reading. This site was at 96% of the URL limit in one file, with no sitemap index splitting it up.

Nothing was broken yet. That is what makes it worth mentioning. A site like this typically got there through product variant URLs, filter combinations, or pagination generating far more addresses than it has actual products, and it will cross the line one catalog update from now. The failure will be silent when it happens.

The law firm handing Google a two-year-old stub

This one is our favorite because the first read of it was wrong.

A firm’s robots.txt named exactly one sitemap. We fetched it. 372 bytes, containing a single URL, the homepage, with a lastmod of January 2023.

The obvious conclusion was that this firm publishes one page. That would have been a great email and a false one.

We probed eight other common sitemap paths. Five returned 200. The real one, generated by their SEO plugin and completely current, sat at a different address and contained 423 posts, 31 attorney pages and 24 other pages. The site is fine. Their content is fine.

What is true is narrower and still worth fixing: the one file they explicitly hand to search engines is a two-year-old stub pointing at the wrong place, while the accurate index is only findable if a crawler guesses at it. Search engines also follow links, so this does not make their attorney bios invisible. It does mean the single most direct signal they control is telling a crawler their site is one page.

The lesson for anyone auditing: probe the alternatives before you write the finding. A missing or wrong sitemap and a site with no sitemap look identical if you only check the path robots.txt names.

The other file, and the mistakes in it

robots.txt produces its own category of quiet failure.

A medical aesthetics practice had a Sitemap: directive with a literal space in the middle of the address. We verified that at the byte level on the raw 131-byte file, specifically because a stray space is exactly the kind of thing that could be an artifact of how a terminal printed it rather than something actually in the file. It was in the file. The address does not resolve, so the directive does nothing.

Another firm’s robots.txt named a sitemap that returns 404 on both the www and non-www versions of the domain. They are pointing crawlers at a file that is not there.

And a method note against ourselves, because we nearly published something wrong off it. On one site we built a sitemap finding from a debug print that had truncated the file at 600 characters, and the Sitemap: directive was the last line. The claim would have been that the directive was missing. It was there. The rule that came out of it is blunt: never build a claim off a truncated dump of a file small enough to read whole. robots.txt is almost always under a kilobyte. Read all of it.

The pages that exist because a platform made them

Injected spam is the dramatic version. The common version is your own platform publishing pages nobody meant to publish.

A firm’s WordPress user sitemap published /author/dev_team/. That URL returned 200, was titled “dev_team”, carried an H1 reading “Nothing Found”, and had no noindex tag on it. It was one of only six URLs the firm published to search engines. One in six of their entire indexable footprint was an empty author archive named after their developer’s shared login.

An event venue’s sitemap listed a /launching-soon page above their homepage. That page was live, indexable, and filled with Lorem ipsum. The business is open and answering its phone. The most prominent URL in their own sitemap tells search engines they have not opened yet.

A contact URL redirected to a “request received” thank-you page that was set to noindex. So the Contact link fed crawlers into a dead end.

None of these were decisions. They are defaults, staging leftovers, and plugin behavior that nobody turned off. That is the entire category: your site is publishing things on your behalf and never mentions it.

How to check yours in five minutes

You do not need a tool for any of this.

Open yourdomain.com/robots.txt in a browser. Read the whole thing, top to bottom. Confirm the Sitemap: line has a complete, correctly spelled, resolving address with no stray characters. Confirm you are not accidentally disallowing something that matters.

Open the sitemap it names. If it is an index, open the children. Look at the URL count. Does it roughly match the number of pages you believe you have? A site with forty pages and a two thousand URL sitemap has a story you need to hear.

Scan the URLs for anything you do not recognize. Slugs in another language, product or category words from an industry you are not in, long strings of numbers, dates far in the future or the past.

Look at the lastmod values. All identical to today means the field is being auto-stamped on deploy and is meaningless. All identical to a date years ago means the sitemap is stale. A spread of real dates is what you want.

Search site:yourdomain.com in Google, and go to the last page of results. This is the single fastest way to find pages you did not know existed. The front pages show you what you expect. The back pages show you the leftovers.

Open Search Console’s page indexing report and read the not-indexed reasons. Unexpected URLs surface there before they surface anywhere else.

Do this after every migration, plugin change, and platform update, and once a quarter regardless.

Why this is worth your afternoon

Everything above is invisible from the front of a website. You can look at your homepage every day for two years while your domain publishes gambling pages, or tells Google you have one page, or points crawlers at a file that does not exist.

There is no notification. Your platform will not email you. Your web designer finished the job and moved on, and none of this was in scope. The first symptom is usually a ranking drop nobody can explain, and by then you are diagnosing backwards from a loss instead of catching it in a text file.

Two files. Five minutes. It is the cheapest possible check on what the internet currently believes your business is.

We wrote up the other half of this problem, the one where search engines never fetch the pages you did mean to publish, in We Published 16 Blog Posts and Got Two Clicks. Our website and SEO service page covers how we handle the technical layer on builds we run, and the free audit reads both files on your site along with everything else.

FAQ

What is the difference between robots.txt and a sitemap?

robots.txt is a set of instructions telling crawlers which areas of your site they may request and where your sitemap is. The sitemap is a list of the URLs you want search engines to know about, optionally with a last modified date for each. One controls access, the other declares inventory. Both live at predictable public addresses and anyone can read yours right now.

How would spam pages get onto my domain?

Most commonly through a compromised content management system, an outdated plugin or theme with a known vulnerability, a weak or reused admin password, or an unsecured file upload path. The injected pages render inside your template because they are being served by your own site. Keeping the platform and plugins current, using strong unique credentials with two-factor authentication, and removing plugins you no longer use closes most of the common routes.

Does Google penalize a site for pages the owner did not create?

Google has an enforcement category for hacked content and it applies at the site level, not just to the affected pages. That is exactly why this matters: the pages are not yours, the consequences are. Search Console will flag it under Security Issues if Google detects it, which is another reason to have the property verified and to check it.

Is a sitemap even necessary if my site is small?

For a small, well-linked site Google will usually find everything through links regardless. A sitemap still helps, particularly for new pages and for anything not reachable in a couple of clicks from the homepage. Where it stops being optional is when you publish regularly, because then you are continuously asking for new URLs to be discovered.

How many URLs can one sitemap hold?

50,000 URLs or 50 MB uncompressed, whichever comes first. Past either limit you split into multiple files and reference them from a sitemap index. Most content management systems handle the split automatically, but ecommerce sites generating URLs from variants and filters can quietly approach the limit in a single file.

Read the two files

Your website is a claim about your business, and these two files are the version of that claim search engines actually read. They are public, they are plain text, and they take five minutes.

Go open yours. The number of URLs alone will tell you whether you have a problem.

Want us to check it properly? Book a 15-minute call. No deck, no fluff. Strategy first. Tactics second. The work works.

Reliable PR & Marketing is a strategy-first marketing agency in Bakersfield, California. We run integrated SEO, PR, web, and content for founder-led companies across Kern County and nationwide. Strategy first. Execution always.