An XML sitemap lists the URLs you want search engines to crawl, with optional lastmod dates that signal freshness. It does not guarantee indexing — it helps discovery. This guide covers creation, splitting, and the coverage checks that catch silent losses.
What a sitemap should contain
Only canonical, indexable URLs returning 200 status. Exclude noindexed pages, redirect chains, and parameter duplicates. Each entry can carry lastmod, and accurate lastmod dates help search engines prioritize recrawls of content you actually changed.
Format and limits
Standard limits: 50,000 URLs per file or 50MB uncompressed. Larger sites split into multiple files and reference them from a sitemap index. Keep the sitemap referenced in robots.txt and submitted in Google Search Console.
Coverage problems
Three failures show up in real audits: URLs declared in the sitemap that are noindexed or non-canonical, important pages missing from the sitemap entirely, and lastmod dates that never change even when content does. Run the comparator to see which of your sitemap URLs are actually indexed.
Sitemaps and AI crawlers
Most AI crawlers read sitemaps too. Fresh lastmod dates in your sitemap are one of the freshness signals answer engines use when deciding which sources to re-fetch.
Generating and maintaining sitemaps
For small sites, a generated sitemap from your CMS or the XML sitemap generator is sufficient. For large or static sites, build the sitemap from your deploy pipeline so it updates with every release. Whatever the source, keep lastmod dates honest — inflated freshness signals train crawlers to distrust your sitemap, which slows discovery of genuinely new pages.
Splitting and indexing large sitemaps
When you exceed 50,000 URLs or 50MB, split by section and reference all parts from a sitemap index file. Keep each part topically coherent — a sitemap of all product pages, another of all blog posts — so crawl prioritization stays sensible. The splitter and merger tools handle both directions: breaking large files apart and consolidating fragments from legacy systems.
Closing the coverage loop
The coverage comparator is the verification step most sites skip. It takes your sitemap URLs, checks them against what Google reports as indexed, and returns three lists: indexed (healthy), submitted but not indexed (quality or canonical problem), and indexed but not submitted (add to sitemap or noindex). Work the middle list monthly — it is where silent traffic losses hide.
Free tools in this hub
- XML sitemap checker — Validates structure, URL count, and lastmod values
- XML sitemap generator — Builds a valid sitemap from a URL list or crawl
- XML sitemap splitter — Splits large sitemaps and creates a sitemap index
- XML sitemap merger — Merges multiple sitemaps and removes duplicate URLs
- Sitemap vs indexed pages comparator — Shows which sitemap URLs Google has actually indexed
- Sitemap lastmod checker — Audits lastmod accuracy across your sitemap
Frequently asked questions
Does a sitemap guarantee my pages get indexed?
No. A sitemap helps discovery and prioritization. Indexing still depends on content quality, internal links, and the page's overall signals.
Should every page be in the sitemap?
Every canonical, indexable page you want ranking should be. Thin, duplicate, or noindexed pages should not be listed.
How often should I regenerate the sitemap?
Whenever you add, remove, or substantially update pages. Automate it if your CMS supports it, and keep lastmod dates honest.
