An XML sitemap is a machine-readable file, written in the Sitemaps protocol, that lists the canonical URLs a website wants search engines to discover and crawl, optionally with the date each page last changed. In crawling, it serves as a discovery and scheduling hint: it tells a web crawler such as Googlebot or Bingbot which pages exist and which have been updated, without forcing any of them to be indexed.
A sitemap complements, rather than replaces, internal linking. Crawlers still find most pages by following links; the sitemap closes gaps for new, deep or weakly linked pages and gives the search engine a declared list of preferred URLs against which it can compare what it actually finds.
How an XML sitemap guides crawling
A crawler fetches the sitemap, parses each url entry and adds the listed addresses to its crawl queue. URLs it has never seen are scheduled for discovery, while known URLs with a newer lastmod date may be prioritized for recrawling. The search engine still decides independently when and whether to fetch each URL, based on crawl budget, perceived importance and server capacity.
Each file is an XML document with a urlset root element and one url element per page. The only required child is loc, which holds the absolute, fully qualified URL. The file must be UTF-8 encoded, and characters such as ampersands must be entity-escaped.
The table below lists the standard elements and how major search engines treat them.
| Element | Required | How search engines use it |
|---|---|---|
| loc | Yes | The canonical URL to crawl |
| lastmod | No | Used when consistently accurate, ignored when unreliable |
| changefreq | No | Ignored by Google, a weak hint at most elsewhere |
| priority | No | Ignored by Google |
Limits, formats and extensions
The protocol sets hard limits per file: at most 50,000 URLs and 50 MB uncompressed. Files may be compressed with gzip, but the uncompressed limit still applies. Larger sites split URLs across several files and list them in a sitemap index, a separate file type that points to individual sitemaps.
| Sitemap type | Purpose | Notable constraint |
|---|---|---|
| Standard URL sitemap | Lists canonical pages | 50,000 URLs or 50 MB per file |
| Sitemap index | Lists other sitemap files | Same per-file limits apply |
| Image extension | Adds image locations to pages | Supplements, not replaces, on-page images |
| Video extension | Adds video metadata such as thumbnail and duration | Requires a thumbnail and description |
| News sitemap | Lists recent news articles | Articles from roughly the last two days |
| Hreflang annotations | Links language and regional alternates | Every alternate must reference the others |
Search engines also accept simpler formats such as a plain text list of URLs or an RSS or Atom feed, but XML remains the standard because it supports dates and extensions.
XML sitemap compared with related crawling tools
An XML sitemap is one of several signals that shape crawling. Confusing their roles is a common source of errors, because a sitemap invites crawling while robots.txt restricts it.
| Concept | Function | Relation to the sitemap |
|---|---|---|
| Robots.txt | Allows or disallows crawling of paths | Can declare sitemap locations with a Sitemap line |
| HTML sitemap | Navigation page for visitors | Helps users and passes internal links |
| Internal linking | Primary discovery path and importance signal | Sitemap fills its gaps |
| IndexNow | Push notification of changed URLs | Faster notice to participating engines |
| Canonical URL | Declares the preferred version of a page | Sitemap should list only canonical versions |
Building and submitting a sitemap correctly
A good sitemap is generated automatically by the content management system or build process, so it updates whenever pages are added, changed or removed. It should list only URLs that return status 200, are indexable and are canonical; redirects, error pages, noindex pages and parameter duplicates send contradictory signals.
Search engines learn about a sitemap in two main ways: through a Sitemap directive in robots.txt, which any crawler can read, or through submission in a webmaster tool such as Google Search Console or Bing Webmaster Tools. Submission adds valuable reporting, showing how many listed URLs were discovered and indexed.
The lastmod value deserves particular care. It should reflect a significant change to the main content, not a regenerated footer or a timestamp updated on every request. Engines learn whether a site’s dates are trustworthy and may ignore them when they are not.
Splitting sitemaps by page type or section, such as products, categories and articles, makes the reporting far more useful. When one file shows a low indexing ratio, the problem is immediately narrowed to a single template, which is much faster to diagnose than one undifferentiated list of every URL.
For very large or fast-changing sites, a sitemap is also a monitoring instrument. Comparing the URLs listed in the sitemap with the URLs crawlers actually request in server logs reveals orphan pages, crawler traps and sections that receive little attention.
Common mistakes and their effects
Sitemap problems rarely block a site entirely, but they waste crawl attention and blur the signals a search engine uses to choose canonical pages.
| Mistake | Effect | Fix |
|---|---|---|
| Listing redirected or broken URLs | Crawl requests spent on dead ends | Include only final 200 URLs |
| Including noindex pages | Mixed signals about indexing intent | Exclude them from the file |
| Every lastmod set to today | Dates treated as unreliable | Update only on real content changes |
| Mixing protocols or hosts | URLs rejected or misattributed | Use one canonical host and HTTPS |
| Stale static file | New pages missing, deleted pages kept | Generate the sitemap dynamically |
A frequent misconception is that a URL in a sitemap will be indexed. Listing a page only makes it known; indexing still depends on quality, duplication and the search engine’s own assessment.
A plain-language summary
An XML sitemap is a list of a website’s important pages, written for search engines rather than for people. It works like a table of contents handed to a librarian. The librarian can still walk the shelves, but the list makes sure no book is missed.
The sitemap can also say when each page last changed. That helps search engines decide what to check again first. It does not guarantee that a page will appear in search results. It only makes sure the page is found.
An example from practice
An online retailer with about 120,000 product pages notices that new products take weeks to appear in search results. Many are reachable only through filtered category pages that crawlers rarely explore deeply.
The development team configures the platform to generate three product sitemaps of up to 50,000 URLs each, plus separate files for categories and editorial guides, all listed in one sitemap index. Only canonical, indexable URLs with status 200 are included, and lastmod changes only when the product description, price or availability changes. The index is declared in robots.txt and submitted in the webmaster tools. Within the following crawl cycles, new products are discovered within days instead of weeks, and the coverage report reveals several thousand discontinued items that are then redirected or removed.