List the page in an XML sitemap that crawlers find on their own
Description
This criterion checks that your site publishes an XML sitemap, that the file is valid and that the analyzed page is listed in it. iPand looks for the sitemap in the Sitemap: lines of robots.txt, then at the /sitemap.xml and /sitemap_index.xml addresses.
The file must follow the sitemaps.org format: at most 50,000 addresses and 50 MB uncompressed. A sitemap index is followed one level deep, up to 10 files from the same site.
Why it matters for AI engines
A sitemap gives search engines the list of pages to crawl, including those few links point to. Bing stresses that it remains a foundational signal for every page of a site to be discovered and indexed, including for Copilot’s answers.
- Without a sitemap, a poorly linked page on a large or new site may be discovered late by crawlers.
- A Sitemap: line in robots.txt lets every crawler find the file on its own, without registering with each search engine.
- Bing usually rereads a declared sitemap at least once a day, and relies on its <lastmod> dates to choose which pages to crawl again.
How to fix it
- 1 Publish an XML sitemap Most CMSs and SEO plugins generate one for you: turn that option on. Otherwise, create a UTF-8 encoded XML file in the sitemaps.org format: a <urlset> element, then one <url> element per page, with its full address in <loc>.
- 2 Declare it in robots.txt Add a Sitemap: line to robots.txt followed by the full address of the file, for example Sitemap: https://www.example.com/sitemap.xml. You can declare several. Also submit it in Search Console and in Bing Webmaster Tools to follow how it is read.
- 3 Stay within the format limits A file must not exceed 50,000 addresses or 50 MB uncompressed, nor contain a DOCTYPE declaration. Beyond that, split the addresses into several sitemaps grouped by an index (<sitemapindex>). iPand reads one level of index and at most 10 sitemaps from the same site: put your important pages in the first ones.
- 4 Add the page to the sitemap List the page’s address exactly as it is published: same protocol (https), same domain (with or without www) and same path. Only include pages that answer 200 and that you want indexed, in other words their canonical addresses.
Key takeaway
A valid sitemap declared in robots.txt points crawlers to each of your important pages. Check that the analyzed page is listed in it, at its correct address.
