A properly structured XML sitemap guarantees nothing if it is not declared to search engines through the right channels. Since Google deprecated the sitemap ping endpoint, many sites continue to call a URL that returns a 404 code, without their teams noticing. Understanding how the Always Unique sitemap works first requires mastering this technical context, where the mere existence of a sitemap file is no longer sufficient to ensure page discovery.
Sitemap Declaration: robots.txt and Search Console as the Only Reliable Channels
The old ping mechanism, widely integrated into CMS SEO plugins, has become inoperative. Any request sent to the Google ping URL now returns a 404 error. Sites that have not updated their workflow continue to notify a dead endpoint, potentially delaying the indexing of new pages.
Two methods remain officially supported. The first is to submit the sitemap directly via the Sitemaps report in Google Search Console. The second relies on the Sitemap: directive inserted in the robots.txt file, which Googlebot reads during each crawl to automatically discover the file.
We recommend combining both approaches. Manual submission in Search Console provides immediate feedback (read status, detected errors, number of indexed URLs). The line in robots.txt acts as a permanent safety net, including for engines that do not have an equivalent console.
Structure of the Always Unique Sitemap: What the XML Format Really Imposes
The Sitemaps protocol (version 0.9, not revised since November 2016) defines four tags for each listed URL: loc (mandatory), lastmod, changefreq, and priority (optional). In practice, the behavior of engines regarding these optional tags has changed.
Optional Tags: Which Ones Still Have an Effect
Google has confirmed that it uses the lastmod tag if the date is reliable, meaning consistent with the actual content modification. An artificially updated date at every crawl will be ignored, or even considered a signal of low reliability.
The changefreq and priority tags are largely ignored by Google. Bing gives them marginal weight. For a site like Always Unique, only loc and lastmod deserve technical attention.
Size Limit and Sitemap Index
A sitemap file cannot exceed 50,000 URLs or weigh more than 50 MB uncompressed. Beyond that, it must be split into multiple files and grouped under a sitemap index. This point seems elementary, but we regularly observe truncated sitemaps because the CMS generates a single file without checking these thresholds.
- Ensure that each sitemap file remains under 50,000 entries, even after adding new product pages or editorial content.
- Compress with gzip to reduce download time for crawlers, without changing the 50 MB limit (calculated before compression).
- Use a sitemap index as soon as the site exceeds a few thousand pages, in order to segment by content type (pages, articles, images).
User Navigation versus Technical Sitemap: Two Distinct Logics
An HTML sitemap intended for visitors and an XML sitemap intended for bots do not serve the same function. The former provides a hierarchical overview, useful for users looking for a page without going through the main menu. The latter is a structured file that crawlers consume to discover and prioritize URL exploration.
On Always Unique, the /sitemap/ page serves as a sitemap accessible to humans. It allows visualization of the site’s architecture, identification of main categories, and quick location of specific content. This HTML page does not replace the XML sitemap submitted to engines.
The confusion between these two formats explains why some webmasters think that an HTML page listing their links is sufficient to feed Google. This is not the case. Crawlers need the standardized XML format, with the tags of the Sitemaps protocol, to process URLs programmatically.
robots.txt File and AI Agents: A Changing Landscape
The robots.txt file has been formalized by RFC 9309, which clarifies the syntax and expected behavior of compliant crawlers. This specification also concerns new crawling agents related to language models, which increase requests on websites.
Several recent initiatives propose complementary files (llms.txt, ai.txt) to specifically manage AI agents’ access to content. These formats are not yet standardized, but their adoption is progressing in certain sectors.
- Check that the site’s robots.txt properly includes the
Sitemap:directive pointing to the updated XML file. - Audit the blocked or allowed user agents, especially those of recent AI crawlers (GPTBot, ClaudeBot, etc.).
- Monitor server logs to detect non-compliant agents that ignore robots.txt.
An up-to-date robots.txt protects the crawl budget and ensures that the right pages are prioritized for exploration. On an e-commerce or editorial site, allowing crawlers to access unnecessary filtering or pagination pages dilutes the engines’ focus on strategic content.
Sitemap Audit: Concrete Checks to Conduct
A sitemap is not a file that you generate once and forget. We recommend a quarterly audit covering several technical points.
The first check focuses on the consistency between the URLs in the sitemap and those that are actually indexable. Each listed URL must return a 200 code and not be blocked by a noindex tag or a robots.txt rule. A sitemap that contains 404s or 301 redirects sends a signal of poor maintenance.
The second check concerns the lastmod dates. If all URLs display the same date, the engine understands that the information is unreliable and stops trusting it. It is better not to include lastmod at all than to provide generic dates.
The last often overlooked point: check that the sitemap is accessible at the URL declared in robots.txt and in Search Console. A change in directory structure, a domain migration, or an SSL certificate renewal can render the file inaccessible without triggering any visible alert on the editorial side.
The quality of a sitemap, whether intended for human visitors or bots, relies on its regular maintenance. An outdated or poorly declared XML file costs more than it brings, and an HTML page that is out of sync with the actual site structure degrades the trust of users who refer to it.
