Localized sitemap index generation ¶

A site with five locales and four hundred pages submits one sitemap. It lists two thousand URLs, several hundred of which redirect, the bare root appears alongside its localized destinations, and every entry carries a lastmod equal to the last deploy. Indexing is slow, incomplete, and impossible to reason about.

A sitemap is a claim about which URLs exist and are worth crawling. Generating it from a route table rather than from what the build emitted turns it into a claim that is partly false.

Root cause: routes are not pages, and one file is not a plan ¶

Two mistakes compound.

Generating from routes. A route table describes what the application can serve; it does not know which locale-parameter combinations were actually built, which pages are excluded, or which URLs redirect. A generator walking routes therefore lists URLs that do not exist, that redirect, or that carry a noindex directive — each of which spends crawl budget and returns nothing.

Keeping one file. A single sitemap for every locale means a change to one language’s content invalidates the whole file, gives no per-locale visibility in reporting, and eventually approaches the format’s size limits. Splitting per locale is not primarily about the limits; it is about being able to see, per language, how many URLs were submitted and how many were indexed.

The lastmod field deserves its own mention because it is nearly always wrong. Set from the build timestamp, it tells a crawler that every page changed on every deploy — which is information the crawler will learn to distrust, and the field then contributes nothing.

Minimal reproducible example ¶

// Generated from routes: lists URLs that were never built
const urls = ROUTES.flatMap((route) =>
  LOCALES.map((l) => ({
    loc: `https://example.com/${l}${route}`,
    lastmod: new Date().toISOString(),        // the build time, for everything
  }))
);

The fix: walk the output, split per locale, index them ¶

import fg from 'fast-glob';
import fs from 'node:fs/promises';

const ORIGIN = 'https://example.com';

// 1. What was actually emitted?
const files = await fg('out/**/index.html');

const entries = await Promise.all(files.map(async (file) => {
  const html = await fs.readFile(file, 'utf8');
  if (/<meta[^>]+noindex/i.test(html)) return null;           // respects the page itself

  const url = '/' + file.replace(/^out\//, '').replace(/index\.html$/, '');
  const canonical = html.match(/<link rel="canonical" href="([^"]+)"/)?.[1];
  if (canonical && canonical !== ORIGIN + url) return null;   // not canonical: skip

  return { url, locale: url.split('/')[1] || DEFAULT_LOCALE, source: sourceFor(url) };
}));

// 2. One child sitemap per locale
const byLocale = groupBy(entries.filter(Boolean), (e) => e.locale);

for (const [locale, list] of Object.entries(byLocale)) {
  await writeSitemap(`out/sitemap-${locale}.xml`, list.map((e) => ({
    loc: ORIGIN + e.url,
    lastmod: gitLastModified(e.source),        // source date, not build date
  })));
}

// 3. The index, written last
await writeIndex('out/sitemap.xml',
  Object.keys(byLocale).map((l) => `${ORIGIN}/sitemap-${l}.xml`));

Three properties make this correct. It reads the emitted HTML, so it cannot list a page that was not built. It honours the page’s own directives, so a noindex page or a non-self canonical is excluded without a second list to maintain. And lastmod comes from the content’s history rather than the build’s.

Whether to put hreflang in the sitemap ¶

The alternate-language annotations can live in the page head, in the sitemap, or both. All three are valid and the choice has practical consequences.

In the head is the most verifiable: viewing a page shows what it claims, and a mismatch is visible to anyone looking. It adds markup to every page, which is negligible for a few locales.

In the sitemap scales better for very large sites and puts the whole annotation set in one place a crawler already fetches. Its weakness is verifiability — a mismatch between sitemap annotations and page reality is invisible without a tool.

Both is not harmful and doubles the surface that can drift. If you do it, generate them from the same source in the same step, so they cannot disagree.

The reciprocity requirement described in locale-aware SEO and hreflang applies identically wherever they live: every URL in a group lists every other, including itself.

Verification ¶

# The index lists one child per locale, and each child exists
curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*</loc>'
for l in en de fr ja ar; do
  curl -sI "https://example.com/sitemap-$l.xml" | head -1
done

# No listed URL redirects — the check that matters most
curl -s https://example.com/sitemap-de.xml \
  | grep -o 'https://[^<]*' \
  | while read -r u; do
      code=$(curl -s -o /dev/null -w '%{http_code}' "$u")
      [ "$code" = 200 ] || echo "$code $u"
    done

That last loop belongs in a scheduled job rather than in a pull-request check — it is slow — and it is the single most effective sitemap test there is. A sitemap whose entries all return 200 is doing its job; one full of redirects is actively wasting the crawl budget it was meant to direct.

Reading submitted-versus-indexed per locale ¶

Splitting the sitemap by locale buys a diagnostic that a single file cannot provide: a per-language count of what was submitted against what was indexed. Three patterns in those numbers point at three different problems.

Submitted high, indexed low, in one locale only. The pages exist and are being rejected. The usual causes are a canonical pointing elsewhere, thin or machine-translated content that the engine has judged not worth indexing, or pages that are near-duplicates of another locale because they were never actually translated.

Submitted low in one locale. The pages were not generated. That is a build problem rather than an indexing one, and comparing the child sitemap’s URL count against the other locales names it immediately — a locale with a fifth of the entries had four fifths of its routes fail to build.

Submitted and indexed both high, traffic low. Everything mechanical is working and the content is not matching what that market searches for. No amount of sitemap work changes that; it is a content question, and it is the good problem to have because it means the pipeline is sound.

The reason this is worth setting up deliberately is that the three look identical in aggregate. A single sitemap reports one submitted number and one indexed number for the whole site, and any of the three above can hide inside it — most often the second, because a locale that failed to build produces no errors anywhere else.

When to escalate ¶

If pages are in the sitemap and not indexed, the sitemap has done its part. Look at the canonical and the annotations next, since a page canonicalising elsewhere will not be indexed regardless of how it is submitted — the failure described in canonical pointing to the default locale.

If one locale indexes far more slowly than another, compare their submitted counts. A locale whose pages were never generated — the static-export trap in next-intl static export locale 404 — has a small sitemap and an obvious explanation.

If the sitemap exceeds the format’s limits, split further by section within a locale. The index can reference any number of children, and there is no requirement that the split be by locale alone.

FAQ ¶

Related ¶

• Locale-Aware SEO & hreflang — the annotation rules the sitemap may also carry.
• Canonical pointing to the default locale — why a submitted page may still not be indexed.
• next-intl static export locale 404 — pages a sitemap claims that were never generated.
• SvelteKit route prefixing for multiple locales — prerendering every URL the sitemap will list.

Part of Locale-Aware SEO & hreflang.