Failing the Build on Untranslated Keys ¶

Failing a build on untranslated keys means a CI step diffs every locale catalog against the source catalog, computes per-locale coverage, and exits nonzero when keys are missing or present-but-empty — while exempting an allowlist of keys you deliberately never translate. The trap most teams hit is binary thinking: either the gate is so strict that a half-finished Polish translation blocks an unrelated English fix, or it is so loose that empty strings ship to production and render as blank buttons. The fix is a coverage gate with a per-locale threshold and a brand allowlist, run as a required check in GitHub Actions.

This page builds the gate from scratch: the comparison algorithm, the allowlist for intentionally-untranslated terms (brand names, product SKUs), and the threshold-per-locale knob that lets a newly-added language sit at 60% while your shipping languages must stay at 100%. It assumes you already have the broader GitHub Actions i18n CI gates wiring and want the completeness check specifically.

Root Cause: Why a Naive Key Count Lies ¶

The instinct is to count keys: if pl/common.json has as many keys as en/common.json, call it done. That count is wrong on three axes, and every one of them produces a green build that ships broken UI.

Present but empty is not translated. Most catalog tooling — i18next, formatjs extraction, gettext msginit — pre-seeds target files with the source string or an empty string so the key exists. A key whose value is "" passes a naive “does the key exist” check yet renders as a blank label at runtime. The gate must treat empty and whitespace-only values as untranslated, not just absent keys.

Source-equals-target is ambiguous. When pl.welcome === en.welcome, it is either an untranslated copy or a legitimately identical string (a proper noun, a units symbol). You cannot reject all of them, which is exactly why the allowlist exists: it is the explicit declaration of which identical strings are intentional.

Nested catalogs hide gaps. i18next and Vue use nested objects (nav.user.settings); a shallow key diff misses leaf keys inside an object that exists in the source but is sparse in the target. The comparison has to flatten to leaf paths first, the same dotted-path model that the broader GitHub Actions i18n CI gates use for placeholder parity.

A correct gate therefore enumerates leaf paths in the source, and for each locale counts a key as translated only if it exists, is non-empty after trimming, and is either different from the source or explicitly allowlisted.

Minimal Reproducible Example ¶

Here is the smallest setup that demonstrates a false pass. The source defines three keys; Polish has all three keys present, so a key-count check is happy — yet two of the three are unusable.

// locales/en/common.json  (source)
{
  "welcome": "Welcome",
  "save": "Save",
  "brand": "Acme"
}

// locales/pl/common.json  (target — naive check passes, runtime broken)
{
  "welcome": "Witaj",
  "save": "",          // empty → renders a blank button
  "brand": "Acme"      // identical → untranslated copy, or intentional brand?
}

A keys(en).length === keys(pl).length check returns true and the build goes green. In production, the Save button is blank and brand is indistinguishable from an oversight. The next section computes real coverage and disambiguates brand with an allowlist.

The Fix: Coverage Script with Allowlist and Per-Locale Thresholds ¶

A single self-contained Node script — no dependencies — does the diff, applies the allowlist, and exits nonzero when any locale is below its threshold. The thresholds and allowlist live in one config file so non-engineers can edit them.

# i18n-coverage.yml — the gate's policy, reviewed like code
source: en
catalogPath: locales/{locale}/common.json
# Keys that are correct when identical to (or absent from) the source.
# Brand names, SKUs, units — never sent to translators.
allowlist:
  - brand
  - "product.sku"
  - "units.kb"
# Default required coverage; per-locale overrides let new langs ramp up.
defaultThreshold: 1.0
thresholds:
  pl: 1.0      # shipping language — must be complete
  uk: 0.6      # newly added — allowed to ramp to 100% over time

// scripts/i18n-coverage.mjs  —  run as `node scripts/i18n-coverage.mjs`
import { readFileSync, readdirSync } from "node:fs";
import { parse } from "yaml";

const cfg = parse(readFileSync("i18n-coverage.yml", "utf8"));
const allow = new Set(cfg.allowlist);          // O(1) membership for brand keys

// Flatten nested catalogs to dotted leaf paths: {nav:{user:"x"}} → "nav.user"
const flatten = (obj, prefix = "") =>
  Object.entries(obj).flatMap(([k, v]) => {
    const path = prefix ? `${prefix}.${k}` : k;
    return v && typeof v === "object" && !Array.isArray(v)
      ? flatten(v, path)
      : [[path, v]];
  });

const load = (locale) =>
  JSON.parse(readFileSync(cfg.catalogPath.replace("{locale}", locale), "utf8"));

const source = new Map(flatten(load(cfg.source)));
// Required keys = every source leaf that is NOT allowlisted.
const required = [...source.keys()].filter((k) => !allow.has(k));

const locales = readdirSync("locales").filter((l) => l !== cfg.source);
let failed = false;

for (const locale of locales) {
  const target = new Map(flatten(load(locale)));
  const untranslated = required.filter((k) => {
    const v = target.get(k);
    if (v == null || String(v).trim() === "") return true;      // missing or empty
    return String(v) === String(source.get(k));                 // identical = not translated
  });

  const coverage = (required.length - untranslated.length) / required.length;
  const threshold = cfg.thresholds?.[locale] ?? cfg.defaultThreshold;
  const ok = coverage >= threshold;
  if (!ok) failed = true;

  const pct = (coverage * 100).toFixed(1);
  console.log(
    `${ok ? "PASS" : "FAIL"} ${locale}: ${pct}% ` +
      `(need ${(threshold * 100).toFixed(0)}%, ${untranslated.length} untranslated)`
  );
  if (!ok) untranslated.slice(0, 10).forEach((k) => console.log(`   - ${k}`));
}

process.exit(failed ? 1 : 0);   // nonzero fails the CI job

The non-obvious lines: the allowlist is subtracted from required before coverage is computed, so brand keys never count against any locale; the String(v) === String(source.get(k)) check catches the source-copy case that the empty-string check alone would miss; and per-locale thresholds let a fresh language ramp without blocking shipping languages — drop uk to 0.6 today, raise it as it fills in. Wire it as a required check exactly like the placeholder-parity step described in GitHub Actions i18n CI gates.

# .github/workflows/i18n-coverage.yml
name: i18n coverage
on:
  pull_request:
    paths: ["locales/**", "i18n-coverage.yml"]
jobs:
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
      - run: node scripts/i18n-coverage.mjs   # exit 1 fails the required check

Configuration Reference ¶

Verification ¶

Prove the gate catches the broken Polish catalog from the example above. With save empty, the script must report less than 100% and exit nonzero:

node scripts/i18n-coverage.mjs; echo "exit: $?"
# FAIL pl: 50.0% (need 100%, 1 untranslated)
#    - save
# exit: 1

Now add the missing translation ("save": "Zapisz") and confirm the gate flips to green, while brand stays exempt because it is allowlisted:

node scripts/i18n-coverage.mjs; echo "exit: $?"
# PASS pl: 100.0% (need 100%, 0 untranslated)
# exit: 0

For a regression test you can run in unit-test CI rather than the catalog itself, assert the coverage math directly:

node -e '
const req = 2, untranslated = 1;          // brand already excluded from req
const coverage = (req - untranslated) / req;
if (coverage !== 0.5) { console.error("coverage math drift"); process.exit(1); }
console.log("coverage assertion ok");'

When to Escalate ¶

This script handles flat-and-nested JSON catalogs. If your translations live in gettext .po files, “untranslated” has a richer meaning — msgstr "", #, fuzzy flags, and plural forms each count differently, and you should compute coverage from the PO metadata rather than string emptiness; that path overlaps with PO / XLIFF format bridging. If failures are not missing keys but mismatched {count} or ICU {n, plural, …} placeholders, that is a parity problem, not a coverage one — keep it in its own gate. And if you are pre-filling gaps with machine translation to clear the threshold automatically, gate that output for quality before it counts as “translated,” as covered in the DeepL pre-translation quality gate. For the full set of checks this gate joins, return to GitHub Actions i18n CI gates.

FAQ ¶

Related ¶

• GitHub Actions i18n CI gates — the full set of completeness, parity, and format checks this coverage gate joins.
• Connecting Crowdin API to GitHub pull requests — the translation PR this gate runs on before merge.
• DeepL pre-translation quality gate — vetting machine-filled strings before they count as translated.
• PO / XLIFF format bridging — computing coverage from gettext fuzzy flags and plural forms.
• Enforcing glossary terms in CI — the complementary check that required terms are used correctly.

Part of GitHub Actions i18n CI Gates.