TMX import losing segments

Forty-eight thousand translation units are exported from the old system. The import reports success. The new memory holds forty-one thousand, and nobody notices for a month — until translators start reporting that strings they translated last year are coming back as new work.

TMX is an interchange format with a permissive specification and strict importers. Segments that do not satisfy an importer’s expectations are usually skipped rather than rejected, and the difference is a number nobody compared.

Unit counts across a TMX migration An export writes a known number of units, the file is parsed on import, the importer skips unusable segments, collapses duplicates and strips unsupported markup, and the memory ends up holding several thousand fewer units than were sent — with no error at any step. Where the count changes Export TMX file Import Memory 48,210 units written parsed skip, collapse, strip 41,884 stored
Comparing the two counts is the entire test, and almost nobody runs it.

Root cause: TMX allows more than most importers accept

The format defines a header carrying defaults and a body of translation units, each with variants per language. Almost everything in it is optional, which is what makes it portable and what makes imports lossy.

Four gaps account for nearly all of the loss.

Missing source language. A unit without srclang is meant to inherit the header’s value. When both are absent — common in files assembled by scripts — the importer cannot tell which variant is the source, and skips the unit.

Duplicate source segments. Two units with the same source text and different translations are legitimate: the same English word takes different translations in different contexts. Many importers deduplicate on source text, keeping one and discarding the rest, which loses exactly the context-specific translations that were most expensive to produce.

Inline markup. Segments carrying <bpt>, <ept> or <ph> elements describe formatting or placeholders inside the text. An importer that does not support them either strips the markup — leaving a segment that no longer matches anything — or skips the unit.

Empty variants. A unit whose target variant has no content is stored as an empty translation rather than as an absent one, which makes the language look translated and produces a blank string later.

Four categories of segment lost on TMX import Segments without a source language, when the header also omits one, are skipped entirely. Duplicate source segments are collapsed to one, losing translations that differed by context. Segments containing inline tags are dropped or stripped when the importer does not support them. And target variants with no content make a language look untranslated even though the unit exists. What a TMX import silently drops Dropped when Consequence Segments with no srclang the header lang is missing too the unit is skipped entirely Duplicate source segments the importer keeps one context-specific translations lost Segments with inline tags tags are unsupported the segment or its markup vanishes Empty target variants a variant has no content the language appears untranslated
All four are silent: the import reports success and a smaller number than you sent.

Minimal reproducible example 

<tmx version="1.4">
  <header creationtool="script" segtype="sentence" adminlang="en"/>
  <!-- no srclang on the header, and none on the unit: this is skipped -->
  <body>
    <tu>
      <tuv xml:lang="en"><seg>Open</seg></tuv>
      <tuv xml:lang="de"><seg>Öffnen</seg></tuv>
    </tu>
    <!-- same source, different context: usually collapsed into the one above -->
    <tu>
      <tuv xml:lang="en"><seg>Open</seg></tuv>
      <tuv xml:lang="de"><seg>Geöffnet</seg></tuv>
    </tu>
  </body>
</tmx>

Two units in, zero or one unit stored, no error emitted.

The fix: make every unit self-describing 

<tmx version="1.4">
  <header srclang="en" adminlang="en" segtype="sentence"
          creationtool="migration" creationtoolversion="1.0"
          datatype="plaintext" o-tmf="json"/>
  <body>
    <tu tuid="ui.open.action" srclang="en">
      <prop type="x-context">button label — verb</prop>
      <tuv xml:lang="en"><seg>Open</seg></tuv>
      <tuv xml:lang="de"><seg>Öffnen</seg></tuv>
    </tu>
    <tu tuid="ui.open.status" srclang="en">
      <prop type="x-context">status label — adjective</prop>
      <tuv xml:lang="en"><seg>Open</seg></tuv>
      <tuv xml:lang="de"><seg>Geöffnet</seg></tuv>
    </tu>
  </body>
</tmx>

Three additions do the work. srclang appears in the header and on each unit, so no inheritance is required. A tuid gives each unit a stable identity, which most importers use in preference to source-text matching and which therefore prevents deduplication. And a context property records why two identical sources have different translations — the same disambiguating role msgctxt plays in PO, described in PO and XLIFF format bridging.

Five steps for a verifiable TMX migration Counting translation units in the file before import establishes the expected number. Declaring the source language in both the header and each unit removes the commonest skip reason. Carrying context as a property keeps duplicate source strings distinct. A first import into an empty memory gives a dry run against real data. And comparing counts, then spot-checking a sample, confirms the result. A migration you can verify 1 Count units in the source file grep the tu elements before importing 2 Declare srclang in the header and per unit never rely on the default alone 3 Preserve context as a property so duplicates stay distinct 4 Import into an empty memory first a dry run with real data 5 Compare counts and spot-check numbers, then twenty random segments
Step four is the one people skip, and it is the only one that lets you retry cheaply.

Migrating in a way you can retry

The property that makes a migration safe is not correctness on the first attempt; it is the ability to try again cheaply. Three habits provide it.

Import into an empty memory first. A dry run against real data reveals the actual loss rate without touching the memory translators are using. If the count is wrong, the fix is to adjust the export and repeat, which costs an hour rather than a rollback.

Keep the source file. The exported TMX is the artifact of record for the migration and should be archived, not regenerated. Regenerating it from a system that has since changed produces a different file, and the difference will be impossible to attribute.

Migrate in batches, per language pair. A single import of every language makes a partial failure hard to interpret. Per-pair imports give per-pair counts, and a discrepancy points at one file rather than at a process.

None of that is specific to TMX. It is what any data migration needs, and translation memories attract the shortcut because the import usually reports success — which reads as confirmation and is only a statement that the parse completed.

What a lost memory actually costs

It is worth putting a number on the loss, because “we lost some segments” does not convey the same thing as the cost it represents, and the cost is what justifies redoing the migration properly.

A translation memory is an asset built from paid work. Every unit in it was produced by a translator and reviewed, and its value is realised each time a similar segment appears and is matched rather than retranslated. A memory of fifty thousand units on a mature product typically delivers a substantial leverage rate on new work — a large share of new segments arrive with a usable match — and that rate is what keeps ongoing localization affordable.

Losing thirteen percent of the units does not reduce leverage by thirteen percent, because the loss is not random. The segments most likely to be dropped are exactly the ones with context properties, inline markup and duplicate sources — which is to say, the complex ones that were most expensive to translate and are most valuable to match. A deduplicated memory keeps the generic “Open” and discards the context-specific variants, so what remains is the cheapest material.

There is a second cost that shows up later. When a previously translated segment returns as new work, translators notice, and what they infer is that the memory is unreliable. A team that stops trusting the memory stops relying on its matches, reviews everything from scratch, and the asset stops paying back regardless of how many units it still contains.

Both costs argue for the same thing: verify the count before anyone works against the new memory, because the window in which a re-import is cheap closes as soon as new translations start landing on top of it.

Verification 

# Units in the file
xmllint --xpath 'count(//tu)' memory.tmx
#   48210

# Units the importer stored (per your system's API or database)
curl -s "$TMS/api/v2/memories/$ID" | jq '.data.segmentsCount'
#   41884   ← the discrepancy this page exists to find

# Which units were dropped: compare the tuid sets
xmllint --xpath '//tu/@tuid' memory.tmx | grep -o '"[^"]*"' | tr -d '"' | sort > sent.txt
curl -s "$TMS/api/v2/memories/$ID/segments" | jq -r '.data[].tuid' | sort > stored.txt
comm -23 sent.txt stored.txt | head -20

The third command is the one that turns a number into an action: it names the units that did not arrive, and their common property is nearly always visible within the first twenty.

When to escalate

If the counts match but leverage is poor — translators seeing few matches on strings that were previously translated — the segments arrived and are not being matched. That is usually a segmentation difference: the old system segmented by sentence and the new one by paragraph, so nothing aligns. Re-exporting with a matching segtype is the fix, and it is why the header field is worth setting explicitly.

If specific languages are empty after import, check for empty target variants in the source file. A unit with an empty <seg> for one language will store as an empty translation, which is worse than absent because it suppresses future matches.

If markup is missing from imported segments, the importer stripped inline elements. Whether that matters depends on your content: for plain UI strings it usually does not, and for formatted marketing copy it destroys the segment’s value.

FAQ

Is TMX still the right interchange format?

For memories, yes — it is the only widely supported one, and every serious system reads and writes it. Its weakness is not the format but the variation in how importers treat optional structures, which is why explicit, self-describing units matter more than the format choice.

Should I deduplicate before importing?

No. Duplicates with different translations are information, and collapsing them is exactly the loss to avoid. Duplicates with identical translations are harmless and can be left alone; the importer will collapse them without cost.

How do I preserve metadata the target system does not understand?

As <prop> elements with an x- prefixed type. Most systems store unknown properties rather than dropping them, so provenance, context and review state survive a round trip even when nothing acts on them.

What if the old system cannot export TMX?

Then the migration goes through whatever it can export, usually a bilingual file per language pair, and the same verification applies: count before, count after, compare identifiers. The format changes and the discipline does not.

Can a partially imported memory be topped up rather than redone?

Yes, and it is usually the better option once translators are working against it. Re-importing the same file with corrected units is additive in most systems: units that already exist are matched by identifier and left alone, and the previously dropped ones are added. What makes this work is having stable identifiers on every unit — without them, a second import creates duplicates instead of filling gaps.

Should the memory be cleaned before migration?

Only of things that are unambiguously worthless: units whose source and target are identical in a language pair where that is impossible, and units referencing products or features that no longer exist. Anything else is worth carrying across, because storage is cheap and a segment that looks useless today matches something next quarter. Cleaning is also a decision that cannot be undone once the old system is gone.

Part of Translation Memory & Glossary Management.