Unicode Text Handling & Collation ¶

Two strings look identical on screen and compare unequal. A list of European names sorts with Österreich after Zypern. A 20-character limit cuts a family emoji into three broken pieces. A Turkish user’s uppercase “i” stops matching their own username.

None of these are localization bugs in the message-catalogue sense — the strings are user data, not translations. They are Unicode bugs, and they surface in every product that accepts text from more than one language.

This page belongs to Core i18n Architecture & Locale Negotiation because it covers the layer beneath the catalogue: how text itself is compared, ordered, and measured once more than one script is in play.

Prerequisites ¶

• A runtime with full ICU data — Intl.Collator and Intl.Segmenter are useless against a slim build (see Intl polyfills and ICU data loading if that is in doubt).
• A decision about where normalization happens — at the edge, in the application, or in the database.
• A database whose collation you can name; “the default” is not an answer.
• At least one non-Latin script and one accented Latin locale in your test data.
• The resolved locale available where sorting and comparison happen, not only where rendering does.

Concept & spec — equivalence, ordering and boundaries ¶

Unicode defines three separate questions about text, and conflating them is what produces the failures above.

Equivalence is defined by the normalization forms in Unicode Annex 15. Canonical equivalence says two sequences represent the same abstract characters: é written as one code point and as e followed by a combining acute are canonically equivalent, and NFC or NFD makes them comparable. Compatibility equivalence goes further, folding a typographic ligature or a fullwidth Latin letter onto its plain form — useful for search, lossy for storage, and expressed by NFKC and NFKD.

Ordering is defined by the Unicode Collation Algorithm in Annex 10 and parameterised per locale by CLDR. It is not a property of the characters alone: Swedish orders ä after z, German orders it with a, and Turkish treats ç as a distinct letter. A code-point sort answers none of that, which is why Array.prototype.sort() is the wrong tool for anything a reader will look at.

Boundaries are defined by Unicode Annex 29 and exposed through Intl.Segmenter. A grapheme cluster is what a reader calls a character — a base letter with its combining marks, a regional-indicator pair forming a flag, or an emoji sequence joined by zero-width joiners. Word and sentence boundaries are locale-dependent too, most visibly in Thai and Japanese, which do not separate words with spaces.

Step-by-step implementation ¶

1. Normalize once, at the boundary ¶

Pick NFC, apply it where text enters the system, and never normalize again. NFC is the form the web platform assumes, it is what most input methods already produce, and it is compact.

// One place: the edge of your API. Everything downstream is already NFC.
export function normalizeInput(value: string): string {
  return value.normalize('NFC').trim();
}

Normalizing on write rather than on read matters because comparisons happen far from the boundary — in database indexes, in cache keys, in equality checks — and none of those can normalize for you.

2. Sort with a collator, and reuse it ¶

// Constructed once per locale: the collator is expensive, comparing is not.
const collators = new Map<string, Intl.Collator>();

export function collatorFor(locale: string): Intl.Collator {
  let c = collators.get(locale);
  if (!c) {
    c = new Intl.Collator(locale, {
      sensitivity: 'base',   // a === A === á for matching
      numeric: true,         // item2 before item10
      usage: 'sort',
    });
    collators.set(locale, c);
  }
  return c;
}

names.sort(collatorFor(locale).compare);

The sensitivity option is the one worth understanding. base ignores case and accents, accent ignores case only, variant distinguishes everything. For a search box you want base; for a displayed sort order you usually want variant with case as a tie-break.

3. Count and cut with a segmenter ¶

const graphemes = new Intl.Segmenter(locale, { granularity: 'grapheme' });

export function visibleLength(text: string): number {
  return [...graphemes.segment(text)].length;
}

export function truncate(text: string, max: number): string {
  const parts = [...graphemes.segment(text)];
  return parts.length <= max
    ? text
    : parts.slice(0, max).map((s) => s.segment).join('') + '…';
}

Every user-visible length limit — a display name, a preview snippet, a character counter under a text field — should use this rather than String.length. A limit expressed in UTF-16 code units is a limit that means something different in every script.

4. Make the database agree with the application ¶

Sorting in the application and sorting in the database must produce the same order, or paginated results will skip and repeat rows. That means naming a collation explicitly on the column rather than inheriting a server default.

-- PostgreSQL: an ICU collation matching the application's locale
CREATE COLLATION de_phone (provider = icu, locale = 'de-u-co-phonebk');
ALTER TABLE users ALTER COLUMN display_name TYPE text COLLATE de_phone;

-- MySQL 8: utf8mb4_0900_ai_ci is accent- and case-insensitive
ALTER TABLE users MODIFY display_name VARCHAR(255)
  CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

If the two cannot be made to agree — a common situation when one database serves many locales — sort in one place only, and let the other side preserve the order it was given.

5. Fold carefully for search ¶

A search index wants aggressive folding: case, accents, ligatures, width. Storage wants none of it. Keeping them separate means the index can be rebuilt with different rules without touching the data.

const searchKey = (s: string) =>
  s.normalize('NFKD')
    .replace(/\p{Diacritic}/gu, '')   // drop combining marks
    .toLocaleLowerCase(locale);        // locale-aware: Turkish differs

Configuration reference ¶

Framework variants ¶

React / Next.js. Build the collator once per locale in a module-level cache and pass it down; constructing one inside a sort callback rebuilds it for every comparison, which turns an O(n log n) sort into something considerably worse. The same caching argument as date and number formatting standards applies, and matters more here because comparisons are called far more often.

Node.js backend. Verify the runtime has full ICU before relying on any of this: a slim build silently falls back to a root collation, so sorting appears to work and is wrong for every locale. process.versions.icu and a spot-check comparison at boot are enough.

Databases. PostgreSQL with the ICU provider gives per-column collations matching CLDR; MySQL 8’s utf8mb4_0900_* collations follow a recent Unicode version. In both cases an index is built for a specific collation, so changing the collation invalidates the index — plan it as a migration, not a configuration tweak.

Search engines. Elasticsearch and its relatives handle folding through analyzers, which means the folding rules live outside your application. Keep them documented next to the application’s own search-key function, or the two will diverge and results will differ between the quick client-side filter and the real search.

Verification ¶

test('names sort the way a German reader expects', () => {
  const names = ['Zypern', 'Österreich', 'Ungarn', 'Ägypten'];
  expect([...names].sort(collatorFor('de').compare))
    .toEqual(['Ägypten', 'Österreich', 'Ungarn', 'Zypern']);
});

test('a family emoji counts as one character', () => {
  expect(visibleLength('👩‍👩‍👧')).toBe(1);
  expect(truncate('👩‍👩‍👧 family', 1)).toBe('👩‍👩‍👧…');
});

test('decomposed and precomposed input compare equal after normalization', () => {
  expect(normalizeInput('e\u0301')).toBe(normalizeInput('é'));
});

Run the first test under a locale sweep. A collation that is right for German and wrong for Swedish will pass a single-locale suite and fail a customer.

Where the boundary between user text and catalogue text sits ¶

A product handles two kinds of text, and almost every rule on this page applies to only one of them.

Catalogue text is written by you and your translators. It arrives through the extraction pipeline, it is reviewed, and it is already normalized by whatever tool produced it. Its problems are the ones covered in string catalog governance: naming, ownership, and keeping the identity stable across edits.

User text is names, addresses, search queries, filenames, tags, messages. It arrives from an input method you do not control, in a script you did not anticipate, in whatever normalization form the operating system happened to produce. It is never reviewed, and it is the text this page is about.

The distinction matters because the two are frequently handled by the same code and need opposite treatment. Catalogue text should never be folded or normalized at runtime — doing so would change a translator’s deliberate choice. User text must be normalized before it is compared, or the comparison is a coin flip.

There is a third category worth naming because it is where the two collide: identifiers derived from user text. A slug, a username, a filename, a cache key. These are user text that has become machine text, and the derivation must be deterministic — same input, same output, on every platform and in every runtime. That means an explicit normalization form, an explicit case-folding locale, and an explicit set of permitted characters, rather than whatever the platform’s default does.

The most common production incident in this area comes from the third category. A user registers with a name normalized one way on a phone and logs in from a desktop that produces the other form, and the lookup misses. Nothing in the logs suggests text encoding; it looks like an authentication bug.

Case folding, and the Turkish i ¶

Lowercasing is not a per-character operation and it is not locale-independent, which is a surprise to nearly everyone the first time it bites.

Turkish and Azeri distinguish a dotted and a dotless i as separate letters. Uppercase I lowercases to the dotless ı in those locales and to i everywhere else; uppercase İ lowercases to i in Turkish and decomposes elsewhere. So 'I'.toLowerCase() produces different results depending on which locale rule is applied, and a username comparison that lowercases without specifying one will match or fail depending on where the code ran.

The rule that holds is to be explicit about which behaviour you want. For a user-facing display transformation, use toLocaleLowerCase(locale) so a Turkish reader sees Turkish casing. For an identifier or a lookup key, use toLowerCase() — the locale-independent form — everywhere, so the same input always produces the same key regardless of who is typing it or where the code runs.

German has a related case: ß uppercases to SS in the traditional mapping, so an uppercase transformation changes the string’s length, and a round trip through upper and lower case does not return the original. Any code assuming case transformations preserve length is wrong for German, and any code assuming they are reversible is wrong for both German and Turkish.

Common pitfalls ¶

• Sorting with Array.prototype.sort(). It compares UTF-16 code units, so accented letters land after z and every uppercase letter precedes every lowercase one.
• Normalizing on read. Comparisons happen in places that cannot normalize — index lookups, cache keys, equality checks. Normalize on write.
• Using NFKC for storage. It folds distinctions the user typed deliberately, and it cannot be undone.
• Measuring length in code units. A limit of 20 means twenty Latin letters, ten emoji halves, or a broken grapheme.
• toLowerCase() without a locale. Turkish and Azeri map the dotted and dotless i differently, and the default mapping breaks both.
• A database collation nobody chose. The application and the database then disagree about order, and pagination skips rows.

Performance, and where it actually goes ¶

Collation and segmentation are more expensive than the operations they replace, and knowing where the cost sits keeps the fix from becoming a regression.

Constructing a collator is the expensive part: it resolves the locale, loads the collation tailoring for it, and builds internal tables. Comparing two strings with an existing collator is cheap — comparable to a locale-independent comparison for typical short strings. That asymmetry is why the cache in the implementation above matters so much: a sort of a thousand names performs roughly ten thousand comparisons, and constructing a collator inside the comparator would perform ten thousand constructions.

Segmentation has a different profile. Each segment() call walks the string once, so the cost is linear in text length rather than in call count, and the segmenter itself is cheap to hold. The trap is calling it inside a render for text that has not changed — a character counter that re-segments the whole field on every keystroke is doing quadratic work. Memoising by input value, or segmenting only the tail that changed, removes it.

Database collation cost is the one that surprises teams at scale. An index built for an ICU collation is larger and slower to build than one built for a byte collation, and a query that sorts by a differently collated expression cannot use the index at all — it falls back to a sort of the whole result set. That is usually the real explanation when a locale-aware ordering makes a page an order of magnitude slower, and the fix is to make the index and the query agree rather than to abandon the collation.

None of this argues for skipping any of it. It argues for constructing once, holding the result, and checking that the database is sorting with the index rather than around it.

FAQ ¶

Related ¶

• Postgres and application sort order disagree — why paginated rows repeat or vanish when two collations are in play.
• Duplicate usernames from Unicode normalization — one name, two byte sequences, two accounts — and the folded key that prevents it.
• Emoji truncation breaking grapheme clusters — counting and cutting text the way a reader sees it.
• Search not matching accented names — folding both sides of the comparison, in a column an index can use.
• Date & Number Formatting Standards — the other family of Intl constructors, with the same caching rules.
• Intl Polyfills & ICU Data Loading — what happens when the locale data these APIs need is missing.
• RTL & Bidirectional Layout Engineering — the rendering counterpart to text handling.
• String Catalog Governance — why user text and catalogue text need different rules.
• Internationalizing Forms & Input — where most of this text enters the system.

Part of Core i18n Architecture & Locale Negotiation.