Duplicate usernames from Unicode normalization ¶

Two accounts exist for josé. The list renders them identically. The uniqueness constraint did not fire, the login form matches whichever was created first, and support cannot explain why one customer’s password “stopped working” on a new device.

The two rows are different byte sequences that display the same: one holds a precomposed é, the other an e followed by a combining acute accent. Everything downstream — the unique index, the login lookup, the mention autocomplete — compared bytes.

Root cause: input methods choose the encoding, not the user ¶

Unicode allows an accented letter to be written as a single precomposed code point or as a base letter plus one or more combining marks. Both are valid, both render identically with any competent font, and which one arrives depends entirely on the software that produced it.

That software varies. Some mobile keyboards and some desktop input methods emit decomposed sequences; most emit precomposed. Copy-and-paste preserves whatever the source used, so a name pasted from a document may differ from the same name typed. Certain platforms normalize filenames to a decomposed form, so a value read from a file differs from the same value typed into a form.

The user is unaware of any of this and has no way to influence it. From their perspective they typed their name twice and got two different answers.

The consequences are not limited to duplicates. A login that matches on the stored form fails when the reader types the other form. An invitation to josé@example.com reaches nobody. A mention autocomplete lists one of the two accounts. A permission check compares an identifier against a list and misses.

Minimal reproducible example ¶

const typed   = 'jose\u0301';   // e + combining acute — what one keyboard sent
const pasted  = 'josé';         // precomposed — what another sent

typed === pasted;               // false
typed.length;                   // 5
pasted.length;                  // 4
[...typed].length !== [...pasted].length;   // true — different code points

typed.normalize('NFC') === pasted.normalize('NFC');   // true

Insert both through a signup form with a unique index on the raw column and the database accepts both, because as far as it is concerned they are different values.

The fix: store two columns and index the derived one ¶

Identity and display are different jobs, and trying to make one column do both is what produces the fork.

// The display name is exactly what the reader typed, normalized only canonically.
// The fold key is a derived value used for every comparison.
export function identityKey(input: string): string {
  return input
    .normalize('NFKC')      // canonical + compatibility: fi → fi, Ａ → A
    .toLowerCase()          // locale-independent: identifiers must be stable
    .trim();
}

export function displayName(input: string): string {
  return input.normalize('NFC').trim();
}

ALTER TABLE users ADD COLUMN identity_key text NOT NULL;
CREATE UNIQUE INDEX users_identity_key ON users (identity_key);
-- The display name carries no constraint: two people may legitimately share one.

Two properties make this work. The unique index is on the derived key, so the database — not the application — enforces uniqueness, and a race between two signups cannot create a duplicate. And the display name is preserved exactly, so the reader sees their own spelling rather than a folded approximation.

Note the deliberate toLowerCase() rather than toLocaleLowerCase(). For an identifier you want the same result regardless of who is typing or where the code runs; the locale-sensitive form gives Turkish readers a different key for the same input, which recreates the problem in a new shape — the case-folding trap described in Unicode text handling and collation.

Deciding how much to fold ¶

NFKC plus case folding is the conservative default. Whether to go further — stripping accents entirely, so jose and josé are one identity — is a product decision with real arguments on both sides.

Folding accents prevents impersonation. An attacker registering josé to be mistaken for jose is a plausible attack on any system where a name conveys trust, and confusable-character attacks extend far beyond accents: Cyrillic а renders identically to Latin a in most fonts.

Folding accents destroys legitimate distinctions. In several languages an accent changes the word, and two people with genuinely different names would be told the name is taken. For a system where the identifier is a person’s actual name rather than a chosen handle, this is a poor trade.

The usual resolution is to fold aggressively for chosen handles — where uniqueness and trust matter and the user picks the string — and to fold only canonically for real names, where the value is descriptive rather than an identifier. Systems that need both keep them as separate fields, which is also the answer to the question of whether the display name may contain characters the handle may not.

Whatever the policy, it must be applied identically at signup, at login, at invitation, and in every lookup. A policy applied at signup only produces accounts that exist and cannot be found.

Retrofitting the key onto a live system ¶

Adding a folded identity key to a table that already has millions of rows, active signups and a login path is a migration with an ordering that matters, because the intermediate states have to keep working.

Add the column and backfill it, without a constraint. The key is derived from data that already exists, so the backfill is a batched update and nothing depends on it yet. Adding the unique index at this point would fail on the first existing duplicate.

Start writing the key on every insert and update. New rows now carry a correct key while old ones are being backfilled, and the two can coexist because nothing reads the column yet.

Report the duplicates and merge them. The grouping query above produces the list. Most entries are the same person twice, and a few are genuinely different people whose names fold together — which is exactly the signal that your folding policy is too aggressive, and it is far better to learn that here than after the constraint is live.

Add the unique index concurrently. On PostgreSQL a concurrent index build avoids locking the table for writes; it will fail if any duplicates remain, which is why the merge comes first.

Switch the lookups over, one path at a time. Login, invitation, mention, search and any administrative tool each query by the key. Doing them individually keeps each change small and reversible.

The step teams are tempted to skip is the third, because merging accounts requires product decisions rather than code. Skipping it means the index build fails repeatedly and someone eventually relaxes the folding to make it pass — which recreates the original bug with extra steps.

Verification ¶

test('normalization variants collapse to one identity', () => {
  const forms = ['josé', 'jose\u0301', 'JOSÉ', 'ｊｏｓé'];
  const keys = new Set(forms.map(identityKey));
  expect(keys.size).toBe(1);
});

test('the display name keeps the reader own spelling', () => {
  expect(displayName('jose\u0301')).toBe('josé');   // NFC, not folded
});

-- Find existing forks before adding the constraint
SELECT lower(normalize(display_name, NFKC)) AS k, count(*), array_agg(id)
FROM users GROUP BY k HAVING count(*) > 1;

That query is the first thing to run on an existing system: the constraint cannot be added until the duplicates it would have prevented are merged, and the list is usually shorter than people fear.

When to escalate ¶

If duplicates persist after the constraint exists, some path is inserting without deriving the key — a data import, an administrative tool, a background job. The derivation belongs in one place that every writer goes through, not in each writer.

If a merge is needed, treat it as an account merge rather than a text problem: which account keeps the identifier, what happens to the other’s data, and how the affected people are told. The text fix prevents new forks and does not resolve old ones.

If the identifier is an email address, remember that the local part is technically case-sensitive by specification and case-insensitive in practice at every major provider. Folding it is what users expect; documenting that you do is what keeps the decision defensible.

FAQ ¶

Related ¶

• Unicode Text Handling & Collation — the normalization and folding model behind the key.
• Search not matching accented names — the same folding applied to search rather than identity.
• Postgres and application sort order disagree — the ordering counterpart of this comparison problem.
• Internationalizing Forms & Input — the boundary where the normalization belongs.

Part of Unicode Text Handling & Collation.