Screenshot diffing RTL layouts in CI

Unit tests cannot see a chevron pointing the wrong way. They cannot see a drop shadow falling on the wrong side, an icon that stayed to the left of its label, or a numeric range whose endpoints now read backwards. Those are pixel facts, and the only automated way to hold them stable is to compare pixels.

This page sets up a right-to-left visual sweep that is reliable enough to gate a merge — which mostly means removing the four things that make screenshot tests flaky before adding the assertions.

Five layout properties that invert under right-to-left A chevron that points forward in a left-to-right layout must flip. Inline start moves from the left edge to the right. The visual order of a numeric range swaps. Icon and label order reverses. Even a drop shadow offset should mirror, or the light source appears to move. What only a mirrored render can catch Left-to-right Right-to-left Chevron direction points forward points backward — must flip Padding side start = left start = right Number range 1–20 reads left to right endpoints swap visually Icon + label order icon then text text then icon Shadow offset offset right offset left, or it looks wrong
None of these are visible in a left-to-right screenshot, which is why the sweep needs a mirrored baseline.

Root cause: mirroring is a rendering property

Setting dir="rtl" at the document root changes how the browser resolves every logical property, the base direction of every text run, and the visual order of inline content. That is a rendering-time transformation. Nothing in the component tree changes, so nothing a DOM-level test inspects changes either: the same elements exist, with the same classes, in the same order.

That is why the defect class is invisible to conventional tests, and why the failures are so consistently the same handful — icons with directional meaning, physical CSS properties that survived a logical-property migration, and shadows or gradients that encode a light source.

How a right-to-left visual sweep runs The runner renders the route in Arabic, stores the actual screenshot, loads the committed baseline for the same locale, compares them band by band, and either passes or emits a diff image naming the changed region. One visual sweep, three renders Test runner App Screenshot store Diff render /ar/settings ar.actual.png load ar.baseline.png compare pixel bands pass, or a diff image
Baselines are per locale — a single English baseline cannot tell you anything about a mirrored layout.

Setting up a sweep that does not flake

Before any assertion, remove the sources of nondeterminism. A screenshot test that fails ten percent of the time will be disabled within a month, and disabled tests catch nothing.

// playwright.config.ts
export default defineConfig({
  use: {
    viewport: { width: 1280, height: 800 },   // fixed, so layout is deterministic
    deviceScaleFactor: 1,
    locale: 'ar',
  },
  projects: [
    { name: 'rtl', use: { locale: 'ar' } },
    { name: 'ltr', use: { locale: 'en' } },
  ],
  expect: {
    toHaveScreenshot: {
      maxDiffPixelRatio: 0.002,   // tolerate antialiasing, not layout shifts
      animations: 'disabled',
    },
  },
});

Then, in the test itself, wait for the two asynchronous things that change glyph metrics: fonts and any deferred content.

import { test, expect } from '@playwright/test';

const ROUTES = ['/settings', '/checkout', '/orders'];

for (const route of ROUTES) {
  test(`${route} mirrors correctly @visual`, async ({ page }) => {
    await page.goto(`/ar${route}`);
    await page.evaluate(() => document.fonts.ready);      // no mid-capture font swap
    await page.waitForLoadState('networkidle');

    // Freeze anything time-dependent so the baseline stays valid tomorrow.
    await page.addInitScript(() => {
      Date.now = () => new Date('2026-01-15T12:00:00Z').getTime();
    });

    await expect(page).toHaveScreenshot(`ar${route.replace(/\//g, '-')}.png`, {
      fullPage: true,
      mask: [page.locator('[data-testid="avatar"]')],     // user content, not layout
    });
  });
}
The four usual sources of screenshot flakiness A web font finishing after the capture changes every glyph, so the capture must wait on document.fonts.ready. A running transition captures a half-finished state, so animations should be disabled in the test build. Scrollbar width differs by platform, so the viewport should be fixed and scrollbars hidden. A rendered date changes daily unless the clock is frozen. Why a screenshot diff goes flaky Cause Remedy Font swap webfont loads mid-capture await document.fonts.ready Animation a transition is still running disable animations in the test build Scrollbar platform-dependent width fixed viewport + hidden scrollbars Date in the UI the value changes daily freeze the clock
Fix all four before trusting a single diff — one unhandled cause makes every baseline suspect.

Reading a diff

A right-to-left diff usually falls into one of three shapes, and recognising the shape tells you what changed without opening the source.

A narrow vertical band at one edge of the image is a padding or margin change — something that was physical became logical, or the reverse. A small isolated blob is almost always an icon that started or stopped mirroring. A whole-page offset, where everything is shifted by a few pixels, is a scrollbar or a font metric difference rather than a layout change, and it means the environment drifted rather than the code.

That last case is the reason to run the sweep in a container image pinned by digest. A baseline captured on a developer’s machine and compared against a CI runner will differ for reasons that have nothing to do with the change under review.

Verification

The sweep is working when a deliberate regression fails it and nothing else does.

# Baseline (reviewed and committed once)
npx playwright test --grep @visual --update-snapshots

# Prove the sweep has teeth: reintroduce a physical property and re-run
#   .back-icon { margin-left: 8px }   ← should now fail in rtl only
npx playwright test --grep @visual --project=rtl

# Expected
#   ✗ /settings mirrors correctly — 412 pixels differ (region: 24,180 40×40)
#   ✓ /settings mirrors correctly [ltr]

A regression that fails in the right-to-left project and passes in the left-to-right one is exactly the signal the sweep exists to produce: the change is direction-dependent.

When to escalate

If diffs appear on every run regardless of the change, the environment is not pinned. Compare the font list and the device scale factor between the two runs before touching any styles.

If a genuine mirroring defect resists a logical-property fix, the element may be drawing its own direction — an inline SVG with a hardcoded transform, or a canvas rendering. Those need explicit handling, because CSS logical properties do not reach inside them. The icon-level decision of what should and should not mirror is covered in fixing mirrored icons and logical CSS properties.

If the layout is correct but the text runs read strangely — a Latin product name inside an Arabic sentence appearing at the wrong end — the problem is bidirectional isolation rather than layout, and it belongs in RTL and bidirectional layout engineering.

Masking: what to exclude and why it matters

A screenshot test compares everything in the frame, which means anything that legitimately varies between runs will fail it. Masking replaces a region with a solid block before comparison, and choosing what to mask is most of what makes a sweep maintainable.

Three categories are worth masking unconditionally. User-generated content — avatars, names, uploaded images — varies by fixture and tells you nothing about layout. Third-party embeds such as maps or payment iframes render on their own schedule and will eventually differ for reasons outside your repository. Anything genuinely random, including generated identifiers rendered into the page for debugging.

One category is worth not masking even though it is tempting: text content. Masking text to avoid churn removes the entire reason for a right-to-left sweep, since text position is what mirroring changes. If text is causing churn, the cause is font loading or a stale baseline, and masking it hides the problem rather than solving it.

await expect(page).toHaveScreenshot('ar-orders.png', {
  mask: [
    page.locator('[data-testid="avatar"]'),
    page.locator('iframe[title="map"]'),
    page.locator('[data-testid="request-id"]'),
  ],
  maskColor: '#ff00ff',   // an obvious colour, so an over-broad mask is visible in review
});

Using a loud mask colour is a small trick with a real payoff: in review, a mask that covers more than intended is immediately obvious, whereas a grey block blends into the design and hides the fact that half the page is no longer being tested.

Keeping the sweep off the critical path

A visual sweep that runs on every push slows every push. Two arrangements keep the feedback without the tax.

The first is a path filter: run the sweep only when files that can affect rendering change — stylesheets, component source, locale files — and skip it for documentation, tests and configuration. Most pull requests in a mature repository touch none of those.

The second is to run it in parallel with the rest of the pipeline rather than after it, and to make it a required check only on the merge queue. Developers get unit results in a minute and the visual verdict arrives before merge, which is the moment it actually needs to be correct. That places it in the same tier as the coverage gate described in GitHub Actions i18n CI gates.

FAQ

Should baselines be committed to the repository?

Yes, and they should be reviewed like code. A baseline update in a pull request is a visual change, and having a human approve the new image is the entire point — an auto-updating baseline records regressions rather than catching them. Keep them small by capturing key routes rather than every page.

How many routes are worth capturing?

Enough to cover each distinct layout, not each page. Most applications have a handful of shells — a form, a table, a dashboard, a modal — and a route per shell finds nearly everything. Twenty screenshots that run in ninety seconds get kept; two hundred that run in fifteen minutes get skipped.

Can this replace testing with a real Arabic build?

It catches layout regressions, which is most of what goes wrong mechanically. It cannot judge whether the Arabic reads well, whether the typeface is appropriate, or whether a term is right. Those need a speaker of the language, and the sweep exists so their time is not spent finding misplaced chevrons.

What tolerance should the diff use?

Small but non-zero. A maxDiffPixelRatio of around 0.002 absorbs antialiasing differences between otherwise identical renders while still failing on a shifted element, because even a small mirroring defect moves far more than two pixels in a thousand. Setting the tolerance to zero produces failures on font hinting; setting it to a percent hides real layout shifts in dense interfaces.

Should the same sweep also cover light and dark themes?

Only if the theme changes layout, which it usually does not. Colour differences produce a total diff on every pixel, so a theme variant doubles the baseline count while testing the same geometry. Cover themes with a contrast check instead, and reserve the screenshot budget for the axes that genuinely move elements: direction, locale and viewport.

Part of Localization Testing & Pseudolocalization.