Files
kandev/apps/web/e2e/tests/i18n/pseudo-coverage.spec.ts
T
2026-08-13 11:26:02 +01:00

885 lines
44 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { type Page } from "@playwright/test";
import { test, expect } from "../../fixtures/test-base";
/**
* Pseudo-locale coverage oracle.
*
* Under the pseudo-locale every EXTRACTED message is accented
* (`Language` → `Ĺàńĝũàĝē`). So any user-facing copy that is still plain ASCII
* is, by definition, a string that was never wrapped in a Lingui macro. This
* spec crawls chrome-heavy screens and reports those leftovers.
*
* It runs TWO passes, because copy reaches the user by two routes:
* - visible text nodes, and
* - `COPY_ATTRIBUTES` — the copy a screen-reader user receives and a sighted
* one never sees. That pass is why the oracle is no longer blind to an
* `aria-label` that was never externalized; see docs/i18n.md.
*
* `SCREENS` mirrors `i18nGuardFiles` in eslint.i18n.options.mjs: a screen belongs
* here once the components that render it have been migrated. The eslint guard
* only sees plain literals in JSX, so it and this spec cover different halves of
* the same question — add to both in the PR that migrates a directory.
*
* THIS IS A HARD CI GATE. It ran behind `KANDEV_I18N_COVERAGE=1` only while the
* migration was mid-flight, because a screen's own copy could be clean while
* shared chrome it renders was not yet. The last live directory has landed and
* the env guard is gone: it now runs unconditionally in the `chromium` project
* alongside every other spec.
*
* pnpm e2e:raw -- e2e/tests/i18n/pseudo-coverage.spec.ts
*
* WHAT IT GUARANTEES: on each screen in `SCREENS`, every rendered text node and
* every value of a `COPY_ATTRIBUTES` attribute that the detector below can see
* either came through `t()` / `<Trans>` (and so renders accented under the
* pseudo-locale), or is on an allowlist that says why it must not be translated.
*
* That guarantee rests on the scan having run against the screen it names, which
* for a while it did not. Each `SCREENS` entry carries a required `anchor` — a
* selector that cannot match until that route rendered — and `waitForScreen`
* blocks on it before anything is inspected. Read its comment before touching
* the readiness logic: the wait it replaced was a one-second settle, and with
* both lazy route chunks blocked so that NO screen rendered, 13-16 of the 21
* tests still reported clean.
*
* It is a floor, not a proof of total coverage. Two limits are deliberate and
* both are documented where they are implemented, because relaxing either
* reddens every screen today:
* - `wordlike` requires a 4-letter ASCII run, so a hardcoded string shorter
* than that ("Add", "of") is not reported. Lowering it to 2 was measured:
* 9 of the 10 tests here go red, on real but out-of-scope copy owned by
* `@kandev/ui` and sonner (e.g. the `alt+T` in sonner's toast-region label).
* - the text pass clears a node on ANY accented character, so an un-migrated
* word sharing one text node with migrated copy is missed. The attribute
* pass does NOT have this limit — it strips allowlisted tokens first and
* reports the remainder as `English frame, migrated value`. docs/i18n.md
* covers the text-node case under "…and it is weakest at an interpolated
* value"; it is why the by-hand pseudo walk is still step 9 of a migration.
*
* IF YOUR PR JUST WENT RED HERE, the failure names the exact strings and the
* screen. Read them: each one is copy your change put on screen without routing
* it through `t()`. This spec is the ONLY gate that can see most of them —
* `i18next/no-literal-string` inspects literals in JSX and nothing else, so copy
* in a variable, a default parameter, a `.ts` helper or a SCREAMING_CASE config
* table is invisible to lint and visible only here. Repeated by-hand sweeps
* found 15-60 such strings that every other gate passed.
*
* The fix is to externalize the string: add it to `src/locales/en/<ns>.json` and
* call `t("<ns>:<key>")`. See docs/i18n.md. Adding an `allow:` entry is NOT the
* fix and is the one failure mode this file has actually shipped twice — an
* entry asserting a path was migrated while it still rendered English. `allow`
* is for text the frontend must not translate but cannot avoid rendering (a
* backend-owned record, a product name), and it must say where the value comes
* from.
*
* BOTH passes walk `document.body`, deliberately, and NOT the page's `<main>`.
* Scoping to the page under test is the obvious reading of what each test claims,
* and it was tried and rejected on one fact: `@kandev/ui` builds Dialog,
* AlertDialog, DropdownMenu, Tooltip and Select on Radix `Portal`, which mounts
* to `document.body`. Every dialog, menu and tooltip in the app therefore renders
* OUTSIDE `<main>`, so a scoped walk would stop checking the surfaces densest in
* copy — and would do it silently, reporting clean. Trading a visible annoyance
* for an invisible blind spot is the wrong direction for an oracle whose whole
* purpose is the strings nothing else can see.
*
* The cost of body-wide is that a screen can fail for chrome it does not own.
* That cost is now small — migrating the sidebar, settings nav and status bar
* (#2214) took the baseline from 28 findings to 5 — and it buys the one thing
* nothing else has: copy that belongs to no directory, like `@kandev/ui`'s
* hardcoded breadcrumb landmark and sonner's toast-container label, is visible
* only because the walk is body-wide. The fix for chrome noise is migrating
* chrome, not narrowing the oracle.
*/
/**
* Migrated screens whose visible text is overwhelmingly UI chrome, not user data.
*
* `allow` extends `ALLOWED` for one screen only. Use it for text the frontend
* must NOT translate but cannot avoid rendering — records the backend owns, or a
* product name — and say where the value comes from, so the exemption stays
* auditable instead of quietly widening into a place to hide missed strings.
*
* `anchor` is REQUIRED and is what makes a green result mean anything; see
* `waitForScreen` below for why a settle timeout did not.
*/
type Screen = {
name: string;
url: string;
/**
* A selector that cannot match until the ROUTE UNDER TEST has rendered. The
* walk does not start until it matches AND the matched element's text is
* accented.
*
* The test it must pass is not "is it unique" but "could the app shell alone
* satisfy it?". `secrets-settings-body` also renders on the workspace-scoped
* secrets page, and that is fine — reaching it still proves a secrets screen
* mounted. What disqualifies a selector is matching something painted before
* the route arrives, which is exactly the bug this field fixes.
*
* The two route trees answer that differently, and the difference is real
* rather than an inconsistency to tidy up:
*
* - SETTINGS screens anchor on a `data-testid` the page's own component
* renders. Seven already had one; `secrets-settings-body` and
* `sprites-connection-card` were added alongside this field.
* - OFFICE screens anchor on `main[data-office-route="<path>"]`, the route
* outlet in src/office-routes.tsx stamped with the RESOLVED path. A
* component-level testid was tried and rejected: the SPA route table
* mounts these pages with empty collections (`initialItems={[]}`), so they
* legitimately render empty states in e2e and any anchor inside the
* populated branch would never match. The outlet is present whichever
* branch a page takes, lives inside the office chunk, is absent while
* `OfficeRouteLoading` holds, and — because the selector pins the VALUE —
* cannot be satisfied by a different office route or by the unknown-route
* fallback.
*
* Note what carries the weight in the Office case: the element is generic, so
* the ACCENTED half of `waitForScreen` is what proves a page actually painted
* into it. An empty outlet has no accented text and times out.
*
* An attribute and not a heading name, deliberately, in both trees: the anchor
* has to be findable BEFORE the pseudo catalog is known to have applied, and
* every accessible name in the app changes under pseudo.
*/
anchor: string;
allow?: string[];
};
const SCREENS: Screen[] = [
{
name: "settings — appearance",
url: "/settings/preferences/appearance",
anchor: "[data-testid=theme-settings-card]",
},
{
name: "settings — notifications",
url: "/settings/preferences/notifications",
anchor: "[data-testid=notification-sound-group]",
// Provider names are rows in the notification_providers table. The backend
// seeds these two (apps/backend/internal/notifications/service/service.go)
// and users name their own Apprise ones, so they are data on the same
// footing as a task title. `Apprise` labels the provider type.
allow: ["Desktop Notifications", "System Notifications", "Apprise"],
},
{
name: "settings — secrets",
url: "/settings/secrets",
anchor: "[data-testid=secrets-settings-body]",
},
{
name: "settings — terminal & editors",
url: "/settings/preferences/terminal-editors",
anchor: "[data-testid=terminal-font-size-card]",
// Every entry is a `label` or a `binary` from `LSP_LANGUAGE_OPTIONS`
// (components/settings/lsp-language-options.ts), whose header states the
// rule these follow: a language name and an executable name are shown
// verbatim, and translating either would name a server that does not exist.
// The qualifier around them ("experimental") and the frame around the
// binaries in the `Auto-start {{language}} language server` aria-labels DO
// go through `t()`, and the attribute pass still checks that frame after
// these tokens are stripped. `Go` is absent on purpose: it is under the
// four-letter `wordlike` threshold, so it is never reported and adding it
// would only widen the mask; `gopls` is over it, so it is listed.
allow: [
"TypeScript / JavaScript",
"typescript-language-server",
"gopls",
"Rust",
"rust-analyzer",
"Python",
"pyright-langserver",
"Kotlin",
"kotlin-lsp",
],
},
// NOT YET: "settings — sprites". The entry existed at `/settings/general/sprites`
// with the `sprites-connection-card` anchor; this restructure folded that card
// into `/settings/executors` (SpritesSettings is rendered by the executors
// page), and that route is excluded below because the fixture's executor
// profile NAMES are user data. The card's copy is covered the moment the
// executors entry is unblocked — it does not need a screen of its own, and
// pointing this entry at `/settings/executors` would just re-import that
// screen's blocker under a name that hides it.
{
name: "settings — layouts",
url: "/settings/preferences/layouts",
anchor: "[data-testid=layout-settings]",
// NOTE: everything here must be a PERSISTED name. This list is load-bearing
// in the wrong direction — a broad token also hides genuinely un-migrated
// copy that happens to match it. "Default" already masked the
// default-action button once, which rendered a raw English "Default" until
// `getDefaultActionState` was fixed to resolve it through the catalog.
//
// It fails in the OTHER direction too, and has: these entries duplicate
// strings that live in `lib/layout/layout-profiles.ts`, with nothing keeping
// the two in sync. #2198 changed the Default profile's description and this
// list kept the old wording, so it manufactured a phantom finding for every
// run between `cc6eb4dd5` and this commit. Re-check these against
// `BUILT_IN_LAYOUT_PROFILES` whenever a built-in name or description moves.
//
// Both groups WERE display strings that are also persisted, and both have
// had the key/persisted-value split this note used to ask for:
// - Built-in profile names/descriptions (lib/layout/layout-profiles.ts)
// carry `nameKey`/`descriptionKey`; `name` stays canonical English
// because `upsertBuiltInLayoutOverride` copies it into the saved record.
// - Dockview panel titles (lib/state/layout-manager/constants.ts) carry
// `titleKey`; `toSerializedDockview` writes the canonical English
// `title` into the stored layout JSON, and `normalizePanel` re-resolves
// the display title on restore. `panel-titles.test.ts` covers both
// directions.
// Only the two product names are still expected to render as English.
allow: ["VS Code", "Agent and VS Code side by side"],
},
{
name: "settings — keyboard shortcuts",
url: "/settings/preferences/keyboard-shortcuts",
anchor: "[data-testid=chat-submit-key-card]",
// Modifier/key names label a physical key and are out of scope for
// translation — the same rule the eslint guard's keyboard pattern encodes.
//
// Shortcut names come from CONFIGURABLE_SHORTCUTS in
// lib/keyboard/shortcut-overrides.ts.
allow: [
"Ctrl",
"Shift",
"Alt",
"Cmd",
"Meta",
"Space",
"Enter",
"Tab",
"Command Panel",
"Command Panel (Alt)",
"File Search",
"Search Task Contents",
"Quick Chat",
"Toggle Bottom Terminal",
"Toggle Sidebar",
"New Task",
"Focus Chat Input",
"Focus CLI Chat Input",
"Toggle Plan Mode",
"Recent Task Switcher",
"Recent Task Switcher (Backward)",
"Reverse Chat Search",
"Open Task Pull Request",
"Open Workspace Picker",
],
},
{
name: "settings — task behavior",
url: "/settings/preferences/task-behavior",
anchor: "[data-testid=archive-confirmation-card]",
},
{
name: "settings — plugins",
url: "/settings/plugins",
anchor: "[data-testid=plugins-tab-installed]",
},
// Office — the last live directory to migrate (#2357 was the batch before it).
// All twelve routes were run under the pseudo-locale before being listed here;
// eleven came back with nothing, and the one `allow` below is the only string
// any of them renders that must stay ASCII.
//
// These twelve reach the browser through a DIFFERENT route tree than the
// settings screens above — their own lazy chunk, loaded by `office-routes`
// rather than `settings-routes` — so they need their own anchors and were
// proved separately; see `waitForScreen`.
// NOT YET: "office — dashboard" (`/office`). The entry existed and passed, and
// it was not testing the dashboard.
//
// `/office` renders the dashboard only for a workspace that has an
// `office_workflow_id` (`hasOfficeWorkspace` in src/office-routes.tsx). The
// shared e2e fixture never seeds one, so `resolveOfficeHomeSetupRedirect`
// sends `/office` to `/office/setup?mode=new` on every run, and the screen the
// walk actually scanned was the SETUP WIZARD — "Set up your Office workspace",
// a different screen from the one the test named. It reported clean because
// the wizard is itself fully migrated, so the pass was real and was about
// something else.
//
// The anchor is what surfaced this: the redirect is invisible to a walk that
// only asks "is anything accented". Restoring real dashboard coverage needs an
// office workflow seeded in the fixture, which is a change to shared
// `test-base` state and belongs with whoever owns that fixture — not an
// allowlist entry and not a renamed screen, either of which would keep the
// false claim alive in a new form.
// NOT YET: "kanban board" (`/`, anchor `[data-testid=kanban-board]`). Probed
// under pseudo against this fixture. The board's own chrome is clean, and the
// one real finding it surfaced — the built-in sidebar view's "All tasks",
// which `sidebar-view-builtins.ts` persists as a user-editable name — is fixed
// and now resolves through `sidebar:viewAllTasks`.
//
// What is left is entirely the fixture's own records: the workflow name
// ("E2E Workflow") and its four step names ("Backlog", "In Progress",
// "Review", "Done"). Those are user data on exactly the footing as the
// workspace and executor-profile names described above, so this entry needs
// the same change — `findUnlocalizedText` learning to skip user data — and not
// an `allow` list of fixture values, which would fix this fixture and leave
// every developer instance red under different workflow names.
//
// `/tasks` was probed with it and cannot be added for a different reason:
// `[data-testid=tasks-list]` never renders in this fixture, so there is no
// anchor that proves the route mounted. It needs a seeded task list first.
{
name: "office — inbox",
url: "/office/inbox",
anchor: 'main[data-office-route="/office/inbox"]',
},
{
name: "office — tasks",
url: "/office/tasks",
anchor: 'main[data-office-route="/office/tasks"]',
},
{
name: "office — agents",
url: "/office/agents",
anchor: 'main[data-office-route="/office/agents"]',
},
{
name: "office — projects",
url: "/office/projects",
anchor: 'main[data-office-route="/office/projects"]',
},
{
name: "office — routines",
url: "/office/routines",
anchor: 'main[data-office-route="/office/routines"]',
},
{
name: "office — workspace costs",
url: "/office/workspace/costs",
anchor: 'main[data-office-route="/office/workspace/costs"]',
},
{
name: "office — workspace activity",
url: "/office/workspace/activity",
anchor: 'main[data-office-route="/office/workspace/activity"]',
},
{
name: "office — workspace routing",
url: "/office/workspace/routing",
anchor: 'main[data-office-route="/office/workspace/routing"]',
},
{
name: "office — workspace skills",
url: "/office/workspace/skills",
anchor: 'main[data-office-route="/office/workspace/skills"]',
},
{
name: "office — workspace org",
url: "/office/workspace/org",
anchor: 'main[data-office-route="/office/workspace/org"]',
},
{
name: "office — workspace settings",
url: "/office/workspace/settings",
anchor: 'main[data-office-route="/office/workspace/settings"]',
// The clone field's placeholder is an example git URL — the SHAPE the user
// is meant to imitate, on the same footing as the email and CSS-function
// placeholders the eslint guard already excludes via its `(https?|ssh|git)://`
// pattern. Accenting it would show a URL that cannot be typed.
allow: ["https://github.com/org/config.git"],
},
// NOT YET: "settings — executors", "settings — account security",
// "settings — account tokens". All three routes' own copy is fully migrated
// and was verified by walking them under pseudo — including every dialog and
// both marketplace tabs, since a static page scan never opens a Radix portal.
// What stops them being entries here is what the fixture renders BESIDE that
// copy, and in each case an `allow` entry would be the wrong fix:
// - `/settings/executors` renders the seeded executor profiles' NAMES, and
// the e2e fixture happens to name them `Local` and `Worktree`. That is
// user data on exactly the footing as the workspace names below, so it
// needs `findUnlocalizedText` to stop treating user data as eligible —
// listing the two values would fix this fixture and leave every developer
// instance broken under different names. The hub cards' own `Docker` and
// `Sprites.dev` labels are brand nouns; they are in the guard's
// `words.exclude` but not in `ALLOWED`, which is a real sync gap in that
// list rather than something this migration should widen it to paper over.
// - Both account routes render `not found` — the router's 404 body, because
// the e2e profile does not register the auth routes, surfaced in the
// sessions/tokens error region. It is a backend diagnostic, English by
// design (docs/i18n.md, "What the backend deliberately does NOT
// translate"), and allowlisting a payload string would hide the next real
// miss in the same region.
// The `plugins` entry above has neither problem and passes with no `allow`.
// Its marketplace entries — names, descriptions, categories, source names and
// index.json URLs — come from the catalog rather than the DOM on load.
//
// NOT YET: "settings — integrations github", "… gitlab", "… jira",
// "… linear", "… sentry". Each page's
// own copy is fully migrated (verified by running this oracle against it —
// every string the integration owns renders accented).
//
// The settings-nav half of this blocker is now GONE: `settings-tree.tsx`,
// `workspaces-group.tsx`, `executors-group.tsx`, `account-group.tsx` and
// `theme-toggle.tsx` are migrated, so "Workspaces", "Integrations",
// "Automations", "Executors", "Utility Agents", "External MCP",
// "Plugins" and "Toggle theme" render accented on every screen. Only "System"
// is left, from `sections/settings/system-group.tsx`, which the System-routes
// migration owns.
//
// What still blocks these five entries is the shared integration chrome, not
// the nav. `components/integrations/**` is now migrated, which cleared
// `drafted-integration-enabled-control.tsx` ("Enabled"/"Disabled"),
// `auth-status-banner.tsx` ("Authenticated", "· checked <relative>") and the
// watcher card's loading state. What is left is `stepPlaceholder`, shared
// with the un-migrated Azure DevOps surface, plus
// `components/integrations/settings-section.tsx` chrome and `@kandev/ui`'s
// built-in dialog "Close" label.
//
// NOT YET: "settings — external mcp", "… prompts",
// "… utility agents". All four pages' own copy is fully migrated and was
// verified by running this oracle against each of them — every string those
// routes own renders accented, including the 43 in the MCP tool catalog
// (`lib/settings/external-mcp-tools.ts`), which is a `.ts` module no lint rule
// inspects. The collapsed tools preview, the utility agent dialog and the
// inference status note were scanned separately, since this spec only sees
// what a route renders on load.
//
// #2214 migrated the settings nav, which was the original blocker, and these
// four now come back clean of it. What still stops them being entries here is
// copy none of them owns:
// - `aria-label="breadcrumb"` from `@kandev/ui`'s Breadcrumb primitive, and
// Sonner's `Notifications alt+T` toast-region label. Both are the shared-UI
// case docs/i18n.md describes: they need a strings-provider seam in the
// package, not a per-route fix.
// - `Configuration Chat` from `components/config-chat/`, not yet migrated.
// Allowlisting either here would hide misses belonging to whoever owns them,
// which is the failure this list's own comment warns about. The workspace
// names these routes also surfaced are no longer among them: #2220 derives
// them from the boot payload and makes them ineligible, which is the right
// fix — they are user data, and listing them by value would have fixed the
// fixture and left every developer's own instance broken.
//
// Each route also renders data that is correctly English and would need an
// `allow` entry: the built-in prompts' names and bodies and the built-in
// utility agents' names and descriptions (both backend-authored), the agent
// product names and config paths on External MCP, and `Ctrl+Shift+M`.
//
// NOT YET: "settings — workspace workflows"
// (`/settings/workspaces/:id/workflows`). The workflow editor's own copy is
// fully migrated and was verified with this oracle against a live instance,
// and the Workspaces branch of the nav it expands is now migrated too. What
// remains is that the route renders the workspace's own name and its
// workflow/step names — user data, on the same footing as the fixture's
// "E2E Workspace". Adding the entry needs `findUnlocalizedText` to stop
// treating user data as eligible, not an allowlist entry per fixture name.
];
/**
* Text that is legitimately un-accented under pseudo: brand/proper nouns, code
* identifiers, and units/symbols. Kept in sync with `words.exclude` in
* apps/web/eslint.i18n.options.mjs.
*/
const ALLOWED = [
"Kandev",
"GitHub",
"GitLab",
"Jira",
"Linear",
"Slack",
"Sentry",
"Azure DevOps",
"ACP",
"MCP",
"SSH",
"URL",
"ID",
"English",
"Pseudo",
"QA",
];
/**
* Any character in the accented range the pseudo-locale transform emits. Shared
* with the in-page detector below, which cannot close over it.
*/
const ACCENTED = /[À-ɏ]/;
async function activatePseudo(page: Page, url: string) {
await page.goto(url);
await page.evaluate(() => {
document.cookie = "kandev_locale=pseudo; path=/; max-age=31536000; SameSite=Lax";
});
await page.reload();
// `<html lang>` proves the SHELL is on the pseudo locale and nothing more: the
// Go shell writes it from the cookie, so it is already correct in the HTML the
// browser parses, before any script has run. Every stronger fact — the route
// rendered, and the catalog reached its copy — is `waitForScreen`'s job, and
// it is per-screen precisely because anything generic enough to assert here is
// something the app shell alone satisfies.
await expect(page.locator("html")).toHaveAttribute("lang", "pseudo", { timeout: 15_000 });
}
/**
* Block until the SCREEN UNDER TEST has rendered under the pseudo locale.
*
* This replaces a `waitForTimeout(1_000)` settle, and the reason is the only
* thing in this file worth reading twice: THE SETTLE LET THE ORACLE REPORT A
* PASS IT HAD NOT EARNED, and it did so in the direction that hurts.
*
* `SCREENS` are lazy routes in two separate chunks — `settings-routes` (2.8 MB)
* and `office-routes`. Nothing in the old wait was tied to either arriving:
* - `activatePseudo`'s `lang="pseudo"` assertion is written server-side by
* shell.go before a single script runs, so it is true immediately;
* - the `inspectedAttributes > 0` control below is satisfied by the sidebar,
* topbar and status bar, which are accented long before the route renders;
* - so the only thing standing between "route rendered" and "scan the DOM"
* was one second of wall clock.
* Measured with BOTH chunks blocked outright, so that not one screen rendered:
* 13-16 of the 21 tests reported clean, varying run to run — because whether a
* screen "passed" came down to a race the settle could not see. The failure mode
* is worse than flakiness — under CI load a slow render makes a green result
* MORE likely, so the gate is at its most permissive exactly when the tree is
* most likely to be broken. With the anchors below the same probe fails all 21,
* every run.
*
* The fix has to be per-screen. Any signal generic enough to share is a signal
* the app shell already satisfies, which is how this happened in the first
* place. So each screen names a selector nothing but that route can satisfy, and
* this waits on two facts about it:
* 1. it is VISIBLE — the lazy route mounted and this screen, specifically, is
* the one on screen;
* 2. its text is ACCENTED — the pseudo catalog has actually been applied to
* copy this route owns. This was written as insurance against a catalog
* that arrives asynchronously; it stopped being hypothetical the moment
* lazy catalog loading landed, and `pseudo` is now a fetched chunk like
* every locale but `en`. Without it the walk could run against a route
* rendered in English fallback and report every string on it as a miss.
* Waiting here is robust to that without weakening anything: a wait can
* only delay the scan, never excuse a finding.
*
* Both are `expect` assertions, not `waitFor`s, so an anchor that never arrives
* FAILS this spec instead of quietly scanning whatever the shell had painted.
* That is the whole point: the gate has to be capable of going red.
*
* Do NOT be tempted to swap this for `waitForLoadState("networkidle")`. Kandev
* holds a live WebSocket for the session stream, so the network is never idle
* and it would hang until timeout — it looks like a stronger wait and is not a
* wait at all.
*/
async function waitForScreen(page: Page, screen: Screen) {
// NOT `.first()`. A selector that matches more than one element is an
// ambiguous anchor, and `.first()` would silently pick one and gate on it —
// the same shape of mistake as the settle this function replaced. Playwright's
// strict mode turns that ambiguity into a failure, which is the direction this
// spec wants. All nine anchors resolve to exactly one element today.
const anchor = page.locator(screen.anchor);
await expect(
anchor,
`The render anchor \`${screen.anchor}\` never appeared on ${screen.name}, so the ` +
`route under test did not render. Scanning now would report the app shell as ` +
`clean and prove nothing about ${screen.name}. If this element was renamed, ` +
`update the \`anchor\` for this screen in SCREENS.`,
).toBeVisible({ timeout: 15_000 });
await expect(
anchor,
`The render anchor \`${screen.anchor}\` rendered on ${screen.name} but its copy ` +
`never went accented, so the pseudo catalog had not been applied to this ` +
`route's copy. Every string on the screen would be reported as a miss. If ` +
`this anchor's own copy was just un-externalized, THAT is the finding — fix ` +
`it rather than moving the anchor.`,
).toHaveText(ACCENTED, { timeout: 15_000 });
}
/**
* A route anchor can be ready while a background panel is still reconciling.
* Do not scan during that window: the loading status is translated, but its
* first render can briefly retain the English fallback while the pseudo
* catalog update reaches the mounted component.
*/
async function waitForTransientLoadingToFinish(page: Page) {
await expect(page.locator('[role="status"][aria-label="Loading"]')).toHaveCount(0, {
timeout: 15_000,
});
}
/**
* Attributes that carry display copy rather than a value the app compares.
* Kept in sync with `jsx-attributes.include` in apps/web/eslint.i18n.options.mjs,
* so the guard and this oracle answer the same question about the same set: on an
* allowlisted path lint flags the literal, and here it is checked at render.
*
* `aria-labelledby` / `aria-describedby` / `aria-controls` are deliberately
* absent — they carry element ids, not prose, and are excluded by the guard too.
*
* `title` is checked on EVERY element, not just interactive ones. A `title` is
* exposed as an accessible name or description whatever the element is, and the
* browser renders its native tooltip on hover for any of them — so "the app never
* shows this one" is not true of a rendered `title`. Restricting it to
* interactive elements would need a tag/role/tabindex heuristic, and that
* heuristic would immediately become the place attribute copy hides. The guard
* checks `title` unconditionally too; the two stay in step.
*/
const COPY_ATTRIBUTES = ["aria-label", "aria-description", "title", "placeholder", "alt"];
/**
* Collect copy that looks un-externalized: plain-English visible text nodes, and
* plain-English values of the five copy-bearing attributes.
*
* Attribute findings are returned prefixed with the attribute and the element
* that carries it (`aria-label on button[data-testid=…]: "More information"`),
* because an attribute string is invisible on screen — a bare value gives you
* nothing to grep for and nowhere to look.
*
* Two counters make an empty `leftovers` mean something, because a selector that
* matched no elements reports exactly as clean as a clean screen:
* - `inspectedAttributes` — non-empty attribute values examined. Asserted per
* screen; this is the one that catches a selector matching nothing.
* - `localizedAttributes` — those that rendered FULLY accented, i.e. migrated
* attribute copy the pass demonstrably reached. Not asserted per screen: a
* page of plain inputs and text can legitimately have none, and secrets,
* terminal and sprites do. Pinned once instead, in its own test below.
*/
async function findUnlocalizedCopy(
page: Page,
allowed: string[],
): Promise<{ leftovers: string[]; localizedAttributes: string[]; inspectedAttributes: number }> {
return page.evaluate(
({ allowedList, copyAttributes }) => {
const skipTags = new Set(["SCRIPT", "STYLE", "NOSCRIPT", "CODE", "PRE", "SVG"]);
// A narrower set for attributes. `skipTags` above encodes "this element's
// CONTENT is not prose", which is a different question from whether its
// LABEL is: an `aria-label` on an <svg> icon, or a `title` on a <code>
// chip, is copy the user receives either way. Only the three tags that
// cannot present a label at all are skipped here.
const attrSkipTags = new Set(["SCRIPT", "STYLE", "NOSCRIPT"]);
const wordlike = /[A-Za-z]{4,}/;
// Same range as `ACCENTED` above, restated because this body is
// serialized into the page and cannot close over module scope.
const accented = /[À-ɏ]/;
// User data is never copy, and a workspace's name is user data on the same
// footing as a task title. Derived from the boot payload rather than
// listed, because listing it fixes one instance and leaves every other
// one broken: `E2E Workspace` is only the fixture's name, and a developer
// running this against their own instance would have hit the identical
// false positive under a different string.
const workspaceNames = (() => {
const payload = (window as unknown as { __KANDEV_BOOT_PAYLOAD__?: unknown })
.__KANDEV_BOOT_PAYLOAD__;
const state = (payload as { initialState?: { workspaces?: { items?: unknown } } })
?.initialState;
const items = state?.workspaces?.items;
if (!Array.isArray(items)) return [];
return items
.map((item) => (item as { name?: unknown })?.name)
.filter((name): name is string => typeof name === "string" && name.trim().length > 0);
})();
// Longest first: stripping "VS Code" before "Agent and VS Code side by
// side" would leave "side by side" behind and report it as a leftover.
const tokens = [...allowedList, ...workspaceNames].sort((a, b) => b.length - a.length);
/** Still word-like ASCII once allowlisted tokens are removed. */
const hasUnmigratedAscii = (text: string) => {
if (!text || !wordlike.test(text)) return false;
let residue = text;
for (const token of tokens) residue = residue.split(token).join(" ");
return wordlike.test(residue);
};
/** The text pass's rule, unchanged: any accented character clears a node. */
const looksUnlocalized = (text: string) => !accented.test(text) && hasUnmigratedAscii(text);
// The text pass drops zero-size nodes because nothing is on screen to read.
// That test is WRONG for an attribute: a visually hidden control with an
// `aria-label` is precisely the case this pass exists to catch, and every
// `sr-only` node measures as good as zero. What actually disqualifies a
// label is the element not being rendered at all — `display: none`,
// `visibility: hidden`, or a skipped `content-visibility` subtree — because
// then no user receives it, sighted or not. `checkVisibility` asks that
// directly; deliberately WITHOUT `opacityProperty`, since an opacity-0
// element is still in the accessibility tree.
const isRendered = (el: Element) =>
typeof el.checkVisibility !== "function" ||
el.checkVisibility({ contentVisibilityAuto: true, visibilityProperty: true });
const collectText = () => {
const found = new Set<string>();
const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT);
let node: Node | null;
while ((node = walker.nextNode())) {
const text = (node.textContent ?? "").trim();
if (!looksUnlocalized(text)) continue;
const el = node.parentElement;
if (!el || skipTags.has(el.tagName)) continue;
// Same rendered-or-not test the attribute pass uses, rather than the
// zero-size rect this used to apply. The rect check kept `sr-only`
// text ONLY because that utility measures 1px rather than 0 — so
// whether screen-reader-only copy was checked came down to a CSS
// implementation detail, and changing `.sr-only` to `clip` at 0×0
// would have silently dropped it. It IS copy, and a deliberately
// in-scope kind: `theme-toggle.tsx`'s "Toggle theme" reached a user
// only this way. `checkVisibility` keeps it for the right reason —
// the element renders and is in the accessibility tree — while
// correctly dropping `visibility: hidden`, which the rect check kept.
if (!isRendered(el)) continue;
found.add(text.slice(0, 120));
}
return [...found];
};
/** Enough to find the element in the DOM and in the source. */
const describe = (el: Element) => {
const tag = el.tagName.toLowerCase();
const testId = el.getAttribute("data-testid");
if (testId) return `${tag}[data-testid=${testId}]`;
if (el.id) return `${tag}#${el.id}`;
const role = el.getAttribute("role");
return role ? `${tag}[role=${role}]` : tag;
};
const seen = new Map<string, { label: string; count: number }>();
// The positive control. An attribute that HAS been migrated renders as
// `Mōŕē ĩńfōŕmàţĩōń`, which contains no 4-letter ASCII run — so it is
// dropped at the `wordlike` gate and never even reaches the `accented`
// test. Soundness is unaffected (a real miss is pure ASCII and is still
// caught), but it means a working pass and a pass whose selector matched
// NOTHING produce identical empty output. Collecting the accented values
// proves the pass actually reached migrated attribute copy on this
// screen, so the caller can tell those two apart.
const localized: string[] = [];
let inspected = 0;
const inspectAttribute = (el: Element, attr: string) => {
// An empty value is never a missed string, and `alt=""` is load-bearing:
// it marks an image as decorative so screen readers skip it. Reporting
// one would push authors into writing alt text for spacers.
const value = (el.getAttribute(attr) ?? "").trim();
if (!value) return;
inspected += 1;
// NOTE the divergence from the text pass, which clears a node the moment
// it contains one accented character. That rule hides a real shape here:
// an `aria-label` built as "Collapse ${label}" renders
// "Collapse Ĝēńēŕàĺ" — an un-migrated English FRAME around a migrated
// value, which accent-clears-all would call done. Allowlist stripping is
// what separates it from the legitimate case, where the ASCII fragment is
// the allowlisted DATA rather than the frame ("Àćţĩōńś ƒōŕ Agent" on the
// layouts screen strips to nothing).
if (!hasUnmigratedAscii(value)) {
// Fully accented, nothing ASCII left: a migrated attribute, and the
// positive control.
if (accented.test(value)) {
localized.push(`${attr} on ${describe(el)}: "${value.slice(0, 60)}"`);
}
return;
}
// Keyed by attribute+value, not by element: one un-migrated shared
// component renders on twenty rows, and twenty identical findings bury
// the other nineteen strings. The first element carrying it is the
// sample you go and look at.
const key = `${attr} ${value}`;
const hit = seen.get(key);
if (hit) {
hit.count += 1;
return;
}
const label = `${attr} on ${describe(el)}: "${value.slice(0, 120)}"`;
seen.set(key, {
label: accented.test(value) ? `${label} - English frame, migrated value` : label,
count: 1,
});
};
const collectAttributes = () => {
const selector = copyAttributes.map((attr) => `[${attr}]`).join(",");
for (const el of Array.from(document.querySelectorAll(selector))) {
if (attrSkipTags.has(el.tagName.toUpperCase()) || !isRendered(el)) continue;
for (const attr of copyAttributes) inspectAttribute(el, attr);
}
const leftovers = [...seen.values()].map(({ label, count }) =>
count > 1 ? `${label} (×${count})` : label,
);
return { leftovers, localized, inspected };
};
const attributes = collectAttributes();
return {
leftovers: [...collectText(), ...attributes.leftovers],
localizedAttributes: attributes.localized,
inspectedAttributes: attributes.inspected,
};
},
{ allowedList: allowed, copyAttributes: COPY_ATTRIBUTES },
);
}
test.describe("i18n pseudo-locale coverage", () => {
for (const screen of SCREENS) {
test(`no un-externalized copy on ${screen.name}`, async ({ testPage }) => {
await activatePseudo(testPage, screen.url);
await waitForScreen(testPage, screen);
await waitForTransientLoadingToFinish(testPage);
const { leftovers, localizedAttributes, inspectedAttributes } = await findUnlocalizedCopy(
testPage,
[...ALLOWED, ...(screen.allow ?? [])],
);
// Control FIRST. A migrated attribute renders accented and so is invisible
// to the leftover check by construction, which means a screen that is
// clean and a selector that matched NOTHING report identically. Assert the
// pass actually read attributes before believing what it did not find.
//
// NOTE what this does and does not establish, because getting that wrong
// is what made this spec passable without looking at anything: it proves
// the pass READ ATTRIBUTES, not that it read THIS SCREEN'S. The sidebar,
// topbar and status bar supply well over a hundred of them on every route,
// so this stays green on a page where the route never mounted. `anchor`
// above is what ties the scan to the screen under test; this remains
// useful only as the narrower check that the selector itself works.
expect(
inspectedAttributes,
`The attribute pass examined no attribute at all on ${screen.name}, so a ` +
`green leftover check proves nothing about attribute copy here.`,
).toBeGreaterThan(0);
expect(
leftovers,
`Un-externalized strings on ${screen.name}:\n${leftovers.map((s) => ` - ${s}`).join("\n")}` +
`\n\nAttribute pass examined ${inspectedAttributes} attribute(s), of which ` +
`${localizedAttributes.length} rendered fully accented.`,
).toEqual([]);
});
}
/**
* The positive control, pinned once rather than per screen.
*
* `inspectedAttributes` above proves the selector matched; it does not prove
* the pass can tell migrated attribute copy apart from a miss. This does, on
* the one element we know is migrated: `language-settings.tsx` renders
* `aria-label={t("settings:displayLanguage")}` on `#language-select`, so under
* pseudo it must come back accented. If this test fails while the screens
* above pass, the accent detection is broken and every green screen is
* meaningless — which is the failure this whole file exists to not have.
*
* A per-screen version of this assertion was tried and dropped: secrets,
* terminal and sprites legitimately render no fully-accented attribute at all,
* so it fired on three screens that were fine.
*/
test("the attribute pass recognizes migrated attribute copy", async ({ testPage }) => {
const appearance = SCREENS.find((screen) => screen.url === "/settings/preferences/appearance");
if (!appearance) throw new Error("the appearance screen is no longer in SCREENS");
await activatePseudo(testPage, appearance.url);
await waitForScreen(testPage, appearance);
await waitForTransientLoadingToFinish(testPage);
const { localizedAttributes } = await findUnlocalizedCopy(testPage, ALLOWED);
expect(
localizedAttributes.join("\n"),
`Expected the migrated aria-label on #language-select to render accented. Got:\n` +
localizedAttributes.join("\n"),
).toContain("language-select");
});
});