kandev/apps/web/e2e/README.md

Kandev Web — E2E Test Suite

Playwright-based end-to-end tests. Each Playwright worker spawns its own real Go backend (no mocks of internal services) on isolated ports; that backend serves the Vite SPA assets and boot data while Playwright drives a real Chromium against it.

Project layout

Folder	What's in it
tests/	Spec files, grouped by feature (chat/, docker/, git/, integrations/, kanban/, pr/, search/, session/, ssh/, layout/, …).
fixtures/	Worker-scoped fixtures that own the backend lifecycle (backend.ts, docker-test-base.ts, ssh-test-base.ts, test-base.ts).
helpers/	Reusable building blocks for specs (api-client.ts, docker.ts, ssh.ts, git-helper.ts, ws-capture.ts, …).
pages/	Page Objects (one class per top-level UI surface — SessionPage, KanbanPage, JiraSettingsPage, SSHSettingsPage, …).
playwright.config.ts	Project definitions, timeouts, sharding config.
global-setup.ts	Pre-flight checks for required binaries (kandev, mock-agent) and the Vite web build.

Playwright projects

The suite is split into five projects. Pick one with --project=<name>.

routing

Runs office-routing-*.spec.ts in an isolated desktop worker. Those specs restart their backend with provider overrides that apply only to the restart/spec that supplies them; the next restart rebuilds the environment from its clean baseline. Keeping these specs separate also keeps their routing-specific provider and agent fixtures away from tests that count agents or assert the active-agent label. Run it directly with:

pnpm e2e:raw --project=routing

auth

Runs tests/auth/** in an isolated desktop worker. These specs restart their backend with authentication enabled, so chromium intentionally excludes them. Select the project explicitly; otherwise Playwright reports No tests found:

pnpm e2e:raw --project=auth tests/auth/auth-lifecycle.spec.ts

chromium (default)

The everyday surface — runs in every CI shard. Excludes the heavyweight containers specs and the mobile specs.

pnpm e2e:raw

mobile-chrome

Same as chromium but on Playwright's Pixel-5 viewport, gated on tests/**/mobile-*.spec.ts. Runs in the same CI shard matrix as chromium.

containers — Docker required

This is the "real-infra heavyweight" project. Despite the name, it covers more than just the Docker executor — it's where any test that needs Docker on the host as a runtime lives:

• Docker executor tests (tests/docker/*.spec.ts) — verify kandev launches real kandev-agent:e2e containers, recovers them across backend restarts, cleans them up on archive/delete, etc.
• SSH executor tests (tests/ssh/*.spec.ts) — verify kandev SSHes into a real kandev-sshd:e2e container, uploads agentctl, runs an agent end-to-end, recovers across backend restarts, etc. The SSH executor's remote target is a Docker container in tests, even though the SSH connection itself is a real SSH connection.

This project:

• Skips entirely when no Docker daemon is reachable. Contributors without Docker can still run chromium + mobile-chrome.
• Builds container images on demand. First run builds kandev-agent:e2e (slim Node base + git) and kandev-sshd:e2e (Alpine + openssh-server + git + pre-baked mock-agent). Subsequent runs hit Docker's layer cache.
• Has a longer per-test timeout (180s vs 60s) because container starts + agent setup are slow.

How to run it locally (requires Docker running):

KANDEV_E2E_CONTAINERS=1 pnpm e2e:raw --project=containers

Or a single spec:

KANDEV_E2E_CONTAINERS=1 pnpm e2e:raw --project=containers tests/ssh/launch-task.spec.ts

Remote-executor fixture contracts

Host-only file:// fixtures are not reachable from an SSH or Docker target. Use a disposable provider-shaped HTTP Git fixture with a target-side URL rewrite; if the spec also uses host-local GitHelper or LSP paths, materialize a local clone at the expected temporary path. Verify both the remote checkout and every host-local fixture consumer.

Why "containers" instead of "docker"?

This project used to be named docker. It was renamed to containers once SSH e2e tests joined it — calling it docker was misleading because SSH tests have nothing to do with the Docker executor; they just happen to use Docker as the runtime that hosts the sshd target.

KANDEV_E2E_DOCKER=1 is still honored as a deprecated alias for KANDEV_E2E_CONTAINERS=1 for one release. Local scripts and stale CI configs won't break, but new code should use the new name.

Commands

e2e:raw, e2e:run, and e2e:ui are defined only in apps/web/package.json. Run them from apps/web (or pnpm --filter @kandev/web e2e:run from elsewhere). From the repo root you get ERR_PNPM_NO_IMPORTER_MANIFEST_FOUND.

Command	What it does
pnpm e2e:raw	Run the default (chromium) project headless.
pnpm e2e:ui	Open Playwright's UI mode for interactive runs.
pnpm e2e:headed	Run headless project but with a visible browser.
pnpm e2e:raw --project=containers	Run container-backed tests (needs Docker).
pnpm e2e:raw --project=mobile-chrome	Run mobile responsive tests.
pnpm e2e:raw --project=routing	Run provider-mutating Office routing tests.
pnpm e2e:raw --project=auth	Run auth-isolated tests.
E2E_DEBUG=1 pnpm e2e:raw	Surface Docker build output + extra logging.

Common flags: --shard=1/4, -g "fragment of test name", --repeat-each=3 (flake hunting).

Duration-aware CI sharding

CI creates an ephemeral manifest for each cohort from the current test catalog. The normal cohort has 14 shards and runs chromium plus mobile-chrome; the container cohort has 6 shards and runs containers. The manifest assigns project/file units with a deterministic longest-processing-time planner. Matrix jobs pass the assigned files to run-planned-shard.sh; they do not use ordinal Playwright --shard selection.

Successful main runs publish e2e-timing-profile, which stores bounded first-attempt passing samples keyed by project, file, and full test title. The next planning job downloads that artifact when available. New files use the count fallback, and changed files are marked warm and receive the conservative multiplier. If the artifact is missing, the planner records profile.mode=count-fallback and still validates complete catalog coverage.

The runner validates the manifest against the checkout before launching Playwright. A missing, stale, overlapping, or incomplete manifest is a hard failure; it cannot silently fall back to ordinal sharding.

To dry-run the planner locally:

cd apps
pnpm exec tsx web/e2e/scripts/plan-shards.ts \
  --web-root "$PWD/web" \
  --output-dir "$PWD/web/e2e/manifests"

To execute one generated shard locally, set E2E_SHARD and pass its manifest:

cd apps/web
E2E_SHARD=1 bash e2e/scripts/run-planned-shard.sh e2e/manifests/normal/1.json

The report job publishes e2e-retry-summary and e2e-timing-diagnostics. The summary includes first-attempt passes, passed-after-retry tests, final failures, timeouts, skipped tests, predicted/actual shard duration deltas, and unknown/warm/stale/fallback planning counters. The profile candidate is uploaded on every report, but only a successful main workflow run is eligible to seed a future plan. Manifests are retained for 3 days; timing profiles for 30 days; retry diagnostics for 14 days.

Dispatch the workflow with fail_on_flaky=true to set failOnFlakyTests: true for a diagnostic run. Normal PR runs retain the existing two-retry policy while the summary makes retry groups visible.

pnpm e2e:run — the managed runner (build + run + teardown)

e2e/scripts/run-e2e.sh (aliased as pnpm e2e:run) handles the build, the run, and cleanup so you don't have to assemble the steps by hand. It auto-selects docker vs host, runs N shards concurrently, enforces strict WS accounting by default (KANDEV_E2E_WS_ASSERT=1, matching CI), and never leaves root-owned artifacts behind.

pnpm e2e:run                                  # auto: docker if the daemon + CI image are available, else host
pnpm e2e:run --shards 3                        # 3 shards concurrently (isolated containers, or host procs with distinct ports)
pnpm e2e:run tests/chat/foo.spec.ts            # extra args pass straight through to Playwright
pnpm e2e:run --project auth tests/auth/auth-lifecycle.spec.ts
pnpm e2e:run --host --no-build -- --grep "x"   # force host, skip rebuild, forward flags after --
pnpm e2e:docker                                # force the docker CI image (full isolation from a host dev instance)
pnpm e2e:clean                                 # remove build/test artifacts, incl. root-owned ones from prior docker runs

Why a script instead of raw docker run: in docker mode it builds the CGO/fts5 backend on the host and runs it in the runtime image — a host glibc that's the same or older than the image's (the usual case; the image tracks recent Ubuntu) is forward-compatible, so no build image is needed. The runner smoke-tests the binary in the image first and only falls back to the build image (KANDEV_CI_BUILD_IMAGE, default …/kandev-ci:build-latest) if your host glibc is newer. It also builds the Vite web assets on the host, points Playwright output at a container-local dir, and cleans up. Run clean if a previous bare docker run left root-owned files you can't delete.

Apple Silicon: the docker path runs the amd64 CI image. Under Docker Desktop's default QEMU emulation the amd64 Go toolchain segfaults (SIGSEGV in modindex.dirHash) during backend build. Use Colima with Rosetta instead: colima start --vm-type=vz --vz-rosetta. QEMU is not viable for the Go build; Rosetta is required for local amd64 E2E repro on arm64.

Office is always enabled in e2e (and dev); only prod gates it off. profiles.yaml sets KANDEV_FEATURES_OFFICE to "true" in the e2e and dev profiles and "false" in prod, and the fixture selects the e2e profile via KANDEV_E2E_MOCK=true. So tests/office/* always have office routes registered — no manual env setup.

This used to break when e2e was launched from a shell that had inherited KANDEV_FEATURES_OFFICE=false (e.g. from a host kandev backend running the prod profile): profiles.ApplyProfile only sets vars that are unset (so launchers/shells win — see docs/decisions/0007-runtime-feature-flags.md), and the fixture spreads process.env into the spawned backend, so the stale prod value won and 404'd every office spec. Fixed at the source: sanitizeInheritedEnv in e2e/fixtures/backend.ts strips all inherited KANDEV_FEATURES_* before spawn, so the e2e profile — not whatever the host exported — decides feature flags. No unset needed.

Profile-managed environment variables: when adding an environment variable with an e2e: or dev: profile default, add it to sanitizeInheritedEnv in e2e/fixtures/backend.ts so inherited shell/task values cannot override the selected profile. Keep explicit backend.restart({ ... }) overrides applied after baseline sanitization, so a spec can still opt into a deliberate per-test value.

Host oversubscription: running >=5 heavy shards concurrently on one machine (each = Go backend + Vite-served SPA assets + Chromium + mock agent) starves CPU/IO and induces timing flakes that CI's isolated runners never see. Use 2-3 concurrent shards locally for a clean signal; see "flake triage" in the /e2e skill.

Backend isolation per worker

Every Playwright worker gets:

• A unique backend port in BACKEND_BASE_PORT + workerIndex (default 18080+).
• A unique frontend port in FRONTEND_BASE_PORT + workerIndex (default 13000+).
• A fresh tmpdir (HOME, KANDEV_HOME_DIR, worktree base, repo clone base — all under that tmpdir).
• A unique agentctl instance port range (30001 + E2E_PORT_OFFSET * 1000 + workerIndex * 200).
• Its own SQLite DB.
• A process-scoped Docker ownership label for backend-created E2E containers and test-created SSH/storage fixtures. Cleanup and storage reporting filter by that label and never sweep another shard's kandev.managed=true containers.

Workers run in parallel across CI manifest shards; within a worker, tests run serially because the testPage fixture calls e2eReset on the shared backend before each test.

Docker image usage remains daemon-wide because images and their layers are shared across shards; only container records, cleanup, and container-based storage reporting are filtered by the process-scoped ownership label.

Mocked vs real

• Mocked: Azure DevOps (KANDEV_MOCK_AZURE_DEVOPS=true), Jira (KANDEV_MOCK_JIRA=true), Linear (KANDEV_MOCK_LINEAR=true), GitHub (KANDEV_MOCK_GITHUB=true), and the agent process itself (KANDEV_MOCK_AGENT=only). These are third-party services or external processes we don't want CI to depend on.
• Real: Everything inside the kandev backend — orchestrator, lifecycle manager, agentctl, SSH/SFTP, Docker SDK, git, worktree manager. The point of e2e is to exercise the real code paths.

The SSH executor specifically has no mock controller. Tests use a real Docker-hosted sshd as the remote target, and fault-injection (host-key rotation, dropped traffic, killed pids) is done by operating on the container itself.

Adding a new spec

1. Pick a directory under tests/ (or create one for a new feature).
2. Decide which project it belongs to. Anything that needs Docker → tests/docker/ or tests/ssh/ (lands in containers). Auth-isolated specs belong in tests/auth/ and need --project=auth; provider-mutating Office routing specs use routing. Anything mobile-specific → name it mobile-*.spec.ts. Otherwise it joins chromium automatically.
3. Import the right test base:
   • import { test, expect } from "../../fixtures/test-base"; for normal tests.
   • import { test, expect } from "../../fixtures/docker-test-base"; for Docker executor tests.
   • import { test, expect } from "../../fixtures/ssh-test-base"; for SSH executor tests.
4. Use getByTestId for selectors. If the surface you're testing lacks stable testids, add them — drift-prone CSS / text selectors are not worth the maintenance cost.

CI

.github/workflows/e2e-tests.yml defines two test cohorts and a report job:

• e2e — a 14-entry matrix executing the generated normal manifests.
• e2e-containers — a 6-entry matrix executing the generated container manifests and requiring Docker.
• e2e-report — merges blob reports and publishes timing/retry artifacts.

The build job uploads e2e-shard-manifests for the current run. Both cohorts upload blob reports that e2e-report merges into a single HTML artifact.

Keep the default at workers: 1 until the controlled worker-concurrency experiment shows a repeatable wall-time improvement without retries. Record worker count, shard wall time, CPU/memory pressure, setup time, and retry count for comparisons. Measure package install, runtime image startup, and browser extraction separately from test-work balance before changing the CI matrix.

For a local worker experiment, run the same selected heavy files twice and save the command output. The second command is diagnostic only; it does not change the checked-in default:

cd apps/web
E2E_SHARD=2 /usr/bin/time -v bash e2e/scripts/run-planned-shard.sh \
  e2e/manifests/normal/2.json -- --workers=1
E2E_SHARD=2 /usr/bin/time -v bash e2e/scripts/run-planned-shard.sh \
  e2e/manifests/normal/2.json -- --workers=2

Record results as { "workers": 2, "wall_seconds": 0, "max_rss_kb": 0, "retries": 0, "backend_errors": 0 }. Compare at least three repetitions with the same build, duration-aware manifest, and profile before considering a default change.