Carlos Florêncio
254 lines
16 KiB
Markdown
254 lines
16 KiB
Markdown
# Kandev Web — E2E Test Suite
|
|
|
|
Playwright-based end-to-end tests. Each Playwright worker spawns its own real Go backend (no mocks of internal services) on isolated ports; that backend serves the Vite SPA assets and boot data while Playwright drives a real Chromium against it.
|
|
|
|
## Project layout
|
|
|
|
| Folder | What's in it |
|
|
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `tests/` | Spec files, grouped by feature (`chat/`, `docker/`, `git/`, `integrations/`, `kanban/`, `pr/`, `search/`, `session/`, `ssh/`, `layout/`, …). |
|
|
| `fixtures/` | Worker-scoped fixtures that own the backend lifecycle (`backend.ts`, `docker-test-base.ts`, `ssh-test-base.ts`, `test-base.ts`). |
|
|
| `helpers/` | Reusable building blocks for specs (`api-client.ts`, `docker.ts`, `ssh.ts`, `git-helper.ts`, `ws-capture.ts`, …). |
|
|
| `pages/` | Page Objects (one class per top-level UI surface — `SessionPage`, `KanbanPage`, `JiraSettingsPage`, `SSHSettingsPage`, …). |
|
|
| `playwright.config.ts` | Project definitions, timeouts, sharding config. |
|
|
| `global-setup.ts` | Pre-flight checks for required binaries (kandev, mock-agent) and the Vite web build. |
|
|
|
|
## Playwright projects
|
|
|
|
The suite is split into five projects. Pick one with `--project=<name>`.
|
|
|
|
### `routing`
|
|
|
|
Runs `office-routing-*.spec.ts` in an isolated desktop worker. Those specs restart
|
|
their backend with provider overrides that apply only to the restart/spec that
|
|
supplies them; the next restart rebuilds the environment from its clean baseline.
|
|
Keeping these specs separate also keeps their routing-specific provider and agent
|
|
fixtures away from tests that count agents or assert the active-agent label. Run
|
|
it directly with:
|
|
|
|
```sh
|
|
pnpm e2e:raw --project=routing
|
|
```
|
|
|
|
### `auth`
|
|
|
|
Runs `tests/auth/**` in an isolated desktop worker. These specs restart their
|
|
backend with authentication enabled, so `chromium` intentionally excludes them.
|
|
Select the project explicitly; otherwise Playwright reports `No tests found`:
|
|
|
|
```sh
|
|
pnpm e2e:raw --project=auth tests/auth/auth-lifecycle.spec.ts
|
|
```
|
|
|
|
### `chromium` (default)
|
|
|
|
The everyday surface — runs in every CI shard. Excludes the heavyweight `containers` specs and the mobile specs.
|
|
|
|
```sh
|
|
pnpm e2e:raw
|
|
```
|
|
|
|
### `mobile-chrome`
|
|
|
|
Same as `chromium` but on Playwright's Pixel-5 viewport, gated on `tests/**/mobile-*.spec.ts`. Runs in the same CI shard matrix as `chromium`.
|
|
|
|
### `containers` — **Docker required**
|
|
|
|
**This is the "real-infra heavyweight" project.** Despite the name, it covers **more than just the Docker executor** — it's where any test that needs Docker on the host as a runtime lives:
|
|
|
|
- **Docker executor tests** (`tests/docker/*.spec.ts`) — verify kandev launches real `kandev-agent:e2e` containers, recovers them across backend restarts, cleans them up on archive/delete, etc.
|
|
- **SSH executor tests** (`tests/ssh/*.spec.ts`) — verify kandev SSHes into a real `kandev-sshd:e2e` container, uploads agentctl, runs an agent end-to-end, recovers across backend restarts, etc. The SSH executor's _remote target_ is a Docker container in tests, even though the SSH connection itself is a real SSH connection.
|
|
|
|
This project:
|
|
|
|
- **Skips entirely** when no Docker daemon is reachable. Contributors without Docker can still run `chromium` + `mobile-chrome`.
|
|
- **Builds container images on demand.** First run builds `kandev-agent:e2e` (slim Node base + git) and `kandev-sshd:e2e` (Alpine + openssh-server + git + pre-baked mock-agent). Subsequent runs hit Docker's layer cache.
|
|
- **Has a longer per-test timeout** (180s vs 60s) because container starts + agent setup are slow.
|
|
|
|
How to run it locally (requires Docker running):
|
|
|
|
```sh
|
|
KANDEV_E2E_CONTAINERS=1 pnpm e2e:raw --project=containers
|
|
```
|
|
|
|
Or a single spec:
|
|
|
|
```sh
|
|
KANDEV_E2E_CONTAINERS=1 pnpm e2e:raw --project=containers tests/ssh/launch-task.spec.ts
|
|
```
|
|
|
|
### Remote-executor fixture contracts
|
|
|
|
Host-only `file://` fixtures are not reachable from an SSH or Docker target. Use a disposable provider-shaped HTTP Git fixture with a target-side URL rewrite; if the spec also uses host-local `GitHelper` or LSP paths, materialize a local clone at the expected temporary path. Verify both the remote checkout and every host-local fixture consumer.
|
|
|
|
### Why "containers" instead of "docker"?
|
|
|
|
This project used to be named `docker`. It was renamed to `containers` once SSH e2e tests joined it — calling it `docker` was misleading because SSH tests have nothing to do with the Docker _executor_; they just happen to use Docker as the runtime that hosts the sshd target.
|
|
|
|
`KANDEV_E2E_DOCKER=1` is still honored as a deprecated alias for `KANDEV_E2E_CONTAINERS=1` for one release. Local scripts and stale CI configs won't break, but new code should use the new name.
|
|
|
|
## Commands
|
|
|
|
`e2e:raw`, `e2e:run`, and `e2e:ui` are defined only in `apps/web/package.json`. Run them from `apps/web` (or `pnpm --filter @kandev/web e2e:run` from elsewhere). From the repo root you get `ERR_PNPM_NO_IMPORTER_MANIFEST_FOUND`.
|
|
|
|
| Command | What it does |
|
|
| -------------------------------------- | ------------------------------------------------ |
|
|
| `pnpm e2e:raw` | Run the default (chromium) project headless. |
|
|
| `pnpm e2e:ui` | Open Playwright's UI mode for interactive runs. |
|
|
| `pnpm e2e:headed` | Run headless project but with a visible browser. |
|
|
| `pnpm e2e:raw --project=containers` | Run container-backed tests (needs Docker). |
|
|
| `pnpm e2e:raw --project=mobile-chrome` | Run mobile responsive tests. |
|
|
| `pnpm e2e:raw --project=routing` | Run provider-mutating Office routing tests. |
|
|
| `pnpm e2e:raw --project=auth` | Run auth-isolated tests. |
|
|
| `E2E_DEBUG=1 pnpm e2e:raw` | Surface Docker build output + extra logging. |
|
|
|
|
Common flags: `--shard=1/4`, `-g "fragment of test name"`, `--repeat-each=3` (flake hunting).
|
|
|
|
### Duration-aware CI sharding
|
|
|
|
CI creates an ephemeral manifest for each cohort from the current test catalog.
|
|
The normal cohort has 14 shards and runs `chromium` plus `mobile-chrome`; the
|
|
container cohort has 6 shards and runs `containers`. The manifest assigns
|
|
project/file units with a deterministic longest-processing-time planner. Matrix
|
|
jobs pass the assigned files to `run-planned-shard.sh`; they do not use ordinal
|
|
Playwright `--shard` selection.
|
|
|
|
Successful `main` runs publish `e2e-timing-profile`, which stores bounded
|
|
first-attempt passing samples keyed by project, file, and full test title. The
|
|
next planning job downloads that artifact when available. New files use the
|
|
count fallback, and changed files are marked warm and receive the conservative
|
|
multiplier. If the artifact is missing, the planner records
|
|
`profile.mode=count-fallback` and still validates complete catalog coverage.
|
|
|
|
The runner validates the manifest against the checkout before launching
|
|
Playwright. A missing, stale, overlapping, or incomplete manifest is a hard
|
|
failure; it cannot silently fall back to ordinal sharding.
|
|
|
|
To dry-run the planner locally:
|
|
|
|
```bash
|
|
cd apps
|
|
pnpm exec tsx web/e2e/scripts/plan-shards.ts \
|
|
--web-root "$PWD/web" \
|
|
--output-dir "$PWD/web/e2e/manifests"
|
|
```
|
|
|
|
To execute one generated shard locally, set `E2E_SHARD` and pass its manifest:
|
|
|
|
```bash
|
|
cd apps/web
|
|
E2E_SHARD=1 bash e2e/scripts/run-planned-shard.sh e2e/manifests/normal/1.json
|
|
```
|
|
|
|
The report job publishes `e2e-retry-summary` and `e2e-timing-diagnostics`.
|
|
The summary includes first-attempt passes, passed-after-retry tests, final
|
|
failures, timeouts, skipped tests, predicted/actual shard duration deltas, and
|
|
unknown/warm/stale/fallback planning counters. The profile candidate is uploaded on
|
|
every report, but only a successful `main` workflow run is eligible to seed a
|
|
future plan. Manifests are retained for 3 days; timing profiles for 30 days;
|
|
retry diagnostics for 14 days.
|
|
|
|
Dispatch the workflow with `fail_on_flaky=true` to set
|
|
`failOnFlakyTests: true` for a diagnostic run. Normal PR runs retain the
|
|
existing two-retry policy while the summary makes retry groups visible.
|
|
|
|
### `pnpm e2e:run` — the managed runner (build + run + teardown)
|
|
|
|
`e2e/scripts/run-e2e.sh` (aliased as `pnpm e2e:run`) handles the build, the run, and cleanup so you don't have to assemble the steps by hand. It **auto-selects docker vs host**, runs **N shards concurrently**, enforces strict WS accounting by default (`KANDEV_E2E_WS_ASSERT=1`, matching CI), and never leaves root-owned artifacts behind.
|
|
|
|
```bash
|
|
pnpm e2e:run # auto: docker if the daemon + CI image are available, else host
|
|
pnpm e2e:run --shards 3 # 3 shards concurrently (isolated containers, or host procs with distinct ports)
|
|
pnpm e2e:run tests/chat/foo.spec.ts # extra args pass straight through to Playwright
|
|
pnpm e2e:run --project auth tests/auth/auth-lifecycle.spec.ts
|
|
pnpm e2e:run --host --no-build -- --grep "x" # force host, skip rebuild, forward flags after --
|
|
pnpm e2e:docker # force the docker CI image (full isolation from a host dev instance)
|
|
pnpm e2e:clean # remove build/test artifacts, incl. root-owned ones from prior docker runs
|
|
```
|
|
|
|
Why a script instead of raw `docker run`: in docker mode it builds the CGO/`fts5` backend on the **host** and runs it in the runtime image — a host glibc that's the same or older than the image's (the usual case; the image tracks recent Ubuntu) is forward-compatible, so no build image is needed. The runner smoke-tests the binary in the image first and only falls back to the build image (`KANDEV_CI_BUILD_IMAGE`, default `…/kandev-ci:build-latest`) if your host glibc is _newer_. It also builds the Vite web assets on the host, points Playwright output at a container-local dir, and cleans up. Run `clean` if a previous bare `docker run` left root-owned files you can't delete.
|
|
|
|
> **Apple Silicon:** the docker path runs the amd64 CI image. Under Docker Desktop's default QEMU emulation the amd64 Go toolchain segfaults (`SIGSEGV` in `modindex.dirHash`) during backend build. Use Colima with Rosetta instead: `colima start --vm-type=vz --vz-rosetta`. QEMU is not viable for the Go build; Rosetta is required for local amd64 E2E repro on arm64.
|
|
|
|
> **Office is always enabled in e2e (and dev); only prod gates it off.** `profiles.yaml` sets `KANDEV_FEATURES_OFFICE` to `"true"` in the `e2e` and `dev` profiles and `"false"` in `prod`, and the fixture selects the e2e profile via `KANDEV_E2E_MOCK=true`. So `tests/office/*` always have office routes registered — no manual env setup.
|
|
>
|
|
> This used to break when e2e was launched from a shell that had inherited `KANDEV_FEATURES_OFFICE=false` (e.g. from a host kandev backend running the prod profile): `profiles.ApplyProfile` only sets vars that are **unset** (so launchers/shells win — see `docs/decisions/0007-runtime-feature-flags.md`), and the fixture spreads `process.env` into the spawned backend, so the stale prod value won and 404'd every office spec. Fixed at the source: `sanitizeInheritedEnv` in `e2e/fixtures/backend.ts` strips all inherited `KANDEV_FEATURES_*` before spawn, so the e2e profile — not whatever the host exported — decides feature flags. No `unset` needed.
|
|
|
|
> **Profile-managed environment variables:** when adding an environment variable with an `e2e:` or `dev:` profile default, add it to `sanitizeInheritedEnv` in `e2e/fixtures/backend.ts` so inherited shell/task values cannot override the selected profile. Keep explicit `backend.restart({ ... })` overrides applied after baseline sanitization, so a spec can still opt into a deliberate per-test value.
|
|
|
|
> **Host oversubscription:** running >=5 heavy shards concurrently on one machine (each = Go backend + Vite-served SPA assets + Chromium + mock agent) starves CPU/IO and induces timing flakes that CI's isolated runners never see. Use 2-3 concurrent shards locally for a clean signal; see "flake triage" in the `/e2e` skill.
|
|
|
|
## Backend isolation per worker
|
|
|
|
Every Playwright worker gets:
|
|
|
|
- A unique backend port in `BACKEND_BASE_PORT + workerIndex` (default `18080+`).
|
|
- A unique frontend port in `FRONTEND_BASE_PORT + workerIndex` (default `13000+`).
|
|
- A fresh tmpdir (`HOME`, `KANDEV_HOME_DIR`, worktree base, repo clone base — all under that tmpdir).
|
|
- A unique agentctl instance port range (`30001 + E2E_PORT_OFFSET * 1000 + workerIndex * 200`).
|
|
- Its own SQLite DB.
|
|
- A process-scoped Docker ownership label for backend-created E2E containers
|
|
and test-created SSH/storage fixtures. Cleanup and storage reporting filter
|
|
by that label and never sweep another shard's `kandev.managed=true`
|
|
containers.
|
|
|
|
Workers run in parallel across CI manifest shards; within a worker, tests run
|
|
serially because the `testPage` fixture calls `e2eReset` on the shared backend
|
|
before each test.
|
|
|
|
Docker image usage remains daemon-wide because images and their layers are
|
|
shared across shards; only container records, cleanup, and container-based
|
|
storage reporting are filtered by the process-scoped ownership label.
|
|
|
|
## Mocked vs real
|
|
|
|
- **Mocked**: Azure DevOps (`KANDEV_MOCK_AZURE_DEVOPS=true`), Jira (`KANDEV_MOCK_JIRA=true`), Linear (`KANDEV_MOCK_LINEAR=true`), GitHub (`KANDEV_MOCK_GITHUB=true`), and the agent process itself (`KANDEV_MOCK_AGENT=only`). These are third-party services or external processes we don't want CI to depend on.
|
|
- **Real**: Everything inside the kandev backend — orchestrator, lifecycle manager, agentctl, SSH/SFTP, Docker SDK, git, worktree manager. The point of e2e is to exercise the real code paths.
|
|
|
|
The SSH executor specifically has no mock controller. Tests use a real Docker-hosted sshd as the remote target, and fault-injection (host-key rotation, dropped traffic, killed pids) is done by operating on the container itself.
|
|
|
|
## Adding a new spec
|
|
|
|
1. Pick a directory under `tests/` (or create one for a new feature).
|
|
2. Decide which project it belongs to. Anything that needs Docker → `tests/docker/` or `tests/ssh/` (lands in `containers`). Auth-isolated specs belong in `tests/auth/` and need `--project=auth`; provider-mutating Office routing specs use `routing`. Anything mobile-specific → name it `mobile-*.spec.ts`. Otherwise it joins `chromium` automatically.
|
|
3. Import the right test base:
|
|
- `import { test, expect } from "../../fixtures/test-base";` for normal tests.
|
|
- `import { test, expect } from "../../fixtures/docker-test-base";` for Docker executor tests.
|
|
- `import { test, expect } from "../../fixtures/ssh-test-base";` for SSH executor tests.
|
|
4. Use `getByTestId` for selectors. If the surface you're testing lacks stable testids, add them — drift-prone CSS / text selectors are not worth the maintenance cost.
|
|
|
|
## CI
|
|
|
|
`.github/workflows/e2e-tests.yml` defines two test cohorts and a report job:
|
|
|
|
- `e2e` — a 14-entry matrix executing the generated normal manifests.
|
|
- `e2e-containers` — a 6-entry matrix executing the generated container
|
|
manifests and requiring Docker.
|
|
- `e2e-report` — merges blob reports and publishes timing/retry artifacts.
|
|
|
|
The build job uploads `e2e-shard-manifests` for the current run. Both cohorts
|
|
upload blob reports that `e2e-report` merges into a single HTML artifact.
|
|
|
|
Keep the default at `workers: 1` until the controlled worker-concurrency
|
|
experiment shows a repeatable wall-time improvement without retries. Record
|
|
worker count, shard wall time, CPU/memory pressure, setup time, and retry count
|
|
for comparisons. Measure package install, runtime image startup, and browser
|
|
extraction separately from test-work balance before changing the CI matrix.
|
|
|
|
For a local worker experiment, run the same selected heavy files twice and
|
|
save the command output. The second command is diagnostic only; it does not
|
|
change the checked-in default:
|
|
|
|
```bash
|
|
cd apps/web
|
|
E2E_SHARD=2 /usr/bin/time -v bash e2e/scripts/run-planned-shard.sh \
|
|
e2e/manifests/normal/2.json -- --workers=1
|
|
E2E_SHARD=2 /usr/bin/time -v bash e2e/scripts/run-planned-shard.sh \
|
|
e2e/manifests/normal/2.json -- --workers=2
|
|
```
|
|
|
|
Record results as `{ "workers": 2, "wall_seconds": 0, "max_rss_kb": 0,
|
|
"retries": 0, "backend_errors": 0 }`. Compare at least three repetitions
|
|
with the same build, duration-aware manifest, and profile before considering a
|
|
default change.
|