dreampiler/kandev
1
0
Fork 0
Code Issues 43 Pull Requests Actions 2 Packages Projects Releases Wiki Activity
Files
607201be79aa6fd5b490f4051815bedca8fc6b19
kandev/apps/web/e2e
T
History
João Salavisa 48a621a566 fix(utility): surface inference-agent probe status + add refresh endpoint (#1287) 
* fix(utility): surface probe status in inference-agents endpoint

GET /api/v1/utility/inference-agents previously filtered out agents whose
host-utility probe didn't reach StatusOK, so a Claude entry with auth or
install issues vanished from the response. The frontend then defaulted
agent_id to "claude-acp", failed to match anything in the list, and
silently rendered an empty Model picker with no explanation.

Include every registered ACP-capable agent in the response with a new
`status` field (and sanitized `status_message`) so the UI can render an
inline note + Refresh button. Add a companion
POST /api/v1/utility/inference-agents/:id/refresh endpoint backed by the
existing hostutility.Manager.Refresh so users can re-probe after fixing
auth or installing the CLI without restarting kandev.

Credential-looking substrings ("api_key=...", "token=...") in probe
errors are stripped before reaching the wire.

Refs #1269, #1263.

* fix(web): show probe status note + Refresh on utility-agent model picker

Previously the Model picker rendered empty and disabled whenever the
selected inference agent was missing from /inference-agents (probe
hadn't reached StatusOK) or advertised no models, with no way for the
user to understand or recover.

Wire the new `status` field through the API client and render an inline
note ("Sign in to Claude to load models.", "Claude CLI is not installed
on this machine.", "Setting up Claude…") with a Refresh button that
calls the new POST /inference-agents/:id/refresh endpoint and merges the
updated agent back into local state — no full refetch, so a slow probe
on a different agent doesn't block the UI.

Both call sites — the create/edit dialog and the Default Model row in
the settings page — render the note. Component-level unit tests cover
the status-to-copy mapping, fallback ("no longer available") copy, and
the in-flight Refresh button disabled state. The e2e test asserts the
auth_required → click Refresh → status:ok state machine end to end.

Refs #1269.

* test(utility): add HTTP integration tests for inference-agents endpoints

End-to-end coverage of the new /api/v1/utility/inference-agents and
/api/v1/utility/inference-agents/:id/refresh routes against a real gin
router with a stateful in-memory host stub.

- ProbeStateMachine: GET (auth_required) → POST refresh → GET (ok+models)
- RefreshUnknownAgent: 404 without invoking the host probe
- RefreshPathFuzz: traversal, null byte, 512-char id — no 5xx, no probe
- ConcurrentRefresh: 50 parallel POSTs under -race, no panic
- StatusMessageSanitizationOnTheWire: 110KB payload + unicode probe
  errors — no secret material reaches the wire, unicode prose preserved

Caught the gap unit tests had: the multi-request GET→POST→GET sequence
on a shared cache, and the wire format (Content-Type, JSON shape)
through the real router.

* fix(utility): tighten credential regex + drop Go 1.22 loop-var shadow

Address two inline review comments from claude bot on PR #1287:

- Drop `ia := ia` shadow in httpRefreshInferenceAgent — pre-Go 1.22
  pointer-capture workaround, dead weight under Go 1.24+ (per
  apps/backend/AGENTS.md).
- Split the credential sanitizer into two patterns so prose without a
  real `=`/`:` separator is no longer mangled. The single regex
  previously turned "access token was revoked" into
  "access token=<redacted> revoked" by consuming the next word as the
  value. New shape:
    - credentialKVPattern — requires `[:=]` (api_key, token, secret,
      password)
    - credentialBearerPattern — allows bare whitespace, anchored to
      "bearer" only
  Existing redaction cases still pass; two new prose cases guard the
  regression.

* fix(utility): address cubic + greptile review findings on PR #1287

Round-2 review feedback:

- P0 (cubic): postEmpty called `t.Fatalf` from `go func()` in
  TestIntegration_ConcurrentRefresh. Go panics on t.Fatalf from non-test
  goroutines. Split into `postEmptyE` (error-returning, goroutine-safe)
  and `postEmpty` (main-goroutine wrapper). Concurrent test now
  forwards the error through the existing `errs` channel.
- P1 (cubic + greptile): tightened the credential sanitizer regex.
    * `api[ _-]?key` now also matches a literal space ("invalid api key=…")
    * separators are restricted to `[^\S\n]` so multi-line stderr like
      "invalid token\ncaused by network" no longer eats words across lines
  Test coverage added for both regressions (and for tab as a valid
  horizontal-whitespace separator).
- P2 (cubic): refresh-failure log now feeds `err.Error()` through
  sanitizeStatusMessage before zapping it — same wire-side guard, but
  applied to server logs too. Closes the "credentials in journalctl"
  hole.
- P2 (cubic): InferenceAgent.status is now optional in the TS type and
  the status note treats an undefined status with `models.length > 0`
  as healthy, so older API responses don't flash a spurious warning.

Deferred (acknowledged in PR replies, not changed here):

- cubic P2 useInferenceAgents drift — the hook is currently called in
  both the dialog and the parent section. Drift is bounded by the
  dialog's own `onSuccess` refetch and only visible inside an open
  dialog. Lifting state to the parent and prop-drilling is the right
  fix but touches multiple files; out of scope for this review round.

* docs(utility): fix stale comments flagged by claude review on PR #1287

Two doc-only items, no code change:

- handlers_integration_test.go: TestIntegration_ConcurrentRefresh
  comment said "we don't assert on call count" but the test DOES
  assert `host.refreshCalls != N` two lines later. Updated to match
  reality (singleflight lives in the real Manager, the stub has none
  → every call must reach the stub).
- utility-agents.spec.ts: the ACP-probe E2E said "backend filters out
  non-StatusOK agents", which is the exact behavior this PR removes.
  Updated to explain the real reason only Mock appears (KANDEV_MOCK_AGENT=only
  registers only `mock-agent`).

---------

Co-authored-by: Kandev Agent <agent@kandev.dev>
2026-06-07 12:38:28 +01:00
..
fixtures
fix(web): send confirm_name when deleting a workspace from settings (#1154)
2026-05-30 14:08:43 +01:00
helpers
feat(web): unified app sidebar (#1165)
2026-06-06 21:58:39 +01:00
pages
feat(web): unified app sidebar (#1165)
2026-06-06 21:58:39 +01:00
scripts
test(e2e): add managed runner script + skill/flake-triage docs (#1170)
2026-05-30 19:16:15 +01:00
tests
fix(utility): surface inference-agent probe status + add refresh endpoint (#1287)
2026-06-07 12:38:28 +01:00
global-setup.ts
feat: web e2e tests (#304)
2026-03-02 08:54:58 +00:00
playwright.config.ts
ci: faster e2e via image split, shard rebalance, and flake fixes (#1126)
2026-05-28 19:16:30 +01:00
README.md
docs: improve agent harness guidance for tests, PR fixup, and debugging (#1235)
2026-06-02 21:23:21 +01:00

README.md

Kandev Web — E2E Test Suite

Playwright-based end-to-end tests. Each Playwright worker spawns its own real Go backend (no mocks of internal services) on isolated ports and a real Next.js standalone frontend, then drives a real Chromium against them.

Project layout

Folder What's in it
tests/ Spec files, grouped by feature (chat/, docker/, git/, integrations/, kanban/, pr/, search/, session/, ssh/, layout/, …).
fixtures/ Worker-scoped fixtures that own the backend lifecycle (backend.ts, docker-test-base.ts, ssh-test-base.ts, test-base.ts).
helpers/ Reusable building blocks for specs (api-client.ts, docker.ts, ssh.ts, git-helper.ts, ws-capture.ts, …).
pages/ Page Objects (one class per top-level UI surface — SessionPage, KanbanPage, JiraSettingsPage, SSHSettingsPage, …).
playwright.config.ts Project definitions, timeouts, sharding config.
global-setup.ts Pre-flight checks for required binaries (kandev, mock-agent, standalone next build).

Playwright projects

The suite is split into three projects. Pick one with --project=<name>.

chromium (default)

The everyday surface — runs in every CI shard. Excludes the heavyweight containers specs and the mobile specs.

pnpm e2e

mobile-chrome

Same as chromium but on Playwright's Pixel-5 viewport, gated on tests/**/mobile-*.spec.ts. Runs in the same CI shard matrix as chromium.

containers — Docker required

This is the "real-infra heavyweight" project. Despite the name, it covers more than just the Docker executor — it's where any test that needs Docker on the host as a runtime lives:

  • Docker executor tests (tests/docker/*.spec.ts) — verify kandev launches real kandev-agent:e2e containers, recovers them across backend restarts, cleans them up on archive/delete, etc.
  • SSH executor tests (tests/ssh/*.spec.ts) — verify kandev SSHes into a real kandev-sshd:e2e container, uploads agentctl, runs an agent end-to-end, recovers across backend restarts, etc. The SSH executor's remote target is a Docker container in tests, even though the SSH connection itself is a real SSH connection.

This project:

  • Skips entirely when no Docker daemon is reachable. Contributors without Docker can still run chromium + mobile-chrome.
  • Builds container images on demand. First run builds kandev-agent:e2e (slim Node base + git) and kandev-sshd:e2e (Alpine + openssh-server + git + pre-baked mock-agent). Subsequent runs hit Docker's layer cache.
  • Has a longer per-test timeout (180s vs 60s) because container starts + agent setup are slow.

How to run it locally (requires Docker running):

KANDEV_E2E_CONTAINERS=1 pnpm e2e --project=containers

Or a single spec:

KANDEV_E2E_CONTAINERS=1 pnpm e2e --project=containers tests/ssh/launch-task.spec.ts

Why "containers" instead of "docker"?

This project used to be named docker. It was renamed to containers once SSH e2e tests joined it — calling it docker was misleading because SSH tests have nothing to do with the Docker executor; they just happen to use Docker as the runtime that hosts the sshd target.

KANDEV_E2E_DOCKER=1 is still honored as a deprecated alias for KANDEV_E2E_CONTAINERS=1 for one release. Local scripts and stale CI configs won't break, but new code should use the new name.

Commands

e2e, e2e:run, and e2e:ui are defined only in apps/web/package.json. Run them from apps/web (or pnpm --filter @kandev/web e2e:run from elsewhere). From the repo root you get ERR_PNPM_NO_IMPORTER_MANIFEST_FOUND.

Command What it does
pnpm e2e Run the default (chromium) project headless.
pnpm e2e:ui Open Playwright's UI mode for interactive runs.
pnpm e2e:headed Run headless project but with a visible browser.
pnpm e2e --project=containers Run container-backed tests (needs Docker).
pnpm e2e --project=mobile-chrome Run mobile responsive tests.
E2E_DEBUG=1 pnpm e2e Surface Docker build output + extra logging.

Common flags: --shard=1/4, -g "fragment of test name", --repeat-each=3 (flake hunting).

pnpm e2e:run — the managed runner (build + run + teardown)

e2e/scripts/run-e2e.sh (aliased as pnpm e2e:run) handles the build, the run, and cleanup so you don't have to assemble the steps by hand. It auto-selects docker vs host, runs N shards concurrently, enforces strict WS accounting by default (KANDEV_E2E_WS_ASSERT=1, matching CI), and never leaves root-owned artifacts behind.

pnpm e2e:run                                  # auto: docker if the daemon + CI image are available, else host
pnpm e2e:run --shards 3                        # 3 shards concurrently (isolated containers, or host procs with distinct ports)
pnpm e2e:run tests/chat/foo.spec.ts            # extra args pass straight through to Playwright
pnpm e2e:run --host --no-build -- --grep "x"   # force host, skip rebuild, forward flags after --
pnpm e2e:docker                                # force the docker CI image (full isolation from a host dev instance)
pnpm e2e:clean                                 # remove build/test artifacts, incl. root-owned ones from prior docker runs

Why a script instead of raw docker run: in docker mode it builds the CGO/fts5 backend on the host and runs it in the runtime image — a host glibc that's the same or older than the image's (the usual case; the image tracks recent Ubuntu) is forward-compatible, so no build image is needed. The runner smoke-tests the binary in the image first and only falls back to the build image (KANDEV_CI_BUILD_IMAGE, default …/kandev-ci:build-latest) if your host glibc is newer. It also builds the FE standalone on the host, pre-creates the standalone symlinks as relative links (so in-container global-setup doesn't recreate them as root), points Playwright output at a container-local dir, and cleans up. Run clean if a previous bare docker run left root-owned files you can't delete.

Office is always enabled in e2e (and dev); only prod gates it off. profiles.yaml sets KANDEV_FEATURES_OFFICE to "true" in the e2e and dev profiles and "false" in prod, and the fixture selects the e2e profile via KANDEV_E2E_MOCK=true. So tests/office/* always have office routes registered — no manual env setup.

This used to break when e2e was launched from a shell that had inherited KANDEV_FEATURES_OFFICE=false (e.g. from a host kandev backend running the prod profile): profiles.ApplyProfile only sets vars that are unset (so launchers/shells win — see docs/decisions/0007-runtime-feature-flags.md), and the fixture spreads process.env into the spawned backend, so the stale prod value won and 404'd every office spec. Fixed at the source: sanitizeInheritedEnv in e2e/fixtures/backend.ts strips all inherited KANDEV_FEATURES_* before spawn, so the e2e profile — not whatever the host exported — decides feature flags. No unset needed.

Host oversubscription: running ≥5 heavy shards concurrently on one machine (each = Go backend + Next standalone + Chromium + mock agent) starves CPU/IO and induces timing flakes that CI's isolated runners never see. Use 2–3 concurrent shards locally for a clean signal; see "flake triage" in the /e2e skill.

Backend isolation per worker

Every Playwright worker gets:

  • A unique backend port in BACKEND_BASE_PORT + workerIndex (default 18080+).
  • A unique frontend port in FRONTEND_BASE_PORT + workerIndex (default 13000+).
  • A fresh tmpdir (HOME, KANDEV_HOME_DIR, worktree base, repo clone base — all under that tmpdir).
  • A unique agentctl instance port range (30001 + E2E_PORT_OFFSET * 1000 + workerIndex * 200).
  • Its own SQLite DB.

Workers run in parallel across CI shards (--shard=N/M); within a worker, tests run serially because the testPage fixture calls e2eReset on the shared backend before each test.

Mocked vs real

  • Mocked: Jira (KANDEV_MOCK_JIRA=true), Linear (KANDEV_MOCK_LINEAR=true), GitHub (KANDEV_MOCK_GITHUB=true), the agent process itself (KANDEV_MOCK_AGENT=only). These are third-party services or external processes we don't want CI to depend on.
  • Real: Everything inside the kandev backend — orchestrator, lifecycle manager, agentctl, SSH/SFTP, Docker SDK, git, worktree manager. The point of e2e is to exercise the real code paths.

The SSH executor specifically has no mock controller. Tests use a real Docker-hosted sshd as the remote target, and fault-injection (host-key rotation, dropped traffic, killed pids) is done by operating on the container itself.

Adding a new spec

  1. Pick a directory under tests/ (or create one for a new feature).
  2. Decide which project it belongs to. Anything that needs Docker → tests/docker/ or tests/ssh/ (lands in containers). Anything mobile-specific → name it mobile-*.spec.ts. Otherwise it joins chromium automatically.
  3. Import the right test base:
    • import { test, expect } from "../../fixtures/test-base"; for normal tests.
    • import { test, expect } from "../../fixtures/docker-test-base"; for Docker executor tests.
    • import { test, expect } from "../../fixtures/ssh-test-base"; for SSH executor tests.
  4. Use getByTestId for selectors. If the surface you're testing lacks stable testids, add them — drift-prone CSS / text selectors are not worth the maintenance cost.

CI

.github/workflows/e2e-tests.yml defines two jobs:

  • e2e — matrixed chromium + mobile-chrome shards.
  • e2e-containers — single job, runs --project=containers, needs Docker.

Both upload blob reports that e2e-report merges into a single HTML artifact.

Reference in New Issue View Git Blame Copy Permalink