kandev/docs/specs/ssh-executor/e2e-plan.md

SSH Executor — E2E Test Plan

This document tracks the Playwright e2e coverage for the SSH executor. Lives next to spec.md so future contributors don't have to reverse-engineer "what's tested" from the test directory.

Approach

Real SSH server, no mocks. Every SSH e2e test runs against a real sshd container the test brings up itself. This keeps test code identical to the production code path (SFTP upload, SSH handshake, port forward, agentctl launch) and lets us trust green tests as evidence the feature works.

For scenarios that need fault injection (host-key rotation, dropped connections, dead agentctl), we operate on the container itself — regenerate keys, drop traffic via iptables, kill processes by pid — rather than mocking those at the backend layer.

Container project (formerly docker)

SSH tests live in the same Playwright project as the existing Docker e2e tests, now renamed from docker → containers. The project gates on Docker being available on the host and skips cleanly when it isn't. See apps/web/e2e/README.md for how to run it.

Env flag: KANDEV_E2E_CONTAINERS=1. The legacy KANDEV_E2E_DOCKER=1 is honored as a deprecated alias for one release.

apps/web/e2e/
├── fixtures/
│   ├── docker-test-base.ts          (existing, untouched)
│   └── ssh-test-base.ts             (new)
├── helpers/
│   ├── docker.ts                    (existing)
│   ├── docker-probe.ts              (existing)
│   ├── ssh.ts                       (new — keygen, container lifecycle, rekey, drop-traffic)
│   ├── ssh-image.ts                 (new — buildE2ESSHImage)
│   └── ssh-bastion.ts               (new — 2-container network for ProxyJump)
├── pages/
│   └── SSHSettingsPage.ts           (new)
└── tests/
    ├── docker/...                   (unchanged)
    └── ssh/                         (new — every SSH spec lands here)

sshd image

kandev-sshd:e2e, built once per CI run. Alpine + openssh-server + openssh-sftp-server + git + bash + sudo + iptables + a pre-baked mock-agent binary at /usr/local/bin/mock-agent. The image does not pre-bake agentctl — every test exercises the SFTP upload path the production code uses.

The image entrypoint:

1. Generates a fresh host key on first start.
2. Reads the worker's public key from a bind-mounted file into /home/kandev/.ssh/authorized_keys.
3. Runs sshd -D -e.

Spec files & coverage

Test naming follows the same concern-based grouping the existing docker/*.spec.ts files use.

File	What it covers
ssh/connection-form.spec.ts	A1–A7: form rendering, field gating, identity-source toggle, default port
ssh/test-result.spec.ts	B1–B7: successful + failed Test Connection paths, step badges, fingerprint surfacing, "cached / will upload"
ssh/trust-gate.spec.ts	C1–C7: Save disabled until trust ticked; edits to host/port/user/identity reset result + trust; fingerprint-change amber warning
ssh/executor-crud.spec.ts	D1–D7: SSH executor CRUD, listing with right icon/label, profile workdir_root round-trip, edit-with-live-sessions modal, delete-with-live-sessions warning
ssh/sessions-card.spec.ts	E1–E6: empty state, row rendering, manual refresh, status badges, truncation, polling pickup
ssh/test-endpoint.spec.ts	F1–F6: HTTP contract for POST /api/v1/ssh/test + WS parity
ssh/sessions-endpoint.spec.ts	G1–G5: HTTP contract for GET /api/v1/ssh/executors/:id/sessions + WS parity
ssh/launch-task.spec.ts	H1–H8: end-to-end task launch on real sshd, agentctl upload + sha256 cache hit on second launch, per-task / per-session dir layout, cleanup on stop
ssh/concurrency.spec.ts	I1–I4: two sessions on same task share workdir, two tasks share connection, single SSH conn for same host, keepalive eviction + reconnect
ssh/recovery.spec.ts	J1–J4: backend restart with live session reconnects; dead remote agentctl handled; two surviving sessions reattach; persisted metadata keys present
ssh/hostkey-rotation.spec.ts	K1–K4: simulate rekey of the container; mismatch surfaces verbatim; re-test + re-trust flow restores function
ssh/auth-methods.spec.ts	L1–L5: file + valid key, file + missing key, file + passphrase-protected (error), agent + agent running, agent without SSH_AUTH_SOCK
ssh/config-inheritance.spec.ts	M1–M6: ~/.ssh/config alias resolution, form-overrides-config precedence, unknown alias fallback
ssh/proxy-jump.spec.ts	N1–N3: direct connect, single bastion ProxyJump (2-container network), chained ProxyJump explicit failure
ssh/workdir-per-profile.spec.ts	P1–P4: different profiles → different remote workdirs; default workdir; profile switch does not move existing task dirs
ssh/error-surfacing.spec.ts	Q1–Q5: TCP refused, auth failed, permission denied (mkdir), ProxyJump unreachable, backend 5xx
ssh/persistence.spec.ts	R1–R4: fingerprint persists across reload, incomplete test does not leak state, mobile project responsive, executor icon/label correct in lists

80 cases across 17 SSH spec files (see apps/web/e2e/tests/ssh/). The full suite runs in ~2.4 min on the containers project.

Helpers exposed to specs

• ssh.ts
  • hasSSHContainerSupport(): boolean — Docker reachable
  • buildE2ESSHImage() — idempotent
  • startSSHServer({ workerIndex }): SSHServerHandle — generate keypair, start container, return { host, port, user, identityFile, hostFingerprint, containerId }
  • stopSSHServer(handle)
  • regenerateHostKey(handle) — simulate rekey
  • dropTrafficToPort22(handle) / restoreTraffic(handle) — simulate connection drop
  • killRemotePid(handle, pid) — fault injection for recovery
  • readRemoteFile(handle, path): string — assertions on uploaded files (sha256, port files)
• ssh-bastion.ts
  • startBastionAndTarget({ workerIndex }): { bastion, target, network } — 2 containers in a private network
  • stopBastionAndTarget(...)
• ssh-test-base.ts
  • sshTest Playwright test fixture: extends dockerTest-style, pre-seeds workspace/workflow/agent profile, brings up one sshd container, exposes a seedData.sshTarget for tests that just need a configured executor.

Page Object

SSHSettingsPage exposes:

• goto(executorId?) — navigate to settings; if executorId omitted, the "new SSH executor" flow.
• fillForm({ name, host, port, user, identitySource, identityFile, proxyJump, hostAlias })
• clickTestConnection() / waitForTestResult()
• expectStep(name, status) — badge assertion per step
• expectFingerprint(fp) / expectFingerprintAny() — accepts any non-empty fingerprint
• tickTrust() / untickTrust()
• clickSave() / confirmRunningSessionsModal() / cancelRunningSessionsModal()
• expectConnectionBadge("trusted" | "unverified")
• sessionsTable.expectEmpty() / sessionsTable.rowFor(sessionId).expectColumns(...)

All locators use data-testid attributes added to ssh-settings.tsx in the same PR.

API-client helpers

apps/web/e2e/helpers/api-client.ts gains:

• createSSHExecutor({ name, config }): Executor
• updateSSHExecutor(id, patch)
• listSSHSessions(executorId): SSHSession[]
• createSSHExecutorProfile(executorId, { name, workdirRoot })
• getExecutorRunning(sessionId) — for asserting persisted SSH metadata after launch

Deferred — revisit later

These are intentionally out of scope for this PR. Each one names what unblocks it.

• Linux arm64 remote host (O1, O2) — needs agentctl-linux-arm64 in the build pipeline and qemu-user-static available in CI to run an arm64 sshd container. Revisit when the Linux arm64 follow-up lands. Until then, the unsupported-platform gate is covered by backend unit tests.
• Chained ProxyJump beyond first hop — v1 spec says single bastion only. N3 just verifies we fail on a chained config rather than silently misroute. A future "chained ProxyJump" feature would add positive cases.
• macOS remote E2E — supported by the runtime, but needs macOS SSH host coverage in CI or a manual runner.
• Windows remote — needs agentctl-windows-* binaries and a Windows SSH host harness.
• Reverse-forward of local MCP servers to the remote agent — not in v1 spec; would need its own test plan when that feature lands.

Phasing (within this PR)

We don't need formal phases since everything is real-SSH-backed and lands together, but the implementation order is:

1. Plan doc (this file).
2. Rename docker project → containers + README explanation.
3. SSH harness: image, helpers (ssh.ts, ssh-image.ts), fixture (ssh-test-base.ts).
4. SSH UI test IDs + Page Object.
5. API client helpers.
6. Spec files in roughly the table order.
7. Bastion helper + proxy-jump.spec.ts last (heaviest infra).