For SSH the agent runs on a remote host whose filesystem doesn't share
paths with the kandev backend's, so the host-absolute mock-agent path
configureMockAgent computes can't apply. Extend the bare-name branch in
MockAgent.BuildCommand to include RuntimeSSH (the SSH e2e image already
bakes mock-agent into /usr/local/bin) and drop the now-redundant SSH e2e
bind-mount that worked around this by mirroring the host path into the
container.
regenerateHostKey was returning the ssh-keyscan fingerprint, but
ssh-keyscan and the Go ssh client can negotiate different host-key
types — meaning the fingerprint kandev observes during a real probe
does not always match what keyscan saw. The keyscan output was used
verbatim as the expected value, which K1 / K3 then asserted against
kandev's pinned-fingerprint result, so the specs flapped depending
on which algorithm sshd advertised.
Drop the return value from regenerateHostKey and have callers hit
/api/v1/ssh/test to learn the canonical fingerprint, matching what
the SSH fixture already does at seeding time.
K2 was polling env.error_message which doesn't exist — the launch
fails at SSH dial time before the executor is recorded on the task
environment. Point the polling at the task-session error_message
instead, where updateTaskSessionState actually persists the dial
failure.
The simple-message scenario completes inside a couple hundred
milliseconds, so the previous "wait for executor_type=ssh, then
query once" pattern raced the row's removal during teardown. Poll
listSSHSessions directly with a 60s window so the row is observed
while the executor is up.
Adds dumpRemoteLogs helper for copying per-session agentctl.log
files out of the sshd container on assertion failure, easing
debugging when the row never appears.
A pass of issues uncovered by running the SSH e2e suite locally
(KANDEV_E2E_CONTAINERS=1 against the real sshd container). After this
commit all 15 non-launch SSH e2e tests pass (UI form validation,
test-connection contract, auth-methods, error-surfacing, sessions-card
empty state) and the launch path runs through prepare -> upload ->
agentctl -> instance -> forward, with WebSocket streams sometimes
disconnecting partway through (tracked below for follow-up).
Image changes (apps/web/e2e/fixtures/ssh-image.ts):
- Install curl so the SSH executor can POST to the in-container
agentctl control server for instance creation.
- Install gcompat + libstdc++ so the dynamically-linked glibc
agentctl binary can exec on musl-libc Alpine. Without it the
kernel returns ENOENT for /lib64/ld-linux-x86-64.so.2.
- Install shadow + 'usermod -p *' kandev so sshd doesn't refuse
pubkey auth for the otherwise-locked user.
- Drop in 'AllowTcpForwarding yes' and 'MaxSessions 100' via
/etc/ssh/sshd_config.d/. Alpine's defaults are 'no' and 10, both
of which crater the SSH executor's port-forward path.
- chown /home/kandev/.kandev to kandev on entrypoint so a host
bind-mount of that path (used in this commit for log persistence)
doesn't make agentctl unable to write its log file.
Image helper (apps/web/e2e/helpers/ssh.ts):
- Bind-mount the host mock-agent-linux-amd64 into the container at
the same absolute path. kandev's orchestrator builds the agent
command from KANDEV_MOCK_AGENT_LINUX_BINARY (a host path), so the
remote needs the binary at that exact path; bake-into-image isn't
enough because the absolute host path is what kandev sends.
- Bind-mount /home/kandev/.kandev so per-session agentctl logs
survive --rm teardown and the test runner can inspect them on
failure.
- Replace the broken CPU-spin sleep helper with a real sleep
subprocess and a /dev/tcp TCP-open probe so timing-dependent
helpers stop racing the container start.
- Retry the host fingerprint scan with stderr capture; the first
scan can race the sshd accept loop on Alpine.
Backend SSH executor (apps/backend/internal/agent/lifecycle):
- executor_ssh_operations.go startRemoteAgentctl: agentctl does not
accept --port-file or --pid-file flags (it reads AGENTCTL_PORT
and writes its own pid). Rewrite the launch wrapper to inject
AGENTCTL_PORT via env, capture $! as the pid, and tail the
on-remote agentctl.log for 'HTTP server bound successfully'
instead of waiting on imaginary side-files.
- executor_ssh_operations.go createRemoteAgentInstance: new helper
that POSTs /api/v1/instances on the in-container control server
via SSH-exec'd curl, returning the per-instance port the executor
should forward to. Mirrors what executor_sprites_operations.go
does inside its sprite. Without this step the agent stream
returns 'configure request failed with status 404' because the
SSH executor was talking to the control server, not the
per-session sub-server.
- executor_ssh.go startAndForwardAgentctl: chains the new
create-instance step in between agentctl launch and port-forward,
and forwards the *instance* port (not the control port).
- executor_ssh.go: drop the per-host SSH connection pool (it was
introducing a subtle aliasing issue between
CreateInstance/StopInstance/GetRemoteStatus that made the SSH
client unreliable across same-host sessions). Each session now
holds its own ssh.Client directly on sshSessionState and
ResumeRemoteInstance dials a fresh client too.
- executor_ssh_operations.go SSHPortForwarder: serialize the
client.Dial calls with a per-forwarder mutex. golang.org/x/crypto/
ssh.Client.Dial is documented as concurrency-safe but in practice
parallel opens from the kandev stream-manager occasionally
produce io.EOF as if the channel-open response was never
received.
Test fix (apps/web/e2e/tests/ssh/error-surfacing.spec.ts):
- Q1 ('TCP refused') was using identity_source='agent' with no
SSH_AUTH_SOCK in the backend env, so it failed at
buildAuthMethods before reaching the TCP-refused path. Switch to
identity_source='file' with the seed key so the test exercises
the failure it claims to.
Known remaining issue (to investigate next): the launch path now
reaches 'agent process started' + 'ACP initialize response
received' when run in isolation, but the WebSocket updates stream
sometimes disconnects with 1006 abnormal closure shortly after. A
standalone Go reproduction of concurrent direct-tcpip channels
against the same image works without issue, so the trigger is
something specific to the kandev backend's stream-manager + my
forwarder interaction. Tracked for the next iteration.
Three bugs surfaced by actually running the SSH e2e suite
(KANDEV_E2E_CONTAINERS=1 against the containers project):
1. Alpine's `adduser -D kandev` leaves the account password-locked
(`!` in /etc/shadow). sshd then refuses every auth method for that
user — including pubkey — with "User kandev not allowed because
account is locked". Fix: `usermod -p '*' kandev` in the Dockerfile
so the account is open for key-based login but still has no valid
password. Also adds the `shadow` apk package for `usermod`.
2. ssh-keyscan reports the ed25519 host fingerprint, but Go's ssh
client may negotiate a different host key type when sshd offers
multiple — meaning the fingerprint the fixture pre-scanned didn't
match the one kandev observed at handshake time. Fix: the fixture
now hits /api/v1/ssh/test with a permissive callback right after
the container is up and uses that fingerprint as the pinned value.
Same code path real users hit; eliminates the negotiation skew by
construction.
3. ssh.ts had a CPU-spin sleep helper and an nc-dependent TCP probe,
both fragile. Replaced with a proper sleep subprocess and a bash
/dev/tcp probe, plus retry-with-stderr on the fingerprint scan so
failures are diagnosable.
Also tightens test-endpoint.spec.ts F4: the test was using
identity_source=agent to "fail fast", but that fails inside
buildAuthMethods before the handshake step ever runs. Switched to
identity_source=file against a refused port so the test exercises
the actual TCP+handshake failure path.
After these fixes 4/6 test-endpoint cases pass against a real sshd.
Lands the building blocks the SSH specs will compose on top of. Every
piece talks to a real sshd container — no mocks at any layer.
- fixtures/ssh-image.ts: builds kandev-sshd:e2e once per machine (Alpine
+ openssh-server + openssh-sftp-server + git + bash + sudo +
iproute2 + iptables, pre-bakes mock-agent at /usr/local/bin/mock-agent,
generates a fresh host key on entrypoint, reads the worker public key
out of a bind-mounted /authorized_keys). The agentctl binary is
deliberately NOT pre-baked so tests exercise the SFTP upload + sha256
cache path the production code uses.
- helpers/ssh.ts: per-worker sshd lifecycle (startSSHServer /
stopSSHServer), keypair generation, host-key fingerprint scanning,
fault injection (regenerateHostKey for host-key-rotation specs,
dropTrafficToPort22 / restoreTraffic for connection-drop tests,
killRemotePid for "agentctl died" recovery), and remote inspection
helpers (readRemoteFile / remotePathExists / listRemoteDir /
countSshdConnections).
- helpers/ssh-bastion.ts: two-container ProxyJump setup. Bastion is on
a private Docker network and has a published port; target is on the
same network with NO host port — any successful connection MUST have
traversed the bastion, which makes the proxy-jump assertion honest.
- fixtures/ssh-test-base.ts: worker-scoped fixture mirroring
docker-test-base. Skips when Docker isn't reachable. Spins up the
sshd container, generates keys, scans the fingerprint, then seeds a
workspace + workflow + bare repo + mock-agent profile + an SSH
executor with the fingerprint already trusted. Specs that need to
exercise the test-then-trust UI gate create their own executor on
top of the same target.
- pages/SSHSettingsPage.ts: Page Object covering the connection form,
test/save buttons, trust checkbox, observed/pinned fingerprint, per
step badges, and the sessions table.
- helpers/api-client.ts: new createSSHExecutor / updateExecutor /
getExecutor / testSSHConnection / listSSHSessions methods plus
matching DTO types.
José Almeida