Files
kandev/apps/web/e2e/fixtures/ssh-image.ts
T
José Almeida 39c574bd57 fix(ssh): make the end-to-end launch path actually work
A pass of issues uncovered by running the SSH e2e suite locally
(KANDEV_E2E_CONTAINERS=1 against the real sshd container). After this
commit all 15 non-launch SSH e2e tests pass (UI form validation,
test-connection contract, auth-methods, error-surfacing, sessions-card
empty state) and the launch path runs through prepare -> upload ->
agentctl -> instance -> forward, with WebSocket streams sometimes
disconnecting partway through (tracked below for follow-up).

Image changes (apps/web/e2e/fixtures/ssh-image.ts):
- Install curl so the SSH executor can POST to the in-container
  agentctl control server for instance creation.
- Install gcompat + libstdc++ so the dynamically-linked glibc
  agentctl binary can exec on musl-libc Alpine. Without it the
  kernel returns ENOENT for /lib64/ld-linux-x86-64.so.2.
- Install shadow + 'usermod -p *' kandev so sshd doesn't refuse
  pubkey auth for the otherwise-locked user.
- Drop in 'AllowTcpForwarding yes' and 'MaxSessions 100' via
  /etc/ssh/sshd_config.d/. Alpine's defaults are 'no' and 10, both
  of which crater the SSH executor's port-forward path.
- chown /home/kandev/.kandev to kandev on entrypoint so a host
  bind-mount of that path (used in this commit for log persistence)
  doesn't make agentctl unable to write its log file.

Image helper (apps/web/e2e/helpers/ssh.ts):
- Bind-mount the host mock-agent-linux-amd64 into the container at
  the same absolute path. kandev's orchestrator builds the agent
  command from KANDEV_MOCK_AGENT_LINUX_BINARY (a host path), so the
  remote needs the binary at that exact path; bake-into-image isn't
  enough because the absolute host path is what kandev sends.
- Bind-mount /home/kandev/.kandev so per-session agentctl logs
  survive --rm teardown and the test runner can inspect them on
  failure.
- Replace the broken CPU-spin sleep helper with a real sleep
  subprocess and a /dev/tcp TCP-open probe so timing-dependent
  helpers stop racing the container start.
- Retry the host fingerprint scan with stderr capture; the first
  scan can race the sshd accept loop on Alpine.

Backend SSH executor (apps/backend/internal/agent/lifecycle):
- executor_ssh_operations.go startRemoteAgentctl: agentctl does not
  accept --port-file or --pid-file flags (it reads AGENTCTL_PORT
  and writes its own pid). Rewrite the launch wrapper to inject
  AGENTCTL_PORT via env, capture $! as the pid, and tail the
  on-remote agentctl.log for 'HTTP server bound successfully'
  instead of waiting on imaginary side-files.
- executor_ssh_operations.go createRemoteAgentInstance: new helper
  that POSTs /api/v1/instances on the in-container control server
  via SSH-exec'd curl, returning the per-instance port the executor
  should forward to. Mirrors what executor_sprites_operations.go
  does inside its sprite. Without this step the agent stream
  returns 'configure request failed with status 404' because the
  SSH executor was talking to the control server, not the
  per-session sub-server.
- executor_ssh.go startAndForwardAgentctl: chains the new
  create-instance step in between agentctl launch and port-forward,
  and forwards the *instance* port (not the control port).
- executor_ssh.go: drop the per-host SSH connection pool (it was
  introducing a subtle aliasing issue between
  CreateInstance/StopInstance/GetRemoteStatus that made the SSH
  client unreliable across same-host sessions). Each session now
  holds its own ssh.Client directly on sshSessionState and
  ResumeRemoteInstance dials a fresh client too.
- executor_ssh_operations.go SSHPortForwarder: serialize the
  client.Dial calls with a per-forwarder mutex. golang.org/x/crypto/
  ssh.Client.Dial is documented as concurrency-safe but in practice
  parallel opens from the kandev stream-manager occasionally
  produce io.EOF as if the channel-open response was never
  received.

Test fix (apps/web/e2e/tests/ssh/error-surfacing.spec.ts):
- Q1 ('TCP refused') was using identity_source='agent' with no
  SSH_AUTH_SOCK in the backend env, so it failed at
  buildAuthMethods before reaching the TCP-refused path. Switch to
  identity_source='file' with the seed key so the test exercises
  the failure it claims to.

Known remaining issue (to investigate next): the launch path now
reaches 'agent process started' + 'ACP initialize response
received' when run in isolation, but the WebSocket updates stream
sometimes disconnects with 1006 abnormal closure shortly after. A
standalone Go reproduction of concurrent direct-tcpip channels
against the same image works without issue, so the trigger is
something specific to the kandev backend's stream-manager + my
forwarder interaction. Tracked for the next iteration.
2026-05-17 23:45:07 +01:00

152 lines
6.1 KiB
TypeScript

import { execFileSync, spawnSync } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
/**
* Image used by the SSH executor's e2e tests. Built once per machine and
* reused. Tag is stable so Docker's layer cache survives across runs.
*
* The image carries:
* - openssh-server + openssh-sftp-server (the remote target)
* - git + bash + sudo + coreutils (for the SSH executor's prepare scripts)
* - iproute2 + iptables (for fault-injection: drop traffic mid-session)
* - a pre-baked `mock-agent` binary at /usr/local/bin/mock-agent so the
* agentctl process the SSH executor uploads has something to spawn when
* the orchestrator asks for an agent subprocess.
*
* The `agentctl` binary is NOT baked in — the e2e tests verify the actual
* SFTP upload + sha256-cache path the production code uses.
*/
export const SSH_E2E_IMAGE_TAG = "kandev-sshd:e2e";
const ENTRYPOINT = `#!/bin/sh
set -e
# Generate a fresh host key on first start so each test run gets a new
# fingerprint. Persisted across container restarts within the same test so
# fingerprint-pinning works end-to-end.
ssh-keygen -A >/dev/null
# Verbose sshd logging so test failures can be diagnosed without rebuilding
# the image — landed via the entrypoint so it always lands.
printf "\\nLogLevel DEBUG2\\n" >> /etc/ssh/sshd_config.d/10-kandev-forwarding.conf
# If the test runner bind-mounted /home/kandev/.kandev to inspect logs from
# the host, the mount inherits host ownership; make it writable by the
# kandev user before sshd starts accepting sessions.
mkdir -p /home/kandev/.kandev
chown -R kandev:kandev /home/kandev/.kandev
# Worker mounts its public key in here; copy into the kandev user's
# authorized_keys with the right perms so sshd accepts it.
if [ -f /authorized_keys ]; then
mkdir -p /home/kandev/.ssh
cp /authorized_keys /home/kandev/.ssh/authorized_keys
chown -R kandev:kandev /home/kandev/.ssh
chmod 700 /home/kandev/.ssh
chmod 600 /home/kandev/.ssh/authorized_keys
fi
# Optional second authorized_keys for a different user (passphrase-protected
# key tests load a second key into ssh-agent and want it accepted too).
if [ -f /authorized_keys.extra ]; then
cat /authorized_keys.extra >> /home/kandev/.ssh/authorized_keys
fi
# Run sshd in the foreground so docker logs and exit-codes propagate.
mkdir -p /var/run/sshd /var/empty
exec /usr/sbin/sshd -D -e
`;
const DOCKERFILE = `FROM alpine:3.20
RUN apk add --no-cache \\
openssh-server \\
openssh-sftp-server \\
git \\
bash \\
sudo \\
shadow \\
coreutils \\
ca-certificates \\
curl \\
iproute2 \\
iptables \\
# agentctl is built dynamically linked against glibc; Alpine is musl. gcompat
# ships a glibc-compat shim that lets glibc binaries exec on Alpine. Without
# it 'nohup /home/kandev/.kandev/bin/agentctl' fails with "No such file or
# directory" (the kernel can't find /lib64/ld-linux-x86-64.so.2).
gcompat \\
libstdc++ \\
# Alpine sshd_config sets AllowTcpForwarding no, which makes sshd reject
# every direct-tcpip channel request — exactly what the SSH executor's
# local port forward needs. Override to yes via a config.d drop-in.
# Also bump MaxSessions: kandev opens several concurrent direct-tcpip
# channels per session (health probe + agent stream + workspace stream +
# follow-up HTTP requests), and the OpenSSH default of 10 isn't enough
# under load — sshd starts EOF'ing channel open requests once exhausted.
&& printf "\\nAllowTcpForwarding yes\\nMaxSessions 100\\n" \\
> /etc/ssh/sshd_config.d/10-kandev-forwarding.conf \\
&& adduser -D -s /bin/bash kandev \\
# Alpine's adduser -D leaves the account "locked" (password = ! in /etc/shadow).
# sshd refuses ALL auth methods — including pubkey — for locked users, so we
# unlock by setting the password hash to '*' (no valid password, but the
# account itself is open for key-based login).
&& usermod -p '*' kandev \\
&& echo "kandev ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers \\
&& mkdir -p /home/kandev/.ssh /var/empty /var/run/sshd \\
&& chown -R kandev:kandev /home/kandev \\
&& rm -f /etc/ssh/ssh_host_* # generated fresh by entrypoint on each first-start
# The mock-agent binary is supplied by the test runner (linux/amd64 build the
# Go suite already produces). Without it agentctl can't spawn an agent
# subprocess; with it, the SSH executor can drive a full task end-to-end.
COPY mock-agent-linux-amd64 /usr/local/bin/mock-agent
RUN chmod +x /usr/local/bin/mock-agent
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
EXPOSE 22
ENTRYPOINT ["/entrypoint.sh"]
`;
/**
* Returns true when a Docker daemon is reachable. Reused from docker-probe;
* duplicated here so callers can ask the "SSH-specific" question without
* implying anything about the Docker executor.
*/
export function hasSSHContainerSupport(): boolean {
return spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
}
/**
* Build the kandev-sshd:e2e image. Idempotent: Docker layer caching makes
* repeated builds near-instant when the Dockerfile and mock-agent haven't
* changed.
*
* Throws if the mock-agent linux/amd64 binary isn't present — that's a
* setup error the global-setup pre-flight should have caught.
*/
export function buildE2ESSHImage(): void {
const mockAgentPath = path.resolve(__dirname, "../../../backend/bin/mock-agent-linux-amd64");
if (!fs.existsSync(mockAgentPath)) {
throw new Error(
`mock-agent-linux-amd64 not found at ${mockAgentPath}; build it with ` +
`'make -C apps/backend build-mock-agent-linux' before running container e2e tests.`,
);
}
const ctxDir = fs.mkdtempSync(path.join(os.tmpdir(), "kandev-e2e-sshd-img-"));
try {
fs.writeFileSync(path.join(ctxDir, "Dockerfile"), DOCKERFILE);
fs.writeFileSync(path.join(ctxDir, "entrypoint.sh"), ENTRYPOINT);
fs.copyFileSync(mockAgentPath, path.join(ctxDir, "mock-agent-linux-amd64"));
execFileSync("docker", ["build", "-t", SSH_E2E_IMAGE_TAG, ctxDir], {
stdio: process.env.E2E_DEBUG ? "inherit" : "ignore",
});
} finally {
fs.rmSync(ctxDir, { recursive: true, force: true });
}
}