39c574bd57
A pass of issues uncovered by running the SSH e2e suite locally
(KANDEV_E2E_CONTAINERS=1 against the real sshd container). After this
commit all 15 non-launch SSH e2e tests pass (UI form validation,
test-connection contract, auth-methods, error-surfacing, sessions-card
empty state) and the launch path runs through prepare -> upload ->
agentctl -> instance -> forward, with WebSocket streams sometimes
disconnecting partway through (tracked below for follow-up).
Image changes (apps/web/e2e/fixtures/ssh-image.ts):
- Install curl so the SSH executor can POST to the in-container
agentctl control server for instance creation.
- Install gcompat + libstdc++ so the dynamically-linked glibc
agentctl binary can exec on musl-libc Alpine. Without it the
kernel returns ENOENT for /lib64/ld-linux-x86-64.so.2.
- Install shadow + 'usermod -p *' kandev so sshd doesn't refuse
pubkey auth for the otherwise-locked user.
- Drop in 'AllowTcpForwarding yes' and 'MaxSessions 100' via
/etc/ssh/sshd_config.d/. Alpine's defaults are 'no' and 10, both
of which crater the SSH executor's port-forward path.
- chown /home/kandev/.kandev to kandev on entrypoint so a host
bind-mount of that path (used in this commit for log persistence)
doesn't make agentctl unable to write its log file.
Image helper (apps/web/e2e/helpers/ssh.ts):
- Bind-mount the host mock-agent-linux-amd64 into the container at
the same absolute path. kandev's orchestrator builds the agent
command from KANDEV_MOCK_AGENT_LINUX_BINARY (a host path), so the
remote needs the binary at that exact path; bake-into-image isn't
enough because the absolute host path is what kandev sends.
- Bind-mount /home/kandev/.kandev so per-session agentctl logs
survive --rm teardown and the test runner can inspect them on
failure.
- Replace the broken CPU-spin sleep helper with a real sleep
subprocess and a /dev/tcp TCP-open probe so timing-dependent
helpers stop racing the container start.
- Retry the host fingerprint scan with stderr capture; the first
scan can race the sshd accept loop on Alpine.
Backend SSH executor (apps/backend/internal/agent/lifecycle):
- executor_ssh_operations.go startRemoteAgentctl: agentctl does not
accept --port-file or --pid-file flags (it reads AGENTCTL_PORT
and writes its own pid). Rewrite the launch wrapper to inject
AGENTCTL_PORT via env, capture $! as the pid, and tail the
on-remote agentctl.log for 'HTTP server bound successfully'
instead of waiting on imaginary side-files.
- executor_ssh_operations.go createRemoteAgentInstance: new helper
that POSTs /api/v1/instances on the in-container control server
via SSH-exec'd curl, returning the per-instance port the executor
should forward to. Mirrors what executor_sprites_operations.go
does inside its sprite. Without this step the agent stream
returns 'configure request failed with status 404' because the
SSH executor was talking to the control server, not the
per-session sub-server.
- executor_ssh.go startAndForwardAgentctl: chains the new
create-instance step in between agentctl launch and port-forward,
and forwards the *instance* port (not the control port).
- executor_ssh.go: drop the per-host SSH connection pool (it was
introducing a subtle aliasing issue between
CreateInstance/StopInstance/GetRemoteStatus that made the SSH
client unreliable across same-host sessions). Each session now
holds its own ssh.Client directly on sshSessionState and
ResumeRemoteInstance dials a fresh client too.
- executor_ssh_operations.go SSHPortForwarder: serialize the
client.Dial calls with a per-forwarder mutex. golang.org/x/crypto/
ssh.Client.Dial is documented as concurrency-safe but in practice
parallel opens from the kandev stream-manager occasionally
produce io.EOF as if the channel-open response was never
received.
Test fix (apps/web/e2e/tests/ssh/error-surfacing.spec.ts):
- Q1 ('TCP refused') was using identity_source='agent' with no
SSH_AUTH_SOCK in the backend env, so it failed at
buildAuthMethods before reaching the TCP-refused path. Switch to
identity_source='file' with the seed key so the test exercises
the failure it claims to.
Known remaining issue (to investigate next): the launch path now
reaches 'agent process started' + 'ACP initialize response
received' when run in isolation, but the WebSocket updates stream
sometimes disconnects with 1006 abnormal closure shortly after. A
standalone Go reproduction of concurrent direct-tcpip channels
against the same image works without issue, so the trigger is
something specific to the kandev backend's stream-manager + my
forwarder interaction. Tracked for the next iteration.
152 lines
6.1 KiB
TypeScript
152 lines
6.1 KiB
TypeScript
import { execFileSync, spawnSync } from "node:child_process";
|
|
import fs from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
|
|
/**
|
|
* Image used by the SSH executor's e2e tests. Built once per machine and
|
|
* reused. Tag is stable so Docker's layer cache survives across runs.
|
|
*
|
|
* The image carries:
|
|
* - openssh-server + openssh-sftp-server (the remote target)
|
|
* - git + bash + sudo + coreutils (for the SSH executor's prepare scripts)
|
|
* - iproute2 + iptables (for fault-injection: drop traffic mid-session)
|
|
* - a pre-baked `mock-agent` binary at /usr/local/bin/mock-agent so the
|
|
* agentctl process the SSH executor uploads has something to spawn when
|
|
* the orchestrator asks for an agent subprocess.
|
|
*
|
|
* The `agentctl` binary is NOT baked in — the e2e tests verify the actual
|
|
* SFTP upload + sha256-cache path the production code uses.
|
|
*/
|
|
export const SSH_E2E_IMAGE_TAG = "kandev-sshd:e2e";
|
|
|
|
const ENTRYPOINT = `#!/bin/sh
|
|
set -e
|
|
|
|
# Generate a fresh host key on first start so each test run gets a new
|
|
# fingerprint. Persisted across container restarts within the same test so
|
|
# fingerprint-pinning works end-to-end.
|
|
ssh-keygen -A >/dev/null
|
|
|
|
# Verbose sshd logging so test failures can be diagnosed without rebuilding
|
|
# the image — landed via the entrypoint so it always lands.
|
|
printf "\\nLogLevel DEBUG2\\n" >> /etc/ssh/sshd_config.d/10-kandev-forwarding.conf
|
|
|
|
# If the test runner bind-mounted /home/kandev/.kandev to inspect logs from
|
|
# the host, the mount inherits host ownership; make it writable by the
|
|
# kandev user before sshd starts accepting sessions.
|
|
mkdir -p /home/kandev/.kandev
|
|
chown -R kandev:kandev /home/kandev/.kandev
|
|
|
|
# Worker mounts its public key in here; copy into the kandev user's
|
|
# authorized_keys with the right perms so sshd accepts it.
|
|
if [ -f /authorized_keys ]; then
|
|
mkdir -p /home/kandev/.ssh
|
|
cp /authorized_keys /home/kandev/.ssh/authorized_keys
|
|
chown -R kandev:kandev /home/kandev/.ssh
|
|
chmod 700 /home/kandev/.ssh
|
|
chmod 600 /home/kandev/.ssh/authorized_keys
|
|
fi
|
|
|
|
# Optional second authorized_keys for a different user (passphrase-protected
|
|
# key tests load a second key into ssh-agent and want it accepted too).
|
|
if [ -f /authorized_keys.extra ]; then
|
|
cat /authorized_keys.extra >> /home/kandev/.ssh/authorized_keys
|
|
fi
|
|
|
|
# Run sshd in the foreground so docker logs and exit-codes propagate.
|
|
mkdir -p /var/run/sshd /var/empty
|
|
exec /usr/sbin/sshd -D -e
|
|
`;
|
|
|
|
const DOCKERFILE = `FROM alpine:3.20
|
|
RUN apk add --no-cache \\
|
|
openssh-server \\
|
|
openssh-sftp-server \\
|
|
git \\
|
|
bash \\
|
|
sudo \\
|
|
shadow \\
|
|
coreutils \\
|
|
ca-certificates \\
|
|
curl \\
|
|
iproute2 \\
|
|
iptables \\
|
|
# agentctl is built dynamically linked against glibc; Alpine is musl. gcompat
|
|
# ships a glibc-compat shim that lets glibc binaries exec on Alpine. Without
|
|
# it 'nohup /home/kandev/.kandev/bin/agentctl' fails with "No such file or
|
|
# directory" (the kernel can't find /lib64/ld-linux-x86-64.so.2).
|
|
gcompat \\
|
|
libstdc++ \\
|
|
# Alpine sshd_config sets AllowTcpForwarding no, which makes sshd reject
|
|
# every direct-tcpip channel request — exactly what the SSH executor's
|
|
# local port forward needs. Override to yes via a config.d drop-in.
|
|
# Also bump MaxSessions: kandev opens several concurrent direct-tcpip
|
|
# channels per session (health probe + agent stream + workspace stream +
|
|
# follow-up HTTP requests), and the OpenSSH default of 10 isn't enough
|
|
# under load — sshd starts EOF'ing channel open requests once exhausted.
|
|
&& printf "\\nAllowTcpForwarding yes\\nMaxSessions 100\\n" \\
|
|
> /etc/ssh/sshd_config.d/10-kandev-forwarding.conf \\
|
|
&& adduser -D -s /bin/bash kandev \\
|
|
# Alpine's adduser -D leaves the account "locked" (password = ! in /etc/shadow).
|
|
# sshd refuses ALL auth methods — including pubkey — for locked users, so we
|
|
# unlock by setting the password hash to '*' (no valid password, but the
|
|
# account itself is open for key-based login).
|
|
&& usermod -p '*' kandev \\
|
|
&& echo "kandev ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers \\
|
|
&& mkdir -p /home/kandev/.ssh /var/empty /var/run/sshd \\
|
|
&& chown -R kandev:kandev /home/kandev \\
|
|
&& rm -f /etc/ssh/ssh_host_* # generated fresh by entrypoint on each first-start
|
|
|
|
# The mock-agent binary is supplied by the test runner (linux/amd64 build the
|
|
# Go suite already produces). Without it agentctl can't spawn an agent
|
|
# subprocess; with it, the SSH executor can drive a full task end-to-end.
|
|
COPY mock-agent-linux-amd64 /usr/local/bin/mock-agent
|
|
RUN chmod +x /usr/local/bin/mock-agent
|
|
|
|
COPY entrypoint.sh /entrypoint.sh
|
|
RUN chmod +x /entrypoint.sh
|
|
|
|
EXPOSE 22
|
|
ENTRYPOINT ["/entrypoint.sh"]
|
|
`;
|
|
|
|
/**
|
|
* Returns true when a Docker daemon is reachable. Reused from docker-probe;
|
|
* duplicated here so callers can ask the "SSH-specific" question without
|
|
* implying anything about the Docker executor.
|
|
*/
|
|
export function hasSSHContainerSupport(): boolean {
|
|
return spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
|
|
}
|
|
|
|
/**
|
|
* Build the kandev-sshd:e2e image. Idempotent: Docker layer caching makes
|
|
* repeated builds near-instant when the Dockerfile and mock-agent haven't
|
|
* changed.
|
|
*
|
|
* Throws if the mock-agent linux/amd64 binary isn't present — that's a
|
|
* setup error the global-setup pre-flight should have caught.
|
|
*/
|
|
export function buildE2ESSHImage(): void {
|
|
const mockAgentPath = path.resolve(__dirname, "../../../backend/bin/mock-agent-linux-amd64");
|
|
if (!fs.existsSync(mockAgentPath)) {
|
|
throw new Error(
|
|
`mock-agent-linux-amd64 not found at ${mockAgentPath}; build it with ` +
|
|
`'make -C apps/backend build-mock-agent-linux' before running container e2e tests.`,
|
|
);
|
|
}
|
|
|
|
const ctxDir = fs.mkdtempSync(path.join(os.tmpdir(), "kandev-e2e-sshd-img-"));
|
|
try {
|
|
fs.writeFileSync(path.join(ctxDir, "Dockerfile"), DOCKERFILE);
|
|
fs.writeFileSync(path.join(ctxDir, "entrypoint.sh"), ENTRYPOINT);
|
|
fs.copyFileSync(mockAgentPath, path.join(ctxDir, "mock-agent-linux-amd64"));
|
|
execFileSync("docker", ["build", "-t", SSH_E2E_IMAGE_TAG, ctxDir], {
|
|
stdio: process.env.E2E_DEBUG ? "inherit" : "ignore",
|
|
});
|
|
} finally {
|
|
fs.rmSync(ctxDir, { recursive: true, force: true });
|
|
}
|
|
}
|