kandev/docs/specs/executors/system-design/ssh-executor.md

status, system, requirements, created, owners

status	system	requirements	created	owners
draft	executors	REQ-EXECUTORS-SSH-EXECUTOR-001	2026-05-16	tbd

SSH Executor System Design

Purpose and boundaries

This design preserves the technical source detail for REQ-EXECUTORS-SSH-EXECUTOR-001 during migration.

Requirement mapping

Requirement	Design section
REQ-EXECUTORS-SSH-EXECUTOR-001	Migrated source detail
AC-EXECUTORS-SSH-EXECUTOR-001.11 and AC-EXECUTORS-SSH-EXECUTOR-001.12	Credential-file conflict policy

Migrated source detail

Why

Today the only ways to run an agent are: (1) locally on the user's own machine (local_pc, local_docker), or (2) in a Sprites-hosted ephemeral sandbox. There is no option for "run on a Linux box I already own" — a user's VPS, homelab server, beefy desktop, EC2/Hetzner instance, or shared team dev box.

That gap matters because:

• Cost control. Sprites are convenient but metered; a $5/mo VPS or an already-paid-for desktop is effectively free capacity for long-running tasks.
• Hardware control. Users who need a specific GPU, a lot of RAM, a particular CPU arch, or pre-installed proprietary tools (databases, CAD, internal services) cannot get there with Sprites.
• Privacy / compliance. Some users cannot send source code through a third-party cloud sandbox but are fine running an agent on a server they fully control.
• "Move my Claude Code to the cloud" use case. Running on a remote host means closing the laptop lid doesn't kill the agent, and the same task is reachable from a phone, tablet, or any browser.

SSH is the lowest-common-denominator transport for reaching a remote Linux box. If we can put an agent on anything reachable over SSH — VPS, homelab, EC2, raspberry pi, lab machine, jump-host-protected private box — we unlock all of the above in one feature without inventing a kandev-specific cloud.

What (v1)

A new executor type ssh joins local_pc / local_docker / sprites in the existing executor framework. From the user's perspective: configure an SSH target once in settings, then pick it as the executor for any task; tasks run on the remote box exactly as they would on a Sprite.

Remote layout: one task workspace per task

Kandev creates one remote workspace per task. The primary repository owns the workspace root so a single-repository agent starts inside a Git checkout; additional repository branches use direct child directories:

~/.kandev/tasks/
├── <task-dir-name-A>/
│   ├── .git/               # primary repository on the task's feature branch
│   └── <repo-2>-<branch>/  # additional repository or branch
└── <task-dir-name-B>/
    └── .git/               # another task's primary repository

The workspace root defaults to ~/.kandev/ on the remote and is configurable per profile (so one host can serve ~/.kandev-team-a/tasks/ and ~/.kandev-team-b/tasks/ for different profiles on the same SSH user). Concurrent sessions on the same task share the task workspace; concurrent tasks on the same host get different <task-dir-name> siblings.

Per-session, not per-task: agentctl process, remote port, local forward

Execution is keyed per session (matching local + Sprites). Each session on an SSH host gets:

• Its own agentctl process, launched on a unique remote port chosen by binding to :0 on the remote and reading back the chosen port.
• Its own SSH local port forward: a fresh 127.0.0.1:0 listener that the kandev backend dials for HTTP + WebSocket streams to that session's agentctl. All forwards ride the single shared SSH connection for the host (see below).
• A session-scoped runtime dir at <workdir_root>/tasks/<task-dir-name>/.kandev/sessions/<session-id>/ for the agentctl PID file, port file, and log — kept under the task dir so cleanup follows the task, not orphaned across the filesystem.

Multiple sessions in the same task share the same worktree on disk (same files, same branch state); they have independent agentctl processes and independent UI streams. Multiple sessions across different tasks on the same host have independent task dirs as well.

Auth, connectivity, and host-key trust

• Auth: SSH key file + ssh-agent. Authentication via golang.org/x/crypto/ssh uses either an explicit IdentityFile or an ssh-agent. A per-host OpenSSH IdentityAgent overrides the backend process's $SSH_AUTH_SOCK, covering 1Password / Secretive / Yubikey / forwarded-agent users whose agent socket differs from the desktop process environment. IdentityAgent none disables agent authentication for that host. Passphrase-protected keys are not handled in kandev — users must load them into ssh-agent themselves. Password and keyboard-interactive auth are not supported in v1.

• OpenSSH config inheritance. When a host_alias is configured, kandev resolves user and system OpenSSH client configuration to inherit HostName, Port, User, IdentityAgent, IdentityFile, and one ProxyJump. IdentityAgent supports ~, ${VAR}, whole-value $VAR, SSH_AUTH_SOCK, and percent tokens %%, %d, %h, %i, %j, %k, %L, %l, %n, %p, %r, and %u; %C is not supported. A user whose terminal already does ssh prod can paste prod into kandev and have it just work.

• Connectivity: direct, ProxyJump, mesh-VPN. Direct TCP to host:port (default 22). ProxyJump (single bastion in v1; chained jumps deferred) implemented natively via the Go SSH client. Tailscale / WireGuard / corporate VPN: "just works" when the kandev backend process is on the same network namespace.

• Create flow: test-connection gate + explicit fingerprint trust. The Sprites executor already gates creation behind a "Test Connection" step. The SSH executor adopts the same pattern and extends it with an explicit host-key trust step:

  1. User fills the form.
  2. User clicks "Test connection". Backend POST /api/v1/ssh/test dials with a permissive host-key callback that records (but does not pin) the observed fingerprint, then probes uname -a, arch, git --version, and the agentctl-cache-or-upload status.
  3. UI shows the result with per-step badges and a prominent fingerprint block with a "Trust this host" checkbox that must be ticked before Save is enabled.
  4. On Save, the trusted fingerprint is written to the executor's Config as host_fingerprint. On every subsequent connection (CreateInstance, RecoverInstances, status polling), a mismatch is a hard error — no silent re-pin.

One SSH connection per host, fanned out to per-session forwards

• One SSH connection per (host, user, identity-source), opened lazily and reused across all the executor's sessions on that host. Each session adds:
  • A local net.Listen("tcp", "127.0.0.1:0") to grab a free local port.
  • A goroutine that, for each accepted local connection, opens an SSH channel of type direct-tcpip to 127.0.0.1:<remoteAgentctlPort> and bidirectionally copies bytes.
  • A teardown path that closes the local listener and any in-flight channels on StopInstance.
• SFTP transfers (agentctl upload, credential push) ride the same SSH connection.
• Connection-level keepalives (~30s) so a dropped connection is noticed quickly and triggers reconnect + re-forwarding of all live sessions' ports.
• Failure model: if the shared connection drops mid-session, all sessions on that host go to a "reconnecting" state simultaneously; one reconnect brings them all back.

Workspace lifecycle on the remote

• On CreateInstance, SSH creates <workdir_root>/tasks/<task-dir-name> and resolves the stored profile prepare script through the same scriptengine providers used by Sprites. An empty stored script falls back to the SSH default, which idempotently materializes the primary repository at the task workspace root, runs the repository setup script, and selects the Kandev task branch.
• The resolved prepare script runs on the SSH target with the selected profile shell and the explicitly configured profile/repository environment. Those values are delivered without placing secrets in remote process arguments or persisted metadata. It completes before the per-session agentctl starts. A non-zero exit, timeout, missing primary checkout, or conflicting origin fails the launch; Kandev does not start agentctl or the agent in an empty or ambiguous workspace.
• Managed preparation text is shell-portable. The profile shell may be sh, bash, or zsh, so every parameter expansion that a : follows is braced (${name}:); zsh otherwise parses $name:r as a modifier and silently corrupts fetch refspecs.
• A profile stores the generated prepare script, and a stored script wins over the managed default, so preparation repairs the one fragment Kandev itself generated before that rule existed: an unbraced refs/heads/$name:refs/remotes/origin/$name is braced when the stored script is resolved. Any other expansion in a stored script is left exactly as its author wrote it.
• On subsequent sessions for the same task on the same host, the default preparation path reuses the matching checkout without deleting local commits or untracked work. The profile prepare script may run again because it is a per-session pre-launch hook and must therefore be idempotent when the workspace is shared.
• agentctl starts with the task workspace as its working directory only after preparation succeeds. Additional repository sources are materialized before the agent process starts.
• Profile cleanup scripts run on the SSH target for terminal task/session archive or deletion, with the same workspace placeholders as preparation. Ordinary Stop and backend restart preserve both the workspace and cleanup hook for resume. Cleanup failure is reported in logs but does not prevent controller teardown.
• The task directory remains after controller teardown so a later resume keeps history. Built-in reclamation is never part of the stop path; terminal cleanup hooks may still run there. Built-in reclamation is an opt-in phase of the durable task-resource cleanup job, specified in Remote task-directory reclamation.

agentctl binary upload with content-hash cache

• On CreateInstance, detect the remote platform via uname -s and uname -m, normalize it to a Go GOOS/GOARCH tuple, and resolve the matching agentctl helper via AgentctlResolver.
• Supported remote platforms are linux/amd64, linux/arm64, darwin/arm64, and darwin/amd64. Unsupported platforms fail with a clear error: unsupported remote platform "<platform>" — SSH executor supports linux/{amd64,arm64} and darwin/{amd64,arm64}.
  • Validation status: linux/amd64 and darwin/arm64 have been validated end-to-end on real hosts (darwin/arm64 live over Tailscale against Apple Silicon Macs). linux/arm64 and darwin/amd64 are wired through the full build/upload/launch path and share the same cross-compiled helper, but have not yet been exercised on real ARM Linux / Intel Mac remotes — treat them as supported-but-unverified until someone confirms a task completes on that hardware.
• Runtime bundles include agentctl-linux-amd64, agentctl-linux-arm64, agentctl-darwin-arm64, and agentctl-darwin-amd64; development builds produce them with make -C apps/backend build-agentctl-remote. The darwin helpers are ad-hoc-signed (Go signs darwin/arm64 at link time; make re-signs both via codesign/rcodesign); bundle validation rejects an unsigned agentctl-darwin-arm64 because Apple Silicon refuses to run it.
• Compute SHA256 locally; check ~/.kandev/bin/agentctl.sha256 on the remote via sha256sum. Upload only if missing or mismatched.
• Upload via SFTP to ~/.kandev/bin/agentctl (chmod 755), then write the sha256 sidecar.
• Binary is shared across all tasks and sessions on the host.

Credential push reuses the existing FileUploader seam

• New sshFileUploader implements FileUploader (write file at path with mode) via SFTP.
• The existing executor_sprites_credentials.go selection + upload logic for remote_credentials / remote_auth_secrets is the reference; SSH v1 wires the uploader and writes credentials to the SSH user's home dir.
• GitHub CLI token: same special case as Sprites (run gh auth token locally, inject as env var).

Credential-file conflict policy

The agent descriptor owns the policy for an existing credential file. The transfer layer must not infer file semantics from a path or an executor type.

OpenCode declares a top-level JSON-object merge for auth.json. The transfer layer reads the target before it writes the source file.

The transfer layer preserves providers that exist only in the target. It adds providers that exist only in the source.

When both files contain a provider, the source entry replaces the target entry. This rule preserves the selected host credential as the copy source.

When the target does not exist, the transfer layer writes the source object. Other agent credential files keep the existing replace policy.

The merge requires two valid JSON objects. An unreadable or malformed target causes a copy error and remains unchanged.

The merge runs in the shared credential-upload path. Persistent transports expose target reads only for this policy and keep ordinary writes unchanged.

An isolated transport has no pre-existing user file. It validates the source object and writes it without a target read.

This design follows ADR-2026-09-05-agent-owned-credential-file-conflicts.

Recovery after backend restart

• Persist in ExecutorRunning.Metadata (allow-listed in persistentMetadataKeys): ssh_host, ssh_port, ssh_user, ssh_host_fingerprint, ssh_remote_task_dir, ssh_remote_session_dir, ssh_remote_agentctl_port, ssh_remote_agentctl_pid, ssh_remote_agentctl_instance_id, ssh_local_forward_port, ssh_workdir_root, ssh_proxy_jump, ssh_identity_source, ssh_identity_file.
• The remote controller retains its launch-time instance ID across backend restarts, independently of each recovered local execution's ID. Same-session recovery preserves ssh_remote_agentctl_instance_id and uses it for authenticated HTTP and WebSocket requests. For legacy rows without that field, the previous execution ID (or current ID when no previous execution exists) provides the initial fallback; recovery records it explicitly for subsequent restarts. This identity is session-scoped: sibling sessions never inherit it, and confirmed-dead controller or broker-refresh replacement clears it before launching a fresh controller with its own ID.
• ResumeRemoteInstance re-opens an SSH connection per surviving session using its full (host, port, user, identity_source, identity_file, proxy_jump, host_fingerprint) tuple — no sharing across sessions, so two profiles on the same host with different keys don't get merged. It re-establishes each session's local port forward to its recorded remote port, verifies the remote agentctl is alive (kill -0 <pid> for liveness, then HTTP probe on the forwarded port), and re-binds the stream manager. If the remote process is confirmed gone, the resume preflight yields to normal creation of a fresh instance.
• The liveness probe distinguishes a remote kill -0 exit, which confirms that the recorded process is unavailable, from an SSH session or transport failure, which leaves liveness unknown. A confirmed-absent process clears only the stale session-runtime metadata (remote session directory, controller PID and port, and local-forward details), preserves the SSH connection configuration and remote task directory, and continues through normal instance creation. The new controller receives the existing ACP resume token so it loads the provider conversation. An unknown liveness result remains a hard resume error and does not create a competing controller.

Settings UI

• New apps/web/components/settings/ssh-settings.tsx exporting two cards:
  • SSHConnectionCard — the form + "Test connection" gate. Fields: name, host (or host_alias from ~/.ssh/config), port, user, identity source (agent | file), optional identity_file, optional proxy_jump. Save disabled until a successful test produced a fingerprint and the "Trust this host" checkbox is ticked.
  • SSHSessionsCard — mirrors SpritesInstancesCard and DockerContainersCard. Table columns: task ID, session ID, host (user@host), remote agentctl port, local forward port, uptime, status badge. Data sourced from GET /api/v1/ssh/executors/{id}/sessions. Refreshes every 90 seconds.
• Per-profile settings: workdir_root stored on ExecutorProfile.Config (so the same host can serve different profiles into different roots).
• SSH profile pages include a direct "Connection Settings" action back to the host-level SSH connection page so users can find Test connection / re-trust without knowing the dedicated executor URL.

Edit-with-live-sessions UX

• On save, if the executor has any running sessions, a confirm modal warns: "This executor has N running session(s). They will keep running on the current host. Only new sessions started after save will use the updated config."
• Backend accepts the save unconditionally — the warning is pure UX. Live sessions retain their ssh_host / ssh_port / ssh_user snapshot in ExecutorRunning.Metadata, which is already how recovery works after a restart, so they're insulated from config changes by construction.

Telemetry & errors

• Same step-progress callback used by Sprites (OnProgress) reports the per-step status: "Connecting → Detecting remote OS → Uploading agent controller → Preparing task directory → Starting agent controller → Connecting to agent controller".
• Connection errors surface verbatim (host unreachable, auth failed, host key mismatch, unsupported platform) — no swallowing into generic "executor failed".

Scenarios

• GIVEN a user has filled the SSH executor form and clicked Test Connection, WHEN the test succeeds, THEN the host fingerprint is shown in the UI, the user must tick "Trust this host" before Save is enabled, and on Save the fingerprint is pinned to the executor — silent first-connect TOFU never happens.

• GIVEN a user has a configured + trusted SSH executor and a provider-backed repository, WHEN they start a task with the default SSH prepare script, THEN Kandev materializes the repository at <workdir_root>/tasks/<task-dir-name>, checks out the task branch, runs repository setup, and starts the agent there so git rev-parse --show-toplevel returns that remote task directory.

• GIVEN an SSH profile has a custom prepare script, WHEN a task starts, THEN the script resolves SSH-supported scriptengine placeholders and completes on the target before the per-session agentctl starts in the prepared workspace.

• GIVEN SSH preparation exits non-zero, times out, leaves an attached primary repository absent, or finds a different origin, WHEN a task starts, THEN launch fails with a preparation step error and neither agentctl nor the agent is started for that session.

• GIVEN an SSH task already has local commits or untracked work in its remote checkout, WHEN a later session starts for the same task, THEN default preparation reuses the checkout without resetting or deleting that work before starting the new session controller.

• GIVEN an SSH profile has a cleanup script and its task or session is archived or deleted, WHEN Kandev tears down the terminal execution, THEN the cleanup script runs on the SSH target with the task workspace as {{workspace.path}}; an ordinary Stop or backend restart does not run it.

• GIVEN a user keeps their SSH key in 1Password (no key file on disk) with their agent running, WHEN they select "ssh-agent" as identity source, THEN the connection succeeds without kandev ever touching key material; passphrase-protected keys without an agent are explicitly rejected with a "load this key into ssh-agent first" message.

• GIVEN a host alias sets IdentityAgent to a working agent socket while the backend's $SSH_AUTH_SOCK is empty or stale, WHEN the user selects "ssh-agent", THEN kandev connects through the alias-specific socket.

• GIVEN a target and its ProxyJump alias select different IdentityAgent sockets, WHEN kandev connects through the bastion, THEN each hop uses its own configured agent; a jump without an auth directive falls back to the target identity.

• GIVEN a literal [user@]host[:port] ProxyJump has no matching host stanza, WHEN kandev connects through it, THEN the jump inherits the target identity.

• GIVEN a task already has session A running on an SSH host, WHEN the user opens a second session B on the same task, THEN kandev reuses the existing task dir (no re-clone), launches a second agentctl on a different remote port with its own local SSH forward over the same SSH connection, and both sessions stream independently against the same worktree.

• GIVEN two different tasks are running against the same SSH host, WHEN the kandev backend restarts, THEN ResumeRemoteInstance opens one SSH connection to the host, re-forwards each session's recorded remote port through that single connection, verifies kill -0 <pid> on each, and the UI reconnects without losing in-progress turns.

• GIVEN the user opens the SSH executor's settings page, WHEN they scroll to "Active sessions", THEN they see each running session on this host (task, session ID, user@host, remote port, local fwd port, uptime, status) — same shape as the Docker containers table and Sprites instances table.

• GIVEN a user is editing an SSH executor profile, WHEN they need to re-test or re-trust the host, THEN the profile page exposes a direct Connection Settings action that opens the SSH connection page with the Test connection button.

• GIVEN a user edits the SSH executor's host field while two sessions are running, WHEN they click Save, THEN a confirm modal warns them the running sessions will keep going on the old host and only new sessions will use the new host. On confirm, the executor row updates and the live sessions are unaffected.

• GIVEN a host is reachable only via a bastion, WHEN the user's ~/.ssh/config has ProxyJump bastion.example.com for that host alias, THEN kandev parses the config, dials through the bastion, and the rest of the flow is unchanged.

• GIVEN the remote host's key changes (re-imaged, MITM, key rotation), WHEN kandev next connects, THEN the connection is refused with a host key changed: expected <pinned> got <new> error, the executor status banner shows "Host key changed", and the user must re-run Test Connection and re-tick "Trust this host" in settings — no silent re-pin.

Out of scope (v1)

• Remote Windows. SSH transport to Windows is out of scope.
• Password / keyboard-interactive auth. Keys + agent only.
• Passphrase-protected keys handled inside kandev. User must load the key into ssh-agent.
• Chained ProxyJump. Single bastion only; multi-hop deferred.
• Pushing local uncommitted changes to the remote. Like Sprites, the remote clones from the git URL — we do not rsync the user's working tree.
• Auto-installing dependencies on the remote. If the user's prepare script needs node/go/python/docker, those must already be on the host. We do not bootstrap toolchains.
• Multi-user-per-host with isolation guarantees. v1 assumes the single SSH user on each host is trusted with everything that user can see; we do not sandbox sessions from each other beyond per-session runtime dirs.
• A background sweep of pre-existing orphaned task dirs. Reclaiming the remote <workdir_root>/tasks/<task-dir-name>/ when a task reaches a terminal outcome is no longer out of scope: it is specified in Remote task-directory reclamation, which supersedes this bullet. What remains out of scope is a scheduled sweep of directories left behind by tasks Kandev no longer has a row for.
• GUI editor for ~/.ssh/config. We read existing config; users edit it in their text editor.
• Bring-your-own-agentctl (skip upload, point at an existing install). Always uploaded + content-hash-cached in v1.
• Kubernetes / pod-exec transport. The k8s executor type is its own future feature.

Future scope

• Hardware validation of linux/arm64 and darwin/amd64 remotes. Both are wired through the full build/upload/launch path and share the tested cross-compiled helper, but neither has been exercised on real hardware yet (see the Validation status note in the support matrix). They stay marked supported-but-unverified until a task is confirmed end-to-end on an ARM Linux host and an Intel Mac.
• Auto-provision mode: kandev runs a one-shot install script on the remote (apt/yum/brew) to set up node/docker/etc. before the first task — turns a bare Hetzner box into a ready-to-use kandev host.
• Multi-user-per-host with per-task Linux user impersonation (sudo -u worker-N) for genuine isolation on shared boxes.
• Chained ProxyJump and ProxyCommand support for esoteric setups.
• "Push working tree" mode for users who want to test uncommitted changes against a remote (rsync the worktree instead of cloning).
• Pooled host capacity: configure 3 SSH hosts as a pool, kandev round-robins / load-balances tasks across them.
• SSH-tunneled MCP servers: expose a user's local MCP servers to a remote-running agent via reverse forward.
• Orphan task-dir sweeper as a backend background job, for directories with no surviving task row. The terminal-outcome reclamation itself is now specified rather than deferred.
• Byte-for-byte live prepare output streaming. SSH reports bounded preparation progress and the final diagnostic tail, but matching Sprites' continuous stdout/stderr streaming can be added later.

Open questions

(All v1 design decisions resolved during planning. None outstanding at spec-approval time.)