kandev/docs/specs/executors/system-design/ssh-executor.md

---
status: draft
system: executors
requirements:
  - REQ-EXECUTORS-SSH-EXECUTOR-001
created: 2026-05-16
owners:
  - tbd
---
# SSH Executor System Design

## Purpose and boundaries

This design preserves the technical source detail for `REQ-EXECUTORS-SSH-EXECUTOR-001` during migration.

## Requirement mapping

| Requirement | Design section |
| --- | --- |
| `REQ-EXECUTORS-SSH-EXECUTOR-001` | [Migrated source detail](#migrated-source-detail) |
| `AC-EXECUTORS-SSH-EXECUTOR-001.11` and `AC-EXECUTORS-SSH-EXECUTOR-001.12` | [Credential-file conflict policy](#credential-file-conflict-policy) |

## Migrated source detail

## Why

Today the only ways to run an agent are: (1) locally on the user's own machine (`local_pc`, `local_docker`), or (2) in a Sprites-hosted ephemeral sandbox. There is no option for **"run on a Linux box I already own"** — a user's VPS, homelab server, beefy desktop, EC2/Hetzner instance, or shared team dev box.

That gap matters because:

- **Cost control.** Sprites are convenient but metered; a $5/mo VPS or an already-paid-for desktop is effectively free capacity for long-running tasks.
- **Hardware control.** Users who need a specific GPU, a lot of RAM, a particular CPU arch, or pre-installed proprietary tools (databases, CAD, internal services) cannot get there with Sprites.
- **Privacy / compliance.** Some users cannot send source code through a third-party cloud sandbox but are fine running an agent on a server they fully control.
- **"Move my Claude Code to the cloud" use case.** Running on a remote host means closing the laptop lid doesn't kill the agent, and the same task is reachable from a phone, tablet, or any browser.

SSH is the lowest-common-denominator transport for reaching a remote Linux box. If we can put an agent on anything reachable over SSH — VPS, homelab, EC2, raspberry pi, lab machine, jump-host-protected private box — we unlock all of the above in one feature without inventing a kandev-specific cloud.

## What (v1)

A new executor type `ssh` joins `local_pc` / `local_docker` / `sprites` in the existing executor framework. From the user's perspective: configure an SSH target once in settings, then pick it as the executor for any task; tasks run on the remote box exactly as they would on a Sprite.

### Remote layout: one task workspace per task

Kandev creates one remote workspace per task. The primary repository owns the workspace root so a
single-repository agent starts inside a Git checkout; additional repository branches use direct
child directories:

```
~/.kandev/tasks/
├── <task-dir-name-A>/
│   ├── .git/               # primary repository on the task's feature branch
│   └── <repo-2>-<branch>/  # additional repository or branch
└── <task-dir-name-B>/
    └── .git/               # another task's primary repository
```

The workspace root defaults to `~/.kandev/` on the remote and is **configurable per profile** (so
one host can serve `~/.kandev-team-a/tasks/` and `~/.kandev-team-b/tasks/` for different profiles on
the same SSH user). Concurrent sessions on the same task share the task workspace; concurrent tasks
on the same host get different `<task-dir-name>` siblings.

### Per-session, not per-task: agentctl process, remote port, local forward

Execution is keyed per session (matching local + Sprites). Each session on an SSH host gets:

- Its **own agentctl process**, launched on a unique remote port chosen by binding to `:0` on the remote and reading back the chosen port.
- Its **own SSH local port forward**: a fresh `127.0.0.1:0` listener that the kandev backend dials for HTTP + WebSocket streams to that session's agentctl. All forwards ride the single shared SSH connection for the host (see below).
- A **session-scoped runtime dir** at `<workdir_root>/tasks/<task-dir-name>/.kandev/sessions/<session-id>/` for the agentctl PID file, port file, and log — kept under the task dir so cleanup follows the task, not orphaned across the filesystem.

Multiple sessions in the *same* task share the same worktree on disk (same files, same branch state); they have independent agentctl processes and independent UI streams. Multiple sessions across *different* tasks on the same host have independent task dirs as well.

### Auth, connectivity, and host-key trust

- **Auth: SSH key file + ssh-agent.** Authentication via `golang.org/x/crypto/ssh` uses either an explicit `IdentityFile` or an ssh-agent. A per-host OpenSSH `IdentityAgent` overrides the backend process's `$SSH_AUTH_SOCK`, covering 1Password / Secretive / Yubikey / forwarded-agent users whose agent socket differs from the desktop process environment. `IdentityAgent none` disables agent authentication for that host. **Passphrase-protected keys are not handled in kandev** — users must load them into `ssh-agent` themselves. Password and keyboard-interactive auth are not supported in v1.

- **OpenSSH config inheritance.** When a `host_alias` is configured, kandev resolves user and system OpenSSH client configuration to inherit `HostName`, `Port`, `User`, `IdentityAgent`, `IdentityFile`, and one `ProxyJump`. `IdentityAgent` supports `~`, `${VAR}`, whole-value `$VAR`, `SSH_AUTH_SOCK`, and percent tokens `%%`, `%d`, `%h`, `%i`, `%j`, `%k`, `%L`, `%l`, `%n`, `%p`, `%r`, and `%u`; `%C` is not supported. A user whose terminal already does `ssh prod` can paste `prod` into kandev and have it just work.

- **Connectivity: direct, ProxyJump, mesh-VPN.** Direct TCP to `host:port` (default 22). `ProxyJump` (single bastion in v1; chained jumps deferred) implemented natively via the Go SSH client. Tailscale / WireGuard / corporate VPN: "just works" when the kandev backend process is on the same network namespace.

- **Create flow: test-connection gate + explicit fingerprint trust.** The Sprites executor already gates creation behind a "Test Connection" step. The SSH executor adopts the same pattern and extends it with an explicit host-key trust step:
  1. **User fills the form.**
  2. **User clicks "Test connection".** Backend `POST /api/v1/ssh/test` dials with a permissive host-key callback that records (but does not pin) the observed fingerprint, then probes `uname -a`, arch, `git --version`, and the agentctl-cache-or-upload status.
  3. **UI shows the result** with per-step badges and a prominent fingerprint block with a **"Trust this host"** checkbox that must be ticked before Save is enabled.
  4. **On Save**, the trusted fingerprint is written to the executor's `Config` as `host_fingerprint`. On every subsequent connection (CreateInstance, RecoverInstances, status polling), a mismatch is a hard error — no silent re-pin.

### One SSH connection per host, fanned out to per-session forwards

- One SSH connection per `(host, user, identity-source)`, opened lazily and reused across all the executor's sessions on that host. Each session adds:
  - A local `net.Listen("tcp", "127.0.0.1:0")` to grab a free local port.
  - A goroutine that, for each accepted local connection, opens an SSH channel of type `direct-tcpip` to `127.0.0.1:<remoteAgentctlPort>` and bidirectionally copies bytes.
  - A teardown path that closes the local listener and any in-flight channels on `StopInstance`.
- SFTP transfers (agentctl upload, credential push) ride the same SSH connection.
- Connection-level keepalives (~30s) so a dropped connection is noticed quickly and triggers reconnect + re-forwarding of all live sessions' ports.
- **Failure model:** if the shared connection drops mid-session, all sessions on that host go to a "reconnecting" state simultaneously; one reconnect brings them all back.

### Workspace lifecycle on the remote

- On `CreateInstance`, SSH creates `<workdir_root>/tasks/<task-dir-name>` and resolves the stored
  profile prepare script through the same scriptengine providers used by Sprites. An empty stored
  script falls back to the SSH default, which idempotently materializes the primary repository at
  the task workspace root, runs the repository setup script, and selects the Kandev task branch.
- The resolved prepare script runs on the SSH target with the selected profile shell and the
  explicitly configured profile/repository environment. Those values are delivered without placing
  secrets in remote process arguments or persisted metadata. It completes before the per-session
  `agentctl` starts. A non-zero exit, timeout, missing primary checkout, or conflicting `origin`
  fails the launch; Kandev does not
  start `agentctl` or the agent in an empty or ambiguous workspace.
- Managed preparation text is shell-portable. The profile shell may be `sh`, `bash`, or `zsh`, so
  every parameter expansion that a `:` follows is braced (`${name}:`); zsh otherwise parses `$name:r`
  as a modifier and silently corrupts fetch refspecs.
- A profile stores the generated prepare script, and a stored script wins over the managed default,
  so preparation repairs the one fragment Kandev itself generated before that rule existed: an
  unbraced `refs/heads/$name:refs/remotes/origin/$name` is braced when the stored script is
  resolved. Any other expansion in a stored script is left exactly as its author wrote it.
- On subsequent sessions for the same task on the same host, the default preparation path reuses the
  matching checkout without deleting local commits or untracked work. The profile prepare script may
  run again because it is a per-session pre-launch hook and must therefore be idempotent when the
  workspace is shared.
- `agentctl` starts with the task workspace as its working directory only after preparation succeeds.
  Additional repository sources are materialized before the agent process starts.
- Profile cleanup scripts run on the SSH target for terminal task/session archive or deletion, with
  the same workspace placeholders as preparation. Ordinary Stop and backend restart preserve both
  the workspace and cleanup hook for resume. Cleanup failure is reported in logs but does not prevent
  controller teardown.
- The task directory remains after controller teardown so a later resume keeps history. Built-in
  reclamation is never part of the stop path; terminal cleanup hooks may still run there. Built-in
  reclamation is an opt-in phase of the durable task-resource cleanup job, specified in
  [Remote task-directory reclamation](requirements/remote-task-directory-reclamation.md).

### agentctl binary upload with content-hash cache

- On `CreateInstance`, detect the remote platform via `uname -s` and `uname -m`, normalize it to a Go `GOOS/GOARCH` tuple, and resolve the matching agentctl helper via `AgentctlResolver`.
- Supported remote platforms are `linux/amd64`, `linux/arm64`, `darwin/arm64`, and `darwin/amd64`. Unsupported platforms fail with a clear error: `unsupported remote platform "<platform>" — SSH executor supports linux/{amd64,arm64} and darwin/{amd64,arm64}`.
  - **Validation status:** `linux/amd64` and `darwin/arm64` have been validated end-to-end on real hosts (`darwin/arm64` live over Tailscale against Apple Silicon Macs). `linux/arm64` and `darwin/amd64` are wired through the full build/upload/launch path and share the same cross-compiled helper, but have **not** yet been exercised on real ARM Linux / Intel Mac remotes — treat them as supported-but-unverified until someone confirms a task completes on that hardware.
- Runtime bundles include `agentctl-linux-amd64`, `agentctl-linux-arm64`, `agentctl-darwin-arm64`, and `agentctl-darwin-amd64`; development builds produce them with `make -C apps/backend build-agentctl-remote`. The darwin helpers are ad-hoc-signed (Go signs darwin/arm64 at link time; `make` re-signs both via `codesign`/`rcodesign`); bundle validation rejects an unsigned `agentctl-darwin-arm64` because Apple Silicon refuses to run it.
- Compute SHA256 locally; check `~/.kandev/bin/agentctl.sha256` on the remote via `sha256sum`. Upload only if missing or mismatched.
- Upload via SFTP to `~/.kandev/bin/agentctl` (chmod 755), then write the sha256 sidecar.
- Binary is shared across all tasks and sessions on the host.

### Credential push reuses the existing `FileUploader` seam

- New `sshFileUploader` implements `FileUploader` (write file at path with mode) via SFTP.
- The existing `executor_sprites_credentials.go` selection + upload logic for `remote_credentials` / `remote_auth_secrets` is the reference; SSH v1 wires the uploader and writes credentials to the SSH user's home dir.
- GitHub CLI token: same special case as Sprites (run `gh auth token` locally, inject as env var).

### Credential-file conflict policy

The agent descriptor owns the policy for an existing credential file. The transfer layer must not infer file semantics from a path or an executor type.

OpenCode declares a top-level JSON-object merge for `auth.json`. The transfer layer reads the target before it writes the source file.

The transfer layer preserves providers that exist only in the target. It adds providers that exist only in the source.

When both files contain a provider, the source entry replaces the target entry. This rule preserves the selected host credential as the copy source.

When the target does not exist, the transfer layer writes the source object. Other agent credential files keep the existing replace policy.

The merge requires two valid JSON objects. An unreadable or malformed target causes a copy error and remains unchanged.

The merge runs in the shared credential-upload path. Persistent transports expose target reads only for this policy and keep ordinary writes unchanged.

An isolated transport has no pre-existing user file. It validates the source object and writes it without a target read.

This design follows [ADR-2026-09-05-agent-owned-credential-file-conflicts](../../../decisions/2026-09-05-agent-owned-credential-file-conflicts.md).

### Recovery after backend restart

- Persist in `ExecutorRunning.Metadata` (allow-listed in `persistentMetadataKeys`): `ssh_host`, `ssh_port`, `ssh_user`, `ssh_host_fingerprint`, `ssh_remote_task_dir`, `ssh_remote_session_dir`, `ssh_remote_agentctl_port`, `ssh_remote_agentctl_pid`, `ssh_remote_agentctl_instance_id`, `ssh_local_forward_port`, `ssh_workdir_root`, `ssh_proxy_jump`, `ssh_identity_source`, `ssh_identity_file`.
- The remote controller retains its launch-time instance ID across backend restarts, independently of each recovered local execution's ID. Same-session recovery preserves `ssh_remote_agentctl_instance_id` and uses it for authenticated HTTP and WebSocket requests. For legacy rows without that field, the previous execution ID (or current ID when no previous execution exists) provides the initial fallback; recovery records it explicitly for subsequent restarts. This identity is session-scoped: sibling sessions never inherit it, and confirmed-dead controller or broker-refresh replacement clears it before launching a fresh controller with its own ID.
- `ResumeRemoteInstance` re-opens an SSH connection per surviving session using its full `(host, port, user, identity_source, identity_file, proxy_jump, host_fingerprint)` tuple — no sharing across sessions, so two profiles on the same host with different keys don't get merged. It re-establishes each session's local port forward to its recorded remote port, verifies the remote agentctl is alive (`kill -0 <pid>` for liveness, then HTTP probe on the forwarded port), and re-binds the stream manager. If the remote process is confirmed gone, the resume preflight yields to normal creation of a fresh instance.
- The liveness probe distinguishes a remote `kill -0` exit, which confirms that the recorded process is unavailable, from an SSH session or transport failure, which leaves liveness unknown. A confirmed-absent process clears only the stale session-runtime metadata (remote session directory, controller PID and port, and local-forward details), preserves the SSH connection configuration and remote task directory, and continues through normal instance creation. The new controller receives the existing ACP resume token so it loads the provider conversation. An unknown liveness result remains a hard resume error and does not create a competing controller.

### Settings UI

- New `apps/web/components/settings/ssh-settings.tsx` exporting two cards:
  - **`SSHConnectionCard`** — the form + "Test connection" gate. Fields: name, `host` (or `host_alias` from `~/.ssh/config`), `port`, `user`, identity source (`agent` | `file`), optional `identity_file`, optional `proxy_jump`. Save disabled until a successful test produced a fingerprint and the "Trust this host" checkbox is ticked.
  - **`SSHSessionsCard`** — mirrors `SpritesInstancesCard` and `DockerContainersCard`. Table columns: task ID, session ID, host (`user@host`), remote agentctl port, local forward port, uptime, status badge. Data sourced from `GET /api/v1/ssh/executors/{id}/sessions`. Refreshes every 90 seconds.
- Per-profile settings: `workdir_root` stored on `ExecutorProfile.Config` (so the same host can serve different profiles into different roots).
- SSH profile pages include a direct "Connection Settings" action back to the host-level SSH connection page so users can find Test connection / re-trust without knowing the dedicated executor URL.

### Edit-with-live-sessions UX

- On save, if the executor has any running sessions, a confirm modal warns: *"This executor has N running session(s). They will keep running on the current host. Only new sessions started after save will use the updated config."*
- Backend accepts the save unconditionally — the warning is pure UX. Live sessions retain their `ssh_host` / `ssh_port` / `ssh_user` snapshot in `ExecutorRunning.Metadata`, which is already how recovery works after a restart, so they're insulated from config changes by construction.

### Telemetry & errors

- Same step-progress callback used by Sprites (`OnProgress`) reports the per-step status: "Connecting → Detecting remote OS → Uploading agent controller → Preparing task directory → Starting agent controller → Connecting to agent controller".
- Connection errors surface verbatim (host unreachable, auth failed, host key mismatch, unsupported platform) — no swallowing into generic "executor failed".

## Scenarios

- **GIVEN** a user has filled the SSH executor form and clicked Test Connection, **WHEN** the test succeeds, **THEN** the host fingerprint is shown in the UI, the user must tick "Trust this host" before Save is enabled, and on Save the fingerprint is pinned to the executor — silent first-connect TOFU never happens.

- **GIVEN** a user has a configured + trusted SSH executor and a provider-backed repository, **WHEN**
  they start a task with the default SSH prepare script, **THEN** Kandev materializes the repository
  at `<workdir_root>/tasks/<task-dir-name>`, checks out the task branch, runs repository setup, and
  starts the agent there so `git rev-parse --show-toplevel` returns that remote task directory.

- **GIVEN** an SSH profile has a custom prepare script, **WHEN** a task starts, **THEN** the script
  resolves SSH-supported scriptengine placeholders and completes on the target before the
  per-session `agentctl` starts in the prepared workspace.

- **GIVEN** SSH preparation exits non-zero, times out, leaves an attached primary repository absent,
  or finds a different `origin`, **WHEN** a task starts, **THEN** launch fails with a preparation step
  error and neither `agentctl` nor the agent is started for that session.

- **GIVEN** an SSH task already has local commits or untracked work in its remote checkout, **WHEN** a
  later session starts for the same task, **THEN** default preparation reuses the checkout without
  resetting or deleting that work before starting the new session controller.

- **GIVEN** an SSH profile has a cleanup script and its task or session is archived or deleted,
  **WHEN** Kandev tears down the terminal execution, **THEN** the cleanup script runs on the SSH target
  with the task workspace as `{{workspace.path}}`; an ordinary Stop or backend restart does not run it.

- **GIVEN** a user keeps their SSH key in 1Password (no key file on disk) with their agent running, **WHEN** they select "ssh-agent" as identity source, **THEN** the connection succeeds without kandev ever touching key material; passphrase-protected keys without an agent are explicitly rejected with a "load this key into ssh-agent first" message.

- **GIVEN** a host alias sets `IdentityAgent` to a working agent socket while the backend's `$SSH_AUTH_SOCK` is empty or stale, **WHEN** the user selects "ssh-agent", **THEN** kandev connects through the alias-specific socket.

- **GIVEN** a target and its ProxyJump alias select different `IdentityAgent` sockets, **WHEN** kandev connects through the bastion, **THEN** each hop uses its own configured agent; a jump without an auth directive falls back to the target identity.

- **GIVEN** a literal `[user@]host[:port]` ProxyJump has no matching host stanza, **WHEN** kandev connects through it, **THEN** the jump inherits the target identity.

- **GIVEN** a task already has session A running on an SSH host, **WHEN** the user opens a second session B on the same task, **THEN** kandev reuses the existing task dir (no re-clone), launches a second agentctl on a different remote port with its own local SSH forward over the same SSH connection, and both sessions stream independently against the same worktree.

- **GIVEN** two different tasks are running against the same SSH host, **WHEN** the kandev backend restarts, **THEN** `ResumeRemoteInstance` opens one SSH connection to the host, re-forwards each session's recorded remote port through that single connection, verifies `kill -0 <pid>` on each, and the UI reconnects without losing in-progress turns.

- **GIVEN** the user opens the SSH executor's settings page, **WHEN** they scroll to "Active sessions", **THEN** they see each running session on this host (task, session ID, user@host, remote port, local fwd port, uptime, status) — same shape as the Docker containers table and Sprites instances table.

- **GIVEN** a user is editing an SSH executor profile, **WHEN** they need to re-test or re-trust the host, **THEN** the profile page exposes a direct Connection Settings action that opens the SSH connection page with the Test connection button.

- **GIVEN** a user edits the SSH executor's `host` field while two sessions are running, **WHEN** they click Save, **THEN** a confirm modal warns them the running sessions will keep going on the old host and only new sessions will use the new host. On confirm, the executor row updates and the live sessions are unaffected.

- **GIVEN** a host is reachable only via a bastion, **WHEN** the user's `~/.ssh/config` has `ProxyJump bastion.example.com` for that host alias, **THEN** kandev parses the config, dials through the bastion, and the rest of the flow is unchanged.

- **GIVEN** the remote host's key changes (re-imaged, MITM, key rotation), **WHEN** kandev next connects, **THEN** the connection is refused with a `host key changed: expected <pinned> got <new>` error, the executor status banner shows "Host key changed", and the user must re-run Test Connection and re-tick "Trust this host" in settings — no silent re-pin.

## Out of scope (v1)

- **Remote Windows.** SSH transport to Windows is out of scope.
- **Password / keyboard-interactive auth.** Keys + agent only.
- **Passphrase-protected keys handled inside kandev.** User must load the key into `ssh-agent`.
- **Chained ProxyJump.** Single bastion only; multi-hop deferred.
- **Pushing local uncommitted changes to the remote.** Like Sprites, the remote clones from the git URL — we do not rsync the user's working tree.
- **Auto-installing dependencies on the remote.** If the user's prepare script needs `node`/`go`/`python`/`docker`, those must already be on the host. We do not bootstrap toolchains.
- **Multi-user-per-host with isolation guarantees.** v1 assumes the single SSH user on each host is trusted with everything that user can see; we do not sandbox sessions from each other beyond per-session runtime dirs.
- **A background sweep of pre-existing orphaned task dirs.** Reclaiming the remote `<workdir_root>/tasks/<task-dir-name>/` when a task reaches a terminal outcome is no longer out of scope: it is specified in [Remote task-directory reclamation](requirements/remote-task-directory-reclamation.md), which supersedes this bullet. What remains out of scope is a scheduled sweep of directories left behind by tasks Kandev no longer has a row for.
- **GUI editor for `~/.ssh/config`.** We read existing config; users edit it in their text editor.
- **Bring-your-own-agentctl** (skip upload, point at an existing install). Always uploaded + content-hash-cached in v1.
- **Kubernetes / pod-exec transport.** The `k8s` executor type is its own future feature.

## Future scope

- **Hardware validation of `linux/arm64` and `darwin/amd64` remotes.** Both are wired through the full build/upload/launch path and share the tested cross-compiled helper, but neither has been exercised on real hardware yet (see the Validation status note in the support matrix). They stay marked supported-but-unverified until a task is confirmed end-to-end on an ARM Linux host and an Intel Mac.
- **Auto-provision mode**: kandev runs a one-shot install script on the remote (apt/yum/brew) to set up node/docker/etc. before the first task — turns a bare Hetzner box into a ready-to-use kandev host.
- **Multi-user-per-host** with per-task Linux user impersonation (`sudo -u worker-N`) for genuine isolation on shared boxes.
- **Chained ProxyJump** and `ProxyCommand` support for esoteric setups.
- **"Push working tree" mode** for users who want to test uncommitted changes against a remote (rsync the worktree instead of cloning).
- **Pooled host capacity**: configure 3 SSH hosts as a pool, kandev round-robins / load-balances tasks across them.
- **SSH-tunneled MCP servers**: expose a user's local MCP servers to a remote-running agent via reverse forward.
- **Orphan task-dir sweeper** as a backend background job, for directories with no surviving task
  row. The terminal-outcome reclamation itself is now specified rather than deferred.
- **Byte-for-byte live prepare output streaming.** SSH reports bounded preparation progress and the
  final diagnostic tail, but matching Sprites' continuous stdout/stderr streaming can be added later.

## Open questions

(All v1 design decisions resolved during planning. None outstanding at spec-approval time.)