Files
kandev/docs/plans/ssh-dead-runtime-resume/plan.md
T
2026-09-03 22:56:03 +00:00

92 lines
3.7 KiB
Markdown

---
created: 2026-09-03
status: done
requirements:
- REQ-EXECUTORS-SSH-EXECUTOR-001
system_design:
- ../../specs/executors/system-design/ssh-executor.md
legacy_specs: []
---
# Implementation Plan: SSH Dead Runtime Resume
## Overview
Issue [#3330](https://github.com/kdlbs/kandev/issues/3330) reports that an SSH
session cannot resume after Kandev intentionally stops its remote controller.
A focused lifecycle test reproduces the failure: the confirmed-dead PID error
escapes the SSH resume preflight before normal instance creation can replace the
runtime.
The correction classifies the remote process probe result and converts only a
confirmed-absent controller into fresh runtime creation. SSH transport and
session failures remain fail-closed.
## Scope
### In scope
- Preserve the task workspace and ACP provider conversation when replacing a
confirmed-dead SSH runtime.
- Clear stale controller metadata before normal SSH instance creation.
- Distinguish a remote non-zero process probe from an indeterminate SSH probe.
- Add focused regression coverage for replacement and fail-closed behavior.
### Out of scope
- Idle-runtime eviction or a background reaper.
- Changes to stop semantics, provider retry policy, or non-SSH executors.
- Database, WebSocket, API, or frontend changes.
## Technical approach
- In `executor_ssh_operations.go`, add a process-liveness probe that returns an
error when SSH cannot establish or complete the remote command. Keep the
existing boolean helper for best-effort status and startup polling callers.
- In `executor_ssh.go`, use the classified probe during resume. On a confirmed
absent process, close the stale SSH client, clear session-runtime metadata
with `clearSSHResumeRuntimeMetadata`, and return successfully so
`CreateInstance` launches a new controller. Preserve the remote task
directory, executor connection metadata, and ACP resume token.
- Keep existing hard-error behavior for connection, authentication, host trust,
and indeterminate process-liveness failures.
## Tests
- `AC-EXECUTORS-SSH-EXECUTOR-001.9`: update
`TestSSHExecutorResumeRemoteInstance/dead_remote_pid` in
`executor_ssh_stop_resume_test.go` to require successful fallback, stale
runtime metadata removal, and preserved task-directory metadata.
- `AC-EXECUTORS-SSH-EXECUTOR-001.10`: add focused probe coverage in
`executor_ssh_operations_remote_test.go` and resume coverage proving a closed
SSH client remains an error.
## Work orders
- [x] [Task 01: Recover from a dead SSH runtime](task-01-recover-dead-ssh-runtime.md)
## Verification results
- RED: `TestSSHExecutorResumeRemoteInstance/dead_remote_pid_falls_back_to_fresh_create`
failed with `ssh resume: agentctl pid 4242 not alive on remote`.
- RED: `TestSSHExecutorResumeRemoteInstance/malformed_remote_pid_remains_fail_closed`
failed because malformed runtime identity incorrectly returned `nil` after
the initial fallback change.
- GREEN: `cd apps/backend && go test -race
./internal/agent/runtime/lifecycle -run
'Test(SSHExecutorResumeRemoteInstance|ProbeRemoteAgentctlLiveness)$'
-count=1` passed with 12 tests.
- Compatibility: the existing liveness and lifecycle preflight tests passed
with 4 tests.
- `python3 scripts/lint-spec-files.test.py` passed with 30 tests.
- `python3 scripts/lint-spec-files.py --all` passed.
- `git diff --check` and `gofmt -l` passed.
## Risks
- Treating all probe errors as absence could start a second controller during a
transient SSH failure. The classified probe must accept only a completed
remote command with a non-zero status as confirmed absence.
- Clearing the remote task directory would lose workspace continuity. The
existing metadata helper must preserve it.