Files
kandev/docs/specs/executors/system-design/remote-task-directory-reclamation.md
T
Henry 8f21873d45 feat(ssh): reclaim remote task directories on terminal archive or delete (#3000)
Co-authored-by: nova28 <17953305+nova28@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Carlos Florencio <carlosmflorencio@gmail.com>
2026-08-25 06:56:59 +00:00

296 lines
15 KiB
Markdown

---
status: current
system: executors
requirements:
- REQ-SSH-TASKDIR-RECLAMATION-001
- REQ-SSH-TASKDIR-RECLAMATION-002
- REQ-SSH-TASKDIR-RECLAMATION-003
- REQ-SSH-TASKDIR-RECLAMATION-004
- REQ-SSH-TASKDIR-RECLAMATION-005
- REQ-SSH-TASKDIR-RECLAMATION-006
- REQ-SSH-TASKDIR-RECLAMATION-007
---
# Remote Task-Directory Reclamation System Design
## Purpose and boundaries
Reclamation is a task-scoped step inside the existing durable task-resource
cleanup job, not a new subsystem and not a new background sweeper. The job
already owns the terminal-outcome trigger, the ordering against runtime stops,
the retry-with-backoff schedule, the cancellation path when an archive is undone,
and the durable `task_resource_cleanup_jobs` row that surfaces failure. This
design adds one phase to that job and one connection-level helper to the SSH
executor package.
The boundary that matters most is *where the removal does not go*.
`SSHExecutor.StopInstance` is the obvious place and is the wrong one, for three
independent reasons:
1. **Scope mismatch.** `StopInstance` is per session; a task directory is per
task. A three-session task would evaluate removal three times, and the first
session's removal would delete the directory out from under the other two
while they are still stopping.
2. **Error laundering.** `stopTaskRuntimeTargets` deliberately swallows stop
errors for targets already marked terminal
(`if target.terminal { ...; continue }`) and for `runtimeStopAlreadyComplete`.
A removal failure raised from `StopInstance` would be discarded on exactly the
paths this feature runs on, producing the false-success shape that
REQ-SSH-TASKDIR-RECLAMATION-005 exists to prevent.
3. **Missing knowledge.** Ownership (REQ-SSH-TASKDIR-RECLAMATION-003) is a
question about tasks, environments, and sibling rows. The executor has an
instance; the task service has the graph.
`StopInstance` therefore keeps its current contract unchanged: it never removes
the task directory. That is what makes AC-SSH-TASKDIR-RECLAMATION-004.1 and
AC-SSH-TASKDIR-RECLAMATION-004.2 structural rather than conditional — an ordinary
stop and a backend shutdown never create a cleanup job, so there is no code path
from them to a removal.
## Requirement mapping
| Requirement | Design section |
| --- | --- |
| `REQ-SSH-TASKDIR-RECLAMATION-001` | [Control flow](#control-flow) |
| `REQ-SSH-TASKDIR-RECLAMATION-002` | [Safety probe](#safety-probe) |
| `REQ-SSH-TASKDIR-RECLAMATION-003` | [Ownership resolution](#ownership-resolution) |
| `REQ-SSH-TASKDIR-RECLAMATION-004` | [Purpose and boundaries](#purpose-and-boundaries) |
| `REQ-SSH-TASKDIR-RECLAMATION-005` | [Failure and recovery](#failure-and-recovery) |
| `REQ-SSH-TASKDIR-RECLAMATION-006` | [Data and contracts](#data-and-contracts) |
| `REQ-SSH-TASKDIR-RECLAMATION-007` | [Control flow](#control-flow) |
## Components and responsibilities
- **`lifecycle` package (`apps/backend/internal/agent/runtime/lifecycle`).** Owns
a new `SSHTaskDirReclaimer` that, given a connection descriptor and a target
directory, opens its own SSH connection, runs the read-only safety probes, and
performs the guarded removal. It reuses `ResolveSSHTarget`, the existing
fingerprint pinning, `runSSHCommand`, and `shellQuote`. It does not touch
`SSHExecutor` session state, because by the time it runs every session has been
stopped and its connection closed.
- **`task/service` package.** Owns the new cleanup-job phase: capturing the
reclamation targets into the job snapshot at prepare time, resolving ownership,
invoking the reclaimer, and folding the outcome into the job's error set.
- **`orchestrator/executor` package.** Owns projecting the new per-profile
setting into launch metadata as an authoritative key.
- **Web settings.** Owns the per-profile toggle and its blast-radius copy.
## Data and contracts
### Per-profile setting
A new executor-profile config key `ssh_reclaim_task_dir` (string `"true"` to
enable; anything else, including absent and empty, disables). It is added to
`profileConfigAuthoritativeKeys` in
`apps/backend/internal/orchestrator/executor/executor_state.go`, not to
`profileConfigPassthroughKeys`. That distinction is the mechanism behind
AC-SSH-TASKDIR-RECLAMATION-006.3: authoritative keys are written unconditionally
from the profile, so a task that supplies `ssh_reclaim_task_dir` in its own
metadata is overwritten rather than honoured. This matches the existing treatment
of `ssh_workdir_root` and `ssh_shell`, which are authoritative for the same
reason — they are redirect vectors.
A matching `MetadataKeySSHReclaimTaskDir` is added to `persistentMetadataKeys` so
the value survives into `ExecutorRunning.Metadata` and is readable by the cleanup
job after a backend restart, alongside the connection tuple that
`ResumeRemoteInstance` already relies on.
Default disabled satisfies AC-SSH-TASKDIR-RECLAMATION-006.1 with no migration:
existing profile rows have no such key, and an absent key is disabled.
### Reclamation target
Captured into `taskResourceCleanupSnapshot` as a new field so the job is
self-contained and survives a restart:
| Field | Meaning |
| --- | --- |
| `host`, `port`, `user` | Connection identity |
| `host_fingerprint` | Pinned fingerprint; a mismatch fails the attempt |
| `identity_source`, `identity_file`, `proxy_jump` | Auth and routing |
| `workdir_root` | Profile workspace root |
| `remote_task_dir` | Absolute remote path recorded at launch |
| `shell` | Login shell used for remote commands |
Targets are gathered from `ListExecutorsRunningByTaskID` at prepare time and
de-duplicated by `(host, port, user, remote_task_dir)`, so a multi-session task
produces one target, and a task spanning two hosts produces two.
## Control flow
Inside `executeTaskResourceCleanupJob`, after the existing stop phase and the
existing `performTaskCleanup`, and only when `len(failedStops) == 0` and no
context cancellation is pending:
1. **Gate on the setting.** Skip targets whose snapshot does not carry
`ssh_reclaim_task_dir == "true"`.
2. **Resolve ownership** (see below). A non-owned or unresolvable target is
skipped and recorded, not failed.
3. **Validate the path shape.** The target must equal
`<workdir_root>/tasks/<segment>` after normalization, with `<segment>`
non-empty and containing no `/`, no `.`, and no `..`. Anything else is
refused. This is a belt-and-braces guard for
AC-SSH-TASKDIR-RECLAMATION-003.4; it makes an empty or corrupted task-dir name
incapable of resolving to `<workdir_root>/tasks/` itself.
The comparison is made against the *expanded* root. `ensureRemoteTaskDir`
builds the recorded directory from `expandRemoteHome(workdir_root)`, while
the profile stores the unexpanded form and stores nothing when the field is
left blank. A literal `~/.kandev` is therefore resolved against the remote
`$HOME` first, and a blank root falls back to the executor's own default, so
the guard bounds the path instead of refusing every default-configured host.
A root that needs no expansion issues no remote command, which is what keeps
a refused path from touching the host at all.
4. **Connect** using the snapshot tuple and pinned fingerprint. An empty pinned
fingerprint is an error: a launch cannot happen without a trusted host key,
so its absence means the recorded connection is not one Kandev established
and dialling would accept whatever key answered.
5. **Probe** (see below). Any at-risk or inconclusive verdict skips and records.
6. **Remove** with `rm -rf -- <shell-quoted path>`, then stat the path to confirm
it is gone. A non-zero exit or a surviving path is an error, not a skip.
The `cleanup_script` ordering required by REQ-SSH-TASKDIR-RECLAMATION-007 is
satisfied by this position rather than by new sequencing code: the script runs
inside `SSHExecutor.StopInstance` during step 0's stop phase, which the job
completes and checks before reaching step 1. AC-SSH-TASKDIR-RECLAMATION-007.2
follows because the probe in step 5 reads the tree the script left behind.
### Ownership resolution
A target is owned by the task when both hold:
- The task's `TaskEnvironment.TaskID` is the job's task, and
`hasActiveOtherTaskSessionsForEnvironment` reports no other live task on that
environment. This is the existing signal that distinguishes a task from an
`inherit_parent` subtask borrowing the parent's workspace, and it is already
consulted for the local worktree path.
- No `executor_running` row belonging to another task carries the same
`(host, port, user, remote_task_dir)`.
The second check is the one that catches the cross-host case the environment
check cannot see, and it is a direct analogue of
`CountActiveWorktreeReferences`. Either check erroring resolves to *not owned*
per AC-SSH-TASKDIR-RECLAMATION-003.3.
### Safety probe
Run against the task-directory root and every direct child directory that
contains a `.git` entry, matching the remote layout the v1 spec defines. Each
checkout must satisfy all four, and every probe is read-only
(AC-SSH-TASKDIR-RECLAMATION-002.6):
| Probe | Command | Safe when |
| --- | --- | --- |
| Is a checkout | `git rev-parse --is-inside-work-tree` | prints `true` |
| Clean tree | `git status --porcelain --untracked-files=all --ignored` | empty output after Kandev-owned `.kandev/` runtime entries are excluded |
| Everything pushed | `git rev-list --count HEAD --branches --not --remotes` | prints `0` |
| No stashes | `git stash list` | empty output |
The third probe is the load-bearing one and the reason a clean `git status` is
not sufficient. It counts commits reachable from `HEAD` or any local branch that
are reachable from no remote-tracking ref — which is exactly "work that would be
lost". Including `HEAD` explicitly covers a detached head, which `--branches`
alone misses.
Remote-tracking refs are a local cache and are not refreshed: the probe answers
"was this pushed at some point", which is the correct bar, and fetching during
cleanup would need network and credentials for a remote that may no longer exist.
The task-directory root not being a checkout is inconclusive, not safe: it means
preparation failed or the layout is unrecognized, and the directory may hold
arbitrary user content. Same for a probe that fails, times out, or returns
unparseable output. All resolve to skip
(AC-SSH-TASKDIR-RECLAMATION-002.5).
Probes run under a bounded per-target timeout derived from the job's context.
## Failure and recovery
The distinction that carries REQ-SSH-TASKDIR-RECLAMATION-005 is **skip versus
error**, and the two are deliberately not the same outcome:
- A **skip** — reclamation disabled, directory shared, work at risk, verdict
inconclusive — is a correct result. It appends no error, the job completes
`succeeded`, and the reason is logged and recorded. Retrying would not change
the verdict, and a permanently-failing job for a directory the user is keeping
on purpose is noise (AC-SSH-TASKDIR-RECLAMATION-002.7).
- An **error** — unreachable host, auth failure, fingerprint mismatch, non-zero
`rm`, directory still present afterwards — appends to the job's `errs` slice
exactly as `cleanup worktrees` and `cleanup quick-chat dir` already do. The
existing `retryTaskResourceCleanupJob` then applies the 1m/5m/15m/1h/3h/6h/12h
backoff, writes `last_error`, and marks the row `failed` after
`taskResourceCleanupMaxAttempts`.
Because the job re-decodes its snapshot and re-runs the phase on every attempt,
AC-SSH-TASKDIR-RECLAMATION-005.3 holds without extra work: no verdict is cached
across attempts.
Reclamation runs after `performTaskCleanup`, so a failure cannot prevent local
row, worktree, and quick-chat cleanup from completing
(AC-SSH-TASKDIR-RECLAMATION-005.4).
`cancelIfTaskUnarchived` already runs before and within the job body and cancels
the whole job when an archive is undone, giving
AC-SSH-TASKDIR-RECLAMATION-004.3.
Backend shutdown cancels the cleanup worker's context mid-job. The phase refuses
to begin on a cancelled context, so an irreversible removal is never started
against a process that is going away; the durable job row survives and the next
attempt runs the phase with a live context. That check lives inside the phase
rather than at its call site so a later reordering of the job body cannot
quietly remove it.
## Persistence
No new tables and no migration. The reclamation targets live inside the existing
`task_resource_cleanup_jobs.resource_snapshot` JSON blob, which is already the
mechanism by which the job survives a backend restart. The new snapshot field is
additive and absent-tolerant: a job row written by an older backend decodes with
an empty target list and performs no reclamation.
The per-profile setting persists in the executor profile's existing `Config` map;
no schema change.
## Security
The removal is the most destructive operation Kandev performs on a machine it
does not own, so the design keeps three independent barriers between a bug and a
user's disk: the profile-authoritative opt-in, the ownership resolution, and the
path-shape guard. Each is sufficient on its own to prevent removal.
Connections reuse the pinned `host_fingerprint` from the snapshot with the same
no-silent-re-pin rule as every other SSH path; a changed host key fails the
attempt rather than proceeding against an unknown machine. The remote path is
shell-quoted through the existing `shellQuote` helper and validated in Go before
it is quoted, so a task-dir name can neither escape its parent nor inject shell.
## Observability
Structured logs at the reclamation phase carry `task_id`, `host`, `remote_task_dir`,
and an `outcome` of `removed`, `skipped`, or `failed`, with a `reason` on the
latter two. `skipped` reasons are enumerated (`disabled`, `shared`,
`dirty_worktree`, `unpushed_commits`, `stash_present`, `not_a_checkout`,
`probe_failed`) so an operator can tell a deliberate preservation from a broken
probe.
The existing `task_resource_cleanup_jobs` row remains the durable record of a
failure, via its `state`, `attempts`, and `last_error` columns. The resource
snapshot in that row also records each target outcome and safety-skip reason,
so a successful preservation is discoverable after restart.
## Testing
Every test for the removal path uses a fake remote-command seam or a temporary
directory. No test executes a real deletion against a real workspace root. The
`SSHTaskDirReclaimer` takes its command execution through an interface so unit
tests never open an SSH connection, and the probe-verdict table is exercised
purely against recorded command outputs.
Regression coverage must include the directory being removed on terminal archive
and delete, and preserved on ordinary stop and on backend shutdown.
## Related decisions
A new ADR records the opt-in default and the placement of the removal in the
cleanup job rather than in `StopInstance`.