Files
kandev/docs/plans/ssh-host-reachability/plan.md
T
2026-09-22 10:31:00 +01:00

541 lines
35 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
created: 2026-09-08
status: draft
requirements:
- REQ-EXECUTORS-SSH-REACHABILITY-001
- REQ-EXECUTORS-SSH-REACHABILITY-002
- REQ-EXECUTORS-SSH-REACHABILITY-003
system_design:
- ../../specs/executors/system-design/ssh-reachability.md
- ../../specs/executors/system-design/ssh-reachability-surfaces.md
legacy_specs: []
---
# Implementation Plan: SSH Host Reachability
## Overview
Kandev gains a continuous, cheap answer to one question per configured SSH
host — can the backend still open an authenticated SSH connection to the host
the user trusted — and puts that answer on the SSH settings page.
The build order is bottom-up so that nothing observable exists until the thing
underneath it is proven. Wave 1 delivers two independent primitives that no
caller can reach: the single-host probe with its classification order in
`lifecycle`, and the `executor_reachability` table with its last-write-wins
upsert in the SQLite repository. Wave 2 is the poller, which is the first code
that can call either one, and which owns the hysteresis rule, the interval
configuration, and the shutdown contract. Wave 3 opens the two producers and
consumers of the record that the poller does not own: the HTTP/event
projection, and the launch path, which must never read a record. Waves 4 and 5 are the two user
surfaces, data layer first. Wave 6 proves the whole thing against a real sshd
container and documents the operator key.
The order matters in one specific way: the launch path (task 05) is the
sharpest edge in this capability, because REQ-003 requires that a reachability
record never gate a launch. Building it after the poller means the record
already exists and is populated, so task 05's own tests can assert the negative
— that `CreateInstance` reads no record — against a store that genuinely holds
one rather than against an empty stub.
**Two flows were cut from this package to fit the specification size budget:**
the task-card reachability indicator and the launch path's record *write*. Both
are recorded in the requirement document's `## Out of scope` with enough detail
to seed their own cards, and their acceptance-criterion IDs
(`AC-…-002.8`, `.9` and `AC-…-003.4` through `.7`) are retired, not reused. The
launch path keeps its two surviving obligations: it never gates on a record, and
its failure message names the host and the reason classified from that launch's
own attempt.
**Reconciled against spec round 5 (2026-09-15, commit `b0481c492`).** Four
rounds of spec review closed all 15 open findings from round 4. The changes
that affect this plan and its work orders:
- A configuration reset is now a dedicated second SQL statement, not a branch
of the observation upsert, and it carries the newly saved `host`. Ownership
of *detecting* a save lives in `internal/task/service` (`CreateExecutor` /
`UpdateExecutor`), which notifies an `ExecutorSaveObserver` the reachability
package implements — not `internal/ssh`, as the pre-round-5 plan assumed.
- `ProbeSSHHost` must bound the SSH handshake on its context, not only the TCP
dial, or a stalled handshake outlives both `probeTimeout` and `Stop`.
- A new criterion, `AC-EXECUTORS-SSH-REACHABILITY-001.29`, requires a bastion
(`ProxyJump`) host-key mismatch to classify as `host_key`, matching a
target-host mismatch. `host` stays the target in every case; a bastion-hop
failure is disambiguated in `message`, never in `host`.
- The client reconciliation key for a pushed or refetched record is
`updated_at`, not `checked_at` — a reset clears `checked_at`, so ordering on
it would make every reset look older than the record it supersedes and get
discarded. The DTO gained an `updated_at` field.
- The GET routes now have a full edge contract: `404` for a nonexistent or
soft-deleted id, `400` only for a wrong executor type (not for an
unresolvable configuration, which is a `200` carrying reason `config`),
`GET /reachability` ordered by `executor_id` ascending and including
inactive executors.
- The settings page runs no refresh timer at all while probing is disabled,
rather than reading a zero effective interval literally.
- The pre-launch warning collapsed from two mechanisms (an interactive prompt
versus a session-recorded non-interactive note) into one: the backend always
emits a `session.launch.warning` event for the launched session, and the
gateway replays the latest warning to a later subscriber while the backend
is running. A client that initiated that launch also renders the same event
inline. There is no "interactive user" branch on the backend. **The system design
attributes the producer to `internal/task/service`; that package is not in
the launch call path.** Per `apps/backend/AGENTS.md`'s Execution Flow
(Orchestrator → Lifecycle Manager → ExecutorBackend) and the verified call
site, the producer belongs in
`internal/agent/runtime/lifecycle/manager_launch.go`'s
`launchBuildExecutorRequest`, immediately before `rt.CreateInstance` at its
existing call site, using the `Manager`'s existing `eventPublisher` (the
same mechanism behind `PublishPrepareProgress`). Tasks 05 and 07 below use
this verified location; the spec's package reference needs a follow-up
correction.
- The operator configuration key's environment and YAML paths diverge: an
unparsable environment value falls back to the default, but an unparsable
`config.yaml` value refuses boot like every other typed catalog key. A
negative whole number decodes cleanly from either source and needs its own
clamp branch (`below 0` → the default).
## Scope
### In scope
- A single-host SSH reachability probe and a total-ordered classification of
its failure into the closed reason set `config`, `timeout`, `host_key`,
`auth`, `network`, `unknown`.
- Persistence of exactly one reachability record per SSH executor, surviving
restart, converging under concurrent writes on the later observation, and
removed when the executor is soft-deleted.
- A bounded-concurrency poller on a configurable interval, with asymmetric
hysteresis (a threshold of consecutive failures down, one success up), an
overlap-skip rule, and a disable switch.
- HTTP and WebSocket projection of the record, an immediate-probe action that
coalesces concurrent callers, and a change-only change event.
- The launch path as a non-gating consumer: it reads no record, and its
dial-failure message names the host and the classified reason.
- The SSH executor settings surface and the pre-launch warning, localized in
all five supported locales.
- Public documentation for the one new operator configuration key.
### Out of scope
- Health probing for any non-SSH executor type. `HealthCheckAll` and every
other runtime's `HealthCheck` are unchanged, including
`SSHExecutor.HealthCheck`, which keeps returning `nil`.
- Repairing what the probe finds: re-resolving a moved host, re-pinning a
changed fingerprint, restarting or migrating a session off an unreachable
host.
- Gating, deferring, queueing, or re-routing a launch on a probe result.
- **Deferred (seeds its own card):** the task-card reachability indicator, and
the launch path as a probe *producer* writing `reachable` on a successful
dial and a classified failure on a failed one. Both are specified in the
requirement document's `## Out of scope`. Until the latter lands the poller
is the only writer, so a host that dies between passes lags the settings page
by up to the failure threshold times the effective interval.
- Alerting outside the product surface (push, email, webhook, chat).
- Probing remote capability on the cadence. Platform detection, `git`
presence, per-agent binary readiness, shell discovery, and the agentctl
cache check stay on the existing manual endpoints.
- Any change to the behavior of the existing `POST /api/v1/ssh/test`
endpoint, which keeps dialing unpinned against a possibly-unsaved form
configuration and keeps writing no record.
- A runtime feature toggle. The interval key's `0` value is the kill switch,
and a toggle would leave a retired identity to carry forever.
- History, trend, uptime percentage, or an incident log. One row per executor.
## Technical approach
### Probe primitive (`internal/agent/runtime/lifecycle`)
A new `executor_ssh_reachability_probe.go` exports the probe and its
classification. The probe takes a resolved `*SSHTarget` and a deadline, calls
the existing `DialSSH` with `PinnedFingerprint` set, closes the client
immediately, and returns a typed outcome. It runs no remote command, so its
cost is one TCP connection plus one SSH handshake.
Classification walks the total order `config`, `timeout`, `host_key`, `auth`,
`network`, `unknown` and takes the first match. `timeout` precedes `network`
because a deadline expiry surfaces through `dialDirect` as a dial error and
would otherwise read as a refused connection. `host_key` precedes `auth`
because both arrive as handshake failures from `ssh.NewClientConn` and only
`host_key` is a security event; the existing `errHostKeyMismatch` is the
discriminator. A bastion (`ProxyJump`) host-key rejection also classifies as
`host_key` per `AC-EXECUTORS-SSH-REACHABILITY-001.29`, even though it arrives
as `ssh: bastion dial: …` rather than `errHostKeyMismatch`, because only the
target hop is fingerprint-pinned — the reason names what failed, not which hop.
`host` always stays the target executor's host, success or failure; a
bastion-hop failure is disambiguated in `message` alone, which names the
bastion and states the failure arose on the jump hop.
`ProbeSSHHost` bounds the handshake phase on its context, not only the TCP
dial: `ssh.NewClientConn` takes no context and `ssh.ClientConfig.Timeout`
covers only `DialSSH`'s own TCP connect, so a host that completes TCP and then
stalls the handshake would otherwise hold its goroutine past `probeTimeout`
and past `Stop`. The probe sets a deadline on the connection covering the
handshake and clears it once the transport is up.
Resolution also validates the pin before any dial: `ResolveSSHTarget` does not
reject an empty `PinnedFingerprint`, and `buildClientConfig`'s host-key
callback accepts any key when the pin is empty, so a probe that skipped this
check would dial unpinned and record a substituted host as `reachable`. The
probe returns reason `config` for an empty pin, exactly as for an
unresolvable configuration, before any network access.
`sshTargetFromExecutorConfig` moves out of `internal/ssh/handlers.go` and is
exported from `lifecycle` as `SSHTargetFromExecutorConfig`, so the poller and
the handlers share one projection of `executors.config` and cannot drift. The
handler delegates to it.
### Persistence (`internal/task/models`, `internal/task/repository`)
One new table, created through the existing idempotent
`r.migrate.Apply("executor_reachability.table", …)` mechanism in
`base_migrations.go`, following the `repository_secret_bindings` precedent, so
it is safe on SQLite, on Postgres, and on an already-migrated database:
```sql
CREATE TABLE IF NOT EXISTS executor_reachability (
executor_id TEXT PRIMARY KEY,
state TEXT NOT NULL DEFAULT 'unknown',
reason TEXT NOT NULL DEFAULT '',
message TEXT NOT NULL DEFAULT '',
consecutive_failures INTEGER NOT NULL DEFAULT 0,
host TEXT NOT NULL DEFAULT '',
checked_at TIMESTAMP,
last_success_at TIMESTAMP,
updated_at TIMESTAMP NOT NULL
);
```
`last_success_at` is distinct from `checked_at` because the pre-launch warning
must name the age of the last *successful* probe, and `checked_at` is
overwritten by every failure. It is `NULL` until the first success and is
cleared only by a connection-configuration reset.
A separate table rather than columns on `executors`: `executors.config` is
user-authored input and `executors.status` is a user-controlled enable switch,
so observed state must not share either row. It also keeps a write path that
fires every interval off the row that every executor read touches.
`UpsertExecutorReachability` carries
`WHERE excluded.checked_at > executor_reachability.checked_at OR
executor_reachability.checked_at IS NULL`, and reads and writes the
consecutive-failure counter inside that same statement's transaction, so a
scheduled pass and an immediate probe racing on one executor converge on the
later observation and the streak cannot be lost to an interleaving.
`DeleteExecutor` deletes the record alongside the soft delete.
A **second** repository method, `ResetExecutorReachability(ctx, executorID,
host string) error`, is a distinct statement, not a branch of the upsert
above: the upsert's `state` CASE has no `unknown` branch and its `WHERE`
admits only a strictly later `checked_at`, which a reset does not carry. It
writes `unknown`, a zero counter, cleared reason/message, the newly saved
`host`, and both timestamps `NULL`, guarded only by `type = 'ssh' AND
deleted_at IS NULL AND status = 'active'` — no `checked_at` or `updated_at`
pin, because a reset is ordered by the save that caused it, not by an
observation clock, and must win over whatever is stored. An executor that is
`ssh` but not `active` is *not* reset (eligibility wins per
`AC-…-001.17`), and the statement lands zero rows for it.
### Poller (`internal/executors/reachability`, new package)
A ticker loop started from `internal/backendapp/main.go` alongside the
integration pollers and registered with `addRuntimeCleanup`. `Start` and
`Stop` are idempotent, `Stop` drains the in-flight pass, and a `goleak`
`TestMain` guards the package — the conventions `healthpoll` established,
adopted without importing it, because `healthpoll.Prober` is shaped around one
integration with many workspaces rather than many independent hosts with
per-host results and bounded concurrency.
One pass lists `type = 'ssh' AND deleted_at IS NULL AND status = 'active'`,
sorts ascending by `executors.id` (the primary key, never reused, so the order
is total and no tiebreak is needed), fans out over a semaphore of
`passConcurrency`, and awaits all workers. A mutex-guarded `passRunning` flag
makes an overlapping tick a no-op — dropped, not queued, because a queued tick
would compound the overload that caused the overlap and carries identical
information.
The write path owns hysteresis: on success, `reachable` with a zero counter
from any prior state; on failure, increment, and become `unreachable` only on
the probe where the counter reaches `failureThreshold`. A sticky failure
(`config` or `host_key`) bypasses the threshold and writes `unreachable` on
that probe, because both repeat until a human changes something: the threshold
would buy no confidence, and a host-key mismatch is a possible interception
that must not wait out an interval.
One new catalog entry, `executors.sshReachabilityIntervalSeconds` /
`KANDEV_EXECUTORS_SSHREACHABILITYINTERVALSECONDS`, default `60`, supported
range 15 to 3600, `0` disables. Adding it requires a matching entry in
`auditedStartupEnvironmentInventory()` in `catalog_test.go`, which is
deliberately independent of `startupCatalog`, plus a new `ExecutorsConfig`
section on `config.Config` and its `SetDefault`. The other three knobs stay
compile-time constants — `probeTimeout` 10s, `passConcurrency` 4,
`failureThreshold` 2 — because no operator scenario needs them independently
and each key is a permanent contract.
**The environment and YAML paths do not behave alike, and the clamp needs a
negative branch.** `applyBoundedIntEnv` reads the environment only and returns
early when it is unset, so a `config.yaml` value never reaches the
fallback-and-bounds logic; a non-integer YAML value instead hits
`decodeConfig`'s ordinary typed-key path and refuses boot, exactly like any
other catalog key. A negative whole number decodes cleanly from either source,
so the package's own clamp (not the catalog's) needs a `below 0` branch
yielding the default `60`, alongside `0` disables, `1`–`14` → `15`, and above
`3600` → `3600`, logged once at startup.
Observability follows the `office_stall_*` precedent: `expvar` counters
`executor_ssh_reachability_probe_total` (by outcome),
`executor_ssh_reachability_state_transitions_total` (by destination state),
`executor_ssh_reachability_pass_skipped_total`, and
`executor_ssh_reachability_probe_duration_ms`, each also emitted as a
structured `zap` log. Transitions log at `Warn` downward and `Info` upward;
an unchanged probe does not log, so a steady host is silent.
### API and events (`internal/ssh`, `internal/events`, `pkg/websocket`)
Reachability is its own resource, not fields on the executor DTO, so a record
that changes every interval does not invalidate the executor list payload:
| Route | Purpose |
| --- | --- |
| `GET /api/v1/ssh/reachability` | Every SSH executor's record in one request. |
| `GET /api/v1/ssh/executors/:id/reachability` | One executor's record. |
| `POST /api/v1/ssh/executors/:id/reachability/probe` | Probe now, persist, return the record. |
The `POST` route coalesces through a `golang.org/x/sync/singleflight` group
keyed by executor id and the SSH connection configuration, so two overlapping
requests for one connection open one connection while a request for a changed
host or pin gets its own probe. It works while the poller is disabled. The
coalesced probe does
**not** run on any caller's request context — a disconnecting caller must
neither cancel it nor fail the other caller waiting on it — but it is also
**not** `context.Background()`: it is registered on the reachability package's
own `WaitGroup` and context, the off-cycle-probe primitive task 03 exposes, so
`Stop` still cancels and awaits it. Status codes are `404` for a nonexistent
or soft-deleted executor id, `400` when the id names a non-`ssh` executor, and
`409` for a probe request against an inactive SSH executor. `resolveSSHTarget`'s existing mapping
must **not** be reused wholesale for the probe route — that helper also maps a
config-resolution failure to `400`, which is exactly the case
`AC-…-002.5` requires to return a `200` outcome carrying reason `config`
instead. `GET` on an executor with no record returns the `unknown` shape with
every timestamp, including `updated_at`, `null`, never `404`.
`GET /reachability` orders its entries by `executor_id` ascending — the same
order the pass uses — and includes executors whose `status` is not `active`,
carrying their retained record.
`events.ExecutorReachabilityChanged = "executor.reachability.changed"` is
published only when `state` or `reason` differs from the stored record, and is
bridged in `task_notifications.go` alongside `events.ExecutorUpdated`.
Publishing every probe result would put one message per executor per interval
on every connected client forever. The record DTO carries `updated_at`
alongside `checked_at`; it is the field a client reconciles two racing
payloads on (see Frontend below), because a reset clears `checked_at` but
still advances `updated_at`.
**Save detection lives in `internal/task/service`, not `internal/ssh`.**
`Service.CreateExecutor` and `Service.UpdateExecutor`
(`service_resources.go`) are the only two call sites that can create a
before/after connection-configuration comparison — `internal/ssh` never sees
an executor write. Each notifies an `ExecutorSaveObserver` interface the
service declares and the `internal/executors/reachability` package
implements, wired at construction. `applyExecutorUpdates` mutates the loaded
executor in place, so `UpdateExecutor` must copy the before-values out ahead
of that call; read afterwards they compare the row against itself and the
reset never fires. On any difference in the fields Terminology names (a
create is a difference by definition), the observer calls
`ResetExecutorReachability` with the newly saved host and dispatches an
out-of-band probe through the same off-cycle primitive the immediate-probe
route uses, if the executor is still eligible.
### Launch interaction (`internal/agent/runtime/lifecycle`)
`SSHExecutor.CreateInstance` reads no record and no code path consults one to
decide whether to proceed. With the producer write deferred, it writes none
either, so this task adds no recorder and no dependency from `lifecycle` onto
the store.
The launch-failure message names the target host and the reason classified from
**that launch's own** dial through `ClassifyDialError`, so a user is not
left holding an agent-generated message about a firewall when the cause was a
stale address. Kandev never intercepts or edits agent output; it adds its own
attribution alongside.
**The pre-launch warning is one mechanism, not two.** There is no branch on
whether an interactive user is present. `Manager.launchBuildExecutorRequest`
(`manager_launch.go`), which already resolves the per-executor-type backend
and calls `rt.CreateInstance` at its single call site for every executor type,
starts a bounded asynchronous read of the target executor's reachability record
through a narrow read-only accessor immediately before that call — never inside
`CreateInstance` itself, so the negative test (`CreateInstance` reads no record)
stays meaningful. When the executor is `ssh`, the record's state is
`unreachable`, and either periodic probing is enabled or the record's `checked_at`
is within three times the *default* interval, it publishes
`session.launch.warning` for the launched session via the
`Manager`'s existing `eventPublisher` (the mechanism behind
`PublishPrepareProgress`), carrying `executor_id`, `host`, `state`, `reason`,
`last_success_at`, and a timestamp. A read failure produces no warning and
cannot delay the launch. Every launch path — WS-initiated, a
dependency chain, a workflow transition, an autostart — converges on this one
call site, so none can diverge from another. A client that initiated the
launch itself renders the same event inline at the point of initiation, in
addition to it appearing in the session-scoped event stream; there is no second
backend code path to keep in step.
### Frontend (`apps/web`)
`lib/types/http-ssh.ts` gains the record shape including `probing_enabled`,
which is `false` when the interval key is `0` so a surface can distinguish
"not probed yet" from "probing is off" without inferring it from an absent
timestamp, `probe_interval_seconds`, the effective interval the staleness rule
and the refresh cadence are both computed from, and `updated_at`.
`lib/api/domains/ssh-api.ts` gains the three calls. The settings slice holds
records keyed by executor id and applies the change action, reconciling a
racing refetch against a pushed event by keeping whichever payload carries the
**later `updated_at`**, not `checked_at` — a reset clears `checked_at`, so
ordering on it would discard the reset in favor of the stale record it just
invalidated.
The settings card renders state, probed host, reason and message when
`unreachable`, the consecutive-failure count, and the age of the last probe;
it marks the result stale past three intervals, states plainly when probing is
off, and offers the probe-now action. Because a steady state is silent on the
wire, the card refetches its own record on mount and on the interval while it
is open — and runs **no timer at all** while `probing_enabled` is `false`,
fetching once on open and again after an immediate probe instead of reading a
zero effective interval as a cadence.
**The pre-launch warning renders a session event, not a derived client
computation.** The backend always publishes `session.launch.warning` for the
launched session (see Launch interaction above); the gateway replays the latest
warning to a later subscriber while the backend is running. The session view
renders it through the existing session-scoped WS-event handler pattern
(the `executor-prepare.ts` shape), and a client that initiated that launch
also renders the same event inline at the initiation point, naming the host
and the age of the last successful probe. Neither surface derives
"unreachable" from the reachability store slice itself — both render the one
event the backend decided to raise, so there is no second code path that could
diverge from the backend's when-to-warn rule. The launch proceeds without a
confirmation step in either rendering.
All copy goes through `t()` in `executors.json` and `tasks.json` across `en`,
`pt-pt`, `zh-cn`, `zh-hk`, and `zh-tw`, with the failure reason rendered as
translated copy rather than the raw token, and state exposed to assistive
technology as text rather than by color alone.
## Tests
Every acceptance criterion below is traced to the current requirements
revision. A criterion owned by two work orders appears once per side of the
split.
| Acceptance criterion | Evidence |
| --- | --- |
| `AC-…-001.4`, `.5`, `.6`, `.7`, `.8` | `internal/agent/runtime/lifecycle/executor_ssh_reachability_probe_test.go` — `TestProbeSucceedsAndClosesWithoutRemoteCommand`, `TestProbeReportsConfigWithoutDialing`, `TestProbeRejectsUnpinnedFingerprintWithoutDialing`, `TestProbeRejectsChangedHostKeyWithoutRepinning`, `TestProbeDeadlineReportsTimeout`, `TestProbeAbandonsStalledHandshakeAtDeadline`, `TestFailureCarriesExactlyOneReasonAndMessage` |
| `AC-…-001.27`, `.29` | `executor_ssh_reachability_probe_test.go` — `TestClassifyDialErrorOrdering`, table-driven, one case per reason plus the deadline-before-network and host-key-before-auth orderings; `TestBastionHostKeyMismatchClassifiesAsHostKey`, `TestBastionFailureNamesTargetHostAndBastionInMessage` |
| `AC-…-001.16`, `.21`, `.22` | `internal/task/repository/sqlite/executor_reachability_test.go` — `TestSoftDeleteRemovesReachabilityRecord`, `TestUpsertKeepsLaterObservationRegardlessOfCommitOrder`, `TestEqualTimestampWriteIsDiscarded`, `TestRecordAndBothTimestampsSurviveRepositoryReopen`; Postgres parity in `executor_reachability_postgres_test.go` |
| `AC-…-001.1`, `.2`, `.3`, `.13`, `.14`, `.15`, `.17`, `.20`, `.26` | `internal/executors/reachability/poller_test.go` — `TestPassProbesEligibleExecutorsInIDOrder`, `TestPassSkipsNonSSHAndInactiveExecutors`, `TestInactiveExecutorRetainsItsRecord`, `TestOverlappingTickIsDroppedNotQueued`, `TestPassConcurrencyIsBounded`, `TestEmptyPassIsSilent`, `TestResultForNowIneligibleExecutorIsDiscarded`, `TestStopDrainsInFlightPass`, `TestStopDiscardsCancelledProbeResults`; `goleak_test.go` |
| `AC-…-001.9`, `.10`, `.11`, `.12` | `internal/executors/reachability/hysteresis_test.go` — `TestNonStickyFailureFlipsOnThresholdProbe`, `TestStickyFailureFlipsOnFirstProbe`, `TestBelowThresholdFailureStoresReasonWithoutChangingState`, `TestSingleSuccessRecoversFromUnreachable`, `TestAbsentRecordReportsUnknown`, each table-driven over probe sequences asserting the exact probe on which the state flips |
| `AC-…-001.23`, `.24`, `.25` | `internal/common/config/config_test.go` — `TestSSHReachabilityIntervalClampsOutOfRangeAndLogsOnce`, `TestAbsentNegativeOrMalformedIntervalUsesDefault`, `TestZeroIntervalDisablesPollerButNegativeDoesNot`; `internal/executors/reachability/poller_test.go` — `TestDisabledPollerRunsNoPassIncludingAtStart` |
| `AC-…-001.18`, `.19`, `.28`, `AC-…-002.7` | `internal/task/service/service_resources_executors_reachability_test.go` — `TestSavingConnectionConfigNotifiesObserverWithBeforeAndAfterFields`, `TestUnchangedSaveNotifiesNothing`, `TestCreateExecutorAlwaysNotifies`; `internal/executors/reachability/save_observer_test.go` — `TestObserverResetsRecordAndSchedulesOffCycleProbe`, `TestObserverSkipsIneligibleExecutor`; `internal/ssh/reachability_handlers_test.go` — `TestInFlightProbeResultDiscardedAfterConfigSave`, `TestImmediateProbeRunsWhilePollerDisabled`, `TestImmediateProbeRefusedForInactiveExecutor` |
| `AC-…-002.2`, `.3` | `internal/executors/reachability/publish_test.go` — `TestChangePublishesEvent`, `TestUnchangedProbePublishesNothing`, `TestResetPublishesEvent`; `internal/gateway/websocket/task_notifications_test.go` bridge assertion |
| `AC-…-002.1`, `.4`, `.5`, `.6`, `.10`, `.11`, `.12` | `internal/ssh/reachability_handlers_test.go` — `TestOverlappingImmediateProbesShareOneProbe`, `TestSharedProbeCompletesWhenStarterDisconnects`, `TestSharedProbeNotOnCallerRequestContext`, `TestGETReturnsUnknownShapeWithNullUpdatedAtForUnprobedExecutor`, `TestGETReachabilityOrderedByExecutorIDAscending`, `TestGET404ForSoftDeletedExecutor`; `apps/web/components/settings/ssh-reachability-card.test.tsx` — field coverage per state, stale past three intervals, probing-off copy with no refresh timer, load failure renders "not known", not-persisted probe result; `apps/web/lib/state/slices/settings/settings-slice.test.ts` — `applies executor.reachability.changed`, `keeps the later updated_at`; `pnpm run i18n:check` |
| `AC-…-003.1` | `internal/agent/runtime/lifecycle/executor_ssh_reachability_launch_test.go` — `TestCreateInstanceReadsNoReachabilityRecord`, asserting against a store fake that fails the test on any read, plus `TestLaunchProceedsAgainstUnreachableExecutor` |
| `AC-…-003.2`, `.3` | `internal/agent/runtime/lifecycle/manager_launch_reachability_warning_test.go` — `TestDialFailureMessageNamesHostAndClassifiedReason`, `TestWarningPublishedForUnreachableExecutorAcrossEveryLaunchPath` (WS-initiated, dependency chain, workflow transition, autostart), `TestNoWarningWhenProbingDisabledAndRecordStale`, `TestWarningStillPublishedWhenProbingDisabledButRecordFresh`; `apps/web/lib/ws/handlers/ssh-launch-warning.test.ts` — session-stream event applied to session-runtime state; `apps/web/components/task/launch-warning.test.tsx` — inline render for the initiating client names host and last-success age, and does not block |
## E2E tests
The `containers` Playwright project already gates real-sshd scenarios on
`KANDEV_E2E_CONTAINERS=1` and owns the `kandev-sshd:e2e` image, so all four
flows live in `apps/web/e2e/tests/ssh/reachability.spec.ts` under that project.
| Flow | Acceptance criteria |
| --- | --- |
| A reachable host renders the settings panel row as reachable. | `AC-…-002.1` |
| The sshd container is stopped mid-test; after the failure threshold the settings panel reports the host unreachable by name. | `AC-…-001.9`, `AC-…-002.1` |
| The container is restarted; the next probe clears the panel on a single success. | `AC-…-001.11`, `AC-…-002.1` |
| A launch started against the stopped host is attempted and fails with a message naming the host, rather than being refused. | `AC-…-003.1`, `AC-…-003.3` |
## Work orders
- [x] [Task 01: Single-host SSH probe and failure classification](task-01-probe-and-classification.md)
- [x] [Task 02: Reachability record persistence](task-02-reachability-persistence.md)
- [x] [Task 03: Reachability poller, hysteresis, and configuration](task-03-reachability-poller.md)
- [x] [Task 04: Reachability API, change event, and immediate probe](task-04-reachability-api-and-events.md)
- [x] [Task 05: Launch-path non-gating and failure attribution](task-05-launch-non-gating.md)
- [x] [Task 06: Reachability data layer and settings surface](task-06-settings-surface.md)
- [x] [Task 07: Pre-launch reachability warning](task-07-pre-launch-warning.md)
- [x] [Task 08: Container E2E coverage and operator documentation](task-08-e2e-and-docs.md)
### Waves
| Wave | Tasks | Parallel-safe |
| --- | --- | --- |
| 1 | 01, 02 | Yes — disjoint packages, only 02 carries a migration |
| 2 | 03 | No — first caller of both primitives |
| 3 | 04, 05 | Yes — disjoint files, both depend only on 03 |
| 4 | 06 | No — consumes the contract task 04 defines |
| 5 | 07 | No — reads the store slice task 06 creates |
| 6 | 08 | No — proves the assembled surfaces |
## Verification results
All 8 work orders are done. Each task's exact verification commands and
results are recorded in its own `## Results` section
(`task-01-probe-and-classification.md` through
`task-08-e2e-and-docs.md`). Task 08's container E2E spec
(`apps/web/e2e/tests/ssh/reachability.spec.ts`) is the plan's only
end-to-end evidence and passes against a real sshd container
(`KANDEV_E2E_CONTAINERS=1 pnpm e2e:run --project=containers
tests/ssh/reachability.spec.ts`, 2 passed).
## Risks
- **The probe's classification depends on error shapes from
`golang.org/x/crypto/ssh` that are not part of that package's stable API.**
A library upgrade could turn a `host_key` failure into `unknown`. Mitigated
by keying `host_key` off Kandev's own `errHostKeyMismatch` rather than an
upstream string, and by table-driven tests over constructed errors; the
residual risk is confined to `auth` versus `network` on exotic errors, where
the fallback is `unknown` rather than a wrong security verdict.
- **A per-interval write path against SQLite adds sustained write traffic that
did not exist before.** At the default 60s interval and a realistic executor
count this is small, but an operator with many SSH executors and a short
interval could contend with the busy `task_sessions` write path. The
supported range's 15s floor bounds this; the separate table keeps the
contention off the executor row.
- **The launch path must read the record for its warning while never reading
it to decide.** `AC-…-003.2` needs the last-success age, so a read exists in
the same flow as `AC-…-003.1`'s prohibition. A future change that moves that
read into `CreateInstance` to "avoid a pointless launch" would silently
violate REQ-003 without breaking any existing test. Mitigated by keeping the
read in `Manager.launchBuildExecutorRequest`, the one call site every
executor-type dispatch already passes through, and by task 05's negative
test, which asserts against a store fake that fails on any read.
- **The system design's stated package for the warning producer
(`internal/task/service`) does not match the verified call path.** Every
launch — WS-initiated, a dependency chain, a workflow transition, an
autostart — converges in `internal/agent/runtime/lifecycle` at
`Manager.launchBuildExecutorRequest`'s single `rt.CreateInstance` call site
(confirmed against `apps/backend/AGENTS.md`'s Execution Flow and
`manager_launch.go`); `internal/task/service` is not in that path. This plan
and tasks 05/07 use the verified location. The spec still needs a follow-up
correction so a future reader isn't sent to the wrong package.
- **Deferring the launch-path write leaves the poller as the only writer.** A
host that dies between passes lags the settings page by up to the failure
threshold times the effective interval, and a failed launch — the strongest
available evidence — updates nothing. Bounded and accepted: `AC-…-003.3`
still names the host and the classified reason on the failure itself, which
is what the 2026-09-06 report actually asked for.
- **Adding a catalog entry touches a deliberately duplicated inventory.**
`auditedStartupEnvironmentInventory()` is independent of `startupCatalog` by
design; forgetting it fails `TestConfigurationCatalogMatchesAuditedEnvironmentInventory`
rather than shipping silently, so the risk is a wasted cycle, not a defect.
- **Five-locale copy gates the build.** `check-i18n-keys.mjs` fails on a
missing key, a dropped placeholder, or a value left identical to English, so
tasks 06 and 07 each carry their own translation work rather than deferring
it.
- **The `containers` E2E project needs a real Docker daemon and is slow.**
Task 08's specs cannot run on a host without Docker, so the rest of the work
package must be verifiable without them; every acceptance criterion above has
non-E2E evidence, and the four flows there re-prove end to end what unit
tests already cover against a host that can actually be stopped.