kandev/docs/plans/ssh-host-reachability/plan.md

created, status, requirements, system_design, legacy_specs

created	status	requirements	system_design	legacy_specs
2026-09-08	draft	REQ-EXECUTORS-SSH-REACHABILITY-001 REQ-EXECUTORS-SSH-REACHABILITY-002 REQ-EXECUTORS-SSH-REACHABILITY-003	../../specs/executors/system-design/ssh-reachability.md ../../specs/executors/system-design/ssh-reachability-surfaces.md

Implementation Plan: SSH Host Reachability

Overview

Kandev gains a continuous, cheap answer to one question per configured SSH host — can the backend still open an authenticated SSH connection to the host the user trusted — and puts that answer on the SSH settings page.

The build order is bottom-up so that nothing observable exists until the thing underneath it is proven. Wave 1 delivers two independent primitives that no caller can reach: the single-host probe with its classification order in lifecycle, and the executor_reachability table with its last-write-wins upsert in the SQLite repository. Wave 2 is the poller, which is the first code that can call either one, and which owns the hysteresis rule, the interval configuration, and the shutdown contract. Wave 3 opens the two producers and consumers of the record that the poller does not own: the HTTP/event projection, and the launch path, which must never read a record. Waves 4 and 5 are the two user surfaces, data layer first. Wave 6 proves the whole thing against a real sshd container and documents the operator key.

The order matters in one specific way: the launch path (task 05) is the sharpest edge in this capability, because REQ-003 requires that a reachability record never gate a launch. Building it after the poller means the record already exists and is populated, so task 05's own tests can assert the negative — that CreateInstance reads no record — against a store that genuinely holds one rather than against an empty stub.

Two flows were cut from this package to fit the specification size budget: the task-card reachability indicator and the launch path's record write. Both are recorded in the requirement document's ## Out of scope with enough detail to seed their own cards, and their acceptance-criterion IDs (AC-…-002.8, .9 and AC-…-003.4 through .7) are retired, not reused. The launch path keeps its two surviving obligations: it never gates on a record, and its failure message names the host and the reason classified from that launch's own attempt.

Reconciled against spec round 5 (2026-09-15, commit b0481c492). Four rounds of spec review closed all 15 open findings from round 4. The changes that affect this plan and its work orders:

• A configuration reset is now a dedicated second SQL statement, not a branch of the observation upsert, and it carries the newly saved host. Ownership of detecting a save lives in internal/task/service (CreateExecutor / UpdateExecutor), which notifies an ExecutorSaveObserver the reachability package implements — not internal/ssh, as the pre-round-5 plan assumed.
• ProbeSSHHost must bound the SSH handshake on its context, not only the TCP dial, or a stalled handshake outlives both probeTimeout and Stop.
• A new criterion, AC-EXECUTORS-SSH-REACHABILITY-001.29, requires a bastion (ProxyJump) host-key mismatch to classify as host_key, matching a target-host mismatch. host stays the target in every case; a bastion-hop failure is disambiguated in message, never in host.
• The client reconciliation key for a pushed or refetched record is updated_at, not checked_at — a reset clears checked_at, so ordering on it would make every reset look older than the record it supersedes and get discarded. The DTO gained an updated_at field.
• The GET routes now have a full edge contract: 404 for a nonexistent or soft-deleted id, 400 only for a wrong executor type (not for an unresolvable configuration, which is a 200 carrying reason config), GET /reachability ordered by executor_id ascending and including inactive executors.
• The settings page runs no refresh timer at all while probing is disabled, rather than reading a zero effective interval literally.
• The pre-launch warning collapsed from two mechanisms (an interactive prompt versus a session-recorded non-interactive note) into one: the backend always emits a session.launch.warning event for the launched session, and the gateway replays the latest warning to a later subscriber while the backend is running. A client that initiated that launch also renders the same event inline. There is no "interactive user" branch on the backend. The system design attributes the producer to internal/task/service; that package is not in the launch call path. Per apps/backend/AGENTS.md's Execution Flow (Orchestrator → Lifecycle Manager → ExecutorBackend) and the verified call site, the producer belongs in internal/agent/runtime/lifecycle/manager_launch.go's launchBuildExecutorRequest, immediately before rt.CreateInstance at its existing call site, using the Manager's existing eventPublisher (the same mechanism behind PublishPrepareProgress). Tasks 05 and 07 below use this verified location; the spec's package reference needs a follow-up correction.
• The operator configuration key's environment and YAML paths diverge: an unparsable environment value falls back to the default, but an unparsable config.yaml value refuses boot like every other typed catalog key. A negative whole number decodes cleanly from either source and needs its own clamp branch (below 0 → the default).

Scope

In scope

• A single-host SSH reachability probe and a total-ordered classification of its failure into the closed reason set config, timeout, host_key, auth, network, unknown.
• Persistence of exactly one reachability record per SSH executor, surviving restart, converging under concurrent writes on the later observation, and removed when the executor is soft-deleted.
• A bounded-concurrency poller on a configurable interval, with asymmetric hysteresis (a threshold of consecutive failures down, one success up), an overlap-skip rule, and a disable switch.
• HTTP and WebSocket projection of the record, an immediate-probe action that coalesces concurrent callers, and a change-only change event.
• The launch path as a non-gating consumer: it reads no record, and its dial-failure message names the host and the classified reason.
• The SSH executor settings surface and the pre-launch warning, localized in all five supported locales.
• Public documentation for the one new operator configuration key.

Out of scope

• Health probing for any non-SSH executor type. HealthCheckAll and every other runtime's HealthCheck are unchanged, including SSHExecutor.HealthCheck, which keeps returning nil.
• Repairing what the probe finds: re-resolving a moved host, re-pinning a changed fingerprint, restarting or migrating a session off an unreachable host.
• Gating, deferring, queueing, or re-routing a launch on a probe result.
• Deferred (seeds its own card): the task-card reachability indicator, and the launch path as a probe producer writing reachable on a successful dial and a classified failure on a failed one. Both are specified in the requirement document's ## Out of scope. Until the latter lands the poller is the only writer, so a host that dies between passes lags the settings page by up to the failure threshold times the effective interval.
• Alerting outside the product surface (push, email, webhook, chat).
• Probing remote capability on the cadence. Platform detection, git presence, per-agent binary readiness, shell discovery, and the agentctl cache check stay on the existing manual endpoints.
• Any change to the behavior of the existing POST /api/v1/ssh/test endpoint, which keeps dialing unpinned against a possibly-unsaved form configuration and keeps writing no record.
• A runtime feature toggle. The interval key's 0 value is the kill switch, and a toggle would leave a retired identity to carry forever.
• History, trend, uptime percentage, or an incident log. One row per executor.

Technical approach

Probe primitive (internal/agent/runtime/lifecycle)

A new executor_ssh_reachability_probe.go exports the probe and its classification. The probe takes a resolved *SSHTarget and a deadline, calls the existing DialSSH with PinnedFingerprint set, closes the client immediately, and returns a typed outcome. It runs no remote command, so its cost is one TCP connection plus one SSH handshake.

Classification walks the total order config, timeout, host_key, auth, network, unknown and takes the first match. timeout precedes network because a deadline expiry surfaces through dialDirect as a dial error and would otherwise read as a refused connection. host_key precedes auth because both arrive as handshake failures from ssh.NewClientConn and only host_key is a security event; the existing errHostKeyMismatch is the discriminator. A bastion (ProxyJump) host-key rejection also classifies as host_key per AC-EXECUTORS-SSH-REACHABILITY-001.29, even though it arrives as ssh: bastion dial: … rather than errHostKeyMismatch, because only the target hop is fingerprint-pinned — the reason names what failed, not which hop. host always stays the target executor's host, success or failure; a bastion-hop failure is disambiguated in message alone, which names the bastion and states the failure arose on the jump hop.

ProbeSSHHost bounds the handshake phase on its context, not only the TCP dial: ssh.NewClientConn takes no context and ssh.ClientConfig.Timeout covers only DialSSH's own TCP connect, so a host that completes TCP and then stalls the handshake would otherwise hold its goroutine past probeTimeout and past Stop. The probe sets a deadline on the connection covering the handshake and clears it once the transport is up.

Resolution also validates the pin before any dial: ResolveSSHTarget does not reject an empty PinnedFingerprint, and buildClientConfig's host-key callback accepts any key when the pin is empty, so a probe that skipped this check would dial unpinned and record a substituted host as reachable. The probe returns reason config for an empty pin, exactly as for an unresolvable configuration, before any network access.

sshTargetFromExecutorConfig moves out of internal/ssh/handlers.go and is exported from lifecycle as SSHTargetFromExecutorConfig, so the poller and the handlers share one projection of executors.config and cannot drift. The handler delegates to it.

Persistence (internal/task/models, internal/task/repository)

One new table, created through the existing idempotent r.migrate.Apply("executor_reachability.table", …) mechanism in base_migrations.go, following the repository_secret_bindings precedent, so it is safe on SQLite, on Postgres, and on an already-migrated database:

CREATE TABLE IF NOT EXISTS executor_reachability (
    executor_id          TEXT PRIMARY KEY,
    state                TEXT NOT NULL DEFAULT 'unknown',
    reason               TEXT NOT NULL DEFAULT '',
    message              TEXT NOT NULL DEFAULT '',
    consecutive_failures INTEGER NOT NULL DEFAULT 0,
    host                 TEXT NOT NULL DEFAULT '',
    checked_at           TIMESTAMP,
    last_success_at      TIMESTAMP,
    updated_at           TIMESTAMP NOT NULL
);

last_success_at is distinct from checked_at because the pre-launch warning must name the age of the last successful probe, and checked_at is overwritten by every failure. It is NULL until the first success and is cleared only by a connection-configuration reset.

A separate table rather than columns on executors: executors.config is user-authored input and executors.status is a user-controlled enable switch, so observed state must not share either row. It also keeps a write path that fires every interval off the row that every executor read touches.

UpsertExecutorReachability carries WHERE excluded.checked_at > executor_reachability.checked_at OR executor_reachability.checked_at IS NULL, and reads and writes the consecutive-failure counter inside that same statement's transaction, so a scheduled pass and an immediate probe racing on one executor converge on the later observation and the streak cannot be lost to an interleaving. DeleteExecutor deletes the record alongside the soft delete.

A second repository method, ResetExecutorReachability(ctx, executorID, host string) error, is a distinct statement, not a branch of the upsert above: the upsert's state CASE has no unknown branch and its WHERE admits only a strictly later checked_at, which a reset does not carry. It writes unknown, a zero counter, cleared reason/message, the newly saved host, and both timestamps NULL, guarded only by type = 'ssh' AND deleted_at IS NULL AND status = 'active' — no checked_at or updated_at pin, because a reset is ordered by the save that caused it, not by an observation clock, and must win over whatever is stored. An executor that is ssh but not active is not reset (eligibility wins per AC-…-001.17), and the statement lands zero rows for it.

Poller (internal/executors/reachability, new package)

A ticker loop started from internal/backendapp/main.go alongside the integration pollers and registered with addRuntimeCleanup. Start and Stop are idempotent, Stop drains the in-flight pass, and a goleak TestMain guards the package — the conventions healthpoll established, adopted without importing it, because healthpoll.Prober is shaped around one integration with many workspaces rather than many independent hosts with per-host results and bounded concurrency.

One pass lists type = 'ssh' AND deleted_at IS NULL AND status = 'active', sorts ascending by executors.id (the primary key, never reused, so the order is total and no tiebreak is needed), fans out over a semaphore of passConcurrency, and awaits all workers. A mutex-guarded passRunning flag makes an overlapping tick a no-op — dropped, not queued, because a queued tick would compound the overload that caused the overlap and carries identical information.

The write path owns hysteresis: on success, reachable with a zero counter from any prior state; on failure, increment, and become unreachable only on the probe where the counter reaches failureThreshold. A sticky failure (config or host_key) bypasses the threshold and writes unreachable on that probe, because both repeat until a human changes something: the threshold would buy no confidence, and a host-key mismatch is a possible interception that must not wait out an interval.

One new catalog entry, executors.sshReachabilityIntervalSeconds / KANDEV_EXECUTORS_SSHREACHABILITYINTERVALSECONDS, default 60, supported range 15 to 3600, 0 disables. Adding it requires a matching entry in auditedStartupEnvironmentInventory() in catalog_test.go, which is deliberately independent of startupCatalog, plus a new ExecutorsConfig section on config.Config and its SetDefault. The other three knobs stay compile-time constants — probeTimeout 10s, passConcurrency 4, failureThreshold 2 — because no operator scenario needs them independently and each key is a permanent contract.

The environment and YAML paths do not behave alike, and the clamp needs a negative branch. applyBoundedIntEnv reads the environment only and returns early when it is unset, so a config.yaml value never reaches the fallback-and-bounds logic; a non-integer YAML value instead hits decodeConfig's ordinary typed-key path and refuses boot, exactly like any other catalog key. A negative whole number decodes cleanly from either source, so the package's own clamp (not the catalog's) needs a below 0 branch yielding the default 60, alongside 0 disables, 1–14 → 15, and above 3600 → 3600, logged once at startup.

Observability follows the office_stall_* precedent: expvar counters executor_ssh_reachability_probe_total (by outcome), executor_ssh_reachability_state_transitions_total (by destination state), executor_ssh_reachability_pass_skipped_total, and executor_ssh_reachability_probe_duration_ms, each also emitted as a structured zap log. Transitions log at Warn downward and Info upward; an unchanged probe does not log, so a steady host is silent.

API and events (internal/ssh, internal/events, pkg/websocket)

Reachability is its own resource, not fields on the executor DTO, so a record that changes every interval does not invalidate the executor list payload:

Route	Purpose
GET /api/v1/ssh/reachability	Every SSH executor's record in one request.
GET /api/v1/ssh/executors/:id/reachability	One executor's record.
POST /api/v1/ssh/executors/:id/reachability/probe	Probe now, persist, return the record.

The POST route coalesces through a golang.org/x/sync/singleflight group keyed by executor id and the SSH connection configuration, so two overlapping requests for one connection open one connection while a request for a changed host or pin gets its own probe. It works while the poller is disabled. The coalesced probe does not run on any caller's request context — a disconnecting caller must neither cancel it nor fail the other caller waiting on it — but it is also not context.Background(): it is registered on the reachability package's own WaitGroup and context, the off-cycle-probe primitive task 03 exposes, so Stop still cancels and awaits it. Status codes are 404 for a nonexistent or soft-deleted executor id, 400 when the id names a non-ssh executor, and 409 for a probe request against an inactive SSH executor. resolveSSHTarget's existing mapping must not be reused wholesale for the probe route — that helper also maps a config-resolution failure to 400, which is exactly the case AC-…-002.5 requires to return a 200 outcome carrying reason config instead. GET on an executor with no record returns the unknown shape with every timestamp, including updated_at, null, never 404. GET /reachability orders its entries by executor_id ascending — the same order the pass uses — and includes executors whose status is not active, carrying their retained record.

events.ExecutorReachabilityChanged = "executor.reachability.changed" is published only when state or reason differs from the stored record, and is bridged in task_notifications.go alongside events.ExecutorUpdated. Publishing every probe result would put one message per executor per interval on every connected client forever. The record DTO carries updated_at alongside checked_at; it is the field a client reconciles two racing payloads on (see Frontend below), because a reset clears checked_at but still advances updated_at.

Save detection lives in internal/task/service, not internal/ssh. Service.CreateExecutor and Service.UpdateExecutor (service_resources.go) are the only two call sites that can create a before/after connection-configuration comparison — internal/ssh never sees an executor write. Each notifies an ExecutorSaveObserver interface the service declares and the internal/executors/reachability package implements, wired at construction. applyExecutorUpdates mutates the loaded executor in place, so UpdateExecutor must copy the before-values out ahead of that call; read afterwards they compare the row against itself and the reset never fires. On any difference in the fields Terminology names (a create is a difference by definition), the observer calls ResetExecutorReachability with the newly saved host and dispatches an out-of-band probe through the same off-cycle primitive the immediate-probe route uses, if the executor is still eligible.

Launch interaction (internal/agent/runtime/lifecycle)

SSHExecutor.CreateInstance reads no record and no code path consults one to decide whether to proceed. With the producer write deferred, it writes none either, so this task adds no recorder and no dependency from lifecycle onto the store.

The launch-failure message names the target host and the reason classified from that launch's own dial through ClassifyDialError, so a user is not left holding an agent-generated message about a firewall when the cause was a stale address. Kandev never intercepts or edits agent output; it adds its own attribution alongside.

The pre-launch warning is one mechanism, not two. There is no branch on whether an interactive user is present. Manager.launchBuildExecutorRequest (manager_launch.go), which already resolves the per-executor-type backend and calls rt.CreateInstance at its single call site for every executor type, starts a bounded asynchronous read of the target executor's reachability record through a narrow read-only accessor immediately before that call — never inside CreateInstance itself, so the negative test (CreateInstance reads no record) stays meaningful. When the executor is ssh, the record's state is unreachable, and either periodic probing is enabled or the record's checked_at is within three times the default interval, it publishes session.launch.warning for the launched session via the Manager's existing eventPublisher (the mechanism behind PublishPrepareProgress), carrying executor_id, host, state, reason, last_success_at, and a timestamp. A read failure produces no warning and cannot delay the launch. Every launch path — WS-initiated, a dependency chain, a workflow transition, an autostart — converges on this one call site, so none can diverge from another. A client that initiated the launch itself renders the same event inline at the point of initiation, in addition to it appearing in the session-scoped event stream; there is no second backend code path to keep in step.

Frontend (apps/web)

lib/types/http-ssh.ts gains the record shape including probing_enabled, which is false when the interval key is 0 so a surface can distinguish "not probed yet" from "probing is off" without inferring it from an absent timestamp, probe_interval_seconds, the effective interval the staleness rule and the refresh cadence are both computed from, and updated_at. lib/api/domains/ssh-api.ts gains the three calls. The settings slice holds records keyed by executor id and applies the change action, reconciling a racing refetch against a pushed event by keeping whichever payload carries the later updated_at, not checked_at — a reset clears checked_at, so ordering on it would discard the reset in favor of the stale record it just invalidated.

The settings card renders state, probed host, reason and message when unreachable, the consecutive-failure count, and the age of the last probe; it marks the result stale past three intervals, states plainly when probing is off, and offers the probe-now action. Because a steady state is silent on the wire, the card refetches its own record on mount and on the interval while it is open — and runs no timer at all while probing_enabled is false, fetching once on open and again after an immediate probe instead of reading a zero effective interval as a cadence.

The pre-launch warning renders a session event, not a derived client computation. The backend always publishes session.launch.warning for the launched session (see Launch interaction above); the gateway replays the latest warning to a later subscriber while the backend is running. The session view renders it through the existing session-scoped WS-event handler pattern (the executor-prepare.ts shape), and a client that initiated that launch also renders the same event inline at the initiation point, naming the host and the age of the last successful probe. Neither surface derives "unreachable" from the reachability store slice itself — both render the one event the backend decided to raise, so there is no second code path that could diverge from the backend's when-to-warn rule. The launch proceeds without a confirmation step in either rendering.

All copy goes through t() in executors.json and tasks.json across en, pt-pt, zh-cn, zh-hk, and zh-tw, with the failure reason rendered as translated copy rather than the raw token, and state exposed to assistive technology as text rather than by color alone.

Tests

Every acceptance criterion below is traced to the current requirements revision. A criterion owned by two work orders appears once per side of the split.

Acceptance criterion	Evidence
AC-…-001.4, .5, .6, .7, .8	internal/agent/runtime/lifecycle/executor_ssh_reachability_probe_test.go — TestProbeSucceedsAndClosesWithoutRemoteCommand, TestProbeReportsConfigWithoutDialing, TestProbeRejectsUnpinnedFingerprintWithoutDialing, TestProbeRejectsChangedHostKeyWithoutRepinning, TestProbeDeadlineReportsTimeout, TestProbeAbandonsStalledHandshakeAtDeadline, TestFailureCarriesExactlyOneReasonAndMessage
AC-…-001.27, .29	executor_ssh_reachability_probe_test.go — TestClassifyDialErrorOrdering, table-driven, one case per reason plus the deadline-before-network and host-key-before-auth orderings; TestBastionHostKeyMismatchClassifiesAsHostKey, TestBastionFailureNamesTargetHostAndBastionInMessage
AC-…-001.16, .21, .22	internal/task/repository/sqlite/executor_reachability_test.go — TestSoftDeleteRemovesReachabilityRecord, TestUpsertKeepsLaterObservationRegardlessOfCommitOrder, TestEqualTimestampWriteIsDiscarded, TestRecordAndBothTimestampsSurviveRepositoryReopen; Postgres parity in executor_reachability_postgres_test.go
AC-…-001.1, .2, .3, .13, .14, .15, .17, .20, .26	internal/executors/reachability/poller_test.go — TestPassProbesEligibleExecutorsInIDOrder, TestPassSkipsNonSSHAndInactiveExecutors, TestInactiveExecutorRetainsItsRecord, TestOverlappingTickIsDroppedNotQueued, TestPassConcurrencyIsBounded, TestEmptyPassIsSilent, TestResultForNowIneligibleExecutorIsDiscarded, TestStopDrainsInFlightPass, TestStopDiscardsCancelledProbeResults; goleak_test.go
AC-…-001.9, .10, .11, .12	internal/executors/reachability/hysteresis_test.go — TestNonStickyFailureFlipsOnThresholdProbe, TestStickyFailureFlipsOnFirstProbe, TestBelowThresholdFailureStoresReasonWithoutChangingState, TestSingleSuccessRecoversFromUnreachable, TestAbsentRecordReportsUnknown, each table-driven over probe sequences asserting the exact probe on which the state flips
AC-…-001.23, .24, .25	internal/common/config/config_test.go — TestSSHReachabilityIntervalClampsOutOfRangeAndLogsOnce, TestAbsentNegativeOrMalformedIntervalUsesDefault, TestZeroIntervalDisablesPollerButNegativeDoesNot; internal/executors/reachability/poller_test.go — TestDisabledPollerRunsNoPassIncludingAtStart
AC-…-001.18, .19, .28, AC-…-002.7	internal/task/service/service_resources_executors_reachability_test.go — TestSavingConnectionConfigNotifiesObserverWithBeforeAndAfterFields, TestUnchangedSaveNotifiesNothing, TestCreateExecutorAlwaysNotifies; internal/executors/reachability/save_observer_test.go — TestObserverResetsRecordAndSchedulesOffCycleProbe, TestObserverSkipsIneligibleExecutor; internal/ssh/reachability_handlers_test.go — TestInFlightProbeResultDiscardedAfterConfigSave, TestImmediateProbeRunsWhilePollerDisabled, TestImmediateProbeRefusedForInactiveExecutor
AC-…-002.2, .3	internal/executors/reachability/publish_test.go — TestChangePublishesEvent, TestUnchangedProbePublishesNothing, TestResetPublishesEvent; internal/gateway/websocket/task_notifications_test.go bridge assertion
AC-…-002.1, .4, .5, .6, .10, .11, .12	internal/ssh/reachability_handlers_test.go — TestOverlappingImmediateProbesShareOneProbe, TestSharedProbeCompletesWhenStarterDisconnects, TestSharedProbeNotOnCallerRequestContext, TestGETReturnsUnknownShapeWithNullUpdatedAtForUnprobedExecutor, TestGETReachabilityOrderedByExecutorIDAscending, TestGET404ForSoftDeletedExecutor; apps/web/components/settings/ssh-reachability-card.test.tsx — field coverage per state, stale past three intervals, probing-off copy with no refresh timer, load failure renders "not known", not-persisted probe result; apps/web/lib/state/slices/settings/settings-slice.test.ts — applies executor.reachability.changed, keeps the later updated_at; pnpm run i18n:check
AC-…-003.1	internal/agent/runtime/lifecycle/executor_ssh_reachability_launch_test.go — TestCreateInstanceReadsNoReachabilityRecord, asserting against a store fake that fails the test on any read, plus TestLaunchProceedsAgainstUnreachableExecutor
AC-…-003.2, .3	internal/agent/runtime/lifecycle/manager_launch_reachability_warning_test.go — TestDialFailureMessageNamesHostAndClassifiedReason, TestWarningPublishedForUnreachableExecutorAcrossEveryLaunchPath (WS-initiated, dependency chain, workflow transition, autostart), TestNoWarningWhenProbingDisabledAndRecordStale, TestWarningStillPublishedWhenProbingDisabledButRecordFresh; apps/web/lib/ws/handlers/ssh-launch-warning.test.ts — session-stream event applied to session-runtime state; apps/web/components/task/launch-warning.test.tsx — inline render for the initiating client names host and last-success age, and does not block

E2E tests

The containers Playwright project already gates real-sshd scenarios on KANDEV_E2E_CONTAINERS=1 and owns the kandev-sshd:e2e image, so all four flows live in apps/web/e2e/tests/ssh/reachability.spec.ts under that project.

Flow	Acceptance criteria
A reachable host renders the settings panel row as reachable.	AC-…-002.1
The sshd container is stopped mid-test; after the failure threshold the settings panel reports the host unreachable by name.	AC-…-001.9, AC-…-002.1
The container is restarted; the next probe clears the panel on a single success.	AC-…-001.11, AC-…-002.1
A launch started against the stopped host is attempted and fails with a message naming the host, rather than being refused.	AC-…-003.1, AC-…-003.3

Work orders

• Task 01: Single-host SSH probe and failure classification
• Task 02: Reachability record persistence
• Task 03: Reachability poller, hysteresis, and configuration
• Task 04: Reachability API, change event, and immediate probe
• Task 05: Launch-path non-gating and failure attribution
• Task 06: Reachability data layer and settings surface
• Task 07: Pre-launch reachability warning
• Task 08: Container E2E coverage and operator documentation

Waves

Wave	Tasks	Parallel-safe
1	01, 02	Yes — disjoint packages, only 02 carries a migration
2	03	No — first caller of both primitives
3	04, 05	Yes — disjoint files, both depend only on 03
4	06	No — consumes the contract task 04 defines
5	07	No — reads the store slice task 06 creates
6	08	No — proves the assembled surfaces

Verification results

All 8 work orders are done. Each task's exact verification commands and results are recorded in its own ## Results section (task-01-probe-and-classification.md through task-08-e2e-and-docs.md). Task 08's container E2E spec (apps/web/e2e/tests/ssh/reachability.spec.ts) is the plan's only end-to-end evidence and passes against a real sshd container (KANDEV_E2E_CONTAINERS=1 pnpm e2e:run --project=containers tests/ssh/reachability.spec.ts, 2 passed).

Risks

• The probe's classification depends on error shapes from golang.org/x/crypto/ssh that are not part of that package's stable API. A library upgrade could turn a host_key failure into unknown. Mitigated by keying host_key off Kandev's own errHostKeyMismatch rather than an upstream string, and by table-driven tests over constructed errors; the residual risk is confined to auth versus network on exotic errors, where the fallback is unknown rather than a wrong security verdict.
• A per-interval write path against SQLite adds sustained write traffic that did not exist before. At the default 60s interval and a realistic executor count this is small, but an operator with many SSH executors and a short interval could contend with the busy task_sessions write path. The supported range's 15s floor bounds this; the separate table keeps the contention off the executor row.
• The launch path must read the record for its warning while never reading it to decide. AC-…-003.2 needs the last-success age, so a read exists in the same flow as AC-…-003.1's prohibition. A future change that moves that read into CreateInstance to "avoid a pointless launch" would silently violate REQ-003 without breaking any existing test. Mitigated by keeping the read in Manager.launchBuildExecutorRequest, the one call site every executor-type dispatch already passes through, and by task 05's negative test, which asserts against a store fake that fails on any read.
• The system design's stated package for the warning producer (internal/task/service) does not match the verified call path. Every launch — WS-initiated, a dependency chain, a workflow transition, an autostart — converges in internal/agent/runtime/lifecycle at Manager.launchBuildExecutorRequest's single rt.CreateInstance call site (confirmed against apps/backend/AGENTS.md's Execution Flow and manager_launch.go); internal/task/service is not in that path. This plan and tasks 05/07 use the verified location. The spec still needs a follow-up correction so a future reader isn't sent to the wrong package.
• Deferring the launch-path write leaves the poller as the only writer. A host that dies between passes lags the settings page by up to the failure threshold times the effective interval, and a failed launch — the strongest available evidence — updates nothing. Bounded and accepted: AC-…-003.3 still names the host and the classified reason on the failure itself, which is what the 2026-09-06 report actually asked for.
• Adding a catalog entry touches a deliberately duplicated inventory. auditedStartupEnvironmentInventory() is independent of startupCatalog by design; forgetting it fails TestConfigurationCatalogMatchesAuditedEnvironmentInventory rather than shipping silently, so the risk is a wasted cycle, not a defect.
• Five-locale copy gates the build. check-i18n-keys.mjs fails on a missing key, a dropped placeholder, or a value left identical to English, so tasks 06 and 07 each carry their own translation work rather than deferring it.
• The containers E2E project needs a real Docker daemon and is slow. Task 08's specs cannot run on a host without Docker, so the rest of the work package must be verifiable without them; every acceptance criterion above has non-E2E evidence, and the four flows there re-prove end to end what unit tests already cover against a host that can actually be stopped.