kandev/docs/specs/executors/system-design/ssh-reachability.md

---
status: draft
system: executors
requirements:
  - REQ-EXECUTORS-SSH-REACHABILITY-001
  - REQ-EXECUTORS-SSH-REACHABILITY-002
  - REQ-EXECUTORS-SSH-REACHABILITY-003
created: 2026-09-07
owners:
  - tbd
---
# SSH Host Reachability System Design

## Purpose and boundaries

The executor system owns SSH lifecycle and executor failure and recovery
contracts, so it owns the path answering "is this SSH host still reachable".
This design covers the poller, the probe, its classification, persistence and
configuration. The API, event and web projections are in
[SSH Host Reachability Surfaces](ssh-reachability-surfaces.md).

Contracts this design uses but does not own:

- `internal/agent/runtime/lifecycle` owns SSH target resolution
  (`ResolveSSHTarget`), dialing (`DialSSH`), fingerprint pinning, `ProxyJump`,
  and the dial error types. The probe consumes those, and does not reimplement.
- The task system owns `executors`, `executor_profiles`, and `task_sessions`,
  and the event bus to WebSocket bridge in `internal/gateway/websocket`.
- `internal/common/config` owns the operator configuration catalog.
- `internal/integrations/healthpoll` is precedent, not a dependency: its
  `Prober` is shaped around one integration with many workspaces, while this
  poller sweeps many independent hosts. Its conventions are kept, not imported.

## Requirement mapping

| Requirement | Design section |
| --- | --- |
| `REQ-EXECUTORS-SSH-REACHABILITY-001` | [The poller](#the-poller), [The probe](#the-probe), [State transitions](#state-transitions), [Persistence](#persistence), [Configuration](#configuration) |
| `REQ-EXECUTORS-SSH-REACHABILITY-002` | [Surfaces design](ssh-reachability-surfaces.md) |
| `REQ-EXECUTORS-SSH-REACHABILITY-003` | [Surfaces design](ssh-reachability-surfaces.md) |

The task-card indicator and the launch-path record write are deferred; see the
requirement document's `## Out of scope`.

## Components and responsibilities

**`internal/agent/runtime/lifecycle` (extended).** `SSHExecutor.HealthCheck`
keeps returning `nil`: the runtime is available even when an individual host is
not, and the registry's `HealthCheckAll` sweep is a per-runtime concept called
once from `Manager.Start`. Conflating per-host reachability with runtime health
there would mark the whole SSH runtime down because one of several hosts moved.

This package gains **three** exported functions and the two types they need:

- `SSHTargetFromExecutorConfig(map[string]string) (*SSHTarget, error)` — lifted
  from `internal/ssh`; projects `executors.config` into the `SSHConnConfig` that
  `ResolveSSHTarget` takes. That function needs the typed struct, not a map, so
  without this the reachability package must import `internal/ssh` to read an
  executor's own config.
- `ProbeSSHHost(ctx, target *SSHTarget, timeout time.Duration) SSHProbeOutcome`
  — dials with the pinned fingerprint, closes immediately, runs no remote
  command, and reports success, cancellation, target host, reason, and a
  sanitized message. **It bounds the handshake on the derived probe context,
  not only the TCP dial.** `ssh.NewClientConn` takes no context and
  `ssh.ClientConfig.Timeout` covers only `ssh.Dial`'s own TCP connect, so a host
  that completes TCP and then stalls the handshake would hold its goroutine
  past `probeTimeout` and past `Stop`. `handshakeWithDeadline` sets a deadline
  when the transport supports it and also closes the owner connection from a
  context cancellation callback, which covers ProxyJump channel connections.
- `ClassifyDialError(err error) SSHReachabilityReason` — maps a dial error to
  exactly one reason from the closed set.

**Classification lives here, not in the reachability package, and that is the
load-bearing decision in this section.** The error types it must distinguish are
`lifecycle`'s own and unexported: `errHostKeyMismatch` is lowercase, so no other
package can `errors.As` it. Exporting the *classifier* rather than the *error
types* keeps that boundary intact and gives the launch path identical
classification without a second implementation.
`Reason` is a string enum whose members are exactly `config`, `network`,
`timeout`, `auth`, `host_key`, `unknown`, plus the empty string for success.

**`internal/executors/reachability` (new).** Owns the poller, the concurrency
limiter, the hysteresis rule and the store interface: it enumerates eligible
executors, orders them, runs a bounded pass, calls `ProbeSSHHost`, applies the
state rule, writes results and publishes change events. It depends on a narrow
repository interface, not the full task repository.

**`internal/task/repository/sqlite` (extended).** Owns the
`executor_reachability` table, its additive migration, its queries and the
explicit record delete in [Persistence](#persistence).

**`internal/task/service` (extended).** Owns save detection, because it is the
only layer holding both the stored executor and the incoming change.
`Service.UpdateExecutor` (`service_resources.go`) loads the row, applies the
request and bumps `UpdatedAt`; `Service.CreateExecutor` is the creation call
site. Each notifies an `ExecutorSaveObserver` — an interface the service
declares and the reachability package implements, wired at construction, so the
service depends on an interface rather than on the reachability package. The
observer is handed the connection-configuration fields as they stood before the
save and as they stand after, compares exactly the fields Terminology names,
and on any difference performs the reset in [Persistence](#persistence) and
schedules the off-cycle probe. A create is a difference by definition.

**`applyExecutorUpdates` mutates the loaded executor in place**, so the before
values must be copied out before it runs. Read afterwards they compare the row
against itself, every save looks like a no-op, the reset never fires, and a
re-pointed executor keeps reporting the previous host's state indefinitely.

**`internal/ssh` and `apps/web`.** The HTTP and WebSocket projection, the web
components and the launch interaction. Specified in the
[surfaces design](ssh-reachability-surfaces.md), which owns them; this document
states no contract for them.

## The poller

A single ticker loop, started where the lifecycle manager and the integration
pollers start and stopped on backend shutdown. Both `Start` and `Stop` are
idempotent. Every probe the package starts is registered on one package-owned
`WaitGroup` and derives from one package context, so `Stop` cancels and awaits
*all* of them — the in-flight pass, an off-cycle probe after a save, a coalesced
immediate probe — not only the pass. That is what makes an immediate probe's
package-owned context safe: it outlives its caller but not the package. Draining
is bounded because every probe carries `probeTimeout` *and* because
`ProbeSSHHost` bounds the handshake phase as well as the TCP dial; without that
second half a stalled handshake outlives both its deadline and `Stop`. A `goleak`
`TestMain` guards the package, matching `healthpoll`.

When the effective interval is `0` the poller does not start: no startup pass
and no ticker. The immediate-probe route still works, not going through the
poller loop. The package context, `WaitGroup` and semaphore are still created
and registered for cleanup; only the pass and ticker are skipped. A `Start`
returning early before creating them leaves immediate probes unowned at
shutdown, breaking the drain contract and the `goleak` guard.

One pass:

1. List eligible executors: `type = 'ssh'`, `deleted_at IS NULL`,
   `status = 'active'`, capturing each row's `updated_at` with its id.
2. Sort ascending by the `executors.id` column. `id` is the primary key and is
   never reused, so the order is total; no tiebreak is needed or defined.
3. Acquire the shared probe semaphore per executor, preserving dispatch order.
4. Await all workers, then release the pass.

A `sync.Mutex`-guarded `passRunning` flag makes an overlapping tick a no-op: the
tick is dropped, not queued, and increments a skipped-pass counter. Queueing it
would compound the overload that caused the overlap, and the next tick carries
identical information.

**The semaphore is process-global, not per-pass.** One `passConcurrency`-slot
semaphore, owned by the package, is acquired by every probe whoever asked for
it: a scheduled pass, an off-cycle probe after a save, an immediate-probe
request. Saving twenty executors in a script therefore opens at most
`passConcurrency` connections, not twenty. A launch's own `DialSSH` is not a
probe and never takes a slot: a launch must not queue behind background probing.

Cost per pass is `N` TCP connections and handshakes for `N` hosts, at most
`passConcurrency` at a time, each bounded by `probeTimeout`. At the defaults
below, one host costs one handshake per minute.

## The probe

Per executor, inside its own `context.WithTimeout(ctx, probeTimeout)`:

1. Project `executors.config` into an `SSHConnConfig` and resolve it with
   `lifecycle.ResolveSSHTarget` — the same projection the existing handler
   performs in `sshTargetFromExecutorConfig`, lifted into a shared helper so the
   poller and the handlers cannot drift. A resolution failure classifies as
   `config` and returns without touching the network. **`ResolveSSHTarget` does
   not validate the pin, so the probe checks it separately** and returns `config`
   when the resolved target's `PinnedFingerprint` is empty, before any dial.
   Leaving this to resolution is a security defect rather than a gap:
   `buildClientConfig`'s host-key callback returns `nil` unconditionally when the
   pin is empty, so the probe would dial unpinned, accept whatever key answered,
   and record a substituted host as `reachable`.
2. `lifecycle.ProbeSSHHost(ctx, target)`, dialing with the pinned fingerprint
   set, so a changed host key fails the handshake rather than being silently
   accepted. That is the substantive difference from the manual test path, which
   dials unpinned to observe a fingerprint for first-time trust.
3. On success the client is closed immediately and the outcome is `reachable`.

`ClassifyDialError` walks a total order and returns the first match. The table
below is written **in that order**, and the order is the contract:

| Order | Reason | Condition |
| --- | --- | --- |
| 1 | `config` | Target resolution failed, or resolved with no pinned fingerprint, before any dial. |
| 2 | `timeout` | The probe deadline expired, or the error unwraps to a net timeout. |
| 3 | `host_key` | The handshake failed host-key verification against the pin. |
| 4 | `auth` | Transport negotiation completed; authentication failed. |
| 5 | `network` | TCP dial failed: refused, no route, DNS, network unreachable. |
| 6 | `unknown` | Any other error. |

`config` is decided before any dial. `timeout` precedes `network` because a
deadline expiry surfaces as a dial error and would otherwise read as a refused
connection. `host_key` precedes `auth` because both arrive as handshake failures
and only `host_key` is a security event. The stored message is the error text,
which `dialDirect` already stamps with host and port.

## State transitions

Three states, and one counter that is part of the contract rather than an
implementation detail:

- `unknown`: no record, or a record explicitly reset. Never inferred.
- `reachable`: the last probe succeeded.
- `unreachable`: a sticky failure, or a transient failure streak that reached
  `failureThreshold`.

**On success:** `state = reachable`, `consecutive_failures = 0`, reason and
message cleared, `last_success_at = checked_at`. One success is enough from any
prior state: a host that just answered is reachable, and delaying that
conclusion only prolongs a false alarm.

**On a sticky failure (`config` or `host_key`):** `state = unreachable`
immediately, whatever the counter says. The counter still increments and
`last_success_at` is left alone. Both conditions are deterministic and will
repeat on the next probe, so the threshold would buy no confidence — it would
only delay a possible host-key compromise by a full interval.

**On a transient failure (`network`, `timeout`, `auth`, `unknown`):**
`consecutive_failures += 1`, and the state becomes `unreachable` on the probe
where the counter reaches `failureThreshold`. Until then the state is left
**exactly as it was**: `reachable` stays `reachable` and `unknown` stays
`unknown`. A failing probe never *promotes* a state, so it cannot turn `unknown`
into `reachable`; it changes the reason and the counter while the state stands
still. That pair, a non-empty reason under `reachable` or `unknown`, is the
intended representation of a below-threshold failure, not a contradiction: one
dropped packet must not paint a healthy host as down, while recovery is reported
at once.

A save that changes the connection configuration resets the record to `unknown`,
then probes out of band if the executor is still eligible, because the prior
streak and success describe a different target. A save that changes nothing
resets nothing; the mechanism is in [Persistence](#persistence).

## Persistence

New table, one row per SSH executor:

```sql
CREATE TABLE IF NOT EXISTS executor_reachability (
    executor_id          TEXT PRIMARY KEY,
    state                TEXT NOT NULL DEFAULT 'unknown',
    reason               TEXT NOT NULL DEFAULT '',
    message              TEXT NOT NULL DEFAULT '',
    consecutive_failures INTEGER NOT NULL DEFAULT 0,
    host                 TEXT NOT NULL DEFAULT '',
    checked_at           TIMESTAMP,
    last_success_at      TIMESTAMP,
    updated_at           TIMESTAMP NOT NULL
);
```

`last_success_at` exists because the pre-launch warning must name the age of the
last *successful* probe, and `checked_at` is overwritten by every failure. It is
`NULL` until the first success, and only the configuration reset clears it.

A separate table rather than columns on `executors`: `config` is user-authored
input and `status` a user-controlled switch, so observed state must not share
either, and a per-interval write stays off the row every read touches.

**Deletion is explicit, not a cascade.** `Repository.DeleteExecutor` is a *soft*
delete (`UPDATE executors SET deleted_at = ?, updated_at = ?`), so no foreign-key
cascade can fire and the table declares none. `DeleteExecutor` issues `DELETE
FROM executor_reachability WHERE executor_id = ?` in the same transaction as the
soft delete, and is the only call site that removes a record.

**One statement decides every observation.** The write is a single upsert; the caller
supplies the observation (reason, message, host, `checked_at`) and the statement
derives the counter and the state from the row's own prior values, so no
read-modify-write in Go can lose a streak to an interleaving:

```sql
INSERT INTO executor_reachability (executor_id, state, reason, message,
       consecutive_failures, host, checked_at, last_success_at, updated_at)
SELECT :id, :state, :reason, :message, :failures, :host, :checked_at,
       :last_success_at, :now
 WHERE EXISTS (SELECT 1 FROM executors e
                WHERE e.id = :id AND e.deleted_at IS NULL
                  AND e.status = 'active' AND e.updated_at = :seen_updated_at)
ON CONFLICT (executor_id) DO UPDATE SET
    consecutive_failures = CASE WHEN :reason = '' THEN 0
                                ELSE executor_reachability.consecutive_failures + 1 END,
    state = CASE WHEN :reason = '' THEN 'reachable'
                 WHEN :reason IN ('config','host_key') THEN 'unreachable'
                 WHEN executor_reachability.consecutive_failures + 1 >= :threshold
                      THEN 'unreachable'
                 ELSE executor_reachability.state END,
    reason = :reason, message = :message, host = :host,
    checked_at = :checked_at,
    last_success_at = CASE WHEN :reason = '' THEN :checked_at
                           ELSE executor_reachability.last_success_at END,
    updated_at = :now
WHERE executor_reachability.checked_at IS NULL
   OR excluded.checked_at > executor_reachability.checked_at;
```

On the insert path there is no prior row, so the caller supplies the initial
`state` and `failures`; every later write derives both in SQL from the stored
row. The `ON CONFLICT` `WHERE` is last-write-wins on `checked_at`: a scheduled
pass and an immediate probe racing on one executor converge on the later
observation rather than on whichever transaction committed second, and an equal
timestamp is not later, so the incoming row loses. `checked_at` is stored at
millisecond resolution or finer, keeping the tie rule a backstop.

**That `>` is a text comparison, so the encoding is part of the contract.**
`TIMESTAMP` takes NUMERIC affinity and the driver binds `time.Time` as text
(`2006-01-02 15:04:05.999999999-07:00`), so SQLite orders these lexically, not
temporally. Two rules make lexical order equal time order, and both are
required: **normalize to UTC** before writing, because the driver keeps whatever
offset it is handed and a DST fall-back then sorts an earlier observation later;
and **bind `time.Time`, never a pre-formatted string**, because RFC3339 renders
UTC as `Z`, which sorts after `.` and would let a whole-second observation beat
a later sub-second one. Postgres compares temporally and is unaffected, which is
why this belongs in the SQLite tests: the parity test passes on Postgres while
SQLite keeps the wrong row.

The eligibility guard is the `EXISTS` above, and `:seen_updated_at` is the
`updated_at` the probe captured before it dialed. Every mutation of the executor
row — configuration save, status change, soft delete — bumps `updated_at`, so one
comparison covers all three: it stops a probe in flight against the *old* target
from landing after a reset, and stops a worker from re-creating a deleted record
or writing one for an executor deactivated mid-pass. The `EXISTS` gates the
insert; the `ON CONFLICT` arm inherits it, because a row can only conflict if the
insert was admitted. A refused write is logged at debug and dropped.

**The reset is a second statement, and deliberately not that one.** A
configuration reset is not an observation; it invalidates one. The upsert above
cannot express it — its `state` CASE has no `unknown` branch, and its `WHERE`
admits only a strictly later `checked_at`, which a reset does not carry. Nor may
it be expressed as a delete: `DeleteExecutor` is the only call site permitted to
remove a row.

```sql
INSERT INTO executor_reachability (executor_id, state, reason, message,
       consecutive_failures, host, checked_at, last_success_at, updated_at)
SELECT :id, 'unknown', '', '', 0, :host, NULL, NULL, :now
 WHERE EXISTS (SELECT 1 FROM executors e
                WHERE e.id = :id AND e.type = 'ssh'
                  AND e.deleted_at IS NULL AND e.status = 'active')
ON CONFLICT (executor_id) DO UPDATE SET
    state = 'unknown', reason = '', message = '', consecutive_failures = 0,
    host = :host, checked_at = NULL, last_success_at = NULL, updated_at = :now;
```

The `EXISTS` carries a decision, not just a safety check: an executor whose
`status` is not `active` is **not** reset by a save, because
`AC-EXECUTORS-SSH-REACHABILITY-001.17` retains its record unchanged and
eligibility wins over the reset trigger. The statement lands zero rows and the
retained record stands until the executor is reactivated and probed.

`:host` is the newly saved host, so the settings page never renders a pre-change
host beside `unknown` — the window this capability exists to illuminate is
exactly the one that would otherwise lie. There is deliberately no `checked_at`
guard and no `:seen_updated_at` pin: a reset is ordered by the save that caused
it, not by an observation clock, so it must win over whatever is stored. A probe
already in flight against the pre-save target cannot overwrite it afterwards,
because that probe's own write still carries the pre-save `updated_at` and the
upsert's `EXISTS` rejects it. That is the mechanism
`AC-EXECUTORS-SSH-REACHABILITY-001.19` requires, and it is why the reset needs no
clock of its own.

The reset advances the row's `updated_at` like any other write, which is what
lets a client order it against a concurrent probe result: see the surfaces
design's reconciliation rule.

Retention is one row per executor. The table survives restart and the surface
reports the preserved record until a new probe replaces it: a blank panel after
every restart would be worse than a timestamped result marked stale.

The migration is a single additive `CREATE TABLE IF NOT EXISTS` applied through
the existing idempotent mechanism in `internal/task/repository/sqlite`, so it is
safe on SQLite and Postgres and on an already-migrated database.

## Configuration

One catalog entry in `internal/common/config/catalog.go`:

| Key | Environment variable | Default |
| --- | --- | --- |
| `executors.sshReachabilityIntervalSeconds` | `KANDEV_EXECUTORS_SSHREACHABILITYINTERVALSECONDS` | `60` |

Supported range 15 to 3600 seconds; `0` disables the poller. **The catalog does
not clamp.** `applyNonNegativeIntEnv` falls back to the *default* — never to a
bound — for a value that is absent, empty, negative, fractional or unparsable, so
every one of those yields `60`. Its minimum must stay `0`: widening it to `15` to
express the supported range would push `0` below the minimum and silently turn
the documented kill switch into a 60-second cadence.

**That normalization is the environment path only, and the two sources do not
behave alike.** `applyBoundedIntEnv`, which `applyNonNegativeIntEnv` wraps, reads
`nonEmptyEnv(env, entry.EnvVars...)` and returns early when no environment
variable is set, so a value written in `config.yaml` never reaches its
fallback-and-bounds logic at all. A YAML value is decoded by `decodeConfig`,
which sets no `WeaklyTypedInput`, and `Load` returns the decode error, so a
non-integer in `config.yaml` refuses boot exactly as it does for every other
typed key in the catalog. That is the documented behavior rather than an
exception carved out for this key, and
`AC-EXECUTORS-SSH-REACHABILITY-001.24` states it that way instead of promising a
leniency the loader does not implement. A *negative whole number* does decode
cleanly and so does reach the package, which is why the clamp below needs a
negative branch of its own.

Clamping is therefore the package's job, applied to whatever value reaches it:
below `0` yields the default `60`, `0` disables, `1`-`14` becomes `15`, above
`3600` becomes `3600`, and a clamp logs once at startup rather than refusing to
boot.

**The clamped value is the effective interval, and every consumer uses it.**
Staleness (three times the interval), the `probing_enabled` projection, and the
settings page's own refresh cadence all read the effective interval, never the
raw configured number. An operator who writes `5` gets a 15-second cadence and a
45-second staleness horizon, consistently across the backend and the UI.

The other three knobs are compile-time constants, not configuration, because no
operator scenario has been identified that needs them independently and each new
key is a permanent contract:

| Constant | Value | Rationale |
| --- | --- | --- |
| `probeTimeout` | 10s | `sshDialTimeout` is 30s, which would let one hung host hold a slot for half the interval. |
| `passConcurrency` | 4 | Bounds simultaneous outbound connections and file descriptors while keeping a pass short. |
| `failureThreshold` | 2 | Smallest value that suppresses a single-probe blip; worst-case detection is two intervals. |

This capability adds no runtime feature toggle: the interval key already
provides the kill switch through the designated operator-startup mechanism, and
a toggle would leave a retired identity to carry forever.

## Control flow

```text
ticker -> poller.pass()        (only if effective interval > 0)
  | list eligible SSH executors by executors.id, with updated_at
  v
global semaphore(passConcurrency)   <- off-cycle and immediate probes
  v
probe()   SSHTargetFromExecutorConfig > ProbeSSHHost > ClassifyDialError
  v
store.Upsert(observation)      guards: eligible, updated_at, checked_at
  |- state or reason changed -> bus.Publish -> WS -> store slice
  '- unchanged -> stop

save -> ExecutorSaveObserver   (do the connection-config fields differ?)
  '- yes -> store.Reset(host)  guard: eligible -> bus.Publish -> off-cycle probe

launch -> session creation     reads record, emits session.launch.warning
  '- SSHExecutor.CreateInstance      (reads no record, writes no record)
       '- failed at its dial -> ClassifyDialError -> error names host
```

## Failure and recovery

The poller is best-effort and never a source of truth for a caller. Every
failure mode degrades to a stale or `unknown` record and a log line:

- Listing executors fails: the pass is abandoned with a warning, nothing is
  written, the next tick retries.
- A probe panics or errors unexpectedly: it classifies `unknown` and the pass
  continues; no other executor is affected.
- A record write fails: logged, the pass completes, the next probe rewrites,
  costing one interval of freshness.
- A write refused by a guard is expected, not an error: logged at debug and
  dropped.
- A probe exceeding `probeTimeout` is abandoned by its context; the client is
  closed on the returning path so it leaks no connection, and its slot released.
- A probe cancelled by `Stop` writes **nothing**, which is not the case above.
  Shutdown cancels the package context, so an in-flight dial returns
  `context.Canceled`, and no guard catches it: a shutdown mutates no executor
  row, so `EXISTS` and `updated_at` still admit the write. Recorded, it would
  increment `consecutive_failures` for every probe in flight — up to
  `passConcurrency` hosts per restart — and at `failureThreshold` a second
  restart would flip a healthy host to `unreachable`, turning a routine deploy
  into exactly the false alarm this capability exists to prevent.
  `ClassifyDialError` must therefore separate the two context errors: only a
  deadline is a `timeout` observation, and a cancellation is not an observation.
- The store is unreachable on an API read: the route errors and the surface
  reports reachability as not known, never `unreachable`.

There are no retries inside a pass: the next tick is the retry, and an immediate
retry against a host that just refused a connection adds cost, not information.

## Security

The probe uses the pinned fingerprint on the dial to the **target**, so a
host-key change there is a detected `host_key` failure and never a silent
re-pin; the stored fingerprint is read-only to this path. `host_key` reports
`unreachable` on its first occurrence rather than waiting out the threshold: a
mismatch is possible interception, and delaying it a full interval is a security
cost with no accuracy benefit. Credentials are unchanged: the probe reuses the
executor's configured identity source and holds no secret of its own.

`message` carries an SSH error string, which can include host, port, username
and identity path — all values the user configured and can already see on the
same settings page, so the record is not sensitive. `ClassifyDialError` must not
place key material or agent socket contents into `message`.

The probe opens no remote shell and runs no remote command, so it grants nothing
beyond what the manual test already exercises, and adds no new trust boundary.

**`ProxyJump` is a second hop with a weaker trust model, and the reason set must
not paper over it.** An executor pins one fingerprint and it belongs to the
target; `dialViaJump` verifies the bastion against `~/.ssh/known_hosts` instead,
accepting an absent host with a logged warning (OpenSSH's
`StrictHostKeyChecking=accept-new`). Two consequences the probe inherits:

- A bastion key *mismatch* is rejected, but surfaces as `ssh: bastion dial: …`,
  not as `errHostKeyMismatch`, so `ClassifyDialError` would call it `network` or
  `unknown` and wait out `failureThreshold` under the wrong reason. The
  classifier must map a bastion host-key rejection to `host_key`, which
  `AC-EXECUTORS-SSH-REACHABILITY-001.29` now requires observably.
- Every bastion-path error is about the *bastion*, while the record's `host` is
  the target. A bastion that is down reports the target unreachable and sends an
  operator to a machine that is fine — the same misattribution as the firewall
  message of 2026-09-06, from a different cause. **`host` stays the target in
  every case**: it identifies the executor's subject, the column keeps one
  meaning for success and failure alike, and `AC-…-001.4` and `AC-…-002.1` stay
  true for a probe that succeeds through a jump. The disambiguation belongs in
  `message`, which must name the bastion and state that the failure arose on the
  jump hop whenever it did. `message` is rendered beside the state wherever an
  unreachable record is shown, so the operator reads "unreachable … via bastion
  X" instead of being sent to a healthy target.

## Observability

Structured `zap` logs plus `expvar` counters under `/debug/vars`, following the
`office_stall_*` and `routing_*` precedent. Every name below carries the prefix
`executor_ssh_reachability_`:

- `probe_total`, labelled by outcome (`reachable` plus each failure reason).
- `state_transitions_total`, by destination state.
- `pass_skipped_total`, when a tick is dropped because a pass is still running:
  the interval is shorter than a pass.
- `write_refused_total`, when a guard drops a write. A rising value means
  configuration churn, not a fault.
- `reset_total`, when a save invalidates a record.
- `probe_discarded_total`, when `Stop` cancelled a probe. `probe_total` is
  labelled by outcome and a cancelled probe has none, so without this the
  shutdown discard is the package's only uncounted drop.
- `probe_duration_ms`, the last pass's aggregate probe duration.

A transition logs at `Warn` for `reachable` to `unreachable` and `Info` for the
reverse, with executor id, host, reason and failure count. Unchanged results do
not log, so a steady host stays silent.

## Prior art and departures

`internal/integrations/healthpoll` is the closest in-repo precedent: an
immediate probe on start, a fixed cadence, a configured-or-skip gate,
best-effort semantics, idempotent `Start`/`Stop`, and a goroutine-leak test.
Those conventions are kept. Its persistence shape — a timestamp, a boolean and an
error string on the config row — is not, in four places, because an SSH host is
not an HTTP API: a **closed reason set** rather than an error string, since five
failures are acted on differently and only a typed reason lets the UI say
"fingerprint changed"; **asymmetric hysteresis** rather than flipping on the
first result, which would let one dropped packet paint a healthy host as down
(see [State transitions](#state-transitions)); a **separate record** rather than
a column, since observed state must not share a row with user-authored input
(see [Persistence](#persistence)); and **changes pushed, results not**, since
pushing every result would put one message per executor per interval on the wire
forever.

## Testing

Strategy only; the plan's `## Tests` table owns the per-criterion evidence.

- Poller tests drive the ticker with `testing/synctest`, matching
  `healthpoll_test.go`, under a `goleak` `TestMain`.
- Classification and hysteresis are table-driven — over constructed errors and
  over probe sequences — so the ordering rules and the exact probe each state
  flips on are asserted, not inferred.
- The upsert's ordering rule is exercised on **both** dialects, including the
  whole-second versus sub-second pair the text-encoding trap turns into a silent
  SQLite-only failure.
- The guards get a negative test each: soft-deleted, deactivated or reconfigured
  mid-probe writes nothing. The reset gets its own: it lands with no
  `checked_at`, and a probe in flight from before the save does not overwrite it.
- The immediate-probe route is tested for caller cancellation and for a
  guard-refused write returning `persisted: false`.
- Web E2E lives in `apps/web/e2e/tests/ssh/` under the `containers` project,
  gated on `KANDEV_E2E_CONTAINERS=1`.

## Related decisions

No new ADR: this design creates no system boundary and no ownership rule, and
extends an existing contract through the repository's designated mechanisms for
configuration, persistence, events and metrics.
([0018](../../../decisions/0018-runtime-settings-overrides.md) covers the
adjacent runtime-override tier.)