kandev/docs/specs/executors/system-design/ssh-reachability.md

status, system, requirements, created, owners

status	system	requirements	created	owners
draft	executors	REQ-EXECUTORS-SSH-REACHABILITY-001 REQ-EXECUTORS-SSH-REACHABILITY-002 REQ-EXECUTORS-SSH-REACHABILITY-003	2026-09-07	tbd

SSH Host Reachability System Design

Purpose and boundaries

The executor system owns SSH lifecycle and executor failure and recovery contracts, so it owns the path answering "is this SSH host still reachable". This design covers the poller, the probe, its classification, persistence and configuration. The API, event and web projections are in SSH Host Reachability Surfaces.

Contracts this design uses but does not own:

• internal/agent/runtime/lifecycle owns SSH target resolution (ResolveSSHTarget), dialing (DialSSH), fingerprint pinning, ProxyJump, and the dial error types. The probe consumes those, and does not reimplement.
• The task system owns executors, executor_profiles, and task_sessions, and the event bus to WebSocket bridge in internal/gateway/websocket.
• internal/common/config owns the operator configuration catalog.
• internal/integrations/healthpoll is precedent, not a dependency: its Prober is shaped around one integration with many workspaces, while this poller sweeps many independent hosts. Its conventions are kept, not imported.

Requirement mapping

Requirement	Design section
REQ-EXECUTORS-SSH-REACHABILITY-001	The poller, The probe, State transitions, Persistence, Configuration
REQ-EXECUTORS-SSH-REACHABILITY-002	Surfaces design
REQ-EXECUTORS-SSH-REACHABILITY-003	Surfaces design

The task-card indicator and the launch-path record write are deferred; see the requirement document's ## Out of scope.

Components and responsibilities

internal/agent/runtime/lifecycle (extended). SSHExecutor.HealthCheck keeps returning nil: the runtime is available even when an individual host is not, and the registry's HealthCheckAll sweep is a per-runtime concept called once from Manager.Start. Conflating per-host reachability with runtime health there would mark the whole SSH runtime down because one of several hosts moved.

This package gains three exported functions and the two types they need:

• SSHTargetFromExecutorConfig(map[string]string) (*SSHTarget, error) — lifted from internal/ssh; projects executors.config into the SSHConnConfig that ResolveSSHTarget takes. That function needs the typed struct, not a map, so without this the reachability package must import internal/ssh to read an executor's own config.
• ProbeSSHHost(ctx, target *SSHTarget, timeout time.Duration) SSHProbeOutcome — dials with the pinned fingerprint, closes immediately, runs no remote command, and reports success, cancellation, target host, reason, and a sanitized message. It bounds the handshake on the derived probe context, not only the TCP dial. ssh.NewClientConn takes no context and ssh.ClientConfig.Timeout covers only ssh.Dial's own TCP connect, so a host that completes TCP and then stalls the handshake would hold its goroutine past probeTimeout and past Stop. handshakeWithDeadline sets a deadline when the transport supports it and also closes the owner connection from a context cancellation callback, which covers ProxyJump channel connections.
• ClassifyDialError(err error) SSHReachabilityReason — maps a dial error to exactly one reason from the closed set.

Classification lives here, not in the reachability package, and that is the load-bearing decision in this section. The error types it must distinguish are lifecycle's own and unexported: errHostKeyMismatch is lowercase, so no other package can errors.As it. Exporting the classifier rather than the error types keeps that boundary intact and gives the launch path identical classification without a second implementation. Reason is a string enum whose members are exactly config, network, timeout, auth, host_key, unknown, plus the empty string for success.

internal/executors/reachability (new). Owns the poller, the concurrency limiter, the hysteresis rule and the store interface: it enumerates eligible executors, orders them, runs a bounded pass, calls ProbeSSHHost, applies the state rule, writes results and publishes change events. It depends on a narrow repository interface, not the full task repository.

internal/task/repository/sqlite (extended). Owns the executor_reachability table, its additive migration, its queries and the explicit record delete in Persistence.

internal/task/service (extended). Owns save detection, because it is the only layer holding both the stored executor and the incoming change. Service.UpdateExecutor (service_resources.go) loads the row, applies the request and bumps UpdatedAt; Service.CreateExecutor is the creation call site. Each notifies an ExecutorSaveObserver — an interface the service declares and the reachability package implements, wired at construction, so the service depends on an interface rather than on the reachability package. The observer is handed the connection-configuration fields as they stood before the save and as they stand after, compares exactly the fields Terminology names, and on any difference performs the reset in Persistence and schedules the off-cycle probe. A create is a difference by definition.

applyExecutorUpdates mutates the loaded executor in place, so the before values must be copied out before it runs. Read afterwards they compare the row against itself, every save looks like a no-op, the reset never fires, and a re-pointed executor keeps reporting the previous host's state indefinitely.

internal/ssh and apps/web. The HTTP and WebSocket projection, the web components and the launch interaction. Specified in the surfaces design, which owns them; this document states no contract for them.

The poller

A single ticker loop, started where the lifecycle manager and the integration pollers start and stopped on backend shutdown. Both Start and Stop are idempotent. Every probe the package starts is registered on one package-owned WaitGroup and derives from one package context, so Stop cancels and awaits all of them — the in-flight pass, an off-cycle probe after a save, a coalesced immediate probe — not only the pass. That is what makes an immediate probe's package-owned context safe: it outlives its caller but not the package. Draining is bounded because every probe carries probeTimeout and because ProbeSSHHost bounds the handshake phase as well as the TCP dial; without that second half a stalled handshake outlives both its deadline and Stop. A goleak TestMain guards the package, matching healthpoll.

When the effective interval is 0 the poller does not start: no startup pass and no ticker. The immediate-probe route still works, not going through the poller loop. The package context, WaitGroup and semaphore are still created and registered for cleanup; only the pass and ticker are skipped. A Start returning early before creating them leaves immediate probes unowned at shutdown, breaking the drain contract and the goleak guard.

One pass:

1. List eligible executors: type = 'ssh', deleted_at IS NULL, status = 'active', capturing each row's updated_at with its id.
2. Sort ascending by the executors.id column. id is the primary key and is never reused, so the order is total; no tiebreak is needed or defined.
3. Acquire the shared probe semaphore per executor, preserving dispatch order.
4. Await all workers, then release the pass.

A sync.Mutex-guarded passRunning flag makes an overlapping tick a no-op: the tick is dropped, not queued, and increments a skipped-pass counter. Queueing it would compound the overload that caused the overlap, and the next tick carries identical information.

The semaphore is process-global, not per-pass. One passConcurrency-slot semaphore, owned by the package, is acquired by every probe whoever asked for it: a scheduled pass, an off-cycle probe after a save, an immediate-probe request. Saving twenty executors in a script therefore opens at most passConcurrency connections, not twenty. A launch's own DialSSH is not a probe and never takes a slot: a launch must not queue behind background probing.

Cost per pass is N TCP connections and handshakes for N hosts, at most passConcurrency at a time, each bounded by probeTimeout. At the defaults below, one host costs one handshake per minute.

The probe

Per executor, inside its own context.WithTimeout(ctx, probeTimeout):

1. Project executors.config into an SSHConnConfig and resolve it with lifecycle.ResolveSSHTarget — the same projection the existing handler performs in sshTargetFromExecutorConfig, lifted into a shared helper so the poller and the handlers cannot drift. A resolution failure classifies as config and returns without touching the network. ResolveSSHTarget does not validate the pin, so the probe checks it separately and returns config when the resolved target's PinnedFingerprint is empty, before any dial. Leaving this to resolution is a security defect rather than a gap: buildClientConfig's host-key callback returns nil unconditionally when the pin is empty, so the probe would dial unpinned, accept whatever key answered, and record a substituted host as reachable.
2. lifecycle.ProbeSSHHost(ctx, target), dialing with the pinned fingerprint set, so a changed host key fails the handshake rather than being silently accepted. That is the substantive difference from the manual test path, which dials unpinned to observe a fingerprint for first-time trust.
3. On success the client is closed immediately and the outcome is reachable.

ClassifyDialError walks a total order and returns the first match. The table below is written in that order, and the order is the contract:

Order	Reason	Condition
1	config	Target resolution failed, or resolved with no pinned fingerprint, before any dial.
2	timeout	The probe deadline expired, or the error unwraps to a net timeout.
3	host_key	The handshake failed host-key verification against the pin.
4	auth	Transport negotiation completed; authentication failed.
5	network	TCP dial failed: refused, no route, DNS, network unreachable.
6	unknown	Any other error.

config is decided before any dial. timeout precedes network because a deadline expiry surfaces as a dial error and would otherwise read as a refused connection. host_key precedes auth because both arrive as handshake failures and only host_key is a security event. The stored message is the error text, which dialDirect already stamps with host and port.

State transitions

Three states, and one counter that is part of the contract rather than an implementation detail:

• unknown: no record, or a record explicitly reset. Never inferred.
• reachable: the last probe succeeded.
• unreachable: a sticky failure, or a transient failure streak that reached failureThreshold.

On success: state = reachable, consecutive_failures = 0, reason and message cleared, last_success_at = checked_at. One success is enough from any prior state: a host that just answered is reachable, and delaying that conclusion only prolongs a false alarm.

On a sticky failure (config or host_key): state = unreachable immediately, whatever the counter says. The counter still increments and last_success_at is left alone. Both conditions are deterministic and will repeat on the next probe, so the threshold would buy no confidence — it would only delay a possible host-key compromise by a full interval.

On a transient failure (network, timeout, auth, unknown): consecutive_failures += 1, and the state becomes unreachable on the probe where the counter reaches failureThreshold. Until then the state is left exactly as it was: reachable stays reachable and unknown stays unknown. A failing probe never promotes a state, so it cannot turn unknown into reachable; it changes the reason and the counter while the state stands still. That pair, a non-empty reason under reachable or unknown, is the intended representation of a below-threshold failure, not a contradiction: one dropped packet must not paint a healthy host as down, while recovery is reported at once.

A save that changes the connection configuration resets the record to unknown, then probes out of band if the executor is still eligible, because the prior streak and success describe a different target. A save that changes nothing resets nothing; the mechanism is in Persistence.

Persistence

New table, one row per SSH executor:

CREATE TABLE IF NOT EXISTS executor_reachability (
    executor_id          TEXT PRIMARY KEY,
    state                TEXT NOT NULL DEFAULT 'unknown',
    reason               TEXT NOT NULL DEFAULT '',
    message              TEXT NOT NULL DEFAULT '',
    consecutive_failures INTEGER NOT NULL DEFAULT 0,
    host                 TEXT NOT NULL DEFAULT '',
    checked_at           TIMESTAMP,
    last_success_at      TIMESTAMP,
    updated_at           TIMESTAMP NOT NULL
);

last_success_at exists because the pre-launch warning must name the age of the last successful probe, and checked_at is overwritten by every failure. It is NULL until the first success, and only the configuration reset clears it.

A separate table rather than columns on executors: config is user-authored input and status a user-controlled switch, so observed state must not share either, and a per-interval write stays off the row every read touches.

Deletion is explicit, not a cascade. Repository.DeleteExecutor is a soft delete (UPDATE executors SET deleted_at = ?, updated_at = ?), so no foreign-key cascade can fire and the table declares none. DeleteExecutor issues DELETE FROM executor_reachability WHERE executor_id = ? in the same transaction as the soft delete, and is the only call site that removes a record.

One statement decides every observation. The write is a single upsert; the caller supplies the observation (reason, message, host, checked_at) and the statement derives the counter and the state from the row's own prior values, so no read-modify-write in Go can lose a streak to an interleaving:

INSERT INTO executor_reachability (executor_id, state, reason, message,
       consecutive_failures, host, checked_at, last_success_at, updated_at)
SELECT :id, :state, :reason, :message, :failures, :host, :checked_at,
       :last_success_at, :now
 WHERE EXISTS (SELECT 1 FROM executors e
                WHERE e.id = :id AND e.deleted_at IS NULL
                  AND e.status = 'active' AND e.updated_at = :seen_updated_at)
ON CONFLICT (executor_id) DO UPDATE SET
    consecutive_failures = CASE WHEN :reason = '' THEN 0
                                ELSE executor_reachability.consecutive_failures + 1 END,
    state = CASE WHEN :reason = '' THEN 'reachable'
                 WHEN :reason IN ('config','host_key') THEN 'unreachable'
                 WHEN executor_reachability.consecutive_failures + 1 >= :threshold
                      THEN 'unreachable'
                 ELSE executor_reachability.state END,
    reason = :reason, message = :message, host = :host,
    checked_at = :checked_at,
    last_success_at = CASE WHEN :reason = '' THEN :checked_at
                           ELSE executor_reachability.last_success_at END,
    updated_at = :now
WHERE executor_reachability.checked_at IS NULL
   OR excluded.checked_at > executor_reachability.checked_at;

On the insert path there is no prior row, so the caller supplies the initial state and failures; every later write derives both in SQL from the stored row. The ON CONFLICT WHERE is last-write-wins on checked_at: a scheduled pass and an immediate probe racing on one executor converge on the later observation rather than on whichever transaction committed second, and an equal timestamp is not later, so the incoming row loses. checked_at is stored at millisecond resolution or finer, keeping the tie rule a backstop.

That > is a text comparison, so the encoding is part of the contract. TIMESTAMP takes NUMERIC affinity and the driver binds time.Time as text (2006-01-02 15:04:05.999999999-07:00), so SQLite orders these lexically, not temporally. Two rules make lexical order equal time order, and both are required: normalize to UTC before writing, because the driver keeps whatever offset it is handed and a DST fall-back then sorts an earlier observation later; and bind time.Time, never a pre-formatted string, because RFC3339 renders UTC as Z, which sorts after . and would let a whole-second observation beat a later sub-second one. Postgres compares temporally and is unaffected, which is why this belongs in the SQLite tests: the parity test passes on Postgres while SQLite keeps the wrong row.

The eligibility guard is the EXISTS above, and :seen_updated_at is the updated_at the probe captured before it dialed. Every mutation of the executor row — configuration save, status change, soft delete — bumps updated_at, so one comparison covers all three: it stops a probe in flight against the old target from landing after a reset, and stops a worker from re-creating a deleted record or writing one for an executor deactivated mid-pass. The EXISTS gates the insert; the ON CONFLICT arm inherits it, because a row can only conflict if the insert was admitted. A refused write is logged at debug and dropped.

The reset is a second statement, and deliberately not that one. A configuration reset is not an observation; it invalidates one. The upsert above cannot express it — its state CASE has no unknown branch, and its WHERE admits only a strictly later checked_at, which a reset does not carry. Nor may it be expressed as a delete: DeleteExecutor is the only call site permitted to remove a row.

INSERT INTO executor_reachability (executor_id, state, reason, message,
       consecutive_failures, host, checked_at, last_success_at, updated_at)
SELECT :id, 'unknown', '', '', 0, :host, NULL, NULL, :now
 WHERE EXISTS (SELECT 1 FROM executors e
                WHERE e.id = :id AND e.type = 'ssh'
                  AND e.deleted_at IS NULL AND e.status = 'active')
ON CONFLICT (executor_id) DO UPDATE SET
    state = 'unknown', reason = '', message = '', consecutive_failures = 0,
    host = :host, checked_at = NULL, last_success_at = NULL, updated_at = :now;

The EXISTS carries a decision, not just a safety check: an executor whose status is not active is not reset by a save, because AC-EXECUTORS-SSH-REACHABILITY-001.17 retains its record unchanged and eligibility wins over the reset trigger. The statement lands zero rows and the retained record stands until the executor is reactivated and probed.

:host is the newly saved host, so the settings page never renders a pre-change host beside unknown — the window this capability exists to illuminate is exactly the one that would otherwise lie. There is deliberately no checked_at guard and no :seen_updated_at pin: a reset is ordered by the save that caused it, not by an observation clock, so it must win over whatever is stored. A probe already in flight against the pre-save target cannot overwrite it afterwards, because that probe's own write still carries the pre-save updated_at and the upsert's EXISTS rejects it. That is the mechanism AC-EXECUTORS-SSH-REACHABILITY-001.19 requires, and it is why the reset needs no clock of its own.

The reset advances the row's updated_at like any other write, which is what lets a client order it against a concurrent probe result: see the surfaces design's reconciliation rule.

Retention is one row per executor. The table survives restart and the surface reports the preserved record until a new probe replaces it: a blank panel after every restart would be worse than a timestamped result marked stale.

The migration is a single additive CREATE TABLE IF NOT EXISTS applied through the existing idempotent mechanism in internal/task/repository/sqlite, so it is safe on SQLite and Postgres and on an already-migrated database.

Configuration

One catalog entry in internal/common/config/catalog.go:

Key	Environment variable	Default
executors.sshReachabilityIntervalSeconds	KANDEV_EXECUTORS_SSHREACHABILITYINTERVALSECONDS	60

Supported range 15 to 3600 seconds; 0 disables the poller. The catalog does not clamp. applyNonNegativeIntEnv falls back to the default — never to a bound — for a value that is absent, empty, negative, fractional or unparsable, so every one of those yields 60. Its minimum must stay 0: widening it to 15 to express the supported range would push 0 below the minimum and silently turn the documented kill switch into a 60-second cadence.

That normalization is the environment path only, and the two sources do not behave alike. applyBoundedIntEnv, which applyNonNegativeIntEnv wraps, reads nonEmptyEnv(env, entry.EnvVars...) and returns early when no environment variable is set, so a value written in config.yaml never reaches its fallback-and-bounds logic at all. A YAML value is decoded by decodeConfig, which sets no WeaklyTypedInput, and Load returns the decode error, so a non-integer in config.yaml refuses boot exactly as it does for every other typed key in the catalog. That is the documented behavior rather than an exception carved out for this key, and AC-EXECUTORS-SSH-REACHABILITY-001.24 states it that way instead of promising a leniency the loader does not implement. A negative whole number does decode cleanly and so does reach the package, which is why the clamp below needs a negative branch of its own.

Clamping is therefore the package's job, applied to whatever value reaches it: below 0 yields the default 60, 0 disables, 1-14 becomes 15, above 3600 becomes 3600, and a clamp logs once at startup rather than refusing to boot.

The clamped value is the effective interval, and every consumer uses it. Staleness (three times the interval), the probing_enabled projection, and the settings page's own refresh cadence all read the effective interval, never the raw configured number. An operator who writes 5 gets a 15-second cadence and a 45-second staleness horizon, consistently across the backend and the UI.

The other three knobs are compile-time constants, not configuration, because no operator scenario has been identified that needs them independently and each new key is a permanent contract:

Constant	Value	Rationale
probeTimeout	10s	sshDialTimeout is 30s, which would let one hung host hold a slot for half the interval.
passConcurrency	4	Bounds simultaneous outbound connections and file descriptors while keeping a pass short.
failureThreshold	2	Smallest value that suppresses a single-probe blip; worst-case detection is two intervals.

This capability adds no runtime feature toggle: the interval key already provides the kill switch through the designated operator-startup mechanism, and a toggle would leave a retired identity to carry forever.

Control flow

ticker -> poller.pass()        (only if effective interval > 0)
  | list eligible SSH executors by executors.id, with updated_at
  v
global semaphore(passConcurrency)   <- off-cycle and immediate probes
  v
probe()   SSHTargetFromExecutorConfig > ProbeSSHHost > ClassifyDialError
  v
store.Upsert(observation)      guards: eligible, updated_at, checked_at
  |- state or reason changed -> bus.Publish -> WS -> store slice
  '- unchanged -> stop

save -> ExecutorSaveObserver   (do the connection-config fields differ?)
  '- yes -> store.Reset(host)  guard: eligible -> bus.Publish -> off-cycle probe

launch -> session creation     reads record, emits session.launch.warning
  '- SSHExecutor.CreateInstance      (reads no record, writes no record)
       '- failed at its dial -> ClassifyDialError -> error names host

Failure and recovery

The poller is best-effort and never a source of truth for a caller. Every failure mode degrades to a stale or unknown record and a log line:

• Listing executors fails: the pass is abandoned with a warning, nothing is written, the next tick retries.
• A probe panics or errors unexpectedly: it classifies unknown and the pass continues; no other executor is affected.
• A record write fails: logged, the pass completes, the next probe rewrites, costing one interval of freshness.
• A write refused by a guard is expected, not an error: logged at debug and dropped.
• A probe exceeding probeTimeout is abandoned by its context; the client is closed on the returning path so it leaks no connection, and its slot released.
• A probe cancelled by Stop writes nothing, which is not the case above. Shutdown cancels the package context, so an in-flight dial returns context.Canceled, and no guard catches it: a shutdown mutates no executor row, so EXISTS and updated_at still admit the write. Recorded, it would increment consecutive_failures for every probe in flight — up to passConcurrency hosts per restart — and at failureThreshold a second restart would flip a healthy host to unreachable, turning a routine deploy into exactly the false alarm this capability exists to prevent. ClassifyDialError must therefore separate the two context errors: only a deadline is a timeout observation, and a cancellation is not an observation.
• The store is unreachable on an API read: the route errors and the surface reports reachability as not known, never unreachable.

There are no retries inside a pass: the next tick is the retry, and an immediate retry against a host that just refused a connection adds cost, not information.

Security

The probe uses the pinned fingerprint on the dial to the target, so a host-key change there is a detected host_key failure and never a silent re-pin; the stored fingerprint is read-only to this path. host_key reports unreachable on its first occurrence rather than waiting out the threshold: a mismatch is possible interception, and delaying it a full interval is a security cost with no accuracy benefit. Credentials are unchanged: the probe reuses the executor's configured identity source and holds no secret of its own.

message carries an SSH error string, which can include host, port, username and identity path — all values the user configured and can already see on the same settings page, so the record is not sensitive. ClassifyDialError must not place key material or agent socket contents into message.

The probe opens no remote shell and runs no remote command, so it grants nothing beyond what the manual test already exercises, and adds no new trust boundary.

ProxyJump is a second hop with a weaker trust model, and the reason set must not paper over it. An executor pins one fingerprint and it belongs to the target; dialViaJump verifies the bastion against ~/.ssh/known_hosts instead, accepting an absent host with a logged warning (OpenSSH's StrictHostKeyChecking=accept-new). Two consequences the probe inherits:

• A bastion key mismatch is rejected, but surfaces as ssh: bastion dial: …, not as errHostKeyMismatch, so ClassifyDialError would call it network or unknown and wait out failureThreshold under the wrong reason. The classifier must map a bastion host-key rejection to host_key, which AC-EXECUTORS-SSH-REACHABILITY-001.29 now requires observably.
• Every bastion-path error is about the bastion, while the record's host is the target. A bastion that is down reports the target unreachable and sends an operator to a machine that is fine — the same misattribution as the firewall message of 2026-09-06, from a different cause. host stays the target in every case: it identifies the executor's subject, the column keeps one meaning for success and failure alike, and AC-…-001.4 and AC-…-002.1 stay true for a probe that succeeds through a jump. The disambiguation belongs in message, which must name the bastion and state that the failure arose on the jump hop whenever it did. message is rendered beside the state wherever an unreachable record is shown, so the operator reads "unreachable … via bastion X" instead of being sent to a healthy target.

Observability

Structured zap logs plus expvar counters under /debug/vars, following the office_stall_* and routing_* precedent. Every name below carries the prefix executor_ssh_reachability_:

• probe_total, labelled by outcome (reachable plus each failure reason).
• state_transitions_total, by destination state.
• pass_skipped_total, when a tick is dropped because a pass is still running: the interval is shorter than a pass.
• write_refused_total, when a guard drops a write. A rising value means configuration churn, not a fault.
• reset_total, when a save invalidates a record.
• probe_discarded_total, when Stop cancelled a probe. probe_total is labelled by outcome and a cancelled probe has none, so without this the shutdown discard is the package's only uncounted drop.
• probe_duration_ms, the last pass's aggregate probe duration.

A transition logs at Warn for reachable to unreachable and Info for the reverse, with executor id, host, reason and failure count. Unchanged results do not log, so a steady host stays silent.

Prior art and departures

internal/integrations/healthpoll is the closest in-repo precedent: an immediate probe on start, a fixed cadence, a configured-or-skip gate, best-effort semantics, idempotent Start/Stop, and a goroutine-leak test. Those conventions are kept. Its persistence shape — a timestamp, a boolean and an error string on the config row — is not, in four places, because an SSH host is not an HTTP API: a closed reason set rather than an error string, since five failures are acted on differently and only a typed reason lets the UI say "fingerprint changed"; asymmetric hysteresis rather than flipping on the first result, which would let one dropped packet paint a healthy host as down (see State transitions); a separate record rather than a column, since observed state must not share a row with user-authored input (see Persistence); and changes pushed, results not, since pushing every result would put one message per executor per interval on the wire forever.

Testing

Strategy only; the plan's ## Tests table owns the per-criterion evidence.

• Poller tests drive the ticker with testing/synctest, matching healthpoll_test.go, under a goleak TestMain.
• Classification and hysteresis are table-driven — over constructed errors and over probe sequences — so the ordering rules and the exact probe each state flips on are asserted, not inferred.
• The upsert's ordering rule is exercised on both dialects, including the whole-second versus sub-second pair the text-encoding trap turns into a silent SQLite-only failure.
• The guards get a negative test each: soft-deleted, deactivated or reconfigured mid-probe writes nothing. The reset gets its own: it lands with no checked_at, and a probe in flight from before the save does not overwrite it.
• The immediate-probe route is tested for caller cancellation and for a guard-refused write returning persisted: false.
• Web E2E lives in apps/web/e2e/tests/ssh/ under the containers project, gated on KANDEV_E2E_CONTAINERS=1.

Related decisions

No new ADR: this design creates no system boundary and no ownership rule, and extends an existing contract through the repository's designated mechanisms for configuration, persistence, events and metrics. (0018 covers the adjacent runtime-override tier.)