kandev/docs/decisions/2026-08-08-agentctl-crash-containment.md

# ADR-2026-08-08-agentctl-crash-containment: Publish Immutable Agent Events and Explicit Runtime Failure State

**Status:** proposed
**Date:** 2026-08-08
**Area:** backend, frontend, protocol, operations

## Context

The standalone `agentctl` process crashed with Go's fatal `concurrent map
iteration and map write` while `NormalizedPayload.MarshalJSON` serialized a tool
update. ACP adapter state is mutated under `Adapter.mu`, but emitted events can
retain pointers to that state after the lock is released. The existing
`cloneSubagentPayload` helper snapshots subagent payloads only; generic
`Input`, `Output`, and miscellaneous `Details` maps can still be updated while
the HTTP stream writer marshals an earlier event.

The backend survives an `agentctl` child exit. Its HTTP server and browser
WebSocket therefore remain reachable even though the local agent runtime has
stopped. Browser connectivity alone cannot identify the outage, and the current
UI gives no durable feedback when task and workspace activity becomes stale.

Restarting only the child is not a local launcher concern. The per-launch auth
token, PID, control clients, lifecycle manager, host utility manager, and active
execution metadata are captured by multiple owners during startup. Replacing
the process without atomically rebinding those owners would create a nominally
healthy but internally split runtime.

## Decision

Kandev contains this failure at two boundaries:

1. `NormalizedPayload` gains a complete immutable snapshot operation. Every ACP
   event snapshots its payload while `Adapter.mu` is held and before the event
   crosses the updates-channel boundary. The snapshot copies all typed pointer
   and slice fields and recursively copies JSON-compatible values held in
   `GenericPayload.Input`, `GenericPayload.Output`, and `MiscPayload.Details`.
   The one-off subagent clone is replaced by this common ownership rule.
2. The launcher reports unexpected child exit through a callback after it has
   distinguished the event from intentional shutdown. A concurrency-safe,
   install-wide availability owner records an `unavailable` snapshot and
   publishes a stable `agentctl_exited` reason. Raw process details remain in
   logs.
3. The current availability snapshot is distributed through the boot/app-state
   payload, a global system WebSocket notification, and replay after user
   subscription. The event is install-wide and intentionally uses the gateway's
   global broadcast path.
4. The application shell renders a persistent, non-dismissible runtime alert
   independent of the optional App status bar. It preserves last-known data and
   reuses the existing supervisor capability and restart flow when available;
   unsupported launch modes receive manual guidance.

Availability is in memory and monotonic within one backend boot. Recovery is a
full supervised Kandev restart, which creates a fresh backend and authenticated
child together. The general backend health endpoint remains unchanged.

The observable behavior is specified in
[`docs/specs/platform/agent-runtime-availability.md`](../specs/platform/agent-runtime-availability.md).

## Consequences

- Stream consumers receive values they own, so later ACP updates cannot race
  their serialization or retroactively change an earlier event.
- An unexpected child exit becomes visible even when browser-to-backend
  connectivity remains healthy, and late clients converge on the same state.
- The first repair avoids a partial child restart that would leave stale
  credentials and process identity in downstream services.
- The unavailable alert can coexist with the WebSocket connectivity warning
  because they report separate backend-child and browser-backend boundaries.
- Snapshotting adds bounded copy work for tool events. Payload size is already
  bounded for stream presentation, and correctness takes precedence at this
  asynchronous ownership boundary.

## Alternatives Considered

### Recover around JSON marshaling

Rejected. Go's concurrent-map runtime failure is fatal and cannot be reliably
recovered. The mutable alias must not reach the serializer.

### Copy only generic maps in `MarshalJSON`

Rejected. Marshaling occurs after adapter ownership has been released, so a
copy attempted there can race the same mutation. It also leaves earlier queued
events mutable and misses typed slices and nested values.

### Keep the subagent-only clone and patch the observed field

Rejected. The bug is an ownership violation shared by every mutable normalized
payload, not a one-field exception. Another payload kind could recreate the
same crash or event-history corruption.

### Restart `agentctl` in process

Deferred. It requires one coordinated owner for child generation, auth token,
PID, clients, lifecycle state, host utility state, and active execution
reconciliation. A future design may supersede this decision once that rebinding
contract exists.

### Exit the backend when the child exits

Rejected for this repair. An arbitrary backend exit is not guaranteed to invoke
the explicit supervisor restart protocol and can terminate a normal CLI launch.
Keeping the backend available preserves saved data, diagnostics, and the
supported user-initiated restart endpoint.

### Rely on the existing connectivity warning, toast, or App status bar

Rejected. The browser WebSocket can remain connected, toasts are transient, and
the status bar is optional. A core-runtime outage needs persistent shell-level
feedback regardless of route state or feature flags.