Files

T

Carlos Florêncio 6c97e0a237 fix(lifecycle): clear stale dispatch gate after cancel (#2825 )

2026-08-19 22:47:38 +01:00

14 KiB

Raw Blame History

status, created, updated, owner

status	created	updated	owner
approved	2026-07-29	2026-08-18	Kandev

Agent Stall Recovery

Decisions:

Implementation plans:

Why

An agent turn can remain RUNNING after it stops emitting events. Silence alone is ambiguous because a legitimate long-running tool may also be quiet, but some agents know that their provider stream has failed and fail to close the ACP prompt. Users need Kandev to distinguish trustworthy terminal evidence from mere inactivity so a failed turn is not presented as healthy work.

What

Advisory inactivity recovery

Kandev checks running prompts for inactivity once per 60 seconds.
After five minutes without agent events, Kandev creates at most one user-visible notice for that prompt generation.
The notice says Kandev is still waiting and includes the active top-level tool's display title or name when available. It does not assert that the tool failed and does not include raw command arguments.
The notice is a single compact inline row with muted neutral copy and a neutral Cancel turn action. It has no warning/error colors, alert icon, tinted background, or alert-card treatment.
On phones, Cancel turn remains inline and content-width rather than becoming a full-width row, while retaining a minimum 44px touch height. Activating it uses the existing agent.cancel request.
The notice remains visible and actionable while the affected prompt's turn_id is the active turn in a RUNNING session, including after a page reload. It is hidden when that turn settles or a later turn becomes active.
Detection after genuine agent activity does not change task state, session state, prompt admission, or process liveness. A current five-minute snapshot with no genuine event since prompt dispatch is a launch failure and moves the session and task to FAILED.
The backend logs the first stall detected for a prompt generation and does not emit another notice or log entry on every watchdog check.

Correlated terminal diagnostics

When an agent emits a terminal provider diagnostic outside ACP, Kandev acts on it only when the diagnostic identifies the current agent type, active provider session, and active foreground prompt.
OpenCode provider diagnostics are collected from the managed opencode acp process's error-only stderr stream. Kandev does not read OpenCode's private files under ~/.local/share/opencode/log.
A correlated OpenCode stream error ends the affected prompt through the existing agent-error lifecycle instead of waiting for the inactivity threshold. The session stops presenting itself as RUNNING and retains the existing recovery actions.
Kandev preserves only allowlisted diagnostic fields needed for recovery: source agent, provider, model, sanitized message, occurrence time, and a reset time when one can be derived. Raw log lines, workspace URLs, account or workspace identifiers, paths, credentials, and unrelated stderr are not persisted or sent to the browser.
A recognized OpenCode usage-limit message is classified as quota_limited. Its recovery surface names the affected model when known, explains when capacity resets, and keeps sanitized technical details collapsed by default.
Kanban interactive recovery treats quota_limited as user-actionable. A reset time is informative and never schedules a Kanban retry. Office may consume the same classification for configured provider fallback and durable scheduler recovery under its separate routing policy.
The user-facing provider-limit copy is localized. Desktop and phone layouts expose the same failure explanation, technical details, and recovery actions.
A diagnostic for a background/title request, another OpenCode session, an earlier prompt generation, or a prompt that has already settled cannot fail the active turn.
An unrecognized, malformed, or unavailable diagnostic stream does not become proof of failure. The prompt continues and the advisory inactivity recovery remains available.

Allowlisted actionable diagnostics

A validated OpenCode remediation URL may cross the agentctl boundary as a separate remediation_url field. It is not appended to the sanitized provider message and is never sourced from an arbitrary raw error string.
The only accepted URL shape is HTTPS on the exact opencode.ai host with a path of /workspace/<safe-workspace-id>/go, with no userinfo, query, or fragment. The workspace identifier and complete URL are bounded before they are retained.
Kandev extracts this field from a structured ACP error field when OpenCode supplies one, or from a correlated structured OpenCode stderr record. If the source does not carry the URL, Kandev keeps the existing short error and does not attempt to reconstruct it from private OpenCode files.
Kanban recovery cards, the persisted last-agent-error notice, and Office's failed-session entry expose the same safe destination as a localized, keyboard- and touch-accessible external link. The sanitized message and collapsed technical details remain unchanged.
A missing, malformed, untrusted, or unsupported URL produces no link and no URL-bearing fallback text. Existing recovery actions and generic error copy remain available.

Diagnostic contract

The internal normalized provider diagnostic contains only:

Field	Meaning
`source`	Stable diagnostic source identifier, initially `opencode_stderr`
`provider_id`	Provider identifier reported by the agent, when present
`model_id`	Model identifier reported by the agent, when present
`message`	Sanitized provider message with URLs and identifiers removed
`occurred_at`	Parsed diagnostic timestamp
`reset_at`	Derived by adding a relative reset duration to `occurred_at`; absolute reset timestamps are not accepted in this version
`remediation_url`	Optional, exact-host OpenCode action URL accepted by the adapter-specific validator

The persisted recovery-message metadata uses failure_kind: provider_quota_limited, plus the safe model and reset fields, to select localized presentation. error_output may contain the same bounded, sanitized diagnostic message for the collapsed technical-details surface.

Failure modes

If active tool identity is unavailable, the generic inactivity notice and cancel action still appear.
If the agent acknowledges cancellation, normal cancellation reconciliation makes the session input-ready.
If the agent does not acknowledge cancellation, the existing bounded cancel-escalation path releases the prompt and reconciles the session.
Cancel escalation clears every prompt-admission barrier for the cancelled turn, including a pending dispatch-only completion. A later prompt can dispatch without a backend restart.
If notice-message persistence fails, the failure is logged without changing or terminating the running session.
If OpenCode does not support error-only stderr emission, changes its log format, or emits a line that fails validation, Kandev ignores the line and retains normal ACP plus inactivity behavior.
If a validated terminal diagnostic races with an ACP prompt response, only one terminal event settles the prompt generation.
If sanitization removes all usable provider text, Kandev surfaces a generic OpenCode provider failure without exposing the raw diagnostic.

Persistence guarantees

Advisory stall notices retain their existing persisted-message behavior.
Sanitized provider failures use the existing session error and recovery message persistence. They survive reloads like other recoverable failures.
A validated remediation_url is persisted only in the structured recovery metadata and remains available after reloads and through the Office session.state_changed snapshot. It is omitted from generic error text and raw stderr.
Raw agent stderr remains a bounded, executor-local in-memory diagnostic. It is not added to session metadata, messages, debug exports, or durable logs by this feature.

Scenarios

GIVEN a running prompt whose top-level shell tool is in_progress, WHEN no agent event arrives for five minutes, THEN one compact neutral notice names that tool and offers Cancel turn while the session remains RUNNING.
GIVEN a running prompt with no known active tool, WHEN no agent event arrives for five minutes, THEN one generic notice offers Cancel turn while the session remains RUNNING.
GIVEN a notice already exists for the active prompt generation, WHEN subsequent watchdog checks observe the same stall, THEN Kandev creates no duplicate notice and emits no repeated stall log.
GIVEN a persisted notice belongs to an earlier prompt generation, WHEN a later prompt is RUNNING, THEN the earlier notice is not rendered and a delayed stall event cannot create a new notice for it.
GIVEN a stall notice is visible, WHEN the user activates Cancel turn, THEN the existing cancellation path settles the turn and the session becomes available for new input without a backend restart.
GIVEN a dispatch-only prompt never reports completion and cancellation escalates, WHEN the user sends a later prompt, THEN Kandev dispatches it without a backend restart.
GIVEN a stall notice is visible on a phone viewport, WHEN the user taps Cancel turn, THEN the same cancellation outcome is reachable through an inline, content-width touch target of at least 44px.
GIVEN a prompt has produced no genuine agent event since dispatch, WHEN the current five-minute watchdog snapshot is handled, THEN Kandev persists a terminal error, moves the session and task to FAILED, and does not render a running-only cancel action.
GIVEN a quiet but legitimate long-running turn, WHEN the inactivity threshold passes and the user does not cancel, THEN Kandev leaves the turn and process running.
GIVEN the active OpenCode prompt emits a stream error for its own ACP session, WHEN the diagnostic says the model's five-hour usage limit was reached, THEN Kandev ends the running state and shows a localized quota-limit recovery message with the model and reset time.
GIVEN that quota-limit recovery is shown, WHEN the user expands technical details, THEN the sanitized provider message is visible and no OpenCode workspace URL or identifier is present.
GIVEN OpenCode emits a title-generation error or an error for another session, WHEN a foreground prompt is active, THEN Kandev ignores that diagnostic and does not settle the foreground prompt.
GIVEN a correlated diagnostic and the ACP response arrive concurrently, WHEN the turn settles, THEN exactly one terminal outcome is emitted for that prompt generation.
GIVEN an OpenCode build without usable diagnostic stderr, WHEN its ACP prompt hangs without further events, THEN the five-minute advisory notice remains the recovery path.
GIVEN a correlated OpenCode diagnostic containing a valid remediation URL, WHEN the recovery surface settles, THEN Kanban and Office show a localized external link while the sanitized message contains no URL or workspace identifier.
GIVEN a URL with the wrong host, scheme, path, query, fragment, or malformed workspace identifier, WHEN it crosses the adapter boundary, THEN Kandev drops the URL and shows the existing generic recovery copy.
GIVEN OpenCode 1.18.5 returns only its current short ACP failure, WHEN Kandev handles the error, THEN it shows the short failure without pretending that the TUI-only URL was received.
GIVEN the provider-limit recovery renders on a phone viewport, WHEN the user reads details or selects a recovery action, THEN the same outcome is available through touch targets of at least 44px with no horizontal page overflow.

Out of scope

Automatically timing out, cancelling, or killing a turn based only on inactivity.
Making the inactivity threshold user-configurable.
Reading, tailing, or exposing an agent vendor's private log files.
Treating arbitrary stderr text as trusted terminal evidence.
Persisting raw stderr or raw OpenCode log records.
Following or exposing arbitrary provider URLs, account URLs, or balance URLs; only the exact allowlisted OpenCode remediation route is in scope.
Recovering a URL that was never present in ACP or managed structured stderr, including by reading or tailing OpenCode's private log files.
Automatically purchasing capacity, changing models, or scheduling a retry at the reset time.
Repairing OpenCode's ACP implementation upstream; OpenCode should still be encouraged to return the provider failure directly from session/prompt.

14 KiB Raw Blame History