fix(orchestrator): fail never-started agent turns instead of reporting healthy (#2776)
Co-authored-by: nova28 <17953305+nova28@users.noreply.github.com> Co-authored-by: Carlos Florencio <carlosmflorencio@gmail.com>
This commit is contained in:
nova28
@@ -1,6 +1,6 @@
|
||||
# ADR-2026-07-29-agent-stall-user-controlled-recovery: Keep Agent Stall Recovery User Controlled
|
||||
|
||||
**Status:** accepted
|
||||
**Status:** superseded by 2026-08-18-never-started-agent-stall-terminal
|
||||
**Date:** 2026-07-29
|
||||
**Area:** backend, frontend, protocol
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# ADR-2026-08-18-never-started-agent-stall-terminal: Treat Never-Started Agent Stalls as Terminal
|
||||
|
||||
**Status:** accepted
|
||||
**Date:** 2026-08-18
|
||||
**Area:** backend, frontend, protocol
|
||||
|
||||
## Context
|
||||
|
||||
The five-minute stall watchdog normally reports advisory inactivity. A prompt
|
||||
that has not produced any genuine agent event is different from a prompt that
|
||||
started work and then became quiet. Without this distinction, a failed launch
|
||||
can remain `RUNNING` and the user cannot recover it from the task view.
|
||||
|
||||
## Decision
|
||||
|
||||
Keep ordinary post-activity stalls advisory. When a current watchdog snapshot
|
||||
shows no genuine turn event after the prompt was accepted, Kandev records the
|
||||
existing launch-failure outcome and moves the session and task to `FAILED`.
|
||||
|
||||
The snapshot carries a prompt activity epoch. The orchestrator validates the
|
||||
execution, prompt generation, and activity epoch before it applies the failure.
|
||||
The terminal message uses settled error rendering and has no running-only cancel
|
||||
action. Running notices keep their existing neutral cancel action.
|
||||
|
||||
## Consequences
|
||||
|
||||
Failed launches become visible and recoverable without waiting for a process
|
||||
restart. A legitimate provider that emits no event before the threshold can be
|
||||
classified as a failed launch, so the rule is limited to zero-event prompts and
|
||||
does not change recovery for turns that already produced activity.
|
||||
|
||||
The lifecycle and orchestrator must keep the activity epoch in memory and must
|
||||
reject stale watchdog events. Frontend rendering must not hide a terminal error
|
||||
because it carries historical running metadata.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- **Keep every stall advisory.** Rejected because a never-started prompt can
|
||||
remain `RUNNING` forever with no user-visible terminal result.
|
||||
- **Fail every quiet prompt.** Rejected because long-running tools can be quiet
|
||||
after they have already started.
|
||||
- **Use only prompt generation.** Rejected because an agent event can arrive
|
||||
after the watchdog snapshot and before the bus consumer handles it.
|
||||
@@ -174,3 +174,4 @@ Read individual ADRs for full context. Create new ones via `/record decision` or
|
||||
| 2026-08-15-office-mode-follows-active-workspace | [Office Mode Follows the Active Workspace](2026-08-15-office-mode-follows-active-workspace.md) | accepted | frontend, backend | 2026-08-15 |
|
||||
| 2026-08-17-separate-task-activity-from-summary-freshness | [Separate Task Activity From Summary Freshness](2026-08-17-separate-task-activity-from-summary-freshness.md) | accepted | backend, frontend, protocol | 2026-08-17 |
|
||||
| 2026-08-17-release-pr-ruleset-bypass | [Give Stable Release PRs an Administrator Token Bypass](2026-08-17-release-pr-ruleset-bypass.md) | accepted | infra, workflow, security | 2026-08-17 |
|
||||
| 2026-08-18-never-started-agent-stall-terminal | [Treat Never-Started Agent Stalls as Terminal](2026-08-18-never-started-agent-stall-terminal.md) | accepted | backend, frontend, protocol | 2026-08-18 |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
spec: docs/specs/agent-stall-recovery/spec.md
|
||||
decision: docs/decisions/2026-07-29-agent-stall-user-controlled-recovery.md
|
||||
decision: docs/decisions/2026-08-18-never-started-agent-stall-terminal.md
|
||||
created: 2026-07-29
|
||||
status: draft
|
||||
---
|
||||
@@ -12,8 +12,9 @@ status: draft
|
||||
Turn the lifecycle watchdog's existing inactivity observation into one
|
||||
generation-scoped internal event, persist that event as an actionable notice,
|
||||
and make the notice visible while the session is still `RUNNING`. The existing
|
||||
cancel-escalation path remains the sole recovery mechanism; no timeout changes
|
||||
session or process state automatically.
|
||||
cancel-escalation path remains the recovery mechanism for turns that already
|
||||
produced activity. A current zero-event snapshot uses the launch failure path
|
||||
described by ADR-2026-08-18-never-started-agent-stall-terminal.
|
||||
|
||||
## Confirmed root cause
|
||||
|
||||
@@ -43,7 +44,8 @@ waits, force-releases a stuck prompt, and reconciles the session to
|
||||
60 seconds, detect five minutes of inactivity, and publish `agent.stalled`
|
||||
once per prompt generation. The payload carries task/session/execution IDs,
|
||||
prompt generation, last-activity time, stalled duration, and optional tool
|
||||
ID/name/title/status.
|
||||
ID/name/title/status. Include an activity epoch so the orchestrator can
|
||||
reject a snapshot that became stale after an agent event arrived.
|
||||
- Add the payload and publisher support in
|
||||
`apps/backend/internal/agent/runtime/lifecycle/event_types.go`,
|
||||
`apps/backend/internal/agent/runtime/lifecycle/events.go`, and
|
||||
@@ -66,6 +68,8 @@ waits, force-releases a stuck prompt, and reconciles the session to
|
||||
the notice the active `turn_id` without lazily creating a successor turn.
|
||||
- If message persistence fails, log the error and leave the running prompt
|
||||
untouched.
|
||||
- Never-started notices use settled error metadata and do not expose the
|
||||
running-only cancel action.
|
||||
|
||||
---
|
||||
|
||||
@@ -191,8 +195,8 @@ requires them; the task-level commands remain the TDD evidence for each change.
|
||||
|
||||
## Risks and non-goals
|
||||
|
||||
- Event silence remains ambiguous, so the notice is advisory and cannot alter
|
||||
lifecycle state by itself.
|
||||
- Event silence remains advisory after genuine activity. A current zero-event
|
||||
snapshot is the explicit launch-failure exception recorded in the ADR.
|
||||
- Tool titles are already user-visible display data; raw command arguments must
|
||||
not be copied into the notice.
|
||||
- The notice is persisted once per prompt generation and assigned the active
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
status: approved
|
||||
created: 2026-07-29
|
||||
updated: 2026-08-08
|
||||
updated: 2026-08-18
|
||||
owner: Kandev
|
||||
---
|
||||
|
||||
@@ -13,6 +13,7 @@ Decisions:
|
||||
- [ADR-2026-08-02-agent-terminal-diagnostics-over-stderr](../../decisions/2026-08-02-agent-terminal-diagnostics-over-stderr.md)
|
||||
- [ADR-2026-08-07-allowlisted-provider-action-links](../../decisions/2026-08-07-allowlisted-provider-action-links.md)
|
||||
- [ADR-2026-08-08-provider-neutral-agent-error-recovery](../../decisions/2026-08-08-provider-neutral-agent-error-recovery.md)
|
||||
- [ADR-2026-08-18-never-started-agent-stall-terminal](../../decisions/2026-08-18-never-started-agent-stall-terminal.md)
|
||||
|
||||
Implementation plans:
|
||||
|
||||
@@ -47,8 +48,10 @@ mere inactivity so a failed turn is not presented as healthy work.
|
||||
- The notice remains visible and actionable while the affected prompt's
|
||||
`turn_id` is the active turn in a `RUNNING` session, including after a page
|
||||
reload. It is hidden when that turn settles or a later turn becomes active.
|
||||
- Detection alone does not change task state, session state, prompt admission,
|
||||
or process liveness.
|
||||
- Detection after genuine agent activity does not change task state, session
|
||||
state, prompt admission, or process liveness. A current five-minute snapshot
|
||||
with no genuine event since prompt dispatch is a launch failure and moves the
|
||||
session and task to `FAILED`.
|
||||
- The backend logs the first stall detected for a prompt generation and does
|
||||
not emit another notice or log entry on every watchdog check.
|
||||
|
||||
@@ -178,6 +181,10 @@ sanitized diagnostic message for the collapsed technical-details surface.
|
||||
- **GIVEN** a stall notice is visible on a phone viewport, **WHEN** the user taps
|
||||
**Cancel turn**, **THEN** the same cancellation outcome is reachable through
|
||||
an inline, content-width touch target of at least 44px.
|
||||
- **GIVEN** a prompt has produced no genuine agent event since dispatch,
|
||||
**WHEN** the current five-minute watchdog snapshot is handled, **THEN** Kandev
|
||||
persists a terminal error, moves the session and task to `FAILED`, and does
|
||||
not render a running-only cancel action.
|
||||
- **GIVEN** a quiet but legitimate long-running turn, **WHEN** the inactivity
|
||||
threshold passes and the user does not cancel, **THEN** Kandev leaves the
|
||||
turn and process running.
|
||||
@@ -214,8 +221,8 @@ sanitized diagnostic message for the collapsed technical-details surface.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Automatically timing out, failing, cancelling, or killing a turn based only
|
||||
on inactivity.
|
||||
- Automatically timing out, cancelling, or killing a turn based only on
|
||||
inactivity.
|
||||
- Making the inactivity threshold user-configurable.
|
||||
- Reading, tailing, or exposing an agent vendor's private log files.
|
||||
- Treating arbitrary stderr text as trusted terminal evidence.
|
||||
|
||||
Reference in New Issue
Block a user