7.7 KiB
spec, created, status
| spec | created | status |
|---|---|---|
| docs/specs/platform/requirements/shutdown-turn-failure-suppression.md | 2026-08-12 | implemented |
Implementation Plan: Do not surface backend-shutdown turn aborts as agent failures
Overview
Gate the lifecycle manager's terminal-failure path on graceful-shutdown state so
an in-flight turn aborted by SIGTERM becomes a benign stop instead of a
user-visible FAILED session. The change is a single-package backend fix in
internal/agent/runtime/lifecycle: guard handleCompleteEventMarkState (the
agent error-event chokepoint) and MarkCompleted (the process-exit chokepoint)
on the already-existing Manager.IsShuttingDown(). When shutting down, mark the
execution STOPPED and skip AgentFailed publication and
classifyAndMaybeRemediate, while still signalling promptDoneCh so waiters
unblock. No frontend or orchestrator change is required, because the orchestrator
already treats AgentStopped as resumable and never sets last_agent_error for
it.
Root cause (confirmed)
Incident task 2a5cad30, execution 5cb0f5e0: backend SIGTERM at 19:03:06 tore
down the agent subprocess mid-turn. The agent emitted an error event carrying
the ACP transport error -32603 ... peer disconnected before response. Flow:
handleErrorEvent -> handleCompleteEvent -> finishPromptCompletion ->
handleCompleteEventMarkState(isError=true) -> MarkCompleted(1, errorMsg),
which set status FAILED, published events.AgentFailed, and ran
classifyAndMaybeRemediate. The classifier had no shutdown context, so it logged
routing_code=agent_runtime_error (confidence=low,
classifier_rule=phase.poststart.unknown) and the orchestrator flipped the
session to a terminal FAILED state that the UI rendered as a red error banner.
Manager.IsShuttingDown() (set at the top of StopAllAgents) was already true at
that moment but was never consulted on the failure path.
Backend
Area 1 — Shutdown-race guard in the terminal transitions
Package: apps/backend/internal/agent/runtime/lifecycle/
-
manager_events.gohandleCompleteEventMarkState(execution, event, isError)(~L108): at the top of theif isErrorbranch, whenm.IsShuttingDown()is true, log WARN"error completion during shutdown, treating as cancellation"(execution_id, task_id, error) and mark the executionSTOPPEDvia a shared helper (see Area 2), thenreturnwithout callingMarkCompleted(1, errorMsg). This is the primary incident path.promptDoneChis already signalled earlier infinishPromptCompletionviahandleCompleteEventSignal, so waiters still unblock — do not move or duplicate that. -
manager_interaction.goMarkCompleted(executionID, exitCode, errorMessage)(~L1397): after computing that the status would beAgentStatusFailed(theelsebranch of the exit-code/error test, ~L1421), whenm.IsShuttingDown()is true, setexec.Status = v1.AgentStatusStoppedinstead ofAgentStatusFailed, keepFinishedAt/ExitCode/ErrorMessagestamping and thepersistExecutorRunning/releaseActivity/trace-span teardown, publishevents.AgentStoppedinstead ofevents.AgentFailed, and do NOT callclassifyAndMaybeRemediate. This covers process-exit paths that reachMarkCompleteddirectly (not only the agenterrorevent). Keep the existing duplicate-terminal guard (~L1405) unchanged.
Area 2 — Shared benign-stop helper (avoid duplicated blocks)
Add one small unexported helper on *Manager, e.g.
markStoppedDuringShutdown(execution *AgentExecution, exitCode int, errorMessage string),
that performs the STOPPED status stamp + terminal teardown + AgentStopped
publish used by both call sites, so the two guards do not duplicate a ≥150-token
block (backend lint limit). handleCompleteEventMarkState calls it; the
MarkCompleted guard reuses the same status/publish decision. Keep each guarded
function under the 80-line / complexity limits by extracting rather than inlining.
Note: AgentStatusStopped and the events.AgentStopped publish already mirror
exactly what StopAgentWithReason does for the StopReasonBackendShutdown path,
so the orchestrator's handleAgentStopped handles it as a known, non-FAILED,
resumable outcome with no new wiring.
Frontend
No user-facing change. Once the backend stops marking the session FAILED, the existing
FAILED-gated red banner (agent-status.tsxSTATE_CONFIG.FAILED) is simply never triggered for this case. No component, client, or store change.
Tests
All backend, in apps/backend/internal/agent/runtime/lifecycle/. Use the
existing helpers: newTestManager(t) / createTestManagerWithTracking() +
createTestExecution(...), and MockEventBus.OnPublish to capture published
subjects.
- What: shutdown-race error completion does not fail the execution.
File: new
manager_events_shutdown_test.go(avoid growing the large existing test file per the 800-line rule). How: build a Manager,mgr.StopAllAgents(ctx)(or set shutting-down) soIsShuttingDown()is true, add a RUNNING execution, callhandleCompleteEventMarkState(exec, errorEvent{is_error:true, error:"...peer disconnected before response"}, true). Assert final status isSTOPPED(notFAILED), assert noevents.AgentFailedsubject was published (capture viaMockEventBus.OnPublish), assertevents.AgentStoppedwas published. - What: classifier is not invoked on shutdown-race abort.
File: same as above. How: inject a recording
remediateNpxCachestub / spy on the classify log or use a seam; at minimum assert noAgentFailedand statusSTOPPED(classify only runs on theAgentFailedbranch, so absence ofAgentFailedis the observable proxy). Prefer asserting via the failure branch not being taken. - What: genuine failure while running normally is unchanged.
File: same file. How: with
IsShuttingDown()false, call the same error completion; assert statusFAILEDandevents.AgentFailedpublished (regression guard mirroring existingTestManager_MarkCompleted_Failure). - What:
MarkCompletedfailure branch honors shutdown. File:manager_lifecycle_test.gosibling or the new file. How: setIsShuttingDown()true,MarkCompleted("id", 1, "boom"); assert statusSTOPPED+AgentStopped. With it false, assertFAILED+AgentFailed(extends existingTestManager_MarkCompleted_Failure). - What: waiter still unblocks on shutdown-race abort (no hang).
File: new file. How: confirm
promptDoneChreceives the completion signal (the signal is emitted before the guard); assert a non-blocking receive succeeds.
Run: cd apps/backend && go test ./internal/agent/runtime/lifecycle/...
(package uses goleak.VerifyTestMain, so ensure cleanupManagerStopCh is used
via newTestManager).
E2E Tests
Skipped: no user-visible UI change (the outcome is the absence of a red banner). The behavior is fully covered by the backend unit tests above; an E2E that kills the backend mid-turn is out of proportion for a logging/state gate.
Verification Results
cd apps/backend && go test ./internal/agent/runtime/lifecycle/->ok(full package pass, ~48s).golangci-lint run ./internal/agent/runtime/lifecycle/... --new-from-rev="$(git merge-base HEAD origin/main)" --timeout=5m->0 issues;gofmt -lon changed files -> clean.- See
task-01-shutdown-failure-guard.md## Resultsfor the full list.
Implementation Waves And Parallel Candidates
Wave 1:
- [x] [task-01-shutdown-failure-guard](task-01-shutdown-failure-guard.md)
Single-task fix; no parallelism. The regression tests land in the same task as the guard so the test fails before and passes after the code change.
Open Questions
(none)