Carlos Florêncio
87 lines
5.1 KiB
Markdown
87 lines
5.1 KiB
Markdown
# 0011: Transient provider errors (529 Overloaded) auto-retry with visible backoff
|
|
|
|
**Status:** superseded by ADR-2026-08-08-provider-neutral-agent-error-recovery
|
|
**Date:** 2026-05-30
|
|
**Area:** backend, frontend
|
|
|
|
Superseded by
|
|
[ADR-2026-08-08-provider-neutral-agent-error-recovery](2026-08-08-provider-neutral-agent-error-recovery.md),
|
|
which retains visible transient retry while generalizing classification and
|
|
recovery policy across agent and provider types.
|
|
|
|
## Context
|
|
|
|
When the Anthropic API returns a transient `529 Overloaded` mid-turn, the agent
|
|
surfaces it over ACP as an error event that the backend turns into `agent.failed`.
|
|
Previously this was handled identically to a terminal failure: a red "Agent has
|
|
encountered an error" banner, the execution torn down, the task dropped to
|
|
`REVIEW`, and (in some paths) an instant silent re-resume — reading as a flickery
|
|
glitch rather than a deliberate retry. The classifier had no rule for 529, so it
|
|
fell through to `phase.poststart.unknown` → `CodeAgentRuntime` (`AutoRetryable=false`).
|
|
|
|
## Decision
|
|
|
|
Transient provider overload is now a first-class, auto-retryable condition with a
|
|
calm, paced, visible retry — separate from generic agent failures.
|
|
|
|
- **Classification** (`internal/agent/runtime/routingerr/`): new runtime rule
|
|
`anthropic.overloaded.529.v1` matches the `529 … overloaded` / `overloaded_error`
|
|
signature and maps it to a dedicated `CodeProviderOverloaded`
|
|
(`AutoRetryable=true, FallbackAllowed=true`). Chose a dedicated code over reusing
|
|
`CodeProviderUnavailable` (HTTP-503 semantics) for distinct UX copy and clean
|
|
`routing_code: provider_overloaded` metrics. Exposes
|
|
`IsTransientProviderError(msg) bool`, mirroring `IsResumeCorrupted`.
|
|
- **Orchestrator** (`internal/orchestrator/event_handlers_transient.go`):
|
|
`handleAgentFailed` routes transient errors to `handleTransientFailure` before
|
|
the red recoverable-failure path. It schedules a backoff retry (5s → 15s → 30s,
|
|
max 3 attempts) via a Service-owned cancellable timer goroutine (mirroring the
|
|
clarification-watchdog pattern; drained in `Stop()`), parks the session in
|
|
`WAITING_FOR_INPUT` (calm, banner-less) without moving the task to `REVIEW` or
|
|
cleaning up, and emits a `variant: "warning"` status message with a Cancel
|
|
action. On each retry it tears down the failed execution and re-drives the
|
|
cached prompt via `PromptTask` (resuming a fresh agent). The prompt is cached
|
|
both in `PromptTask` (follow-ups) and in the `LaunchPreparedSession` initial
|
|
paths (`startTask`, `StartCreatedSession`) so a 529 on the very first turn can
|
|
retry too; if no prompt is cached, the retry clears itself and surfaces the
|
|
recovery banner rather than parking. After the budget is exhausted, or on
|
|
Cancel (`session.recover` action `cancel_retry`), it falls through to the
|
|
existing red Resume / Start-fresh banner (with friendly "stayed overloaded
|
|
after several retries" copy instead of the raw 529). Gated to non-office tasks
|
|
(`isOfficeTask`, i.e. authoritative Office ownership plus an assigned runner)
|
|
so genuine office tasks keep
|
|
their structured error UI. The transient branch runs *before* run-mode
|
|
automation finalization in `handleAgentFailed` — it's the only non-terminal
|
|
failure path, so finalizing (which reaps the ephemeral worktree) must wait
|
|
until the budget is actually spent. The synchronous prompt-error path
|
|
(`handlePromptError`) also skips the `REVIEW` transition for transient errors.
|
|
`httpStatusToCode` maps a structured `529` to `CodeProviderOverloaded` too,
|
|
for adapters that surface the status separately from the body.
|
|
- **Frontend** (`components/task/chat/messages/action-message.tsx`): the existing
|
|
`variant: "warning"` rendering (amber, not red) drives the retry card; the
|
|
bottom `AgentStatus` banner is never `FAILED` during retry, so there is no red
|
|
flicker. The Cancel button dispatches the `cancel_retry` `ws_request`.
|
|
- **Repro** (`cmd/mock-agent/`): a `/overloaded[:N]` scenario returns a real
|
|
prompt-time ACP error carrying the production 529 string for the first N
|
|
prompts (file-backed counter), then recovers.
|
|
|
|
## Consequences
|
|
|
|
- A 529 mid-turn now reads as "Provider overloaded — retrying in Ns (attempt X/Y)"
|
|
with a Cancel button instead of a red error flash, and self-heals when the
|
|
provider recovers, without user intervention.
|
|
- Adds a per-session retry-loop owner the Service must drain on shutdown
|
|
(goleak-covered).
|
|
- Re-driving via `PromptTask` re-sends the user prompt to the resumed agent; for a
|
|
real agent this could append a duplicate user turn in agent-side history — an
|
|
accepted limitation for the transient-recovery case.
|
|
|
|
## Alternatives Considered
|
|
|
|
- **Reuse `CodeProviderUnavailable`** — rejected: conflates HTTP-503 "unavailable"
|
|
with "temporarily overloaded, retry"; muddies metrics and UX copy.
|
|
- **Keep the agent process alive and re-prompt in place** — rejected: the lifecycle
|
|
marks the execution FAILED on a prompt error and rejects further prompts, so the
|
|
retry must tear down and resume a fresh execution.
|
|
- **Office-style FAILED state for transient errors** — rejected: the goal is an
|
|
uninterrupted working→retrying→working chat, not a terminal failure surface.
|