Provider Error Recovery Requirements

Overview

A provider can also abandon a partially streamed response and retry it inside the same prompt. That retry must replace the abandoned attempt without leaving duplicate transcript rows or stale recovery context.

Terminology

Provider response attempt: One provider generation attempt inside a Kandev prompt. A provider can retry this attempt without ending the prompt.
Response-attempt retry boundary: Structured, current-prompt evidence that the provider abandoned its active response attempt and will generate a replacement.

Requirements

REQ-PLATFORM-PROVIDER-ERROR-RECOVERY-001: Provider Error Recovery

Intent: Agent CLIs and providers report equivalent failures through different ACP frames, HTTP metadata, process exits, and diagnostic strings. Capacity, network, subscription, and quota failures need different recovery behavior, but orchestration code must not branch on provider names or raw prose.

Acceptance criteria

AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.1: Agent adapters collect bounded evidence from structured ACP errors, ordered ACP updates, managed stderr, process exits, and structured HTTP metadata.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.2: Evidence is correlated to the active invocation, prompt generation, and lifecycle phase before it can authorize recovery.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.3: A shared deterministic classifier maps evidence to a stable semantic code, policy class, confidence, scope, classifier rule ID, and validated timing hints.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.4: Provider-specific signatures live in adapter extractors and a central fixture-driven catalogue. Kanban, Office, lifecycle, and UI code do not inspect provider names or raw error strings to choose behavior.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.5: Structured metadata and exact signatures outrank broad text patterns. Collision tests enforce deterministic priority.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.6: Adding a known provider message normally adds a sanitized fixture and catalogue rule. It does not add an orchestration or UI branch.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.7: Classification is deterministic in this version. Calling a model to classify an error or extract timing data is deferred.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.8: Automatic retry, reset waiting, or fallback requires evidence tied to the current invocation and a failure boundary that is known to be pre-result and effect-safe.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.9: While an eligible interactive transient retry is pending, desktop and mobile task chat shall show the retry reason, provider, attempt count, countdown, and Cancel action only while the backend owns that retry.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.10: When an interactive transient retry succeeds, is exhausted, reaches a terminal state, is stopped, or is cancelled, the system shall attempt to durably retire every outstanding retry notice for the session so that a successful cleanup prevents the notice from reappearing after the session becomes idle, the task changes, the page reloads, or another viewer observes the session. If listing or deletion fails, the system shall log and swallow the failure so a later authorized cleanup can retry the remaining notices.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.11: After authorization, cancelling an interactive transient retry shall attempt to retire outstanding retry notices even when the retry loop has already ended. Cancelling an active loop shall also expose manual Resume and Start fresh recovery actions. Listing or deletion failures follow AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.10. A denied or foreign task-session pair shall not expose whether retry state or retry notices exist.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.12: A current Cursor frame beginning, after trimming, with case-insensitive Error: RetriableError: and a non-empty suffix of at most 256 UTF-8 bytes shall be suppressed when no later progress occurs, then yield one structured failure after the barrier. [unavailable] PING timed out and Connection stalled are valid. Cancellation, deadline, and retry-escalation signatures remain ordinary output. Prose before the prefix, empty/partial markers, stale evidence, and non-Cursor output stay ordinary output.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.13: When Cursor resumes generation before the prompt settles, Kandev shall treat the earlier matching frame as internal retry telemetry. Kandev shall not start a retry from the superseded frame. A later matching terminal frame for the same prompt can become new failure evidence.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.14: A terminal Cursor diagnostic beginning with case-insensitive Error: RetriableError: and carrying a non-empty suffix of at most 256 UTF-8 bytes shall classify as transient agent_transport_lost. Both layers share Unicode trim and byte bound. [canceled] stays eligible; context cancellation, deadline, and retry escalation retain the veto.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.15: Kandev shall automatically recover a terminal Cursor RetriableError only when current-turn evidence proves that the attempt had no output or tool activity. Eligible recovery shall retain the selected provider. It shall use the existing same-provider retry owner, attempt limit, and backoff. Missing evidence, output, or tool activity shall stop automatic replay and expose manual recovery.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.16: When an ACP adapter emits a current-prompt diagnostic message and then returns a matching structured session/prompt provider error, Kandev shall retain the diagnostic for the transcript but treat it as pre-result evidence only when both messages classify to the same high-confidence, fallback-eligible semantic code and no later output or tool activity exists. A mismatch, stale generation, low-confidence classification, later output, or tool activity shall keep the diagnostic as ordinary output and stop automatic recovery.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.17: The catalogue shall carry a raw ACP replay fixture for every required cell of the replay matrix. The gateway enumeration is claude-direct, teamclaude, openrouter, litellm. The case enumeration is matched, mismatched, later-output, queued, prose-matched, prose-tool, prose-output. The first four cases are required for every gateway. The three prose- cases are required only for a gateway the matrix declares classifying, because a prose case asserts nothing when the terminal envelope does not classify. A required cell with no fixture shall fail the fixture test, and a test shall not skip a required cell. Two fixtures sharing a case shall not carry equivalent frames, compared as canonically re-encoded values rather than as raw file bytes.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.18: Each replay fixture shall declare its gateway, its correlation case, the agent identity it replays through, whether its gateway is classifying, its capture mode as either recorded from a live ACP session or reconstructed from a cited published contract, and the replay identity the recovery-evidence layer requires: a corpus-unique session identifier, an execution identifier, and a non-zero prompt generation. The replayed payload shall contain no credential, bearer token, account or organization identifier, URL, or absolute host path. The system design shall enumerate the forbidden identifier shapes, and that enumeration shall be the enforced definition rather than an illustrative sample. Provenance fields shall carry no credential, no account or organization identifier, and no private host, and may cite a public documentation URL. Replaying a fixture shall be deterministic and shall not depend on wall-clock timing.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.19: Replaying a fixture shall produce the fixture's declared ordered event sequence, its declared classified diagnostic code, and its declared recovery disposition. The event sequence and the diagnostic code shall be observed at the ACP transport layer, where every token before the terminal failure is observed on the adapter's event stream and the terminal failure token is observed as the sanitized provider projection built from the error the prompt call returned. Every preceding token shall be on the event stream before the prompt call returns. The recovery disposition shall be observed at the recovery-evidence layer. Both layers shall consume the same fixture file, and a fixture shall not be satisfied by one layer alone. For the queued-diagnostic case, the diagnostic message chunk shall reach the event stream before the terminal failure event regardless of update-worker scheduling.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.20: The provider-diagnostic candidate marker shall be decided once, at the ACP conversion boundary, and carried unchanged on every streaming, lifecycle, and evidence hop. Carried means the field survives on the in-flight event value that each hop receives. Message accumulation and flush shall neither read, act on, nor persist the marker, and no message-level column shall be added for it. A consumer shall not re-derive the marker from message text. An event that reaches a consumer without the marker shall be treated as ordinary model output.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.21: A marked diagnostic chunk shall stay visible in the transcript, shall not advance turn progress, and shall not clear a recorded diagnostic for its own generation. An unmarked non-empty assistant or thought chunk in the same generation shall clear the recorded diagnostic and restore the ordinary output fence. Tool activity in the same generation shall not clear the recorded diagnostic; it shall instead fail the attempt through the effect fence, so an attempt with tool activity is never pre-result safe.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.22: A provider diagnostic projected from a terminal ACP prompt error shall carry only allowlisted structured metadata: the JSON-RPC error code, the adapter-declared error kind, the negotiated provider identity, and the session model identity. The session model identity shall be carried only when the adapter has settled a model for that session, meaning it has emitted a session configuration or model event carrying a non-empty current model identity; otherwise the field shall be omitted. Raw JSON-RPC error data shall never cross the boundary. A malformed, oversized, or unrecognized field shall be dropped and shall not invalidate the projection, and the presence or absence of this metadata shall not affect whether a projection is valid. A model settled after a projection is built shall not be added to it retroactively. This metadata shall not by itself authorize recovery in this version.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.23: A recorded diagnostic shall authorize pre-result recovery only when its normalized text is contained in the normalized message of the terminal provider failure, in addition to both classifying to the same high-confidence, fallback-eligible semantic code. Normalization shall trim leading and trailing whitespace and collapse internal whitespace runs to a single space, and comparison shall be case-sensitive. An empty normalized diagnostic shall never satisfy containment. The normalized diagnostic text shall be retained with the recorded diagnostic code, shall be cleared whenever that code is cleared, and shall not be overwritten by a later diagnostic in the same generation. A diagnostic that classifies to the terminal code but whose normalized text is not contained in the terminal message shall keep the output fence and expose manual recovery.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.24: When a terminal ACP prompt error carries its transient signature only in RequestError.Data, Kandev shall present the failure as terminal and expose manual recovery on every occurrence, including repeated failures in the same session, and shall neither show a transient retry notice nor start an automatic retry. This criterion applies to the terminal failure whether or not a diagnostic preceded it, and it does not outrank acceptance criteria .16 and .23: when a diagnostic did precede, those criteria reach the same manual-recovery outcome, because a terminal message that classifies to nothing can match no recorded diagnostic code. The existing session end-to-end specification that drives the mock agent's /transport-lost command shall be the surface that observes this outcome: its automatic-retry assertions shall be retargeted to manual recovery, and its retry-cancellation sub-test shall be retired rather than rewritten, because no retry loop exists on this path to cancel. The mock command's envelope shall keep the SDK's own peer-disconnect shape, Message Internal error with the signature only in Data, so the specification keeps exercising a shape that occurs in production. Repository documentation describing that command, or the recovery timeout it drives, shall not state that this path enters the automatic retry ladder. Losing end-to-end coverage of the automatic retry ladder for the transport-loss cause is accepted in this version; the catalogue rule keeps its unit coverage and its remaining surfaces.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.25: During one interactive transient retry loop, desktop and mobile chat shall show at most one current retry banner. Each subsequent attempt shall update that banner's reason, provider, attempt count, deadline, and Cancel action. Reloading the session shall retain the latest persisted attempt. Existing duplicate retry notices shall consolidate on the next successful retry-notice write. Unrelated messages and other sessions shall remain unchanged. Storage failures shall remain non-fatal and shall not trigger creation of an additional notice when an existing notice cannot be read or updated.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.26: When a supported adapter receives exact structured evidence that the active provider abandoned the current response attempt and will retry it internally, Kandev shall accept a response-attempt retry boundary within that prompt generation. One prompt generation may contain multiple sequential boundaries; each boundary applies only to assistant and thinking records created after the preceding boundary or commit boundary. Malformed evidence, a false retry flag, an unsupported adapter, a zero or stale prompt generation, and terminal provider errors shall not create that boundary or remove transcript content.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.27: After Kandev accepts a response-attempt retry boundary, it shall durably remove assistant and thinking messages created by the abandoned attempt. Connected desktop and mobile chats shall remove those messages before later replacement output, and a reload or another viewer shall see only the replacement response.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.28: Response-attempt cleanup shall preserve messages, tool activity, permission state, and turns committed before the active provider attempt. A repeated retry boundary with no new output shall be a no-op, and abandoned assistant text that has not crossed a history boundary shall not enter later fallback resume context. The boundary shall not erase prompt-level output or effect evidence used to decide whether a separate Kandev-owned replay is safe.
AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.29: The primary launch prompt shall use credential-only, unbounded redaction on every attempt, including attempt 0. Provider output and FailureReason shall use the diagnostic tier.

Out of scope

Classifier rules keyed on the structured error kind or JSON-RPC code. The metadata is transported and observable in this version; making it outrank the message-text catalogue is a separate contract change, because it would let a gateway authorize dynamic recovery through a field no fixture yet constrains.
Automatic transient retry for a generic ACP prompt error whose only classification signature lives in RequestError.Data. Acceptance criterion .22's boundary guarantee is preferred over that classification path. The generic projection derives its message from RequestError.Message alone, so a signature carried only in Data is never observable to the classifier, the terminal failure does not classify as transient, and Kandev exposes manual recovery instead of retrying. The worked example is ACP peer-disconnect surfaced only through session/prompt, where the adapter reports Message Internal error and carries peer disconnected before response only in Data, so acp.transport_lost.v1 no longer fires on that surface. The catalogue rule itself is unchanged and still fires wherever the same text reaches the classifier through another surface, including managed stderr, an ACP message chunk, and an error that is not an acp.RequestError; an adapter that reports the signature in Message classifies normally, though no error the ACP SDK constructs takes that shape, so that carve-out belongs to adapters that build their own error rather than to the SDK path. Those surfaces are not equivalent: the message-chunk surface classifies but can never authorize recovery for this cause, because the transport-loss cause is not fallback-eligible and so never marks a diagnostic candidate. It is a surface where the rule fires, not a route to restoring automatic retry. This is the same fail-closed shape as a version-skewed remote executor that omits the diagnostic marker: an accepted loss of one automatic path, not a silent misclassification. Restoring automatic retry here needs a separate contract change, either the deferred error-kind-keyed rule above or a classification channel carrying unsanitized text that reaches no display, log, or persistence path. A follow-up needs: the chosen channel, a fixture covering a Data-only signature, and the paired negative case proving the channel cannot reach the transcript.
New catalogue signatures for the OpenRouter and LiteLLM error envelopes. Neither gateway has a message-text rule today, so both are declared non-classifying and their matched and queued fixtures assert an unclassified disposition and manual recovery. A follow-up needs: a real captured envelope per gateway, a rule added beside acp.gateway_server_failure.v1, the paired negative-case prose test that catalogue convention requires, and a flip of the gateway's classifying declaration, which then makes its three prose- cells required.
Tightening the existing overloaded signature. The reported false positive reaches the fence through anthropic.overloaded.529.v1, whose pattern matches a numeric code adjacent to the word overloaded on one line; the anchored gateway signature already rejects equivalent prose. Narrowing that pattern would change stderr classification everywhere it is used, so this version closes the hole at correlation instead, through AC-PLATFORM-PROVIDER-ERROR-RECOVERY-001.23.
Live gateway calls during test execution. Fixtures are committed artifacts; the capture path is a developer-run script.
Rendering the diagnostic candidate marker in the web or desktop UI. The marker is backend recovery evidence; the diagnostic text itself already renders as ordinary transcript content.
Changes to retry policy values, attempt limits, backoff, or candidate ordering.
Gateway usage adapters beyond the existing TeamClaude client (docs/plans/teamclaude-proxy-usage/ tasks 02 through 04).

System design

The migrated technical source is split into part 1, part 2 (replay fixture matrix and harness) and part 3 (marker propagation, diagnostic text correlation, allowlisted prompt-error metadata), and part 4 (provider-owned response-attempt replacement and transcript retraction).

20 KiB Raw Blame History