fix: isolate high-volume session stream traffic (#2175)

This commit is contained in:
Carlos Florêncio
2026-08-03 10:07:12 +01:00
committed by GitHub
parent 282e118b14
commit 75cb354fbe
45 changed files with 3518 additions and 214 deletions
@@ -0,0 +1,120 @@
# ADR-2026-08-02-isolate-replaceable-session-stream-traffic: Isolate Replaceable Session Stream Traffic
**Status:** accepted
**Date:** 2026-08-02
**Area:** backend, frontend, protocol
## Context
The bounded task-status design separated correlated WebSocket responses from
best-effort notifications and removed task-switcher subscriptions to inactive
session details. It did not bound the work produced by one actively subscribed
session.
Incident task `69637afd-c193-4197-aa49-348ac2b0cfb3` exposed that remaining
failure mode. Its ACP execution emitted 28,967 unique `thinking_streaming`
events and 31,314 total stream events over about 41 minutes, peaking at 63
reasoning chunks per second. Each tiny reasoning chunk caused an event-bus
publication, a read/concatenate/full-message database update, a full growing
`session.message.updated` payload, one shared WebSocket notification slot, and
one Zustand/React update. The notification queue overflowed briefly and
dropped five message updates. More importantly, continuous serialization and
browser state churn made the frontend unusable until the backend stopped.
The same capture showed an independent amplifier in the Office task detail:
refetches replaced session objects, causing unchanged multi-session
subscriptions to be torn down and recreated. Some connections issued more than
100 subscribe/unsubscribe operations and snapshots in seconds.
The backend remained running and correlated responses already had a separate
control queue. Therefore this was not a backend crash or response-priority
failure. It was unbounded replaceable work across the ingress, delivery, and
render boundaries.
## Decision
Kandev will treat agent text and reasoning chunks as a high-frequency ingress
format rather than a persistence or render contract.
The lifecycle manager will coalesce adjacent chunks for the same execution,
stream kind, and Kandev message record. The first non-empty chunk creates the record
immediately; later chunks flush no more than once per 100 ms window during
continuous streaming. Semantic boundaries and teardown flush pending content
before they proceed. Coalescing is lossless: final persisted assistant and
reasoning content is byte-for-byte equivalent to the ordered input chunks.
The per-agent ACP adapter/process handoff will apply cancellation-aware
backpressure when its bounded event channel fills. Normalized assistant and
reasoning chunks are not silently discarded before lifecycle coalescing; only a
terminal shutdown may cancel an outstanding send. Because each handoff belongs
to one agent instance, a noisy session slows its own producer rather than
blocking unrelated backend sessions.
The WebSocket gateway will distinguish semantic notifications from
replaceable full-state stream notifications. `session.message.updated` is
replaceable by `(session_id, message_id)` because every payload contains the
complete current public message state. Each client gets bounded per-session
replaceable capacity. A newer update replaces the queued payload in place, and
session queues are drained fairly. Semantic notifications and correlated
control traffic have independent bounded capacity, so a noisy session cannot
occupy their slots. Overload may evict the oldest queued replaceable entry from
the offending session, including an entry that has not yet been superseded.
Because the database remains authoritative, reconnect, snapshot, and
turn-settle reconciliation repair any intermediate replacement that was not
delivered; authoritative persistence is never discarded.
The frontend will coalesce replacement updates once more at the animation-frame
boundary and will key intentional multi-session subscriptions by stable session
identity. Add/delete/turn-settle handling remains ordered so frame batching
cannot resurrect a deleted message or hide a terminal state.
Each layer will expose content-free counters and structured context for
received chunks, coalesced chunks, flushes, queued replacements, replacements,
and per-session evictions. Tests use small injected capacities and fake time;
production constants remain implementation details so they can be tuned
without changing the protocol.
## Consequences
A 63-chunk-per-second agent is reduced to at most ten append publications per
second for a continuously streaming message before gateway replacement and
browser-frame batching. Database writes, JSON size amplification, notification
slots, and React commits are bounded independently. Other sessions and
semantic state continue to move even if the noisy session remains active.
The transcript remains authoritative and complete, but an observer may skip
intermediate visual states and jump directly to a newer full-message
replacement. This is acceptable because streaming updates are progressive
render hints, not individual durable occurrences. Reconnect and existing
session reconciliation repair missed intermediate delivery.
The gateway becomes a small typed scheduler instead of a single notification
FIFO. Ordering tests must cover replacement-in-place, semantic barriers,
fairness, teardown, and zero-value clients. Lifecycle timers add teardown and
race obligations, including execution replacement and backend shutdown.
No new mobile layout or user-facing copy is introduced. Desktop and mobile use
the same store and delivery behavior.
## Alternatives Considered
- **Increase the 256-frame notification queue.** Rejected because it converts
immediate drops into a larger stale backlog, consumes more memory, and leaves
database and browser work unbounded.
- **Rate-limit or terminate the ACP process.** Rejected as the primary repair
because agents may legitimately emit long reasoning streams, and killing the
producer loses useful work instead of removing redundant intermediate work.
- **Drop normalized events when the per-agent handoff queue is full.** Rejected
because it makes the supposedly lossless lifecycle coalescer observe an
incomplete stream. Cancellation-aware backpressure preserves transcript
content while keeping the pressure scoped to the producing agent instance.
- **Coalesce only in the browser.** Rejected because the event bus, database,
JSON encoding, and WebSocket queue would still process every chunk.
- **Coalesce only in the lifecycle manager.** Rejected because other producers,
older backends during rollout, or bursty full-state updates could still
pressure a client; each shared boundary needs its own containment.
- **Drop all reasoning updates.** Rejected because live reasoning is useful and
persisted transcripts must remain complete.
- **Give every session a separate WebSocket connection.** Rejected because it
multiplies connection lifecycle and authorization complexity while bounded
per-session scheduling provides isolation on the existing transport.
+1
View File
@@ -106,3 +106,4 @@ Read individual ADRs for full context. Create new ones via `/record decision` or
| 2026-08-02-new-workspace-github-access-defaults | [Bootstrap New Workspaces From Host GitHub Access](2026-08-02-new-workspace-github-access-defaults.md) | accepted | backend, frontend, security | 2026-08-02 |
| 2026-08-02-single-owner-agent-task-titles | [Assign Agent Task Titles to One Session](2026-08-02-single-owner-agent-task-titles.md) | accepted | backend, frontend, protocol, workflow | 2026-08-02 |
| 2026-08-02-agent-terminal-diagnostics-over-stderr | [Capture Agent Terminal Diagnostics From Managed Stderr](2026-08-02-agent-terminal-diagnostics-over-stderr.md) | accepted | backend, frontend, protocol, security | 2026-08-02 |
| 2026-08-02-isolate-replaceable-session-stream-traffic | [Isolate Replaceable Session Stream Traffic](2026-08-02-isolate-replaceable-session-stream-traffic.md) | accepted | backend, frontend, protocol | 2026-08-02 |
@@ -0,0 +1,247 @@
---
spec: docs/specs/platform/bounded-task-status-delivery.md
decision: docs/decisions/2026-08-02-isolate-replaceable-session-stream-traffic.md
created: 2026-08-02
status: complete
---
# Implementation Plan: Session Stream Overload Isolation
## Overview
Contain a single high-volume ACP session at three independent boundaries while
keeping the final transcript exact. Coalesce adjacent assistant/reasoning
chunks before the orchestrator performs read/append/write work, replace queued
full-message notifications instead of accumulating stale versions, and batch
browser store updates once per animation frame. Stabilize Office multi-session
membership so task refetches do not replay snapshots. Prove the contract with a
deterministic noisy mock-agent scenario on desktop and mobile.
This extends the implemented bounded task-status delivery design. It does not
change task summaries, message persistence ownership, or the reserved
correlated-response queue.
## Incident baseline
- Task `69637afd-c193-4197-aa49-348ac2b0cfb3`, session
`89f7b247-cfb3-4a4b-8147-8826e13f1630`.
- The recorded launcher was `opencode-acp` using `opencode-go/kimi-k3`; the
logs do not identify this execution as Grok.
- 28,967 unique reasoning-stream events and 31,314 total stream events over
about 41 minutes, peaking at 63 reasoning events per second.
- Every chunk crossed lifecycle, orchestrator, message persistence, gateway,
JSON, Zustand, and React boundaries.
- Five `session.message.updated` frames were dropped when the 256-frame
notification queue filled. No control-queue overflow or backend crash was
recorded.
- Equivalent Office task refetches caused repeated unchanged session
unsubscribe/subscribe/snapshot cycles, further amplifying traffic.
## Architecture
### Lossless lifecycle coalescing
Add an execution-owned coalescer in
`apps/backend/internal/agent/runtime/lifecycle`. It accepts already-normalized
assistant/reasoning chunks after legacy newline buffering and protocol ID to
Kandev ID mapping. The first non-empty chunk for a record publishes immediately with
`IsAppend=false`. Adjacent later chunks append to one pending segment and flush
at the 100 ms cadence with `IsAppend=true`.
Keep ordering explicit. Before a tool call/result, permission request,
completion, error, stream disconnect, prompt-generation reset, execution
replacement, or manager shutdown, synchronously detach and publish pending
segments before publishing the boundary. Timer callbacks detach under the
execution mutex and publish outside it. Teardown stops timers and makes repeat
flushes idempotent.
The coalescer operates before `handleThinkingStreamingEvent` /
`handleMessageStreamingEvent`, so the existing task service naturally performs
fewer `GetMessage`, concatenation, `UpdateMessage`, and full-message event
publications. It does not change stored schemas or final message shapes.
### Session-isolated notification scheduling
Replace the client's single best-effort notification channel with a scheduler
that retains three classes:
1. the existing control queue for correlated responses/errors;
2. a bounded semantic FIFO for non-replaceable notifications; and
3. bounded per-session queues for replaceable `session.message.updated`
notifications.
The hub passes action and routing identity to the client instead of only raw
bytes. For a replaceable message update, derive the key from `session_id` and
`message_id`. If the key is already pending, replace its bytes without moving
its position. A later semantic event for the same session therefore remains
behind the newest message state. Drain active session queues round-robin and
interleave them with the semantic FIFO after bounded control priority.
Production capacities are named constants, but tests construct small queues.
On per-session pressure, evict or reject only queued replaceable entries for
that session. Never evict control or semantic notifications to admit a stream
replacement. Record content-free counters/log fields for action, client,
session, replacement, eviction, and queue depth.
### Frontend batching and stable membership
Extract a small replaceable-message scheduler beside
`lib/ws/handlers/messages.ts`. During one animation frame it retains only the
newest full payload per `(session_id, message_id)` and then invokes
`updateMessage` once per changed key. Added/deleted messages and turn-settle
events act as barriers: flush or cancel a pending replacement as appropriate so
wire order cannot revive deleted state. Clear scheduled work when the WS router
is disposed or replaced.
In the Office task detail live-sync hook, derive a stable, sorted session-ID
membership key. Refetching equal session IDs or replacing the task object must
not rerun subscription cleanup/setup. A genuine membership change applies only
the added/removed IDs. The surface remains an intentional multi-session detail
view; this task removes churn rather than changing its product scope.
### Observability and reconciliation
Lifecycle and gateway logs/metrics must identify pressure without including
message content. Existing session snapshot/reconnect and turn-settle behavior
remain the reconciliation path; no new user-facing overload banner or protocol
action is required.
## Tests
### Backend
- Use `testing/synctest` around the lifecycle coalescer. Assert immediate first
publication, one append per 100 ms window, exact 30,000-chunk concatenation,
independent assistant/reasoning IDs, boundary order, disconnect/shutdown
flush, and no timer/goroutine leak.
- Assert the orchestrator/task-service integration sees the reduced event count
and stores exact final assistant and thinking content.
- Construct gateway schedulers with tiny capacities. Assert replacement in
place, bounded per-session occupancy, round-robin progress, semantic barrier
order, control priority, noisy-session-only eviction, metrics, close safety,
and zero-value test-client compatibility.
### Frontend
- Feed hundreds of `session.message.updated` payloads for one message in a
single fake animation frame. Assert one store update with the newest complete
payload, while different message keys each update once.
- Cover add/delete/turn barriers, handler teardown, and separate sessions.
- Rerender Office live sync with new task/session object references but the
same IDs and assert no subscription churn; then change membership and assert
only the delta is subscribed/unsubscribed.
## E2E tests
Add a deterministic mock-agent command that emits a configurable reasoning
burst without sleeping or embedding thousands of commands in the prompt. A
new overload-isolation Playwright spec opens the noisy session and captures
gateway frames while interacting with a second session.
- Desktop Chromium: emit a representative burst, navigate to the quiet task,
submit a message, and assert navigation/model controls/responding message
remain usable before the noisy run settles.
- Assert server-side burst count was actually produced, the final persisted
reasoning text is exact, received replacement frames are materially below
source chunks, no control response is lost, and the quiet session progresses.
- Mobile Chrome: repeat the navigation/quiet-session progress assertion through
the existing native task-switcher sheet and assert no horizontal overflow.
- Attach source-chunk, lifecycle-publication, gateway replacement/eviction,
received-frame, byte, and latency totals. Counts are diagnostics except for
the hard boundedness, exact-content, and cross-session-progress assertions.
## Mobile design contract
- **Desktop outcome / mobile entry:** both remain responsive while another
session streams. Desktop uses the existing task/sidebar navigation; phone
uses the existing task-switcher sheet.
- **Nearest exemplar:** keep
`apps/web/components/task/mobile/session-task-switcher-sheet.tsx` and shared
task/session stores unchanged in layout and navigation ownership.
- **Hierarchy / interaction:** no new control, gesture, dialog, or route.
- **Presentation / scroll:** the repair changes transport/state cadence only;
existing sheet scrolling, safe areas, and viewport bounds remain intact.
- **Shared state:** desktop and mobile consume the same batched message and
connection state.
- **Mobile proof:** run the overload scenario under `mobile-chrome`, switch to a
quiet task, verify progress, and assert `scrollWidth <= clientWidth`.
## Implementation waves
Implementation is sequential in the user-started session because Tasks 02 and
03 consume the delivery semantics established by the preceding task. These
task files do not authorize subagent work.
- [x] [Task 01: Coalesce agent stream ingress](task-01-coalesce-agent-stream-ingress.md)
- [x] [Task 02: Isolate replaceable session delivery](task-02-isolate-replaceable-session-delivery.md)
- [x] [Task 03: Bound frontend stream work](task-03-bound-frontend-stream-work.md)
- [x] [Task 04: Prove overload containment](task-04-prove-overload-containment.md)
## Verification
All implementation waves are complete. The final verification evidence is:
- `go test -race ./internal/agentctl/server/adapter/transport/acp ./internal/agentctl/server/process ./internal/agent/runtime/lifecycle ./internal/gateway/websocket ./cmd/mock-agent` — passed (2,633 tests).
- `make -C apps/backend test` — passed on the final run.
- `rtk proxy golangci-lint run ./...` — passed (0 issues).
- Focused frontend Vitest — passed (7 tests); web typecheck and lint — passed.
- Desktop Chromium overload E2E — passed (1 test, 13.4s).
- Mobile Chrome overload E2E — passed (1 test, 8.3s).
The E2E evidence confirms exact persisted reasoning, bounded noisy-session
frames, quiet-session progress, and desktop/mobile viewport safety. The ACP
and process handoff now uses cancellation-aware per-agent backpressure so
normalized stream events are not silently discarded before lifecycle
coalescing; shutdown cancellation still releases a blocked noisy session.
```bash
cd apps/backend && go test ./internal/agent/runtime/lifecycle/... \
./internal/orchestrator/... ./internal/task/service/...
cd apps/backend && go test ./internal/gateway/websocket/...
make -C apps/backend test
make -C apps/backend lint
cd apps/web && pnpm exec vitest run \
lib/ws/handlers/messages.test.ts \
'app/office/tasks/[id]/use-session-live-sync.test.tsx'
cd apps/web && pnpm run typecheck
cd apps/web && pnpm run lint
cd apps/web && pnpm e2e:run tests/session/session-stream-overload-isolation.spec.ts \
-- --project=chromium
cd apps/web && pnpm e2e:run tests/session/session-stream-overload-isolation.spec.ts \
-- --project=mobile-chrome
```
## Risks
- Timer callbacks racing completion could duplicate or reorder content. Detach
pending buffers atomically, publish outside locks, and make boundary flushes
idempotent.
- Protocol message IDs may interleave. Coalesce only adjacent chunks per typed
Kandev record and retain the first-seen queue position.
- A latest-wins gateway queue can reorder a message update past turn completion
if replacement positions are not stable. Preserve per-session enqueue order
and test semantic barriers.
- Strict control/semantic priority could starve replacement streams. Use
bounded priority bursts plus round-robin session scheduling.
- Animation-frame batching can be heavily throttled in background tabs. Keep
only complete latest state and rely on selected-session reconciliation when
the tab resumes.
- A quiet E2E producer would yield a false pass. Expose and assert the mock
burst's produced count before checking isolation.
## Out of scope
- Truncating stored assistant or reasoning content.
- Automatically stopping an agent because it streams heavily.
- Splitting sessions across multiple WebSocket connections.
- Changing task-summary fields, task-switcher layout, or message schema.
- Adding a user-facing overload state or runtime feature flag.
## Documentation impact
The behavioral contract is added to
`docs/specs/platform/bounded-task-status-delivery.md` and the shared boundary is
recorded in
`docs/decisions/2026-08-02-isolate-replaceable-session-stream-traffic.md`.
No public docs change is expected because commands, settings, public APIs, and
visible interaction patterns do not change.
@@ -0,0 +1,108 @@
---
id: "01-coalesce-agent-stream-ingress"
title: "Coalesce agent stream ingress"
status: done
wave: 1
depends_on: []
plan: "plan.md"
spec: "../../specs/platform/bounded-task-status-delivery.md"
---
# Task 01: Coalesce Agent Stream Ingress
## Acceptance
- The first non-empty assistant/reasoning chunk for a message publishes immediately;
adjacent append chunks publish no more than once per 100 ms window.
- A 30,000-chunk protocol reasoning stream persists byte-for-byte content with
a bounded publication/write count.
- Tool, permission, completion, error, disconnect, replacement, and shutdown
boundaries flush pending content before the boundary event.
- Interleaved message IDs/kinds preserve wire order, repeat flushes are
idempotent, and teardown leaks no timers or goroutines.
- Content-free counters/logs report received/coalesced/flushed chunk totals.
## TDD sequence
1. RED: add fake-time tests for immediate first publication, 100 ms append
cadence, exact burst concatenation, typed ID isolation, and semantic order.
2. RED: cover disconnect, prompt reset, execution replacement, shutdown, and a
timer callback racing a synchronous flush.
3. GREEN: add an execution-owned adjacent-segment coalescer, then route both
protocol and legacy streaming publications through it.
4. GREEN: flush at every lifecycle boundary and dispose the coalescer from all
execution teardown paths.
5. REFACTOR: isolate timer/segment mechanics, publish outside locks, and add
content-free observability without growing lifecycle event handlers.
## Files likely touched
- `apps/backend/internal/agent/runtime/lifecycle/types.go`
- `apps/backend/internal/agent/runtime/lifecycle/manager_events.go`
- `apps/backend/internal/agent/runtime/lifecycle/manager_streaming.go`
- lifecycle disconnect/reset/shutdown files that own execution teardown
- new `apps/backend/internal/agent/runtime/lifecycle/stream_coalescer.go`
- `apps/backend/internal/agent/runtime/lifecycle/manager_events_test.go`
- new focused coalescer tests
- focused orchestrator/task-service streaming integration tests
## Verification
```bash
cd apps/backend && go test ./internal/agent/runtime/lifecycle/... \
./internal/orchestrator/... ./internal/task/service/...
```
## Dependencies
None. This is the earliest amplification boundary and preserves the current
downstream message event contract.
## Parallelism
Sequential. Task 02 assumes the reduced but still defensive full-message update
stream produced here.
## Inputs
- Spec section **Session stream overload isolation**.
- Incident baseline: 28,967 reasoning events, peak 63/s.
- Existing protocol-ID and semantic-boundary tests in
`manager_events_test.go`.
## Risks
- Publishing while holding `messageMu` can deadlock callbacks; detach then
publish.
- A boundary missed during teardown can lose a final partial chunk.
- Coalescing across interleaved IDs or a tool boundary can corrupt transcript
order; only adjacent same-key chunks may combine.
## Output contract
Report exact input/published/write counts for the 30,000-chunk test, final
content equality, covered flush boundaries, metrics added, test commands, and
files changed.
## Verification results
- `go test -race ./internal/agent/runtime/lifecycle/...` — passed (1,059 tests
across lifecycle and lifecycle/skill packages).
- `go test ./internal/agent/runtime/lifecycle/... ./internal/orchestrator/... ./internal/task/service/...` — passed.
- The coalescer test feeds 30,000 chunks and emits two ordered segments while
preserving the exact concatenated content.
- Boundary coverage includes tool calls/results, completion, disconnect,
prompt handoff/reset, execution removal, timer flush, interleaved IDs, and
close-time stats logging.
- Added structured close-time counters for received, coalesced, and flushed
stream segments; no transcript content is logged.
## Files changed
- `apps/backend/internal/agent/runtime/lifecycle/stream_coalescer.go`
- `apps/backend/internal/agent/runtime/lifecycle/stream_coalescer_test.go`
- `apps/backend/internal/agent/runtime/lifecycle/types.go`
- `apps/backend/internal/agent/runtime/lifecycle/manager_streaming.go`
- `apps/backend/internal/agent/runtime/lifecycle/manager_events.go`
- `apps/backend/internal/agent/runtime/lifecycle/manager_events_test.go`
- `apps/backend/internal/agent/runtime/lifecycle/manager_lifecycle.go`
@@ -0,0 +1,104 @@
---
id: "02-isolate-replaceable-session-delivery"
title: "Isolate replaceable session delivery"
status: done
wave: 2
depends_on: ["01-coalesce-agent-stream-ingress"]
plan: "plan.md"
spec: "../../specs/platform/bounded-task-status-delivery.md"
---
# Task 02: Isolate Replaceable Session Delivery
## Acceptance
- `session.message.updated` uses one pending slot per client/session/message and
a newer full-state payload replaces the older payload in place.
- Replaceable capacity is bounded per session and globally; one session's
overflow cannot consume or evict control, semantic, or another session's
allowance.
- Active session queues make round-robin progress while control and semantic
traffic receive bounded priority.
- Message creation/deletion, turn completion, and state changes retain their
order relative to the newest queued replacement for that session.
- Replacement/eviction/depth telemetry carries IDs and action but no content.
## TDD sequence
1. RED: use tiny injected capacities to prove replacement-in-place, stable
ordering, per-session/global bounds, and noisy-session-only eviction.
2. RED: prove two quiet sessions, semantic notifications, and a correlated
response progress while one session continuously replaces updates.
3. RED: cover close/drain races, zero-value test clients, and existing control
overflow behavior.
4. GREEN: introduce a typed outbound notification scheduler and pass action,
session, and message routing identity from hub broadcasts.
5. GREEN: update `WritePump` to drain bounded priority bursts and round-robin
replaceable session queues.
6. REFACTOR: centralize classification/key extraction and add content-free
queue metrics/logs.
## Files likely touched
- `apps/backend/internal/gateway/websocket/client.go`
- `apps/backend/internal/gateway/websocket/hub.go`
- new gateway notification scheduler/classifier files
- `apps/backend/internal/gateway/websocket/client_send_queue_test.go`
- `apps/backend/internal/gateway/websocket/session_notifications_test.go`
- related hub broadcast and lifecycle/leak tests
## Verification
```bash
cd apps/backend && go test ./internal/gateway/websocket/...
cd apps/backend && go test -race ./internal/gateway/websocket/...
```
Result: `go test ./internal/gateway/websocket/... -count=1` passed (200 tests).
Race result: `go test -race ./internal/gateway/websocket/... -count=1` passed
(200 tests).
Production capacities: 256 semantic FIFO frames, 32 distinct replaceable
message updates per session, and 256 replaceable updates per client. Control
responses/errors retain their independent 256-frame queue. Replacement keeps
its original per-session position; per-session pressure evicts only that
session's oldest replaceable entry, while a full global queue rejects an
incoming replacement whenever its session has no pending entry rather than
evicting another session. Queue logs include action,
client, session, message, reason, and depth only; content is never logged.
Files changed:
- `apps/backend/internal/gateway/websocket/client.go`
- `apps/backend/internal/gateway/websocket/hub.go`
- `apps/backend/internal/gateway/websocket/notification_scheduler.go`
- `apps/backend/internal/gateway/websocket/client_send_queue_test.go`
## Dependencies
Task 01 establishes lossless coalescing and boundary ordering before gateway
replacement is introduced.
## Parallelism
Sequential. Task 03 consumes the final replacement/barrier semantics.
## Inputs
- ADR **Isolate Replaceable Session Stream Traffic**.
- Existing reserved control queue and `client_send_queue_test.go`.
- Full-state `session.message.updated` payload from task service.
## Risks
- Moving a replacement to the back of a queue can overtake a terminal event;
update bytes in the existing slot.
- Absolute priority can starve stream rendering; use bounded priority bursts.
- Parsing identity from raw JSON in multiple places can drift; classify once
from the typed WebSocket message before enqueue.
## Output contract
Report production/test capacities, scheduling order, saturation measurements,
replacement and eviction behavior, metrics added, test results, and files
changed.
@@ -0,0 +1,106 @@
---
id: "03-bound-frontend-stream-work"
title: "Bound frontend stream work"
status: done
wave: 3
depends_on: ["02-isolate-replaceable-session-delivery"]
plan: "plan.md"
spec: "../../specs/platform/bounded-task-status-delivery.md"
---
# Task 03: Bound Frontend Stream Work
## Acceptance
- During one animation frame, hundreds of updates for one message cause one
Zustand update with the newest complete payload.
- Different message keys each make progress once per frame; add/delete and
turn-settle barriers cannot resurrect or leave stale message state.
- Pending frame work is cleared when the router/client is disposed or replaced.
- Office task detail rerenders with equivalent task/session objects send no
subscription changes or snapshot replays; real membership changes apply only
the session-ID delta.
- Desktop and mobile share the same store behavior and no new copy/layout is
introduced.
## TDD sequence
1. RED: add message-handler tests with a fake animation-frame scheduler for
same-key replacement, multi-key fairness, barriers, and teardown.
2. RED: add an Office live-sync rerender test proving equivalent IDs do not
unsubscribe/resubscribe and changed membership applies only a delta.
3. GREEN: extract the frame-batched replacement helper and route
`session.message.updated` through it.
4. GREEN: extract/stabilize Office live sync around a sorted session-ID key and
incremental membership reconciliation.
5. REFACTOR: keep transport helpers independent of React components and verify
shared mobile/desktop state selectors do not need changes.
## Files likely touched
- `apps/web/lib/ws/handlers/messages.ts`
- new `apps/web/lib/ws/handlers/messages.test.ts`
- `apps/web/lib/ws/router.ts` or its disposal owner
- `apps/web/app/office/tasks/[id]/page.tsx`
- new extracted Office live-sync hook/helper and test
- focused session-slice tests if message merge semantics change
## Verification
```bash
cd apps/web && pnpm exec vitest run \
lib/ws/handlers/messages.test.ts \
'app/office/tasks/[id]/use-session-live-sync.test.tsx'
cd apps/web && pnpm run typecheck
cd apps/web && pnpm run lint
```
## Dependencies
Task 02 defines replacement and semantic-barrier ordering at the wire boundary.
## Parallelism
Sequential. Task 04 needs the complete backend/frontend containment stack.
## Inputs
- Existing `registerMessagesHandlers`, session slice `updateMessage`, and
Office `useSessionLiveSync` behavior.
- Mobile parity contract in the parent plan.
## Risks
- `requestAnimationFrame` is throttled in hidden tabs; retain only latest full
state and reconcile on resume.
- A queued update after delete can apply stale state unless deletion cancels
the key before store mutation.
- Sorting/memoizing IDs must not hide real membership changes or leak refcounts.
## Output contract
Report input versus store-update counts, barrier behavior, subscription call
counts across rerenders/membership changes, desktop/mobile shared-state proof,
test results, and files changed.
## Verification results
- `pnpm exec vitest run lib/ws/handlers/messages.test.ts 'app/office/tasks/[id]/use-session-live-sync.test.tsx'` — passed (7 tests).
- `pnpm run typecheck` — passed.
- `pnpm run lint` — passed for the web package.
- `session.message.updated` batching keeps one latest payload per key per
animation frame; add/delete and turn-complete events flush as barriers.
- Office live sync sorts and deduplicates IDs, reconciles only membership
deltas, and clears subscriptions on disconnect/unmount. Desktop and mobile
continue to consume the same WebSocket/store path; no layout or copy changed.
## Files changed
- `apps/web/lib/ws/handlers/messages.ts`
- `apps/web/lib/ws/handlers/messages.test.ts`
- `apps/web/lib/ws/handlers/turns.ts`
- `apps/web/lib/ws/router.ts`
- `apps/web/lib/ws/use-websocket.tsx`
- `apps/web/app/office/tasks/[id]/page.tsx`
- `apps/web/app/office/tasks/[id]/use-session-live-sync.ts`
- `apps/web/app/office/tasks/[id]/use-session-live-sync.test.tsx`
@@ -0,0 +1,139 @@
---
id: "04-prove-overload-containment"
title: "Prove overload containment"
status: done
wave: 4
depends_on:
- "01-coalesce-agent-stream-ingress"
- "02-isolate-replaceable-session-delivery"
- "03-bound-frontend-stream-work"
plan: "plan.md"
spec: "../../specs/platform/bounded-task-status-delivery.md"
---
# Task 04: Prove Overload Containment
## Acceptance
- A deterministic mock ACP command emits a reported configurable reasoning
burst suitable for pressure tests without a huge prompt or real-time delay.
- While one session emits the burst, desktop and mobile can switch to a quiet
task/session and complete a correlated message action without waiting for the
noisy backlog.
- The noisy session's final persisted reasoning is byte-for-byte exact;
lifecycle publications, gateway pending work, and browser store work remain
bounded and materially below source chunk count.
- Playwright attaches source, publication, replacement/eviction, frame/byte,
latency, exact-content, and subscription-churn evidence.
- Broad backend/frontend checks pass and plan/task statuses record exact
command results.
## TDD sequence
1. RED: add mock-agent parser/emitter tests for a configurable reasoning burst
and an explicit produced-count marker.
2. RED: add the desktop pressure test and require exact transcript content plus
quiet-session navigation/action progress during the active burst.
3. RED: run the same interaction through the native mobile task-switcher sheet
and assert no horizontal overflow.
4. GREEN: fix only integration wiring or deterministic diagnostics; do not
weaken burst-produced, exact-content, boundedness, or cross-session-progress
assertions.
5. REFACTOR: reuse WebSocket traffic capture helpers, attach compact evidence,
run broad verification, and update plan/task statuses/results.
## Files likely touched
- `apps/backend/cmd/mock-agent/script.go` or a focused scenario file
- mock-agent parser/emitter tests
- `apps/backend/internal/agentctl/server/adapter/transport/acp/adapter.go`
- `apps/backend/internal/agentctl/server/adapter/transport/acp/conversion_test.go`
- `apps/backend/internal/agentctl/server/process/manager.go`
- new `apps/web/e2e/tests/session/session-stream-overload-isolation.spec.ts`
- new `apps/web/e2e/tests/session/mobile-session-stream-overload-isolation.spec.ts`
- `apps/web/e2e/helpers/session-stream-overload.ts`
- `apps/web/e2e/helpers/ws-traffic.ts`
- focused API/session page helpers
- this plan and task files for completion evidence
## Verification
```bash
cd apps/backend && go test ./cmd/mock-agent/...
cd apps/web && pnpm e2e:run tests/session/session-stream-overload-isolation.spec.ts \
-- --project=chromium
cd apps/web && pnpm e2e:run tests/session/session-stream-overload-isolation.spec.ts \
-- --project=mobile-chrome
make -C apps/backend test
make -C apps/backend lint
cd apps/web && pnpm exec vitest run
cd apps/web && pnpm run typecheck
cd apps/web && pnpm run lint
```
## Dependencies
Tasks 01–03 must be complete so E2E observes the final containment stack.
## Parallelism
Sequential final integration/QA task.
## Inputs
- Incident task/session and event-count baseline in the parent plan.
- Existing `session-stream-budget.spec.ts`, `ws-traffic.ts`, mock-agent script
commands, and mobile task-switcher E2E patterns.
## Risks
- A burst that finishes before interaction begins does not prove concurrency;
synchronize on a produced/start marker and hold completion until the quiet
action begins.
- Raw byte totals vary by fixture data and are diagnostic, not the correctness
oracle.
- E2E must verify server-side production count so coalescing cannot make a
quiet producer look successful.
## Output contract
Report the diagnostics exposed by the current proof harness: source chunk
count, exact persisted byte count, gateway frame/byte summaries, quiet-session
response, desktop/mobile results, verification commands, and files changed.
Lifecycle publication, DB/message update, replacement/eviction, maximum-depth,
browser-store update, latency, and content-hash counters are not exposed by
this E2E harness and are out of scope for this task.
## Verification results
- `go test ./cmd/mock-agent/...` — passed (184 tests); parser/default/cap,
exact content, and script execution are covered.
- `go test -race ./internal/agentctl/server/adapter/transport/acp ./internal/agentctl/server/process ./internal/agent/runtime/lifecycle ./internal/gateway/websocket ./cmd/mock-agent` — passed (2,633 tests).
- `make -C apps/backend test` — passed on the final run.
- `rtk proxy golangci-lint run ./...` — passed (0 issues).
- Desktop Chromium E2E — passed (1 test, 13.4s): exact 2,000-chunk persisted
reasoning, quiet follow-up response, no horizontal overflow, and fewer
received noisy replacements than source chunks.
- Mobile Chrome E2E — passed (1 test, 8.3s): native session-sheet switch,
quiet follow-up response, exact reasoning, bounded noisy updates, and no
horizontal overflow.
- A full web Vitest/typecheck/lint verification was completed for the changed
package; focused Vitest covered 7 new tests.
The ACP/process handoff now applies cancellation-aware per-agent backpressure
instead of dropping normalized stream events when its bounded queue fills. The
E2E evidence attaches source chunk count, exact persisted byte count, gateway
frame/byte summaries, and quiet-session progress without transcript content.
## Files changed
- `apps/backend/cmd/mock-agent/reasoning_burst.go`
- `apps/backend/cmd/mock-agent/reasoning_burst_test.go`
- `apps/backend/cmd/mock-agent/script.go`
- `apps/backend/internal/agentctl/server/adapter/transport/acp/adapter.go`
- `apps/backend/internal/agentctl/server/adapter/transport/acp/conversion_test.go`
- `apps/backend/internal/agentctl/server/process/manager.go`
- `apps/web/e2e/helpers/api-client.ts`
- `apps/web/e2e/helpers/session-stream-overload.ts`
- `apps/web/e2e/tests/session/session-stream-overload-isolation.spec.ts`
- `apps/web/e2e/tests/session/mobile-session-stream-overload-isolation.spec.ts`
@@ -1,6 +1,7 @@
---
status: implemented
status: approved
created: 2026-08-01
updated: 2026-08-02
owner: kandev
---
@@ -34,6 +35,9 @@ detail surface.
refresh has a targeted action that does not alter focus.
- Correlated WebSocket responses and errors are prioritized over unsolicited
notifications and cannot be silently dropped by notification pressure.
- One session's streaming text or reasoning cannot monopolize persistence,
notification delivery, or browser state work. Intermediate replacement
updates are bounded while the final transcript remains lossless.
- `message.add` uses a stable client-generated message ID so an uncertain send
can be reconciled or retried without creating or dispatching a duplicate.
- Existing desktop and mobile status/icon precedence remains unchanged.
@@ -122,6 +126,57 @@ remain during migration, but switchers use the summary when present.
notification hydration. Duplicate notifications and responses upsert the
same frontend message.
## Session stream overload isolation
Agent stream chunks are an ingress format, not a required persistence or
browser-render cadence. The lifecycle boundary coalesces adjacent assistant or
reasoning chunks for the same Kandev message record before publishing them to
the orchestrator:
- the first non-empty chunk creates the streaming record without waiting for a
coalescing interval;
- subsequent adjacent chunks for that record publish at most once per 100 ms
window while streaming continues;
- concatenation preserves byte order and content exactly; coalescing may not
truncate, summarize, or rewrite agent output; and
- pending content flushes before a tool call/result, permission request,
completion, error, stream disconnect, execution replacement, or backend
shutdown so persisted transcript order remains authoritative.
The per-agent ACP adapter and process handoff apply cancellation-aware
backpressure when their bounded handoff queues fill. They must not silently
drop normalized assistant/reasoning chunks before the lifecycle coalescer; a
noisy session may slow its own agent process, and shutdown cancellation may
discard only work after that session is terminal. This handoff is scoped to the
agent instance and must not block unrelated backend sessions.
The WebSocket gateway treats `session.message.updated` as a replaceable full-
state notification keyed by client, session, and message. At most one unsent
replacement for a key occupies delivery capacity. A newer update replaces the
older payload in place. Replaceable stream notifications use bounded
per-session capacity and fair scheduling, separate from correlated control
traffic and semantic notifications such as message creation/deletion, task or
session state changes, pending-input occurrences, and turn completion. When a
session exceeds its replaceable allowance, that session's oldest queued
replaceable entry may be evicted; it cannot consume another session's
allowance or the semantic-notification lane. Reconnect, snapshot, and
turn-settle reconciliation repair any intermediate replacement that was not
delivered because the database remains authoritative.
The frontend applies the same defense at the render boundary. It keeps only the
newest `session.message.updated` payload for each message during one animation
frame and performs one store update per changed message when the frame flushes.
Message add/delete and turn-settle events remain ordered barriers. Intentional
multi-session detail surfaces may subscribe to every session they display, but
rerendering or refetching equivalent session objects must not tear down and
recreate unchanged subscription membership.
Overload handling is observable. Structured metrics/logs identify the client,
session, action, queue class, coalesced chunk count, replacement count, and
eviction count without logging transcript content. Reconnect, a selected-
session snapshot, or turn-settle reconciliation repairs any skipped
intermediate replacement.
## Failure modes
- If projection fails, the last valid summary remains visible, the failure is
@@ -138,6 +193,10 @@ remain during migration, but switchers use the summary when present.
reconciliation cannot determine acceptance.
- Notification overload may coalesce or drop replaceable notifications, but
it cannot silently discard a correlated response/error.
- If one session exceeds its replaceable stream allowance, other sessions and
semantic notifications continue to make progress. The noisy session retains
exact persisted content and converges through its newest replacement or
normal reconciliation.
- If `status_summary` is temporarily absent during rollout, task rows use the
existing coarse task fields and omit unavailable decorations; they do not
subscribe to inactive sessions.
@@ -166,6 +225,18 @@ remain during migration, but switchers use the summary when present.
- **GIVEN** many background sessions are active, **WHEN** the selected agent
publishes model configuration, **THEN** the selected chat retains its model
selector and can submit a follow-up message.
- **GIVEN** one ACP session emits 30,000 tiny reasoning chunks for one message,
**WHEN** the stream crosses the lifecycle, persistence, gateway, and browser
boundaries, **THEN** the final reasoning content is byte-for-byte complete,
append publications occur no more than once per 100 ms window, and pending
replacement work remains bounded.
- **GIVEN** a noisy session is continuously streaming, **WHEN** the user opens
another task or submits a message in another selected session, **THEN** that
navigation and its correlated response complete without waiting for the
noisy session's backlog.
- **GIVEN** an Office task detail refetch returns new objects for the same set
of session IDs, **WHEN** React rerenders the page, **THEN** the client sends no
unsubscribe, subscribe, or initial-snapshot replay for unchanged membership.
- **GIVEN** a phone viewport, **WHEN** task summaries change, **THEN** the
existing task-switcher sheet shows the same badges and precedence as the
desktop sidebar without new navigation, scroll, or touch behavior.
@@ -182,7 +253,14 @@ remain during migration, but switchers use the summary when present.
backend process crash; this contract prevents duplicate handler dispatch on
client retry.
- Treating a larger timeout or queue capacity as the fix.
- Limiting how much reasoning or assistant content an agent may ultimately
produce. This contract bounds intermediate work, not transcript length.
- Automatically stopping or penalizing an agent solely because it emits many
valid chunks.
## Implementation plan
See [`../../plans/bounded-task-status-delivery/plan.md`](../../plans/bounded-task-status-delivery/plan.md).
- Original delivery redesign:
[`../../plans/bounded-task-status-delivery/plan.md`](../../plans/bounded-task-status-delivery/plan.md)
- Stream overload repair:
[`../../plans/session-stream-overload-isolation/plan.md`](../../plans/session-stream-overload-isolation/plan.md)