Files
kandev/docs/plans/session-stream-overload-isolation/plan.md
T
2026-08-24 11:09:03 +00:00

248 lines
12 KiB
Markdown

---
spec: docs/specs/platform/requirements/bounded-task-status-delivery.md
decision: docs/decisions/2026-08-02-isolate-replaceable-session-stream-traffic.md
created: 2026-08-02
status: complete
---
# Implementation Plan: Session Stream Overload Isolation
## Overview
Contain a single high-volume ACP session at three independent boundaries while
keeping the final transcript exact. Coalesce adjacent assistant/reasoning
chunks before the orchestrator performs read/append/write work, replace queued
full-message notifications instead of accumulating stale versions, and batch
browser store updates once per animation frame. Stabilize Office multi-session
membership so task refetches do not replay snapshots. Prove the contract with a
deterministic noisy mock-agent scenario on desktop and mobile.
This extends the implemented bounded task-status delivery design. It does not
change task summaries, message persistence ownership, or the reserved
correlated-response queue.
## Incident baseline
- Task `69637afd-c193-4197-aa49-348ac2b0cfb3`, session
`89f7b247-cfb3-4a4b-8147-8826e13f1630`.
- The recorded launcher was `opencode-acp` using `opencode-go/kimi-k3`; the
logs do not identify this execution as Grok.
- 28,967 unique reasoning-stream events and 31,314 total stream events over
about 41 minutes, peaking at 63 reasoning events per second.
- Every chunk crossed lifecycle, orchestrator, message persistence, gateway,
JSON, Zustand, and React boundaries.
- Five `session.message.updated` frames were dropped when the 256-frame
notification queue filled. No control-queue overflow or backend crash was
recorded.
- Equivalent Office task refetches caused repeated unchanged session
unsubscribe/subscribe/snapshot cycles, further amplifying traffic.
## Architecture
### Lossless lifecycle coalescing
Add an execution-owned coalescer in
`apps/backend/internal/agent/runtime/lifecycle`. It accepts already-normalized
assistant/reasoning chunks after legacy newline buffering and protocol ID to
Kandev ID mapping. The first non-empty chunk for a record publishes immediately with
`IsAppend=false`. Adjacent later chunks append to one pending segment and flush
at the 100 ms cadence with `IsAppend=true`.
Keep ordering explicit. Before a tool call/result, permission request,
completion, error, stream disconnect, prompt-generation reset, execution
replacement, or manager shutdown, synchronously detach and publish pending
segments before publishing the boundary. Timer callbacks detach under the
execution mutex and publish outside it. Teardown stops timers and makes repeat
flushes idempotent.
The coalescer operates before `handleThinkingStreamingEvent` /
`handleMessageStreamingEvent`, so the existing task service naturally performs
fewer `GetMessage`, concatenation, `UpdateMessage`, and full-message event
publications. It does not change stored schemas or final message shapes.
### Session-isolated notification scheduling
Replace the client's single best-effort notification channel with a scheduler
that retains three classes:
1. the existing control queue for correlated responses/errors;
2. a bounded semantic FIFO for non-replaceable notifications; and
3. bounded per-session queues for replaceable `session.message.updated`
notifications.
The hub passes action and routing identity to the client instead of only raw
bytes. For a replaceable message update, derive the key from `session_id` and
`message_id`. If the key is already pending, replace its bytes without moving
its position. A later semantic event for the same session therefore remains
behind the newest message state. Drain active session queues round-robin and
interleave them with the semantic FIFO after bounded control priority.
Production capacities are named constants, but tests construct small queues.
On per-session pressure, evict or reject only queued replaceable entries for
that session. Never evict control or semantic notifications to admit a stream
replacement. Record content-free counters/log fields for action, client,
session, replacement, eviction, and queue depth.
### Frontend batching and stable membership
Extract a small replaceable-message scheduler beside
`lib/ws/handlers/messages.ts`. During one animation frame it retains only the
newest full payload per `(session_id, message_id)` and then invokes
`updateMessage` once per changed key. Added/deleted messages and turn-settle
events act as barriers: flush or cancel a pending replacement as appropriate so
wire order cannot revive deleted state. Clear scheduled work when the WS router
is disposed or replaced.
In the Office task detail live-sync hook, derive a stable, sorted session-ID
membership key. Refetching equal session IDs or replacing the task object must
not rerun subscription cleanup/setup. A genuine membership change applies only
the added/removed IDs. The surface remains an intentional multi-session detail
view; this task removes churn rather than changing its product scope.
### Observability and reconciliation
Lifecycle and gateway logs/metrics must identify pressure without including
message content. Existing session snapshot/reconnect and turn-settle behavior
remain the reconciliation path; no new user-facing overload banner or protocol
action is required.
## Tests
### Backend
- Use `testing/synctest` around the lifecycle coalescer. Assert immediate first
publication, one append per 100 ms window, exact 30,000-chunk concatenation,
independent assistant/reasoning IDs, boundary order, disconnect/shutdown
flush, and no timer/goroutine leak.
- Assert the orchestrator/task-service integration sees the reduced event count
and stores exact final assistant and thinking content.
- Construct gateway schedulers with tiny capacities. Assert replacement in
place, bounded per-session occupancy, round-robin progress, semantic barrier
order, control priority, noisy-session-only eviction, metrics, close safety,
and zero-value test-client compatibility.
### Frontend
- Feed hundreds of `session.message.updated` payloads for one message in a
single fake animation frame. Assert one store update with the newest complete
payload, while different message keys each update once.
- Cover add/delete/turn barriers, handler teardown, and separate sessions.
- Rerender Office live sync with new task/session object references but the
same IDs and assert no subscription churn; then change membership and assert
only the delta is subscribed/unsubscribed.
## E2E tests
Add a deterministic mock-agent command that emits a configurable reasoning
burst without sleeping or embedding thousands of commands in the prompt. A
new overload-isolation Playwright spec opens the noisy session and captures
gateway frames while interacting with a second session.
- Desktop Chromium: emit a representative burst, navigate to the quiet task,
submit a message, and assert navigation/model controls/responding message
remain usable before the noisy run settles.
- Assert server-side burst count was actually produced, the final persisted
reasoning text is exact, received replacement frames are materially below
source chunks, no control response is lost, and the quiet session progresses.
- Mobile Chrome: repeat the navigation/quiet-session progress assertion through
the existing native task-switcher sheet and assert no horizontal overflow.
- Attach source-chunk, lifecycle-publication, gateway replacement/eviction,
received-frame, byte, and latency totals. Counts are diagnostics except for
the hard boundedness, exact-content, and cross-session-progress assertions.
## Mobile design contract
- **Desktop outcome / mobile entry:** both remain responsive while another
session streams. Desktop uses the existing task/sidebar navigation; phone
uses the existing task-switcher sheet.
- **Nearest exemplar:** keep
`apps/web/components/task/mobile/session-task-switcher-sheet.tsx` and shared
task/session stores unchanged in layout and navigation ownership.
- **Hierarchy / interaction:** no new control, gesture, dialog, or route.
- **Presentation / scroll:** the repair changes transport/state cadence only;
existing sheet scrolling, safe areas, and viewport bounds remain intact.
- **Shared state:** desktop and mobile consume the same batched message and
connection state.
- **Mobile proof:** run the overload scenario under `mobile-chrome`, switch to a
quiet task, verify progress, and assert `scrollWidth <= clientWidth`.
## Implementation waves
Implementation is sequential in the user-started session because Tasks 02 and
03 consume the delivery semantics established by the preceding task. These
task files do not authorize subagent work.
- [x] [Task 01: Coalesce agent stream ingress](task-01-coalesce-agent-stream-ingress.md)
- [x] [Task 02: Isolate replaceable session delivery](task-02-isolate-replaceable-session-delivery.md)
- [x] [Task 03: Bound frontend stream work](task-03-bound-frontend-stream-work.md)
- [x] [Task 04: Prove overload containment](task-04-prove-overload-containment.md)
## Verification
All implementation waves are complete. The final verification evidence is:
- `go test -race ./internal/agentctl/server/adapter/transport/acp ./internal/agentctl/server/process ./internal/agent/runtime/lifecycle ./internal/gateway/websocket ./cmd/mock-agent` — passed (2,633 tests).
- `make -C apps/backend test` — passed on the final run.
- `rtk proxy golangci-lint run ./...` — passed (0 issues).
- Focused frontend Vitest — passed (7 tests); web typecheck and lint — passed.
- Desktop Chromium overload E2E — passed (1 test, 13.4s).
- Mobile Chrome overload E2E — passed (1 test, 8.3s).
The E2E evidence confirms exact persisted reasoning, bounded noisy-session
frames, quiet-session progress, and desktop/mobile viewport safety. The ACP
and process handoff now uses cancellation-aware per-agent backpressure so
normalized stream events are not silently discarded before lifecycle
coalescing; shutdown cancellation still releases a blocked noisy session.
```bash
cd apps/backend && go test ./internal/agent/runtime/lifecycle/... \
./internal/orchestrator/... ./internal/task/service/...
cd apps/backend && go test ./internal/gateway/websocket/...
make -C apps/backend test
make -C apps/backend lint
cd apps/web && pnpm exec vitest run \
lib/ws/handlers/messages.test.ts \
'app/office/tasks/[id]/use-session-live-sync.test.tsx'
cd apps/web && pnpm run typecheck
cd apps/web && pnpm run lint
cd apps/web && pnpm e2e:run tests/session/session-stream-overload-isolation.spec.ts \
-- --project=chromium
cd apps/web && pnpm e2e:run tests/session/session-stream-overload-isolation.spec.ts \
-- --project=mobile-chrome
```
## Risks
- Timer callbacks racing completion could duplicate or reorder content. Detach
pending buffers atomically, publish outside locks, and make boundary flushes
idempotent.
- Protocol message IDs may interleave. Coalesce only adjacent chunks per typed
Kandev record and retain the first-seen queue position.
- A latest-wins gateway queue can reorder a message update past turn completion
if replacement positions are not stable. Preserve per-session enqueue order
and test semantic barriers.
- Strict control/semantic priority could starve replacement streams. Use
bounded priority bursts plus round-robin session scheduling.
- Animation-frame batching can be heavily throttled in background tabs. Keep
only complete latest state and rely on selected-session reconciliation when
the tab resumes.
- A quiet E2E producer would yield a false pass. Expose and assert the mock
burst's produced count before checking isolation.
## Out of scope
- Truncating stored assistant or reasoning content.
- Automatically stopping an agent because it streams heavily.
- Splitting sessions across multiple WebSocket connections.
- Changing task-summary fields, task-switcher layout, or message schema.
- Adding a user-facing overload state or runtime feature flag.
## Documentation impact
The behavioral contract is added to
`docs/specs/platform/requirements/bounded-task-status-delivery.md` and the shared boundary is
recorded in
`docs/decisions/2026-08-02-isolate-replaceable-session-stream-traffic.md`.
No public docs change is expected because commands, settings, public APIs, and
visible interaction patterns do not change.