--- spec: docs/specs/agents/requirements/subagent-context-persistence.md created: 2026-08-13 status: draft --- # Implementation Plan: Subagent context persistence > **Current implementation boundary (2026-08-15).** The execution-aware > uniqueness key is the initial schema contract. This plan retains the original > design notes below, but the supported implementation does not include a > compatibility migration for the unshipped two-column table shape. See > [ADR-2026-08-15-subagent-context-schema-boundary](../../decisions/2026-08-15-subagent-context-schema-boundary.md). > The load-bearing differences from the original draft are: > > - **Uniqueness key** is `(task_session_id, agent_execution_id, tool_call_id)`, > not `(task_session_id, tool_call_id)`. `task_session_subagents` carries a > new `agent_execution_id TEXT NOT NULL DEFAULT 'unknown'` column (AC-31). > A live write with no execution identity available stores the `'unknown'` > sentinel and the service increments `unknown_execution`. > - The final table is created by `initSubagentContextSchema`; `runMigrations()` > only performs the historical-message backfill and writes its two activation > keys. There is no execution-schema activation key or shape probe. > - **The backfill and the activation-key writes are one transaction**, not > three independent `r.migrate.Apply(...)` calls: `tx.Beginx()` / > `defer tx.Rollback()` wraps the `INSERT ... SELECT`, the `capture_since` > write, and the `backfill_through` write, then `tx.Commit()`. A key is > therefore never written after a swallowed backfill failure — the whole > transaction rolls back together, so this is already the atomic-with-success > behavior a docs-only staleness report once flagged as missing. > - **`RecordSubagentContextRequest` carries `ExecutionID string`**, propagated > from the orchestrator's frame handling through to the repository upsert's > conflict key. > > The "Schema (task 01)" through "Orchestrator wiring (task 04)" sections below > are retained as the historical design record; the boundary above is > authoritative for the current implementation. ## Overview Add one new table, `task_session_subagents`, and a write path that fills it from the two orchestrator frame handlers that already parse the subagent payload. The order is forced by dependency: table DDL and the one-time backfill first, then the repository upsert that encodes every merge rule, then the service-level writer that owns normalization and the `expvar` counters, then the orchestrator call sites, and finally the Postgres parity and health-query coverage. Nothing user-facing changes, so there is no frontend and no E2E work. The whole feature is additive telemetry. The governing constraint from the spec is that it must be structurally impossible for this writer to break the product path (AC-27), and that an unreported measurement must never be stored as `0` (AC-7, and the regression named in the spec's *The one shape a builder must not get wrong*). --- ## Backend ### Schema (task 01) New table created in the schema-init DDL, in `apps/backend/internal/task/repository/sqlite/base_schema.go`, as a new `initSubagentContextSchema()` called from `initSessionSchema()` immediately after `initMessageTurnSchema()`. **On the spec's "DDL *and* migration" wording.** `apps/backend/AGENTS.md` § *Schema & migrations* is explicit that `initSchema()` runs every `CREATE TABLE IF NOT EXISTS` step **before** `runMigrations()`, on every boot. A *new table* is therefore fully created on existing databases by the schema-init DDL alone, which satisfies AC-17. The "add it only via a migration" rule in that document governs **new columns on existing tables**, which this feature has none of. Do **not** duplicate the `CREATE TABLE` into `base_migrations.go` — the migration entry for this feature is the backfill and the activation keys, and nothing else. Columns exactly as the spec's *Data model* table. The three measurement columns carry **no `DEFAULT`**: ```sql CREATE TABLE IF NOT EXISTS task_session_subagents ( id TEXT PRIMARY KEY, task_session_id TEXT NOT NULL, task_id TEXT NOT NULL, turn_id TEXT, tool_call_id TEXT NOT NULL, parent_tool_call_id TEXT, subagent_type TEXT, description TEXT, agent_id TEXT, child_session_id TEXT, model TEXT, agent_status TEXT, tool_status TEXT, is_async INTEGER NOT NULL DEFAULT 0, total_tokens INTEGER, tool_use_count INTEGER, duration_ms INTEGER, source TEXT NOT NULL DEFAULT 'live', observed_at TIMESTAMP NOT NULL, settled_at TIMESTAMP, updated_at TIMESTAMP NOT NULL, UNIQUE (task_session_id, tool_call_id), FOREIGN KEY (task_session_id) REFERENCES task_sessions(id) ON DELETE CASCADE ); CREATE INDEX IF NOT EXISTS idx_subagents_session_id ON task_session_subagents(task_session_id); CREATE INDEX IF NOT EXISTS idx_subagents_task_id ON task_session_subagents(task_id); CREATE INDEX IF NOT EXISTS idx_subagents_turn_id ON task_session_subagents(turn_id); ``` No `FOREIGN KEY` on `turn_id` — deliberate, per the spec: a turn deletion must not delete the fan-out record. ### Migration: backfill and activation keys (task 01) `apps/backend/internal/task/repository/sqlite/base_migrations.go`, a new `migrateSubagentContextBackfill()` called from `runMigrations()` **after** the existing statements. Three `r.migrate.Apply(...)` calls, all argument-free SQL (`MigrateLogger.Apply` takes no bind args, and its swallow-plus-`WARN` contract at `internal/db/migratelog.go:33` is what AC-20 is written against): 1. `task_session_subagents.backfill` — one unbounded `INSERT INTO task_session_subagents (...) SELECT ... FROM task_session_messages m WHERE = 'subagent_task' AND ... ON CONFLICT (task_session_id, tool_call_id) DO NOTHING`. No `LIMIT`, no batching (AC-23a). 2. `kandev_meta.subagent_context_capture_since` — write-once `INSERT ... ON CONFLICT(key) DO NOTHING` with an `fmt.Sprintf`-inlined `time.Now().UTC().Format(time.RFC3339)` literal. 3. `kandev_meta.subagent_context_backfill_through` — write-once, value from a subselect `COALESCE(MAX(m.created_at), '')` over the same predicate as (1). `kandev_meta` is guaranteed to exist: `persistence.Provide` calls `ensureMetaTable` (`internal/persistence/provider.go:72`) before `repository.Provide` builds the task repo (`internal/backendapp/storage.go:41`). Backfill derivations, all via dialect-aware helpers in the same file: | Target | Source | Notes | |---|---|---| | `id` | `m.id` | The message's own UUID. Deterministic, so a replay maps to the same row and `DO NOTHING` is a true no-op. | | `task_session_id`, `task_id` | `m.task_session_id`, `m.task_id` | Rows with `task_id = ''` are excluded by the `WHERE` (AC-2). | | `turn_id` | `NULLIF(m.turn_id, '')` | | | `tool_call_id` | `metadata.tool_call_id` | Excluded by `WHERE` when empty (AC-2). | | `parent_tool_call_id` | `metadata.parent_tool_call_id` | Top-level metadata key, not inside `normalized`. | | `agent_status` | `metadata.normalized.subagent_task.status` | | | `tool_status` | `metadata.status` | The ACP status the message row already stores. | | `observed_at` | `m.created_at` | | | `settled_at` | `m.updated_at` when the derived `tool_status` is terminal, else NULL | Terminal set mirrors `isTerminalToolStatus`. | | `source` | literal `'backfill'` | | Three new helpers beside `jsonColumn` / `jsonText` (`base_migrations.go:319-335`), each reusing `jsonColumn`'s empty/`'null'` normalization so one malformed metadata document cannot abort the statement (AC-23b): - `jsonKey(postgres, column, key string) string` — a **top-level** text extract (`#>> '{key}'` / `json_extract(..., '$.key')`), needed for `tool_call_id`, `parent_tool_call_id`, `status`, and `normalized.kind`. - `jsonInt(postgres, column, parent, key string) string` — `CAST` of the text extract to `BIGINT` (Postgres) / `INTEGER` (SQLite), wrapped so an empty extract and a **negative** value both yield NULL (AC-9, AC-23). - `jsonBoolToInt(postgres, column, parent, key string) string` — `1` / `0`. Postgres `#>>` yields `'true'`; SQLite `json_extract` yields `1`. The helper must accept both spellings, which is the whole reason it exists. `total_tokens` and `duration_ms` are `omitempty int64` on the payload, so a reported `0` never reaches the JSON and absent-means-NULL falls out for free. `tool_use_count` is `*int` with `omitempty`, so it is present in the JSON exactly when the agent reported it — including a genuine `0` (AC-8). ### Model and repository upsert (task 02) `apps/backend/internal/task/models/subagent_context.go`: ```go type SubagentContext struct { ID, TaskSessionID, TaskID, ToolCallID string TurnID, ParentToolCallID, SubagentType, Description *string AgentID, ChildSessionID, Model, AgentStatus, ToolStatus *string IsAsync bool TotalTokens *int64 ToolUseCount *int64 DurationMs *int64 Source string ObservedAt time.Time SettledAt *time.Time UpdatedAt time.Time } ``` Every nullable column is a pointer. This is not style: `ToolUseCount` must distinguish a reported `0` from "not reported", and a non-pointer `int64` cannot. `apps/backend/internal/task/repository/sqlite/subagent_context.go`: - `UpsertSubagentContext(ctx context.Context, sc *models.SubagentContext) error` — a **single** `INSERT ... ON CONFLICT (task_session_id, tool_call_id) DO UPDATE SET ...`. No read-then-write anywhere (AC-14). Every merge rule lives in the conflict clause: | Column | Conflict expression | AC | |---|---|---| | `turn_id` | `COALESCE(task_session_subagents.turn_id, excluded.turn_id)` | AC-2a — the *original* turn wins; a later turn never re-attributes | | text fields | `COALESCE(excluded., task_session_subagents.)` | AC-4, AC-5 — fill-forward, never blank | | `tool_status` | `CASE WHEN task_session_subagents.settled_at IS NOT NULL THEN task_session_subagents.tool_status ELSE COALESCE(excluded.tool_status, ...) END` | AC-12 — frozen at the terminal value | | `is_async` | `CASE WHEN excluded.is_async = 1 THEN 1 ELSE task_session_subagents.is_async END` | sticky, never reset to 0 | | metrics | `COALESCE(excluded., task_session_subagents.)` | AC-4, AC-7 | | `settled_at` | `COALESCE(task_session_subagents.settled_at, excluded.settled_at)` | AC-11 — write-once | | `updated_at` | `excluded.updated_at` | | | `source`, `observed_at`, `id` | **absent from the SET list** | write-once: a backfilled row is never relabelled `live` (AC-22), and first observation wins (AC-1a) | Both dialects accept `excluded.` on the right and unqualified names on the left; the self-qualified `task_session_subagents.` reads are valid on both. Statement built once as a package-level `const` and passed through `r.db.Rebind`, matching `review.go:27` and `document.go:291`. - `ListSubagentContextsBySession(ctx, sessionID string) ([]*models.SubagentContext, error)` and `ListSubagentContextsByTurn(ctx, turnID string)` — both `ORDER BY observed_at ASC, tool_call_id ASC` (AC-13). These are the read seam AC-13 is written against and the seam the tests assert ordering through. They are repository-level only: no service method, no DTO, no route. The spec's *Out of scope* forbids a read API, not a repository accessor. ### Service writer and counters (task 03) `apps/backend/internal/task/service/subagent_context.go`, one entry point: ```go type RecordSubagentContextRequest struct { TaskSessionID, TaskID, TurnID string ToolCallID, ParentToolCallID string ToolStatus string Payload *streams.SubagentTaskPayload ObservedAt time.Time } func (s *Service) RecordSubagentContext(ctx context.Context, req RecordSubagentContextRequest) ``` **It returns nothing.** That is the mechanism by which AC-27 holds: a writer with no error return cannot fail the enclosing message write, turn, or stream, and no future caller can accidentally propagate one. It owns: - Identity gate — empty `TaskSessionID`, `TaskID`, or `ToolCallID` ⇒ no write, `skipped_no_identity`++, return (AC-2). - `nilIfEmpty(string) *string` applied to every nullable TEXT field, so `""` becomes NULL and `COUNT(model)` counts models (spec § *Column-level normalization rules*). - Metric normalization: `DurationMs`/`TotalTokens` map to `*int64` and are left nil when zero-or-absent; `ToolUseCount` maps to `*int64` **only** when the payload pointer is non-nil, preserving a reported `0` (AC-7, AC-8). Any negative value ⇒ that field nil **and** `anomalous_value`++, other fields unaffected (AC-9). - Terminality: `settled_at = &req.ObservedAt` only when `orchestrator`-equivalent terminal classification of `req.ToolStatus` is true. `agent_status` is stored verbatim and never consulted (AC-10a, AC-11). The classifier is unexported in `internal/orchestrator`, so the service gets its own `isTerminalToolStatus` over the identical set (`complete`, `completed`, `success`, `error`, `failed`, `cancelled`) with a comment naming `event_handlers_streaming.go:762` as the peer, plus a test that pins the two lists together. - Failure handling: repository error ⇒ `WARN` with session id and tool call id, `failed`++, return (AC-27). - `attempted`++ on entry past the nil-payload check; `persisted`++ on success. `apps/backend/internal/task/service/subagent_context_metrics.go` — one `expvar.NewMap("subagent_context_total")` with the five keys AC-26 names (`attempted`, `persisted`, `skipped_no_identity`, `anomalous_value`, `failed`), following `internal/office/scheduler/metrics_vars.go` (package-level `NewMap`, counters only, no gauges). ### Orchestrator wiring (task 04) `apps/backend/internal/orchestrator/service.go` — a narrow interface beside `MessageCreator`: ```go type SubagentContextRecorder interface { RecordSubagentContext(ctx context.Context, req taskservice.RecordSubagentContextRequest) } ``` held as an optional `subagentContexts` field on `Service`, nil-checked at both call sites exactly as `messageCreator` is. Two call sites in `event_handlers_streaming.go`, each already holding every input: - `handleToolCallEvent` (~line 312) — after the `CreateToolCallMessage` block, using `s.getActiveTurnID(payload.SessionID)`. - `persistToolUpdateMessage` (~line 565) — after `UpdateToolCallMessage`, using the `turnID` that function already resolved (`peekActiveTurnID` for terminal frames, `getActiveTurnID` otherwise). Both guarded by `payload.Data.Normalized.Kind() == streams.ToolKindSubagentTask` and a non-nil `Normalized.SubagentTask()`. A new `recordSubagentContextFromFrame(ctx, payload, turnID)` helper holds the guard and the request build so neither handler grows past the package's `funlen` limit of 80 lines / 50 statements. `payload.SessionID` is the Kandev task session id: `messageCreatorAdapter` passes the same value straight into `CreateMessageRequest.TaskSessionID` (`internal/backendapp/adapters.go:879`), despite the `agentSessionID` parameter name. `apps/backend/internal/backendapp/adapters.go` — a `subagentContextAdapter` over `*taskservice.Service`, wired where `messageCreatorAdapter` is wired. --- ## Frontend None. The spec's *What this feature is, and is not* rules out any frontend surface: the subagent card keeps rendering from message metadata, unchanged. No DTO field, no WebSocket event, no store slice. --- ## Tests ### Migration and backfill — `subagent_context_migration_test.go` (task 01) Patterned on `task_external_id_migration_test.go`. - **Fresh DB creates the table, indexes, and UNIQUE constraint.** File: `apps/backend/internal/task/repository/sqlite/subagent_context_migration_test.go`. How: `NewWithDB` over a temp SQLite file, then assert `idx_subagents_session_id`, `idx_subagents_task_id`, `idx_subagents_turn_id` exist in `sqlite_master` and that a duplicate `(task_session_id, tool_call_id)` insert is rejected. (AC-17) - **Replay is a no-op.** How: `repo.runMigrations()` twice; assert no error and an unchanged row count. (AC-18) - **Backfill inserts one row per subagent message.** How: seed a session, a turn, and message rows whose `metadata.normalized.kind = 'subagent_task'`, then `runMigrations()`; assert one row each with `source = 'backfill'` and `observed_at = m.created_at`. (AC-21) - **`async_launched` shape stores NULL, not `0`** — the spec's named must-not-get-wrong regression. How: seed a message whose `subagent_task` object has `status: "async_launched"` and no `total_tokens` / `tool_use_count` / `duration_ms`; assert all three columns `IS NULL` and that none is `0`. (AC-7) - **A reported `tool_use_count` of `0` survives as `0`.** (AC-8) - **A negative reported metric backfills as NULL.** (AC-9, AC-23) - **Empty JSON strings become NULL.** How: a payload with `model: ""`; assert `model IS NULL`, not `''`. - **Malformed metadata does not abort the statement.** How: seed one message row with `metadata = ''` and another with `metadata = 'null'` alongside a valid subagent row; assert the valid row still lands. (AC-23b) - **A live row wins over the backfill.** How: pre-insert a `source = 'live'` row for a `(session, tool_call_id)` that also exists as a message, run `runMigrations()`, assert `source` is still `'live'` and there is exactly one row. (AC-22) - **Rows with no `task_id` or no `tool_call_id` are not backfilled.** (AC-2) - **Activation keys are written once.** How: assert both `kandev_meta` keys are present and RFC3339-parseable (`backfill_through` may be `''`), capture their values, `runMigrations()` again, assert both unchanged. (AC-24) - **Fresh DB with no subagent messages sets `backfill_through` to `''`.** (AC-24) ### Repository upsert — `subagent_context_test.go` (task 02) File: `apps/backend/internal/task/repository/sqlite/subagent_context_test.go`. - **First upsert inserts.** (AC-1) - **Second upsert for the same key updates in place, one row.** (AC-3) - **Fill-forward: a frame with empty fields does not blank learned values.** Table-driven over every nullable column. (AC-4) - **A non-empty value replaces the stored value.** (AC-5) - **`settled_at` is write-once**: a second terminal frame with a later timestamp leaves the first value. (AC-11) - **A non-terminal frame after settling leaves `settled_at` and `tool_status` alone but still fills NULL columns.** (AC-12) - **Never-terminal row exists with `settled_at IS NULL`.** (AC-10) - **`turn_id` from the first frame survives a later frame carrying a different turn.** (AC-2a) - **`is_async` is sticky** across a later frame reporting false. - **`source` and `observed_at` are not overwritten** by a later upsert. - **Nested `parent_tool_call_id` is stored and the row is not suppressed.** (AC-6) - **Concurrent upserts of the same key** — N goroutines, `errgroup`; assert exactly one row and that the invariants above hold regardless of commit order. (AC-14) - **Concurrent upserts of different keys in one turn** — the observed 3-, 4-, 5- and 8-way fan-outs; assert every row lands. (AC-15) - **Deleting the parent session cascades the rows away.** (AC-16) Also add `task_session_subagents` to `assertNoWorkspaceCascadeDependents` in `workspace_test.go:317`, so the workspace cascade test covers the new table. - **Read ordering is `observed_at ASC, tool_call_id ASC`** — seed rows sharing an `observed_at` and assert the `tool_call_id` tiebreak, per AC-13's explicit rejection of `id` as a tiebreak. ### Service writer — `subagent_context_test.go` (task 03) File: `apps/backend/internal/task/service/subagent_context_test.go`. - **Table-driven identity gate**: each of empty session id, empty task id, empty tool call id ⇒ no repository call, `skipped_no_identity` incremented by one. (AC-2) - **Table-driven normalization**: `""` ⇒ NULL for every nullable TEXT column; absent metric ⇒ nil; reported `0` `ToolUseCount` ⇒ `*int64(0)`; negative ⇒ nil plus `anomalous_value`++ with the other fields of the same frame intact. (AC-7, AC-8, AC-9) - **Terminal `tool_status` sets `settled_at`; `agent_status = "async_launched"` alone does not.** (AC-10a, AC-11) - **An unrecognised `agent_status` is stored verbatim and does not settle.** (AC-10a) - **A repository error logs `WARN`, increments `failed`, and the call still returns normally.** How: a stub repo returning an error plus `zaptest`/observed-logs. (AC-27) - **`attempted` and `persisted` move as specified** on the happy path. (AC-26) - **The service's terminal-status set equals the orchestrator's** — pin both lists so a future edit to one breaks the test. (AC-11) ### Orchestrator integration — `event_handlers_streaming_subagent_test.go` (task 04) File: `apps/backend/internal/orchestrator/event_handlers_streaming_subagent_test.go`. - **`handleToolCallEvent` with a `subagent_task` payload records a context** with the active turn id. (AC-1) - **`handleToolUpdateEvent` for the same tool call records again** so the merge path runs; assert one recorded request per frame with the right turn id. (AC-3) - **A terminal update carries the terminal `tool_status` through.** (AC-11) - **A non-`subagent_task` normalized kind records nothing.** - **A nil `subagentContexts` field is safe** — no panic, message write still happens. (AC-27) - **A nested frame (`ParentToolCallID` set) is recorded, not dropped.** (AC-6) - **Integration through the real service and repository**, using `newServiceBackedMessageCreator(repo)` as `event_handlers_streaming_test.go:906` does, so one test exercises handler → service → repo end to end. ### Postgres parity and the health query (task 05) - **Fresh Postgres schema plus migration replay**, including the backfill and both activation keys. File: `apps/backend/internal/task/repository/sqlite/subagent_context_postgres_test.go`, gated on `KANDEV_TEST_POSTGRES_DSN` via `testutil.OpenIsolatedPostgres`. (AC-19) - **The upsert's conflict clause behaves identically on Postgres** — fill-forward, `settled_at` write-once, `is_async` sticky, `turn_id` preserved. This is the ADR-0027 requirement that schema replay alone is insufficient for a dialect-sensitive method. (AC-19) - **The backfill's JSON helpers work on `jsonb`**, including the `''`/`'null'` metadata rows and the boolean spelling difference. (AC-23b) - **AC-28 health query** on a seeded store: assert the `task_session_messages` `subagent_task` count equals `COUNT(*) FROM task_session_subagents` after live writes, and that seeding a message the writer never saw makes the two diverge. File: `apps/backend/internal/task/repository/sqlite/subagent_context_health_test.go`. (AC-28, AC-29) --- ## E2E Tests None. The spec's *Out of scope* states no UI surface changes, therefore "no E2E coverage is implied". Adding one would test nothing this feature changes. --- ## Verification Results Pending. On completion, synchronize this section with each task's `## Results`: record exact commands and outcomes/counts, generated artifact paths, and cleanup/teardown evidence. --- ## Implementation Waves And Parallel Candidates Every task shares the `internal/task` subtree and each depends on the previous one's schema or seam, so the chain is sequential. Nothing here is parallel-safe: tasks 01 and 02 both edit the same package and task 01 owns shared schema DDL. ``` Wave 1: - [ ] [task-01-schema-and-backfill](task-01-schema-and-backfill.md) Wave 2: - [ ] [task-02-repository-upsert](task-02-repository-upsert.md) Wave 3: - [ ] [task-03-service-writer](task-03-service-writer.md) Wave 4: - [ ] [task-04-orchestrator-wiring](task-04-orchestrator-wiring.md) Wave 5: - [ ] [task-05-postgres-and-health](task-05-postgres-and-health.md) ``` ## Open Questions None. Two spec ambiguities were resolved against repository convention rather than left open, and both are recorded above so a builder does not re-litigate them: 1. **"Schema-init DDL *and* an idempotent migration"** for a new table would be a duplicated `CREATE TABLE`. `apps/backend/AGENTS.md` settles it: schema init runs before `runMigrations()` on every boot, so the DDL alone satisfies AC-17. See *Schema (task 01)*. 2. **AC-13 mandates a read ordering** while *Out of scope* forbids a read API. Resolved as two repository-level list methods and no service, DTO, or route. See *Model and repository upsert (task 02)*.