kandev/docs/plans/subagent-context-persistence/plan.md

spec, created, status

spec	created	status
docs/specs/subagent-context-persistence/spec.md	2026-08-13	draft

Implementation Plan: Subagent context persistence

Current implementation boundary (2026-08-15). The execution-aware uniqueness key is the initial schema contract. This plan retains the original design notes below, but the supported implementation does not include a compatibility migration for the unshipped two-column table shape. See ADR-2026-08-15-subagent-context-schema-boundary.

The load-bearing differences from the original draft are:

• Uniqueness key is (task_session_id, agent_execution_id, tool_call_id), not (task_session_id, tool_call_id). task_session_subagents carries a new agent_execution_id TEXT NOT NULL DEFAULT 'unknown' column (AC-31). A live write with no execution identity available stores the 'unknown' sentinel and the service increments unknown_execution.
• The final table is created by initSubagentContextSchema; runMigrations() only performs the historical-message backfill and writes its two activation keys. There is no execution-schema activation key or shape probe.
• The backfill and the activation-key writes are one transaction, not three independent r.migrate.Apply(...) calls: tx.Beginx() / defer tx.Rollback() wraps the INSERT ... SELECT, the capture_since write, and the backfill_through write, then tx.Commit(). A key is therefore never written after a swallowed backfill failure — the whole transaction rolls back together, so this is already the atomic-with-success behavior a docs-only staleness report once flagged as missing.
• RecordSubagentContextRequest carries ExecutionID string, propagated from the orchestrator's frame handling through to the repository upsert's conflict key.

The "Schema (task 01)" through "Orchestrator wiring (task 04)" sections below are retained as the historical design record; the boundary above is authoritative for the current implementation.

Overview

Add one new table, task_session_subagents, and a write path that fills it from the two orchestrator frame handlers that already parse the subagent payload. The order is forced by dependency: table DDL and the one-time backfill first, then the repository upsert that encodes every merge rule, then the service-level writer that owns normalization and the expvar counters, then the orchestrator call sites, and finally the Postgres parity and health-query coverage. Nothing user-facing changes, so there is no frontend and no E2E work.

The whole feature is additive telemetry. The governing constraint from the spec is that it must be structurally impossible for this writer to break the product path (AC-27), and that an unreported measurement must never be stored as 0 (AC-7, and the regression named in the spec's The one shape a builder must not get wrong).

---

Backend

Schema (task 01)

New table created in the schema-init DDL, in apps/backend/internal/task/repository/sqlite/base_schema.go, as a new initSubagentContextSchema() called from initSessionSchema() immediately after initMessageTurnSchema().

On the spec's "DDL and migration" wording. apps/backend/AGENTS.md § Schema & migrations is explicit that initSchema() runs every CREATE TABLE IF NOT EXISTS step before runMigrations(), on every boot. A new table is therefore fully created on existing databases by the schema-init DDL alone, which satisfies AC-17. The "add it only via a migration" rule in that document governs new columns on existing tables, which this feature has none of. Do not duplicate the CREATE TABLE into base_migrations.go — the migration entry for this feature is the backfill and the activation keys, and nothing else.

Columns exactly as the spec's Data model table. The three measurement columns carry no DEFAULT:

CREATE TABLE IF NOT EXISTS task_session_subagents (
    id                  TEXT PRIMARY KEY,
    task_session_id     TEXT NOT NULL,
    task_id             TEXT NOT NULL,
    turn_id             TEXT,
    tool_call_id        TEXT NOT NULL,
    parent_tool_call_id TEXT,
    subagent_type       TEXT,
    description         TEXT,
    agent_id            TEXT,
    child_session_id    TEXT,
    model               TEXT,
    agent_status        TEXT,
    tool_status         TEXT,
    is_async            INTEGER NOT NULL DEFAULT 0,
    total_tokens        INTEGER,
    tool_use_count      INTEGER,
    duration_ms         INTEGER,
    source              TEXT NOT NULL DEFAULT 'live',
    observed_at         TIMESTAMP NOT NULL,
    settled_at          TIMESTAMP,
    updated_at          TIMESTAMP NOT NULL,
    UNIQUE (task_session_id, tool_call_id),
    FOREIGN KEY (task_session_id) REFERENCES task_sessions(id) ON DELETE CASCADE
);

CREATE INDEX IF NOT EXISTS idx_subagents_session_id ON task_session_subagents(task_session_id);
CREATE INDEX IF NOT EXISTS idx_subagents_task_id    ON task_session_subagents(task_id);
CREATE INDEX IF NOT EXISTS idx_subagents_turn_id    ON task_session_subagents(turn_id);

No FOREIGN KEY on turn_id — deliberate, per the spec: a turn deletion must not delete the fan-out record.

Migration: backfill and activation keys (task 01)

apps/backend/internal/task/repository/sqlite/base_migrations.go, a new migrateSubagentContextBackfill() called from runMigrations() after the existing statements. Three r.migrate.Apply(...) calls, all argument-free SQL (MigrateLogger.Apply takes no bind args, and its swallow-plus-WARN contract at internal/db/migratelog.go:33 is what AC-20 is written against):

1. task_session_subagents.backfill — one unbounded INSERT INTO task_session_subagents (...) SELECT ... FROM task_session_messages m WHERE <kind> = 'subagent_task' AND ... ON CONFLICT (task_session_id, tool_call_id) DO NOTHING. No LIMIT, no batching (AC-23a).
2. kandev_meta.subagent_context_capture_since — write-once INSERT ... ON CONFLICT(key) DO NOTHING with an fmt.Sprintf-inlined time.Now().UTC().Format(time.RFC3339) literal.
3. kandev_meta.subagent_context_backfill_through — write-once, value from a subselect COALESCE(MAX(m.created_at), '') over the same predicate as (1).

kandev_meta is guaranteed to exist: persistence.Provide calls ensureMetaTable (internal/persistence/provider.go:72) before repository.Provide builds the task repo (internal/backendapp/storage.go:41).

Backfill derivations, all via dialect-aware helpers in the same file:

Target	Source	Notes
id	m.id	The message's own UUID. Deterministic, so a replay maps to the same row and DO NOTHING is a true no-op.
task_session_id, task_id	m.task_session_id, m.task_id	Rows with task_id = '' are excluded by the WHERE (AC-2).
turn_id	NULLIF(m.turn_id, '')
tool_call_id	metadata.tool_call_id	Excluded by WHERE when empty (AC-2).
parent_tool_call_id	metadata.parent_tool_call_id	Top-level metadata key, not inside normalized.
agent_status	metadata.normalized.subagent_task.status
tool_status	metadata.status	The ACP status the message row already stores.
observed_at	m.created_at
settled_at	m.updated_at when the derived tool_status is terminal, else NULL	Terminal set mirrors isTerminalToolStatus.
source	literal 'backfill'

Three new helpers beside jsonColumn / jsonText (base_migrations.go:319-335), each reusing jsonColumn's empty/'null' normalization so one malformed metadata document cannot abort the statement (AC-23b):

• jsonKey(postgres, column, key string) string — a top-level text extract (#>> '{key}' / json_extract(..., '$.key')), needed for tool_call_id, parent_tool_call_id, status, and normalized.kind.
• jsonInt(postgres, column, parent, key string) string — CAST of the text extract to BIGINT (Postgres) / INTEGER (SQLite), wrapped so an empty extract and a negative value both yield NULL (AC-9, AC-23).
• jsonBoolToInt(postgres, column, parent, key string) string — 1 / 0. Postgres #>> yields 'true'; SQLite json_extract yields 1. The helper must accept both spellings, which is the whole reason it exists.

total_tokens and duration_ms are omitempty int64 on the payload, so a reported 0 never reaches the JSON and absent-means-NULL falls out for free. tool_use_count is *int with omitempty, so it is present in the JSON exactly when the agent reported it — including a genuine 0 (AC-8).

Model and repository upsert (task 02)

apps/backend/internal/task/models/subagent_context.go:

type SubagentContext struct {
    ID, TaskSessionID, TaskID, ToolCallID string
    TurnID, ParentToolCallID, SubagentType, Description *string
    AgentID, ChildSessionID, Model, AgentStatus, ToolStatus *string
    IsAsync       bool
    TotalTokens   *int64
    ToolUseCount  *int64
    DurationMs    *int64
    Source        string
    ObservedAt    time.Time
    SettledAt     *time.Time
    UpdatedAt     time.Time
}

Every nullable column is a pointer. This is not style: ToolUseCount must distinguish a reported 0 from "not reported", and a non-pointer int64 cannot.

apps/backend/internal/task/repository/sqlite/subagent_context.go:

• UpsertSubagentContext(ctx context.Context, sc *models.SubagentContext) error — a single INSERT ... ON CONFLICT (task_session_id, tool_call_id) DO UPDATE SET .... No read-then-write anywhere (AC-14). Every merge rule lives in the conflict clause:

  Column	Conflict expression	AC
  turn_id	COALESCE(task_session_subagents.turn_id, excluded.turn_id)	AC-2a — the original turn wins; a later turn never re-attributes
  text fields	COALESCE(excluded.<c>, task_session_subagents.<c>)	AC-4, AC-5 — fill-forward, never blank
  tool_status	CASE WHEN task_session_subagents.settled_at IS NOT NULL THEN task_session_subagents.tool_status ELSE COALESCE(excluded.tool_status, ...) END	AC-12 — frozen at the terminal value
  is_async	CASE WHEN excluded.is_async = 1 THEN 1 ELSE task_session_subagents.is_async END	sticky, never reset to 0
  metrics	COALESCE(excluded.<c>, task_session_subagents.<c>)	AC-4, AC-7
  settled_at	COALESCE(task_session_subagents.settled_at, excluded.settled_at)	AC-11 — write-once
  updated_at	excluded.updated_at
  source, observed_at, id	absent from the SET list	write-once: a backfilled row is never relabelled live (AC-22), and first observation wins (AC-1a)

  Both dialects accept excluded. on the right and unqualified names on the left; the self-qualified task_session_subagents.<c> reads are valid on both. Statement built once as a package-level const and passed through r.db.Rebind, matching review.go:27 and document.go:291.

• ListSubagentContextsBySession(ctx, sessionID string) ([]*models.SubagentContext, error) and ListSubagentContextsByTurn(ctx, turnID string) — both ORDER BY observed_at ASC, tool_call_id ASC (AC-13). These are the read seam AC-13 is written against and the seam the tests assert ordering through. They are repository-level only: no service method, no DTO, no route. The spec's Out of scope forbids a read API, not a repository accessor.

Service writer and counters (task 03)

apps/backend/internal/task/service/subagent_context.go, one entry point:

type RecordSubagentContextRequest struct {
    TaskSessionID, TaskID, TurnID string
    ToolCallID, ParentToolCallID  string
    ToolStatus                    string
    Payload                       *streams.SubagentTaskPayload
    ObservedAt                    time.Time
}

func (s *Service) RecordSubagentContext(ctx context.Context, req RecordSubagentContextRequest)

It returns nothing. That is the mechanism by which AC-27 holds: a writer with no error return cannot fail the enclosing message write, turn, or stream, and no future caller can accidentally propagate one. It owns:

• Identity gate — empty TaskSessionID, TaskID, or ToolCallID ⇒ no write, skipped_no_identity++, return (AC-2).
• nilIfEmpty(string) *string applied to every nullable TEXT field, so "" becomes NULL and COUNT(model) counts models (spec § Column-level normalization rules).
• Metric normalization: DurationMs/TotalTokens map to *int64 and are left nil when zero-or-absent; ToolUseCount maps to *int64 only when the payload pointer is non-nil, preserving a reported 0 (AC-7, AC-8). Any negative value ⇒ that field nil and anomalous_value++, other fields unaffected (AC-9).
• Terminality: settled_at = &req.ObservedAt only when orchestrator-equivalent terminal classification of req.ToolStatus is true. agent_status is stored verbatim and never consulted (AC-10a, AC-11). The classifier is unexported in internal/orchestrator, so the service gets its own isTerminalToolStatus over the identical set (complete, completed, success, error, failed, cancelled) with a comment naming event_handlers_streaming.go:762 as the peer, plus a test that pins the two lists together.
• Failure handling: repository error ⇒ WARN with session id and tool call id, failed++, return (AC-27).
• attempted++ on entry past the nil-payload check; persisted++ on success.

apps/backend/internal/task/service/subagent_context_metrics.go — one expvar.NewMap("subagent_context_total") with the five keys AC-26 names (attempted, persisted, skipped_no_identity, anomalous_value, failed), following internal/office/scheduler/metrics_vars.go (package-level NewMap, counters only, no gauges).

Orchestrator wiring (task 04)

apps/backend/internal/orchestrator/service.go — a narrow interface beside MessageCreator:

type SubagentContextRecorder interface {
    RecordSubagentContext(ctx context.Context, req taskservice.RecordSubagentContextRequest)
}

held as an optional subagentContexts field on Service, nil-checked at both call sites exactly as messageCreator is.

Two call sites in event_handlers_streaming.go, each already holding every input:

• handleToolCallEvent (~line 312) — after the CreateToolCallMessage block, using s.getActiveTurnID(payload.SessionID).
• persistToolUpdateMessage (~line 565) — after UpdateToolCallMessage, using the turnID that function already resolved (peekActiveTurnID for terminal frames, getActiveTurnID otherwise).

Both guarded by payload.Data.Normalized.Kind() == streams.ToolKindSubagentTask and a non-nil Normalized.SubagentTask(). A new recordSubagentContextFromFrame(ctx, payload, turnID) helper holds the guard and the request build so neither handler grows past the package's funlen limit of 80 lines / 50 statements.

payload.SessionID is the Kandev task session id: messageCreatorAdapter passes the same value straight into CreateMessageRequest.TaskSessionID (internal/backendapp/adapters.go:879), despite the agentSessionID parameter name.

apps/backend/internal/backendapp/adapters.go — a subagentContextAdapter over *taskservice.Service, wired where messageCreatorAdapter is wired.

---

Frontend

None. The spec's What this feature is, and is not rules out any frontend surface: the subagent card keeps rendering from message metadata, unchanged. No DTO field, no WebSocket event, no store slice.

---

Tests

Migration and backfill — subagent_context_migration_test.go (task 01)

Patterned on task_external_id_migration_test.go.

• Fresh DB creates the table, indexes, and UNIQUE constraint. File: apps/backend/internal/task/repository/sqlite/subagent_context_migration_test.go. How: NewWithDB over a temp SQLite file, then assert idx_subagents_session_id, idx_subagents_task_id, idx_subagents_turn_id exist in sqlite_master and that a duplicate (task_session_id, tool_call_id) insert is rejected. (AC-17)
• Replay is a no-op. How: repo.runMigrations() twice; assert no error and an unchanged row count. (AC-18)
• Backfill inserts one row per subagent message. How: seed a session, a turn, and message rows whose metadata.normalized.kind = 'subagent_task', then runMigrations(); assert one row each with source = 'backfill' and observed_at = m.created_at. (AC-21)
• async_launched shape stores NULL, not 0 — the spec's named must-not-get-wrong regression. How: seed a message whose subagent_task object has status: "async_launched" and no total_tokens / tool_use_count / duration_ms; assert all three columns IS NULL and that none is 0. (AC-7)
• A reported tool_use_count of 0 survives as 0. (AC-8)
• A negative reported metric backfills as NULL. (AC-9, AC-23)
• Empty JSON strings become NULL. How: a payload with model: ""; assert model IS NULL, not ''.
• Malformed metadata does not abort the statement. How: seed one message row with metadata = '' and another with metadata = 'null' alongside a valid subagent row; assert the valid row still lands. (AC-23b)
• A live row wins over the backfill. How: pre-insert a source = 'live' row for a (session, tool_call_id) that also exists as a message, run runMigrations(), assert source is still 'live' and there is exactly one row. (AC-22)
• Rows with no task_id or no tool_call_id are not backfilled. (AC-2)
• Activation keys are written once. How: assert both kandev_meta keys are present and RFC3339-parseable (backfill_through may be ''), capture their values, runMigrations() again, assert both unchanged. (AC-24)
• Fresh DB with no subagent messages sets backfill_through to ''. (AC-24)

Repository upsert — subagent_context_test.go (task 02)

File: apps/backend/internal/task/repository/sqlite/subagent_context_test.go.

• First upsert inserts. (AC-1)
• Second upsert for the same key updates in place, one row. (AC-3)
• Fill-forward: a frame with empty fields does not blank learned values. Table-driven over every nullable column. (AC-4)
• A non-empty value replaces the stored value. (AC-5)
• settled_at is write-once: a second terminal frame with a later timestamp leaves the first value. (AC-11)
• A non-terminal frame after settling leaves settled_at and tool_status alone but still fills NULL columns. (AC-12)
• Never-terminal row exists with settled_at IS NULL. (AC-10)
• turn_id from the first frame survives a later frame carrying a different turn. (AC-2a)
• is_async is sticky across a later frame reporting false.
• source and observed_at are not overwritten by a later upsert.
• Nested parent_tool_call_id is stored and the row is not suppressed. (AC-6)
• Concurrent upserts of the same key — N goroutines, errgroup; assert exactly one row and that the invariants above hold regardless of commit order. (AC-14)
• Concurrent upserts of different keys in one turn — the observed 3-, 4-, 5- and 8-way fan-outs; assert every row lands. (AC-15)
• Deleting the parent session cascades the rows away. (AC-16) Also add task_session_subagents to assertNoWorkspaceCascadeDependents in workspace_test.go:317, so the workspace cascade test covers the new table.
• Read ordering is observed_at ASC, tool_call_id ASC — seed rows sharing an observed_at and assert the tool_call_id tiebreak, per AC-13's explicit rejection of id as a tiebreak.

Service writer — subagent_context_test.go (task 03)

File: apps/backend/internal/task/service/subagent_context_test.go.

• Table-driven identity gate: each of empty session id, empty task id, empty tool call id ⇒ no repository call, skipped_no_identity incremented by one. (AC-2)
• Table-driven normalization: "" ⇒ NULL for every nullable TEXT column; absent metric ⇒ nil; reported 0 ToolUseCount ⇒ *int64(0); negative ⇒ nil plus anomalous_value++ with the other fields of the same frame intact. (AC-7, AC-8, AC-9)
• Terminal tool_status sets settled_at; agent_status = "async_launched" alone does not. (AC-10a, AC-11)
• An unrecognised agent_status is stored verbatim and does not settle. (AC-10a)
• A repository error logs WARN, increments failed, and the call still returns normally. How: a stub repo returning an error plus zaptest/observed-logs. (AC-27)
• attempted and persisted move as specified on the happy path. (AC-26)
• The service's terminal-status set equals the orchestrator's — pin both lists so a future edit to one breaks the test. (AC-11)

Orchestrator integration — event_handlers_streaming_subagent_test.go (task 04)

File: apps/backend/internal/orchestrator/event_handlers_streaming_subagent_test.go.

• handleToolCallEvent with a subagent_task payload records a context with the active turn id. (AC-1)
• handleToolUpdateEvent for the same tool call records again so the merge path runs; assert one recorded request per frame with the right turn id. (AC-3)
• A terminal update carries the terminal tool_status through. (AC-11)
• A non-subagent_task normalized kind records nothing.
• A nil subagentContexts field is safe — no panic, message write still happens. (AC-27)
• A nested frame (ParentToolCallID set) is recorded, not dropped. (AC-6)
• Integration through the real service and repository, using newServiceBackedMessageCreator(repo) as event_handlers_streaming_test.go:906 does, so one test exercises handler → service → repo end to end.

Postgres parity and the health query (task 05)

• Fresh Postgres schema plus migration replay, including the backfill and both activation keys. File: apps/backend/internal/task/repository/sqlite/subagent_context_postgres_test.go, gated on KANDEV_TEST_POSTGRES_DSN via testutil.OpenIsolatedPostgres. (AC-19)
• The upsert's conflict clause behaves identically on Postgres — fill-forward, settled_at write-once, is_async sticky, turn_id preserved. This is the ADR-0027 requirement that schema replay alone is insufficient for a dialect-sensitive method. (AC-19)
• The backfill's JSON helpers work on jsonb, including the ''/'null' metadata rows and the boolean spelling difference. (AC-23b)
• AC-28 health query on a seeded store: assert the task_session_messages subagent_task count equals COUNT(*) FROM task_session_subagents after live writes, and that seeding a message the writer never saw makes the two diverge. File: apps/backend/internal/task/repository/sqlite/subagent_context_health_test.go. (AC-28, AC-29)

---

E2E Tests

None. The spec's Out of scope states no UI surface changes, therefore "no E2E coverage is implied". Adding one would test nothing this feature changes.

---

Verification Results

Pending. On completion, synchronize this section with each task's ## Results: record exact commands and outcomes/counts, generated artifact paths, and cleanup/teardown evidence.

---

Implementation Waves And Parallel Candidates

Every task shares the internal/task subtree and each depends on the previous one's schema or seam, so the chain is sequential. Nothing here is parallel-safe: tasks 01 and 02 both edit the same package and task 01 owns shared schema DDL.

Wave 1:
- [ ] [task-01-schema-and-backfill](task-01-schema-and-backfill.md)

Wave 2:
- [ ] [task-02-repository-upsert](task-02-repository-upsert.md)

Wave 3:
- [ ] [task-03-service-writer](task-03-service-writer.md)

Wave 4:
- [ ] [task-04-orchestrator-wiring](task-04-orchestrator-wiring.md)

Wave 5:
- [ ] [task-05-postgres-and-health](task-05-postgres-and-health.md)

Open Questions

None. Two spec ambiguities were resolved against repository convention rather than left open, and both are recorded above so a builder does not re-litigate them:

1. "Schema-init DDL and an idempotent migration" for a new table would be a duplicated CREATE TABLE. apps/backend/AGENTS.md settles it: schema init runs before runMigrations() on every boot, so the DDL alone satisfies AC-17. See Schema (task 01).
2. AC-13 mandates a read ordering while Out of scope forbids a read API. Resolved as two repository-level list methods and no service, DTO, or route. See Model and repository upsert (task 02).