22 KiB
status, system, requirements, created, updated, owners
| status | system | requirements | created | updated | owners | ||
|---|---|---|---|---|---|---|---|
| draft | agents |
|
2026-08-12 | 2026-08-14 |
|
Subagent context persistence System Design Part 6
Purpose and boundaries
This design preserves the technical source detail for REQ-AGENTS-SUBAGENT-CONTEXT-PERSISTENCE-001 during migration.
Requirement mapping
| Requirement | Design section |
|---|---|
REQ-AGENTS-SUBAGENT-CONTEXT-PERSISTENCE-001 |
Migrated source detail |
Migrated source detail
Writer health
AC-26 (amended by Amendment 1; unit defined in revision 1) — The system
SHALL expose counters, under the existing expvar convention used by routing_*
and subproc_*, for: attempted, persisted, skipped_no_identity,
anomalous_value, failed, and unknown_execution (AC-31).
The counting unit is one observed frame, not one row and not one SQL
statement. Every recognized subagent_task frame that reaches the writer
increments exactly one of attempted's outcomes, and a frame that merges onto an
existing row increments persisted exactly as a frame that creates one does.
Stated because the unit is otherwise ambiguous and the two readings differ by
roughly the fan-out's frame count:
attempted= frames that reached the writer carrying a recognized payload.skipped_no_identity= of those, the ones rejected by AC-2 before any SQL ran.persisted= of those, the ones whose upsert committed. This counts writes, not rows, sopersistedis expected to exceedCOUNT(*)on the table by a wide margin (each subagent is typically observed on several frames), and the two SHALL NOT be reconciled against each other. AC-28's row-count comparison is the reconciliation; the counters are not.failed= of those, the ones whose upsert returned an error (AC-27).anomalous_value= frames carrying at least one negative metric (AC-9). A qualifier, not an outcome. (revision 2) It is counted only for frames that reached the write, i.e. not for frames AC-2 skipped: AC-2 rejects on identity before any metric is examined, so a skipped frame incrementsskipped_no_identityand nothing else, whatever its payload contained.unknown_execution= live frames written with the'unknown'sentinel (AC-31). A qualifier, not an outcome.
So attempted = skipped_no_identity + persisted + failed, exactly, while
anomalous_value and unknown_execution overlap the others freely.
AC-27 — WHEN a persist attempt fails, THEN the system SHALL log at WARN
with the session id and tool-call id, SHALL increment the failed counter, and
SHALL NOT fail the enclosing message write, the turn, or the agent stream.
Telemetry never breaks the product path.
AC-28 — The expected-versus-observed health check SHALL be answerable from the database alone, with no counter access, as:
(queries corrected in revision 1 — they now carry the activation predicate the prose always required, and the PostgreSQL form)
(revision 3) Which formulation is normative. This criterion carries three renderings of the same idea, accumulated across revisions, and only one of them is the contract. The two directed anti-joins in Two directions, never netted are normative — those are the queries to implement, test and alarm on. The two
COUNTpairs immediately below establish what is being compared (distinct(session, tool_call_id)keys on each side, scoped by:since) and are retained as the explanation; the filtered expected count later in this criterion shows which messages qualify, and its predicate is carried verbatim into the shortfall anti-join. A builder implementing theCOUNTpair as the check has implemented a signed difference, which revision 3 exists to replace — see the banking argument below. Stated explicitly because a criterion that shows three queries and marks none of them authoritative is exactly the defect the stale Failure modes row was.(revision 4) Two consequences of that hierarchy. First, only the normative anti-joins carry the
msgCTE; the explanatory queries below are written unwrapped for readability. That is a presentation choice, not an exemption — any query from this criterion that is actually EXECUTED SHALL normalizemetadatathe way the anti-joins do, because a''metadata row raises on both dialects (see Two directions, never netted). Second, what Build ships for this criterion is the two anti-joins as executable queries exercised by tests on both dialects — that is the surface AC-28a's "SHALL NOT be run" branch and AC-24c's key-state assertions act on. No production endpoint, scheduled check or alerting integration is in scope; wiring a monitor to the shortfall query is a deployment concern, and § Out of scope already excludes a read API.
-- SQLite. :since is kandev_meta.subagent_context_capture_since.
-- Both sides count DISTINCT (session, tool_call_id) keys (revision 2).
SELECT COUNT(DISTINCT task_session_id || ':' || json_extract(metadata,'$.tool_call_id'))
FROM task_session_messages
WHERE json_extract(metadata,'$.normalized.kind') = 'subagent_task'
AND created_at >= :since;
-- versus (revision 2: DISTINCT on the execution-agnostic key, see below)
SELECT COUNT(DISTINCT task_session_id || ':' || tool_call_id)
FROM task_session_subagents
WHERE observed_at >= :since;
-- PostgreSQL: same two counts, dialect-appropriate JSON extraction.
SELECT COUNT(DISTINCT task_session_id || ':' || (metadata::jsonb #>> '{tool_call_id}'))
FROM task_session_messages
WHERE metadata::jsonb #>> '{normalized,kind}' = 'subagent_task'
AND created_at >= :since;
-- versus (revision 2: DISTINCT on the execution-agnostic key, see below)
SELECT COUNT(DISTINCT task_session_id || ':' || tool_call_id)
FROM task_session_subagents
WHERE observed_at >= :since;
The :since predicate is part of the contract, not decoration. Without it the
comparison spans history the backfill could not see — sessions deleted before the
upgrade left no message row and no subagent row, but sessions deleted between
the two counts, and messages whose sessions cascaded away, skew the sides
differently — so an unscoped comparison drifts permanently and reads as writer
failure. AC-19's dialect-parity expectation applies to this check as much as to
the migration: json_extract does not exist on PostgreSQL, so a SQLite-only
health query is a health query that cannot run on half the supported deployments.
(amended by Amendment 1; comparison corrected in revision 2; direction split in
revision 3) These two counts SHALL agree for activity at or after
subagent_context_capture_since, up to the named allowances below.
(revision 3) The two directions SHALL be measured separately, as directed set differences, and SHALL NOT be reduced to one signed number. The actionable signal is the SHORTFALL set — expected keys with no matching row — and it is the only one that alarms. The EXCESS set is attributed and does not alarm. See Two directions, never netted below for the queries and the reason.
Why the observed side counts DISTINCT keys rather than rows. Tool-call
messages are looked up by (session_id, tool_call_id) only
(getToolCallMessageWithRetry, internal/task/service/service_messages.go:483), so
a cross-execution collision updates one message row, while this table correctly holds
one row per execution (AC-32). Counting rows therefore makes the two sides
structurally incomparable, and Amendment 1 is what introduced the asymmetry.
Revision 1 handled that by relaxing the check to
COUNT(task_session_subagents) >= COUNT(messages) and treating only a shortfall as
evidence of a stopped writer. (revision 2) That relaxation was unsound and is
replaced. Every cross-execution collision banks one row of permanent headroom, and
headroom absorbs later missed writes: with N banked rows the writer can stop dead and
the comparison stays "healthy" for the next N subagent messages. AC-29 promises the
comparison diverges when the writer stops while message writes continue, and under
the >= rule that promise was false in exactly the configuration Amendment 1 exists
to create — the hole growing in proportion to how often the defect being recorded
actually occurs.
Collapsing the observed side to COUNT(DISTINCT task_session_id || ':' || tool_call_id) removes the excess from the comparison at the source: one subagent
counts once however many executions recorded it, which is exactly one per message
row. Equality is restored as the expectation, a shortfall regains its meaning, and
AC-29 becomes true again. The ':' separator is not required for correctness — both
operands are fixed-length UUIDs and cannot collide — but it costs nothing and removes
the question permanently.
Both sides count distinct keys, not rows, and the symmetry is required. The
message side can over-count for the mirror-image reason: nothing constrains
task_session_messages to be unique on (task_session_id, tool_call_id) (AC-21b),
so two messages can describe one subagent. If only the observed side were collapsed,
every such duplicate would present as a permanent one-row shortfall on a perfectly
healthy writer — the same false-alarm failure that got the >= relaxation introduced
in the first place. Counting distinct keys on both sides makes the comparison
symmetric: each side answers "how many distinct subagents does this table know
about", and those are the numbers that must agree.
Two directions, never netted (revision 3)
Revision 2 restored equality as the expectation but named allowances only on the
shortfall side, leaving what a persistent excess means undefined. It is not an
academic state: AC-27 deliberately keeps the subagent write independent of the message
write, so a failed message write with a successful upsert leaves a permanent +1; and
AC-1a puts recognition on a later tool_call_update, so a message created just before
capture_since whose recognizing frame lands just after it counts on one side only.
Under a bare "SHALL agree" such an installation reads unhealthy forever, and a check that
fires on correct behaviour gets muted — the exact failure the >= relaxation was
reaching for, arriving from the other direction.
The fix is not to tolerate excess. Tolerating it as slack in a single signed difference would reintroduce SR19's banking hole verbatim: with N banked excess keys the writer could stop dead and the difference would read zero for the next N subagents. Instead the comparison is split, so neither direction can cancel the other:
-- SQLite. :since is kandev_meta.subagent_context_capture_since, bound as a timestamp
-- (AC-24e). The `msg` CTE is AC-23b's normalization: it maps NULL / '' / 'null' metadata
-- to '{}' so one legacy row cannot abort the scan (revision 4). The CTE is byte-identical
-- on PostgreSQL; only the extraction changes (AC-19).
WITH msg AS (
SELECT m.task_session_id, m.task_id, m.created_at,
(CASE WHEN m.metadata IS NULL OR m.metadata = '' OR m.metadata = 'null'
THEN '{}' ELSE m.metadata END) AS meta
FROM task_session_messages m
)
-- SHORTFALL — expected keys with no subagent row. THIS is the alarm.
SELECT COUNT(*) FROM (
SELECT msg.task_session_id, json_extract(msg.meta,'$.tool_call_id') AS tool_call_id
FROM msg
WHERE json_extract(msg.meta,'$.normalized.kind') = 'subagent_task'
AND msg.created_at >= :since
AND COALESCE(msg.task_session_id,'') <> ''
AND COALESCE(msg.task_id,'') <> ''
AND COALESCE(json_extract(msg.meta,'$.tool_call_id'),'') <> ''
GROUP BY 1, 2
EXCEPT
SELECT s.task_session_id, s.tool_call_id
FROM task_session_subagents s
WHERE s.observed_at >= :since
) AS shortfall;
-- EXCESS — subagent keys with no accounting message. Attributed, does NOT alarm.
WITH msg AS (
SELECT m.task_session_id, m.created_at,
(CASE WHEN m.metadata IS NULL OR m.metadata = '' OR m.metadata = 'null'
THEN '{}' ELSE m.metadata END) AS meta
FROM task_session_messages m
)
SELECT COUNT(*) FROM (
SELECT s.task_session_id, s.tool_call_id
FROM task_session_subagents s
WHERE s.observed_at >= :since
GROUP BY 1, 2
EXCEPT
SELECT msg.task_session_id, json_extract(msg.meta,'$.tool_call_id')
FROM msg
WHERE json_extract(msg.meta,'$.normalized.kind') = 'subagent_task'
AND msg.created_at >= :since
) AS excess;
EXCEPT is set-based and deduplicates on both dialects, so this subsumes revision 2's
distinct-key collapse rather than replacing it: a cross-execution collision (AC-32)
contributes one key however many rows recorded it, and two messages describing one
subagent (AC-21b) contribute one key however many messages exist. The PostgreSQL form is
identical with msg.meta::jsonb #>> '{tool_call_id}' and #>> '{normalized,kind}'
substituted for json_extract, per AC-19.
(revision 4) Three things about the SQL above are load-bearing and were wrong or unstated before.
The derived tables are ALIASED, and they have to be. PostgreSQL rejects a subquery in
FROM without an alias, so revision 3's SELECT COUNT(*) FROM ( … EXCEPT … ); did not
parse on PostgreSQL at all — while the surrounding prose claimed the PostgreSQL form was
"identical with the extraction substituted", which was therefore false of the normative
queries themselves. AS shortfall / AS excess are accepted by both dialects, so the
identical-modulo-extraction claim above is now true rather than aspirational.
Every metadata reference goes through the msg CTE, for exactly AC-23b's reason. The
repository writes NULL, '' and 'null' for "no metadata" — that is what jsonColumn
(base_migrations.go) exists to absorb — and an unwrapped extraction raises on such a row:
json_extract('') is malformed JSON on SQLite and ''::jsonb errors on PostgreSQL. A
single legacy row would therefore disable the health check on either dialect. That is a
worse failure than any it detects, because the check whose job is noticing that this writer
stopped would itself stop, silently, in the same direction. The CTE is the same
normalization AC-23b already requires of the backfill, applied to the same rows.
The excess direction's message half deliberately does NOT carry the three identity
predicates the shortfall half carries. This asymmetry is intentional, not an oversight.
The shortfall side asks "which messages should have produced a row", so it must exclude
exactly what AC-2 and AC-21a exclude. The excess side asks the opposite question — "is
there any message that accounts for this row" — so it must subtract the widest message
set available. Filtering it would report excess for a subagent row whose message exists but
whose own task_id column is empty for an unrelated reason, which is a real row correctly
written from a frame that did carry a task id. A builder SHALL NOT mirror the filter onto
the excess side for symmetry's sake.
A non-zero SHORTFALL is evidence of a stopped or failing writer and is the check. A non-zero EXCESS is expected on any installation that has taken a message-write failure or straddled the activation boundary; it SHALL be reported separately for attribution and SHALL NOT be raised as writer failure, and it can never mask a shortfall because it is counted in the other direction. Cross-execution collisions no longer appear in either direction at all, which is what makes AC-29's guarantee unconditional.
Where a deployment wants the collision count itself — rows minus distinct subagents, the quantity revision 2 published — it remains available and is orthogonal to both directions above:
-- Cross-execution collisions (AC-32). Expected > 0 wherever a tool_call_id was reused.
SELECT COUNT(*) - COUNT(DISTINCT task_session_id || ':' || tool_call_id)
FROM task_session_subagents
WHERE observed_at >= :since;
AC-28a (added by revision 2) — WHEN subagent_context_capture_since is
absent, THEN the AC-28 comparison SHALL NOT be run, and the key's absence SHALL
itself be the health signal, read per AC-24c.
This branch has to exist because revision 1 created the state and left the query
undefined in it. AC-20 promises that a failed migration is made detectable by the
health signals; AC-24a requires capture_since to be absent after a failed
backfill; and AC-28's own prose disowns the unscoped comparison as one that "drifts
permanently and reads as writer failure." So in precisely the failure state AC-20
points at, the prescribed query had no value to bind to :since and either could not
run or had to fall back to the form the spec rejects. The resolution is that the
detection in that state is the missing key, not a shortfall — which is a stronger
signal anyway, since it is present from the first boot after the failure rather than
accumulating.
AC-28b (added by revision 2) — WHEN a shortfall is observed, THEN it SHALL be confirmed by a second evaluation before being treated as evidence of a failing writer, OR both counts SHALL be taken within a single read transaction.
AC-28's claim that a shortfall means a stopped writer "and nothing else" is not true of two independently-executed statements against a live store: rows and messages are written continuously and a session cascade-deleting between the two counts skews them differently, so a healthy writer can present a transient shortfall.
(revision 3) The anti-join form largely retires this hazard rather than merely mitigating it: each direction is now a single statement, so both of its sides are read under one snapshot and the two-statement skew this criterion was written against cannot arise within a direction. The confirm-on-second-evaluation rule is retained for the residual case — a session cascade-deleting mid-statement, and any deployment whose monitor still evaluates the two directions separately — but a builder using the queries as written satisfies it by construction with the single-statement reading. A check that fires on correct behaviour gets muted, and a muted check is the same as no check — which is the outcome this feature exists to prevent.
(revision 1) The shortfall side needs its own allowance, for the mirror-image reason. A frame carrying a session id but no task id writes its message and is then correctly skipped by AC-2, so it produces a permanent, benign shortfall that is indistinguishable from a dead writer if the expected count is taken naively. The expected count SHALL therefore exclude what AC-2 and AC-21a exclude, which is answerable from the database alone:
-- SQLite: expected count, AC-2 skips removed. Same shape on PostgreSQL with
-- metadata::jsonb #>> '{...}' substituted for json_extract.
SELECT COUNT(DISTINCT task_session_id || ':' || json_extract(metadata,'$.tool_call_id'))
FROM task_session_messages
WHERE json_extract(metadata,'$.normalized.kind') = 'subagent_task'
AND created_at >= :since
AND COALESCE(task_session_id,'') <> ''
AND COALESCE(task_id,'') <> ''
AND COALESCE(json_extract(metadata,'$.tool_call_id'),'') <> '';
With that expected count, a shortfall is evidence of a stopped or failing writer and nothing else, which is what makes the check actionable. A health check that fires on correct behaviour gets muted, and a muted check is the same as no check — which is the outcome this feature exists to prevent.
The message count remains a valid independent expectation because the message write is load-bearing for the UI: a broken message write is noticed by a human within one turn, whereas a broken context write is noticed by nobody. That asymmetry is what makes the comparison a real check rather than two readings of the same failure.
AC-29 (scope corrected in revision 2) — WHEN the writer has stopped while
message writes continue, THEN the AC-28 comparison SHALL diverge, and the divergence
SHALL be attributable in time via MAX(observed_at) on task_session_subagents. No
separate heartbeat row is written, precisely so that the health signal cannot itself
be the thing that silently stops.
(revision 2) This guarantee is made against AC-28's distinct-key count, and it
holds only because of that. Under revision 1's >= comparison it was false: banked
cross-execution excess absorbed the first N missed writes, so a stopped writer
produced no divergence at all until the backlog was consumed. The distinct-key count
does not accumulate excess, so the first missed subagent is the first divergence.
(revision 3) "Diverge" means the SHORTFALL direction becomes non-zero — the first anti-join in Two directions, never netted. Naming the direction is what makes this guarantee unconditional: because the shortfall is a directed set difference rather than a signed count, no amount of banked excess from any source (cross-execution collisions, a failed message write, an activation-boundary straddle) can offset it, so the first missed subagent is the first divergence on every installation rather than only on one with no excess. A monitor SHALL alarm on the shortfall query, not on the equality of two counts. This criterion is the reason AC-28's comparison changed, rather than the other way round — the requirement that a stopped writer be noticed is the fixed point, and the query is what had to move.
(revision 4) The attribution half is scoped, and its empty result has a defined
meaning. The MAX(observed_at) SHALL be taken over rows at or after
subagent_context_capture_since — the same :since and the same timestamp comparison the
shortfall query uses (AC-24e) — not over the whole table. Unscoped it returns the newest of
whatever exists, so on an installation whose only rows are backfilled or legacy it reports
an instant that never established live-writer health at all, and reads as a healthy writer.
WHEN that scoped MAX returns NULL — no row has been observed since activation — THEN
it SHALL be read as "the writer has produced nothing since this installation activated",
which is the strongest available divergence signal rather than an absence of information.
It SHALL NOT be reported as "no divergence", "unknown", or a healthy state. This is the
reachable case where the very first post-activation subagent is the one missed, and it is
exactly the case a naive MAX reports most reassuringly.