Files
kandev/docs/plans/clarification-response-reliability/plan.md
T
2026-09-06 18:52:30 +01:00

148 lines
7.6 KiB
Markdown

---
spec: docs/specs/tasks/requirements/clarification-response-reliability.md
related_specs:
- docs/specs/tasks/system-design/clarification-response-reliability.md
- docs/specs/tasks/requirements/clarification-active-lifecycle.md
created: 2026-09-05
status: implemented
---
# Implementation Plan: Clarification response reliability
## Overview
Make answer and Skip requests independent of unrelated transcript size. Bound
the complete browser wait and preserve a safe retry path. The change adds one
pending-ID-leading expression index for SQLite and PostgreSQL. It extends the
five-second claim budget to all pre-claim work. It maps that timeout to a
retryable 503. The shared web hook gets a 40-second deadline.
The plan preserves current-turn authority, detached-resume behavior, durable
delivery intents, and the existing idempotent loser reconstruction. It does not
add a clarification table, rewrite message rows, or expand PostgreSQL support
to multiple backend replicas.
## Confirmed cause
- The affected SQLite database held about 904,000 message rows and was 4.44 GB.
- `FindMessagesByPendingID` and the bundle claim filter by pending ID without a
session ID.
- `idx_messages_metadata_pending_id` is ordered by session ID before the JSON
pending-ID expression, so SQLite cannot use it for that query shape.
- The live query plan was a full `task_session_messages` scan. Submit and Skip
requests took 37 to 55 seconds and several ended with
`claim active clarification bundle: context deadline exceeded`.
- Only the atomic claim currently receives the five-second claim context. The
preceding identity lookup uses the unbounded request context.
- The web hook awaits `fetch` without an abort deadline, so a slow backend or
proxy leaves the controls in `submitting` indefinitely.
## Design decisions
### Persistence access path
- Add `idx_messages_metadata_pending_id_lookup_ordered` with extracted pending
ID first, followed by `created_at` and `id`, and restrict it to non-null
pending IDs. Retain the existing bare pending-ID lookup index for compatible
upgrades from the current main branch.
- Generate the expression and DDL through `internal/db/dialect`. Query and
index expressions must remain textually aligned for both database drivers.
- Retain the existing session-first index for session-scoped consumers.
- Use the current startup-critical `CREATE INDEX IF NOT EXISTS` path. A new
index name upgrades existing installations and is replay-safe.
- Use real SQLite and environment-gated PostgreSQL query-plan tests. PostgreSQL
remains under the documented one-replica, stop-the-writer upgrade contract.
this package does not introduce concurrent index management.
### Bounded server result
- Apply one fresh five-second context to identity resolution, authorization
inputs, validation reads, and the atomic claim.
- Convert exhaustion of that internal pre-claim budget into a typed resolver
error and HTTP 503 with code `temporarily_unavailable`.
- Preserve the existing post-claim 30-second delivery/recovery budget and all
durable compensation rules.
- Add low-cardinality timeout counters by phase and structured phase-duration
logs without placing pending IDs in metrics.
### Desktop and phone recovery
- Give `postClarification` its own 40-second `AbortController` deadline and
clear the timer on every path.
- Reuse the existing error banner and idempotent Retry action. Preserve answers
and release the in-flight guard after timeout, network failure, 503, or 5xx.
- Keep collapse/dismiss local and Skip as the explicit rejection. Both become
available again after a retryable failure.
- Share the same hook, state, and banner across desktop and phone. Preserve the
existing 44-pixel phone Retry target and add phone outcome coverage rather
than a separate mobile component.
## Work order
### Wave 1: Indexed lookup foundation
- [x] [Task 01: Index pending-ID bundle access](task-01-index-pending-id-bundle-access.md)
### Wave 2: Bounded backend contract
- [x] [Task 02: Bound clarification response resolution](task-02-bound-clarification-response-resolution.md) - depends on Task 01.
### Wave 3: Client recovery and viewport coverage
- [x] [Task 03: Bound and recover clarification submission](task-03-bound-and-recover-clarification-submission.md) - depends on Task 02.
Implementation remains sequential in the primary conversation unless the user
explicitly authorizes implementation sessions.
## Verification strategy
- SQLite schema and query-plan tests prove fresh creation, replay, upgrade, and
index use with substantial unrelated message history.
- Environment-gated real PostgreSQL tests prove the native expression index,
replay, pending-ID read eligibility, and the exact clarification claim
statement, including its malformed-bundle guard.
- Resolver and handler tests use controlled blocking repositories and fake
clocks/channels rather than wall-clock sleeps. They prove the internal
timeout classification, no-delivery boundary, 503 envelope, and unchanged
post-claim outcomes.
- Hook tests use fake timers and a non-resolving fetch to prove the 40-second
deadline, timer cleanup, answer preservation, mutex release, and safe retry.
- Desktop and Pixel 5 Playwright scenarios inject a retryable backend result,
prove that controls recover and complete the same saved answer after Retry.
## Risks and mitigations
- **Index build time and disk headroom:** A large existing message table may
lengthen the first startup. Keep the migration startup-critical, retain the
current operator backup/maintenance requirements, and prove the new partial
index against a representative large SQLite database before release.
- **PostgreSQL planner variation:** Do not assert a universal plan for tiny
tables. Seed enough unrelated rows, run `ANALYZE`, and prove the expression
index is eligible in the real PostgreSQL suite.
- **Client deadline drift:** Keep the browser deadline at 40 seconds. Pin the
browser and server constants in focused tests.
- **Duplicate delivery after an ambiguous response:** Reuse the existing durable
claim, delivery intent, and winner reconstruction. Do not add client-side
optimistic success for timeouts or 503 responses.
## Package validation commands
- `cd apps/backend && go test ./internal/db/dialect ./internal/task/repository/sqlite ./internal/clarification -count=1`
- `cd apps/backend && go test -race ./internal/clarification -count=1`
- `cd apps/backend && KANDEV_TEST_POSTGRES_DSN="$KANDEV_TEST_POSTGRES_DSN" go test ./internal/task/repository/sqlite ./internal/clarification -run 'Postgres.*Clarification|Clarification.*Postgres|PendingIDLookup' -count=1`
- `cd apps && pnpm --filter @kandev/web test -- --run hooks/domains/session/use-clarification-group.test.ts hooks/domains/session/use-clarification-group.timeout.test.ts hooks/domains/session/use-clarification-group.regressions.test.ts components/task/chat/clarification-input-overlay.test.tsx`
- `cd apps/web && pnpm run typecheck`
- `cd apps/web && pnpm run lint`
- `cd apps/web && pnpm run i18n:check && pnpm run i18n:ratchet`
- `cd apps/web && pnpm e2e:raw --project=chromium tests/chat/clarification-submit-failure.spec.ts`
- `cd apps/web && pnpm e2e:raw --project=mobile-chrome tests/chat/mobile-clarification.spec.ts`
## Results
Implemented across the SQLite/PostgreSQL persistence layer, clarification
resolver and HTTP handler, and shared desktop/mobile web hook. Verification
includes focused and race-enabled backend tests, frontend unit/type/lint/
localization checks, and the managed Chromium (2 passed) and Pixel 5 (8
passed) E2E scenarios. The PostgreSQL planner test is environment-gated and
was skipped because no `KANDEV_TEST_POSTGRES_DSN` was available.