12 KiB
created, updated, status, requirements, system_design, legacy_specs
| created | updated | status | requirements | system_design | legacy_specs | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| 2026-09-08 | 2026-09-09 | blocked |
|
|
Implementation Plan: Reduce Frontend Log Capture Latency
Overview
Review the available capture evidence before the implementation starts. The required active-profile evidence is not available, so Task 01 remains blocked. Task 02 is implemented as a bounded mitigation based on the available trace and isolated benchmark. It does not claim root-cause confirmation.
It then uploads one chronological page before it reads the next page. The backend supplies a relative duration for a monotonic local deadline.
Scope
In scope
- Phase timing for the frontend capture and backend upload handler.
- A fixed staging watermark and a one-second persistence-flush limit.
- Receipt-time memory fallback that does not stop normal persistence.
- Chronological paging through the existing IndexedDB timestamp index.
- A 128 KiB first page and the existing 800 KiB target for later pages.
- A relative capture duration and an abort signal for the active upload.
- Regression tests for ordering, bounds, clock skew, timeout, and fallback.
Out of scope
- An IndexedDB schema change or a new write index.
- Changes to console levels, producer frequency, retention, or staging limits.
- Continuous frontend telemetry or background upload retries.
- A longer backend collection window or a larger request-body limit.
- A backend lock refactor without timing evidence from Task 01.
- User-facing UI or copy changes.
Evidence and attribution gap
The backend sent both observed capture notifications immediately. The first frontend request reached the server 11.6 seconds later. The second request reached the server 14.8 seconds later.
The server then spent 24.8 seconds and 16.2 seconds in those requests. The HTTP timer includes request-body decoding. The handler decodes the body before it enters the diagnostic bundle service lock.
An isolated Chromium benchmark used 10,000 entries with approximately 20 MiB
of data. The current getAll, sort, and chunk preparation took 1.08 seconds on
the first read. Four later reads took 0.20 to 0.54 seconds.
The eager snapshot is a bounded cost, but this benchmark did not reproduce the reported delay. The current evidence does not separate these possible costs:
- The unbounded staging flush.
- IndexedDB contention in the active browser profile.
- Request-body serialization and transfer.
- Backend lock wait and file writes.
Task 01 remains blocked because the required phase tables, 90 percent timing accounting, request sizes and counts, and backend gate results are missing. The phrase "confirmed root cause" does not apply until this evidence is complete.
Requirement conformance
This package amends the existing platform requirements. It does not create a new capability owner.
- Console calls remain synchronous and do not wait for persistence or network work.
- A capture includes only entries that exist at notification receipt.
- Later console entries do not extend the capture flush.
- Identity isolation, chronological order, and all retention limits remain.
- The backend remains authoritative for job expiry.
- The relative duration removes dependence on the browser wall clock.
- A partial capture does not retry or write through the console interceptor.
No new ADR is necessary. The privacy, ownership, persistence, and transport boundaries remain the same. The added relative duration is backward compatible.
Technical approach
Phase measurement
Task 01 adds temporary, non-console timing around each capture phase. Browser
timing uses performance.now(). Backend timing uses the existing structured
logger. The task removes this temporary code after it records the results.
The frontend phase table includes these intervals:
- Notification receipt to flush start.
- The current persistence flush.
- The IndexedDB request.
- Snapshot filtering and ordering.
- Chunk preparation and request-body serialization.
- Each upload request.
The backend phase table includes request-body decoding, stream claim, service lock wait, validation, file writes, and response completion. Task 01 compares an active profile with an isolated maximum-size profile.
The frontend intervals do not overlap. Their total covers notification receipt through the fetch response. The backend intervals subdivide the server request and do not add to the frontend total.
The backend gate is 250 milliseconds and 10 percent of request time for lock wait and file writes. Task 01 has not applied this gate because its backend phase table is missing. No backend lock work order can be ruled out.
Fixed capture watermark
In apps/web/lib/logger/runtime.ts, assign a local sequence number to each
staged entry. Each drain request owns a fixed maximum sequence number. Entries
that arrive later start a later drain.
At notification receipt, save the current sequence number and a bounded memory
snapshot. Start a serialized IndexedDB boundary transaction before the
receipt-prefix drain. If the boundary cannot be proven, use the memory
snapshot. Join the active drain and persist only through that sequence number.
Wait at most one second. If the wait expires, use the saved memory snapshot and
let persistence continue. Report memory storage for this capture and include
flush_timeout: true in its final metadata.
Existing-index snapshot pages
In apps/web/lib/logger/indexeddb-store.ts, keep database version 2. Start the
receipt boundary on the already-open database connection. Pass its upper key
and the exact keys returned by receipt-prefix append batches to every page.
Read the existing timestamp_ms index with a continuation pair of timestamp
and primary key. Use continuePrimaryKey() for equal-timestamp continuation.
Filter the requested identity while the cursor scans the globally bounded
store.
Return one page and close its readonly transaction before the uploader asks for
the next page. Use stored prepared-byte counts for page limits. Do not use
getAll or sort the complete identity partition.
Start the fixed boundary transaction at receipt, before the prefix flush. Pass the combined boundary key to every page read so entries written after receipt cannot enter the capture between page transactions. Use receipt memory when the boundary transaction cannot be proven.
Upload budget and page sizes
In apps/backend/internal/system/logbundle/service.go, add
capture_timeout_ms to system.logs.capture_requested. Calculate it from the
backend clock when the notification is created. Keep capture_deadline for
backend authority and compatibility.
In apps/web/lib/logger/capture.ts, use performance.now() to create the local
deadline. Use the absolute deadline only when an older backend omits the
duration. Check the local deadline before each page and upload.
Use 128 KiB of entry data for the first page. Use the existing 800 KiB target
for later pages. This change adds at most one request to a maximum-size capture.
Use an AbortController to cancel the active request when the local budget
expires.
If no production caller remains, remove chunkEntries. Replace its direct test
with tests for the streaming capture behavior.
Tests
| Acceptance criterion | Test evidence |
|---|---|
AC-PLATFORM-DIAGNOSTIC-LOGGING-001.10 |
capture.test.ts holds the second page and proves that the first upload starts first. |
AC-PLATFORM-DIAGNOSTIC-LOGGING-001.11 |
Backend notification tests cover both deadline fields. Frontend fake-time tests prove monotonic behavior with wall-clock skew. |
AC-PLATFORM-BROWSER-CONSOLE-RETENTION-001.1 |
Existing store tests retain the three-day, 10,000-entry, and 20 MiB limits. |
AC-PLATFORM-BROWSER-CONSOLE-RETENTION-001.3 |
Existing concurrent-store tests retain shared transaction totals. |
AC-PLATFORM-BROWSER-CONSOLE-RETENTION-001.5 |
Runtime tests prove that persistence uses one active drain. |
AC-PLATFORM-BROWSER-CONSOLE-RETENTION-001.6 |
Runtime tests prove that capture fallback does not change persistence mode or erase staging. |
AC-PLATFORM-BROWSER-CONSOLE-RETENTION-001.9 |
Runtime tests freeze a sequence watermark and add later entries while the drain is held. |
AC-PLATFORM-BROWSER-CONSOLE-RETENTION-001.10 |
Store tests cover timestamp ties, identity filtering, continuation, page bounds, and bounded cursor visits. |
AC-PLATFORM-BROWSER-CONSOLE-RETENTION-001.11 |
Fake-time runtime tests prove the one-second memory fallback and continued persistence. |
AC-PLATFORM-BROWSER-CONSOLE-RETENTION-001.12 |
Capture tests prove that budget expiry stops new pages and cancels an active upload. |
Work orders
Task 02 has an implementation and focused verification, but its work order remains blocked by the missing Task 01 evidence. The active-profile phase table could not be reproduced because this workspace had no attached browser session.
Verification results
pnpm install --frozen-lockfilecompleted fromapps.- The focused frontend logger suite passed: 4 files and 45 tests.
pnpm run typecheckpassed fromapps/web.- The log-bundle package tests passed: 23 tests.
- The log-bundle package race tests passed: 23 tests.
- The focused frontend ESLint and Prettier checks passed.
gofmt -lreported no changed Go files, andgit diff --checkpassed.- The repository-wide backend test target was run twice. Both runs failed in unrelated process-probe, configuration-discovery, launcher, and Office migration tests. The first run also used the workspace home configuration.
- The required active-profile phase tables, 90 percent accounting, request sizes and counts, and backend 250 millisecond / 10 percent gate results remain uncollected.
Execution TODO
- Collect two active-profile captures and the isolated-profile phase tables.
- Account for at least 90 percent of frontend and backend timing.
- Apply the backend 250 millisecond and 10 percent gate.
- Implement Task 02 bounded, paged capture with regression tests.
- Run the exact task-defined checks and update work-order/plan status.
- Commit the current remediation with hooks and push the feature branch.
- Run PR fixup for the current remediation head.
Previous PR Fixup Result (superseded)
- PR #3540: https://github.com/kdlbs/kandev/pull/3540
- Fixup commit:
105549d3f79f3527719275669b5f0c4f178e12d4 - Final exact-head CI: 54 passed, 0 failed, 0 pending; snapshot complete.
- Final review state: 0 unresolved threads, 0 hidden unresolved threads, and 0 actionable issue comments. All six original threads were explicitly resolved.
- Final mergeability:
MERGEABLE / CLEAN. Themainbase advanced during CI; localgit merge-tree --write-treevalidation completed without conflicts.
Current remediation
- Replaced the post-flush high-watermark read with a receipt-started serialized boundary and receipt-prefix primary-key tracking.
- Replaced equal-timestamp rescans with
continuePrimaryKey()and added a cursor-visit bound regression. - Reopened Task 01 and this plan. The missing phase evidence remains an explicit blocker, and Task 02 is a bounded mitigation rather than a root cause claim.
- Local remediation tests passed. Remote CI and review checks for the new head are pending.
Risks
- A timestamp cursor can skip or duplicate entries when timestamps are equal.
- An IndexedDB boundary can be unavailable when its connection is not open.
- A slow active write can continue after capture selects its memory fallback.
- A delayed notification can leave less server time than the relative client budget. The backend can reject the late request.
- A 128 KiB first request can still exceed the budget on a very slow link.
- Request-body transfer can remain the largest cost after frontend paging.
Mobile parity
This change does not alter layout, navigation, touch behavior, scrolling, or viewport-specific controls. Desktop and mobile browsers use the same logger runtime, capture protocol, and tests. Focused unit and protocol tests are sufficient. No new mobile Playwright test is necessary.
Documentation impact
No public documentation changes are necessary. The visible bundle workflow is unchanged. The platform requirements and designs own the changed contract.