Carlos Florêncio
8.3 KiB
created, status, requirements, system_design, legacy_specs
| created | status | requirements | system_design | legacy_specs | ||
|---|---|---|---|---|---|---|
| 2026-09-05 | complete |
|
|
Implementation Plan: Progressive Storage Analysis
Overview
Reduce scan time with bounded filesystem concurrency. Then make overview reads nonblocking and stream coarse progress to the Storage page.
The order protects the cache contract. The scanner lands first, the backend state contract follows, and the web work consumes that contract.
The temporary storage visibility and cleanup package extends this completed source and progress contract with an informational system temporary-folder source.
Scope
In scope
- Bounded, context-aware byte measurement for read-only storage analysis.
- Nonblocking overview reads with stale-while-revalidate behavior.
- Coarse progress notifications with a polling fallback.
- First-scan partial results and atomic replacement of older snapshots.
- Scan duration, cache lifetime, and next-refresh details.
- Desktop and phone access to the timing disclosure.
- Focused public operator documentation.
Out of scope
- Changes to destructive cleanup or quarantine rules.
- A persistent overview snapshot across backend restarts.
- A user-configurable scan worker count or cache lifetime.
- File-level progress events or exact time-remaining estimates.
- Remote-executor disk analysis.
- Automatic scans when no client reads the Storage overview.
Technical approach
Bounded filesystem measurement
Add apps/backend/internal/system/storage/filescan. Its shared limiter will permit four active
directory partitions across workspace, Go-cache, and temporary-artifact analysis.
Partition each root by immediate children. Preserve provider-specific symlink rejection, marker exclusion, warning order, and path ownership.
Inject the limiter through storageOverview provider construction. Keep destructive provider paths
on their current serial code.
Progressive cache and API
Extend OverviewCache with process-local analysis state and a nonblocking read. Keep Refresh as
the tracked manual path and join every concurrent request to one scan generation.
Add source progress callbacks to storageOverview. Keep the last successful Summary atomic and
store cold-scan partial results separately.
Extend GET /api/v1/system/storage with nullable first-scan snapshot fields and an analysis
object. Include scan timestamps, duration, cache lifetime, refresh time, source progress, and a
partial summary.
Publish system.storage.analysis.updated with generation and state only. Add the event constant,
gateway action, subscription, and frontend revision field.
Web state and presentation
Update storage API types and the Zustand system slice for analysis state and revision ordering.
Refactor useStorageMaintenance so WebSocket revisions reload only the overview. Poll every 1.5
seconds while analysis is active and stop in a terminal state.
Schedule one read at refresh_due_at while the Storage page remains mounted. Preserve request
generation checks and terminal Analyze refresh behavior.
Update StorageOverviewCard to show cold-scan source progress and counted-so-far totals. Keep a
stale completed snapshot visible during revalidation.
Add an information icon beside the completed snapshot time. Reuse StorageSettingHelp interaction
and permit structured content for the three timing values.
Documentation and observability
Update docs/public/operations.md with the asynchronous scan, 15-minute cache, and Analyze behavior.
Add structured scan completion logs and storage scan metrics. Do not include task paths in logs or metrics.
Tests
| Acceptance criteria | Evidence |
|---|---|
AC-SYSTEM-PAGE-STORAGE-OVERVIEW-PARALLEL-SCAN-002.1, .10 |
Cache and handler tests block the provider and prove immediate, single-flight reads. |
.2, .3 |
Provider, cache, hook, and card tests publish sources progressively and mark partial totals. |
.4, .9 |
Cache tests keep the last snapshot through expired and failed refreshes. |
.5 |
Event, hook, and fake-timer tests cover WebSocket refresh and the 1.5-second fallback. |
.6, .7 |
Component and Playwright tests cover timing values, focus, hover, tap, and touch size. |
.8 |
Hook fake-timer tests request at refresh_due_at. Analyze tests retain forced refresh. |
The filesystem scanner tests will cover exact bytes, no-follow behavior, deterministic warnings, cancellation, and a four-worker ceiling.
The provider tests will use barriers instead of elapsed-time assertions. They will prove that two partitions enter before either partition is released.
E2E tests
Update apps/web/e2e/tests/system/storage-maintenance.spec.ts for progressive cold-scan values,
stale refresh, and pointer or keyboard timing disclosure.
Update apps/web/e2e/tests/system/mobile-storage-maintenance.spec.ts for tap disclosure, a 44-pixel
trigger, progressive values, and no horizontal overflow.
Both flows map to AC-SYSTEM-PAGE-STORAGE-OVERVIEW-PARALLEL-SCAN-002.2 through .8.
Mobile design contract
Desktop and phone users enter through /settings/system/storage. Both receive the same snapshot,
progress, and timing data.
The existing Storage page and StorageSettingHelp are the nearest shipped examples. They provide
the one-column card flow and the hover, focus, and pinned-tap help interaction.
Storage analysis remains an inline card in the page scroll. No drawer or full-height surface is necessary because the timing content contains three short values.
The information trigger will use the existing 44-pixel phone hit area and compact desktop size. The page keeps one scroll owner and no horizontal overflow.
Business state, progress derivation, and refresh scheduling remain shared across viewports. Only the responsive trigger size differs.
Work orders
- Task 01: Build bounded filesystem scanner
- Task 02: Add progressive overview state
- Task 03: Render live analysis details
- Task 04: Prove progressive storage behavior
Dependency order
- Task 01 creates the scan and progress primitives.
- Task 02 uses those primitives for the cache, API, events, and observability.
- Task 03 consumes the backend contract and updates public operator documentation.
- Task 04 proves the integrated desktop and phone behavior.
All tasks are sequential. They share contracts, fixtures, and storage test files.
Verification results
- Task 01:
go test -race ./internal/system/storage/filescan ./internal/system/storage/workspaces ./internal/system/storage/gocache ./internal/system/storage/tempartifacts ./internal/backendapppassed (853 tests). - Task 01 benchmark:
go test ./internal/system/storage/filescan -run '^$' -bench 'BenchmarkMeasureTrees' -benchmempassed at 81,604 ns/op and 11,367 B/op. - Task 02:
go test -race ./internal/system/storage ./internal/backendapp ./internal/gateway/websocketpassed (1,423 tests). - Task 03: the focused web suite passed (46 tests), the extracted terminal-refresh suite passed (2 tests), typecheck, lint, i18n checks, and public-doc validation passed.
- Task 04:
make build-web,make build-backend, Chromium storage E2E (8 tests), and mobile Chromium storage E2E (6 tests) passed. - Changed-file Go lint passed with
golangci-lint run ./... --new-from-rev=481359ddb282fd7d3e8c9cdff99da6f9bdd24c7c --timeout=5m. - Specification lint passed with
python3 scripts/lint-spec-files.py --all.
Risks
- Parallel directory reads can increase contention on rotating disks. The shared four-walker limit bounds this risk, and timing evidence will record regressions.
- Partial results can look final. The API separates them from completed snapshots, and the UI marks aggregates as counted so far.
- Late progress can corrupt a new scan. Generation checks must guard every progress and completion write.
- Broadcast events can disclose storage facts. The event carries only generation and state.
- A missed event can leave stale UI. The active-scan poll provides recovery.
- A page timer can drift after sleep. The timer callback reads the server state instead of assuming that a scan started.