Files
kandev/docs/plans/workspace-git-status-scalability/plan.md
T
2026-07-20 11:04:59 +01:00

10 KiB

spec, created, status
spec created status
docs/specs/platform/workspace-git-status.md 2026-07-19 done

Implementation Plan: Workspace Git Status Scalability

Overview

First keep reusable verification caches shared and all verification storage outside Git worktrees, ignore legacy worktree-local cache paths, and reclaim each owned agent temp root during permanent instance teardown. Then make diff enrichment linear and context-aware while preserving complete changed-path metadata and all existing limits. Next coalesce overlapping live observations through the workspace tracker using Kandev's established detached, bounded singleflight pattern. Finish with HTTP-boundary concurrency and multi-repository regression coverage.

The incident that motivated this work involved 13,869 untracked Go cache files. Diff enrichment repeatedly recomputed total diff bytes by scanning the complete file map inside per-file loops, producing quadratic work. Six concurrent fresh multi-repository requests then repeated that computation and held agentctl near full CPU for roughly three minutes.

Backend

Verification cache safeguard

  • Add /.verify-cache/ and /.tmp/ to the repository root .gitignore so legacy or misconfigured local verification runs cannot add thousands of cache entries to Git status.
  • Strengthen .agents/skills/verify/SKILL.md to preserve shared managed caches, isolate only invocation scratch, and keep every fallback outside the repository and all worktrees.
  • Keep the ignore pattern root-scoped; do not hide similarly named paths that a fixture or nested repository may intentionally track.
  • Verify the rule with git check-ignore and confirm the guidance still uses an external mktemp root.

Agent temp teardown

  • Track the exact session temp directory created by ensureAgentTempEnv on process.Manager.
  • After shell, VS Code, workspace, adapter, and agent process teardown completes for permanent instance deletion, remove only that validated session directory. A later resume creates a new instance and does not depend on prior scratch. Apply the same cleanup when teardown finds the main agent already stopped so natural process exit cannot strand the directory.
  • Reject cleanup targets that are empty, equal to the shared kandev-agent root, or outside that root. Preserve sibling session directories.
  • Return cleanup failures to the instance teardown caller and add deterministic tests using an isolated test TMPDIR.

Linear and cancellable diff enrichment

  • Replace repeated totalDiffBytes(update) scans in the unstaged, staged, and untracked loops in workspace_git_diff.go with one request-local diff-budget accumulator initialized once and incremented whenever diff content is stored.
  • Preserve the existing 10 MiB source-file limit, 256 KiB per-file output limit, 2 MiB per-status limit, overshoot semantics, binary handling, and skip reasons.
  • Continue retaining every changed path after the total budget is exhausted. Avoid further file reads or diff synthesis for paths marked budget_exceeded.
  • Check ctx.Err() before each per-file filesystem or diff operation and propagate cancellation so callers cannot publish or cache a partial successful snapshot.
  • Add deterministic accumulator and cancellation tests. Use budget state plus structural review to guard the request-local accounting, and benchmark representative input sizes rather than asserting filesystem wall-clock thresholds in unit tests.

Coalesced live observations

  • Add a per-WorkspaceTracker singleflight.Group; golang.org/x/sync/singleflight is already used in the repository.
  • Route fresh reads, empty-cache fallback, and poll or explicit refresh computation through the same per-tracker flight so no live path bypasses the overlap guard.
  • Keep updateMu for the existing polling-skip versus explicit-refresh blocking policy. Only the expensive observation is shared.
  • Use DoChan and let each caller select on its own context. A cancelled caller exits promptly without cancelling work needed by other callers.
  • Run the shared body on a tracker-owned context with an independent deadline no greater than 60 seconds, matching the established detached-singleflight pattern in internal/agent/handlers/git_handlers.go. Root it in tracker lifetime so WorkspaceTracker.Stop cancels it.
  • Check and propagate the shared context error at observation phase boundaries. A cancelled or timed-out computation must not return, publish, or cache partial success.
  • Preserve cache ownership: a standalone fresh read does not update currentStatus; a poll that joins or initiates the shared observation may cache the completed result through its existing update path.

API regression coverage

  • Exercise concurrent GET /api/v1/git/status/multi?fresh=true requests against the same manager and tracker.
  • Assert one underlying observation per repository, shared capture data for overlapping callers, independent waiter cancellation, and repository-scoped results.
  • Retain partial success when one repository is invalid and other repositories succeed.
  • Prefer channel-synchronized Git shims or a focused unexported loader seam over elapsed-time assertions.

Frontend

No frontend changes. Existing Git-status response shapes and Changes/Review rendering remain unchanged.

Tests

  • What: Diff budget accounting is constant-time per changed entry and preserves existing output and skip semantics. File: apps/backend/internal/agentctl/server/process/workspace_git_diff_test.go How: Test accumulator boundaries, content accounting, and behavior after budget exhaustion. Retain existing oversized, binary, truncated, and budget tests.
  • What: Cancellation stops untracked enrichment before remaining filesystem operations. File: apps/backend/internal/agentctl/server/process/workspace_git_diff_test.go How: Cancel a context at a deterministic operation boundary and assert no later files are opened or enriched.
  • What: Same-repository live observations coalesce without coupling caller cancellation. File: apps/backend/internal/agentctl/server/process/workspace_git_status_concurrency_test.go How: Coordinate callers with channels; assert one loader invocation and identical completed results, while one cancelled waiter exits independently.
  • What: Tracker shutdown or the shared deadline stops the underlying observation without updating the cache. File: apps/backend/internal/agentctl/server/process/workspace_git_status_concurrency_test.go How: Block at a controlled phase, cancel tracker lifetime or expire an injected deadline, and assert cancellation plus unchanged cached state.
  • What: Existing fresh-versus-cached semantics remain unchanged. File: apps/backend/internal/agentctl/server/process/workspace_git_status_test.go How: Keep TestGetGitStatus_FreshBypassesStaleCache passing and cover empty-cache fallback through the shared observation path.
  • What: Concurrent fresh HTTP requests preserve coalescing, repository identity, and partial success. File: apps/backend/internal/agentctl/server/api/git_status_fresh_concurrency_test.go How: Issue overlapping multi-repository requests through the handler and assert invocation counts and response entries rather than timing alone.
  • What: Large untracked-set enrichment scales linearly. File: apps/backend/internal/agentctl/server/process/workspace_git_diff_test.go How: Add a benchmark at representative 1,000- and 10,000-entry sizes, deterministic unit assertions for request-local budget accounting, and structurally verify no whole-map scan occurs inside a per-file loop. Do not impose a filesystem wall-clock threshold in unit tests.
  • What: Verification storage cannot appear as root-level Git-status entries and managed caches remain shared. Files: .gitignore, .agents/skills/verify/SKILL.md How: Assert git check-ignore -q .verify-cache/go-cache/probe and git check-ignore -q .tmp/gocache/probe, then inspect the verification skill for shared-cache preservation and external scratch requirements.
  • What: Permanent instance teardown removes only its owned agent temp directory after subprocess teardown. Files: apps/backend/internal/agentctl/server/process/manager.go, manager_temp_test.go How: Use an isolated temporary root; create session and sibling sentinels, stop an already-stopped manager and a normally stopping manager, and assert only the owned session directory is removed. Cover invalid/root targets and cleanup errors.

E2E Tests

No browser E2E changes. The API payload and user-visible rendering are unchanged; focused process and handler tests cover the affected behavior.

Implementation Waves

Wave 0:

Wave 1 (parallel):

Wave 2 (depends on Wave 1):

Wave 3 (depends on Wave 2):

Verification

rtk git check-ignore -q .verify-cache/go-cache/probe
rtk git check-ignore -q .tmp/gocache/probe
rtk go test ./internal/agentctl/server/process -run 'TestManager_.*Temp'
rtk go test ./internal/agentctl/server/process -run 'Test(DiffBudget|EnrichUntrackedFileDiffs|GetGitStatus)'
rtk go test ./internal/agentctl/server/api -run 'Test.*GitStatus.*Fresh'
rtk go test -race ./internal/agentctl/server/process ./internal/agentctl/server/api
rtk make -C apps/backend fmt
rtk make -C apps/backend test
rtk make -C apps/backend lint

Run the go test commands from apps/backend. Run formatting before the full test and lint commands.

Risks

  • The shared body cannot inherit the first caller's context; otherwise that caller can cancel results still needed by peers.
  • Detached work needs both tracker-lifetime cancellation and a bounded deadline so a timed-out HTTP caller cannot leave unbounded work behind.
  • Pollers and fresh readers share computation but retain different cache-write behavior. Coalescing must not make fresh reads cache owners.
  • Map iteration order determines which files receive the remaining diff budget today. The change must preserve semantics without claiming stable ordering.

Open Questions

None.