dreampiler/kandev
1
0
Fork 0
Code Issues 43 Pull Requests Actions 6 Packages Projects Releases Wiki Activity

feat(office): bound run history growth with a scheduled retention sweep (#3566)

Browse Source 
Co-authored-by: nova28 <17953305+nova28@users.noreply.github.com>
Co-authored-by: Carlos Florencio <carlosmflorencio@gmail.com>
This commit is contained in:
Henry
2026-09-14 04:43:44 +08:00
committed by GitHub
parent e75c8b37c8
commit f529d4b757
69 changed files with 9872 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/run-history-retention/plan.md
+64
View File
@@ -0,0 +1,64 @@
---
created: 2026-09-09
status: complete
requirements:
- REQ-OFFICE-RUN-HISTORY-RETENTION-001
- REQ-OFFICE-RUN-HISTORY-RETENTION-002
- REQ-OFFICE-RUN-HISTORY-RETENTION-003
- REQ-OFFICE-RUN-HISTORY-RETENTION-004
- REQ-OFFICE-RUN-HISTORY-RETENTION-005
system_design:
- ../../specs/office/system-design/run-history-retention.md
- ../../specs/office/system-design/run-history-retention-operations.md
legacy_specs: []
---
# Implementation Plan: Office Run History Retention
## Overview
Office writes `office_routine_runs` and `run_events` rows on every routine firing and
run lifecycle transition, and nothing ever deletes them on a schedule. A `*/5 * * * *`
routine produces roughly 105,000 `office_routine_runs` rows a year; one reference
install had already accumulated 323 consecutive `coalesced` rows from a routine that
was doing nothing useful. This plan adds one scheduled sweep, on its own interval
(never the 5s Office tick), that bounds `office_routine_runs`, `runs`, and their
satellites (`run_events`, `office_run_route_attempts`, `office_run_skills`) by age,
with a per-owner floor, identical behavior on SQLite and PostgreSQL, and an operator
surface (Settings > System > Data & Logs) that reports policy, counts, previews, and
warnings before and while rows are deleted.
The two halves of the contract are split the same way the specs are split: the sweep
itself, its eligibility rules, and engine parity are
[run history retention](../../specs/office/system-design/run-history-retention.md)
(REQ-001, REQ-002, REQ-005); the settings record, preview marker, health warnings, and
System page surface are
[run history retention operations](../../specs/office/system-design/run-history-retention-operations.md)
(REQ-003, REQ-004).
## Scope
### In scope
- A `internal/office/retention` package: settings store, eligibility/count/delete
queries shared by preview and delete, a session-scoped PostgreSQL advisory lock with
a SQLite single-process equivalent, a scheduler goroutine on its own interval, and an
HTTP handler for `GET`/`PUT /api/v1/system/retention`.
- Status-only classification of history vs. live state for both `office_routine_runs`
and `runs`, a per-owner floor, oldest-first chunked batch deletion, and satellite rows
deleted in the same transaction as their parent run.
- A per-table preview (report, delete nothing) on each table's first evaluation, and
`health.Issue` warnings before the cap and on sweep/count failure.
- A `RetentionSettingsCard` on Settings > System > Data & Logs showing policy, retained
counts, preview state, last sweep, and backlog/error warnings.
- Expression indexes serving the sweep's filter/order on both engines.
### Out of scope
- Filesystem/container cleanup (owned by storage maintenance).
- Routine or workspace deletion (already deletes runs structurally; unaffected).
- Any change to run lifecycle, routine dispatch, or task/session/checkout data.
## Tasks
- [x] [Task 01: Bound run history with a scheduled retention sweep](task-01-bound-run-history-with-retention-sweep.md)
docs/plans/run-history-retention/task-01-bound-run-history-with-retention-sweep.md
+183
View File
@@ -0,0 +1,183 @@
---
id: "01-bound-run-history-with-retention-sweep"
title: "Bound Office run history with a scheduled retention sweep"
status: done
wave: 1
depends_on: []
plan: "plan.md"
requirements:
- REQ-OFFICE-RUN-HISTORY-RETENTION-001
- REQ-OFFICE-RUN-HISTORY-RETENTION-002
- REQ-OFFICE-RUN-HISTORY-RETENTION-003
- REQ-OFFICE-RUN-HISTORY-RETENTION-004
- REQ-OFFICE-RUN-HISTORY-RETENTION-005
acceptance_criteria:
- AC-OFFICE-RUN-HISTORY-RETENTION-001.1
- AC-OFFICE-RUN-HISTORY-RETENTION-001.2
- AC-OFFICE-RUN-HISTORY-RETENTION-001.3
- AC-OFFICE-RUN-HISTORY-RETENTION-001.4
- AC-OFFICE-RUN-HISTORY-RETENTION-001.5
- AC-OFFICE-RUN-HISTORY-RETENTION-001.6
- AC-OFFICE-RUN-HISTORY-RETENTION-001.7
- AC-OFFICE-RUN-HISTORY-RETENTION-001.8
- AC-OFFICE-RUN-HISTORY-RETENTION-001.9
- AC-OFFICE-RUN-HISTORY-RETENTION-001.10
- AC-OFFICE-RUN-HISTORY-RETENTION-002.1
- AC-OFFICE-RUN-HISTORY-RETENTION-002.2
- AC-OFFICE-RUN-HISTORY-RETENTION-002.3
- AC-OFFICE-RUN-HISTORY-RETENTION-002.4
- AC-OFFICE-RUN-HISTORY-RETENTION-002.5
- AC-OFFICE-RUN-HISTORY-RETENTION-002.6
- AC-OFFICE-RUN-HISTORY-RETENTION-002.7
- AC-OFFICE-RUN-HISTORY-RETENTION-002.8
- AC-OFFICE-RUN-HISTORY-RETENTION-002.9
- AC-OFFICE-RUN-HISTORY-RETENTION-002.10
- AC-OFFICE-RUN-HISTORY-RETENTION-002.11
- AC-OFFICE-RUN-HISTORY-RETENTION-002.12
- AC-OFFICE-RUN-HISTORY-RETENTION-002.13
- AC-OFFICE-RUN-HISTORY-RETENTION-003.1
- AC-OFFICE-RUN-HISTORY-RETENTION-003.2
- AC-OFFICE-RUN-HISTORY-RETENTION-003.3
- AC-OFFICE-RUN-HISTORY-RETENTION-003.4
- AC-OFFICE-RUN-HISTORY-RETENTION-003.5
- AC-OFFICE-RUN-HISTORY-RETENTION-003.6
- AC-OFFICE-RUN-HISTORY-RETENTION-003.7
- AC-OFFICE-RUN-HISTORY-RETENTION-003.8
- AC-OFFICE-RUN-HISTORY-RETENTION-003.9
- AC-OFFICE-RUN-HISTORY-RETENTION-003.10
- AC-OFFICE-RUN-HISTORY-RETENTION-003.11
- AC-OFFICE-RUN-HISTORY-RETENTION-004.1
- AC-OFFICE-RUN-HISTORY-RETENTION-004.2
- AC-OFFICE-RUN-HISTORY-RETENTION-004.3
- AC-OFFICE-RUN-HISTORY-RETENTION-004.4
- AC-OFFICE-RUN-HISTORY-RETENTION-004.5
- AC-OFFICE-RUN-HISTORY-RETENTION-004.6
- AC-OFFICE-RUN-HISTORY-RETENTION-004.7
- AC-OFFICE-RUN-HISTORY-RETENTION-004.8
- AC-OFFICE-RUN-HISTORY-RETENTION-004.9
- AC-OFFICE-RUN-HISTORY-RETENTION-005.1
- AC-OFFICE-RUN-HISTORY-RETENTION-005.2
- AC-OFFICE-RUN-HISTORY-RETENTION-005.3
- AC-OFFICE-RUN-HISTORY-RETENTION-005.4
- AC-OFFICE-RUN-HISTORY-RETENTION-005.5
system_design:
- ../../specs/office/system-design/run-history-retention.md
- ../../specs/office/system-design/run-history-retention-operations.md
---
# Task 01: Bound Office Run History with a Scheduled Retention Sweep
## Summary
Add `internal/office/retention`: a settings-backed sweep that ages out
`office_routine_runs` and `runs` history (plus their satellites) on its own
interval, a per-owner floor, status-only live/history classification, a preview
pass before first deletion, engine-identical behavior on SQLite and PostgreSQL,
and a `GET`/`PUT /api/v1/system/retention` operator surface with a matching
Settings > System > Data & Logs card.
## In scope
- Eligibility, count, and delete queries shared by preview and the real sweep,
keyed on `COALESCE(completed_at, created_at)` / `COALESCE(finished_at,
requested_at)`, oldest-first, chunked across statements.
- A session-scoped PostgreSQL advisory lock (with a SQLite single-process
equivalent) so only one backend sweeps at a time.
- A per-owner floor (newest N rows retained regardless of age) applied to
count, preview, and delete identically, re-asserted in the DELETE statement.
- Satellite deletion (`run_events`, `office_run_route_attempts`,
`office_run_skills`) in the same transaction as their parent run.
- `health.Issue` warnings for approaching the backlog cap and for count/sweep
failure; a one-time preview per table before any row is deleted.
- `RetentionSettingsCard` on Settings > System > Data & Logs: policy, retained
counts, preview/backlog state, last sweep, errors.
- Expression indexes serving the sweep's filter/order on both engines.
## Out of scope
- Filesystem/container artifact cleanup (storage maintenance owns that).
- Routine/workspace deletion cascades (already delete runs structurally).
- Any run lifecycle, routing, or task/session/checkout behavior change.
## Acceptance
- History rows (terminal `office_routine_runs`/`runs` statuses) older than the
configured window are deleted in batches; live-state rows (`received`,
`task_created`, `queued`, `claimed`) are never age-pruned; an unrecognized
status fails safe as live state and warns.
- The newest `floor` rows per owner survive regardless of age.
- Each table's first sweep is a preview: it reports would-delete counts and
deletes nothing; a later sweep performs real deletion.
- Behavior, including the advisory lock and batch chunking, is identical on
SQLite and PostgreSQL.
- `GET`/`PUT /api/v1/system/retention` read/write policy and report current
counts, preview state, last sweep outcome, and backlog/error warnings.
## Verification
```bash
cd apps/backend && go test -race -count=1 ./internal/office/retention/... ./internal/office/repository/sqlite/...
cd apps/backend && KANDEV_TEST_POSTGRES_DSN=<dsn> go test -race -count=1 -v ./internal/office/retention/...
cd apps/backend && golangci-lint run ./internal/office/retention/...
cd apps/web && pnpm run typecheck
cd apps && pnpm --filter @kandev/web test -- --run retention-settings-card system-route-copy
cd apps/web && pnpm e2e:run -- --project chromium --grep "System retention settings"
```
## Files likely touched
- `apps/backend/internal/office/retention/*.go`
- `apps/backend/internal/office/repository/sqlite/base_migrations.go`,
`retention_indexes_test.go`, `retention_indexes_postgres_test.go`
- `apps/backend/internal/backendapp/*` (scheduler/handler wiring)
- `apps/web/src/**/retention-settings-card.tsx` and its tests
- `docs/specs/office/requirements/run-history-retention*.md`,
`docs/specs/office/system-design/run-history-retention*.md`
- `docs/public/operations.md`
## Dependencies
None.
## Risks
- A dialect-sensitive query bug that only reproduces on PostgreSQL (mitigated
by an environment-gated Postgres suite covering the advisory lock, the
EvalPlanQual TOCTOU window, and batch chunking).
- A too-aggressive window or floor deleting rows an operator still needed
(mitigated by the default 30-day window, the per-owner floor, and the
preview-before-delete behavior).
## Parallelism
`sequential`
## Inputs
- `REQ-OFFICE-RUN-HISTORY-RETENTION-001` through `-005`.
- [run history retention](../../specs/office/system-design/run-history-retention.md)
and
[run history retention operations](../../specs/office/system-design/run-history-retention-operations.md).
- The reference install's 323 consecutive `coalesced` routine-run rows (28
days, one bricked routine) cited in the requirements' Overview.
## Results
- Implemented `internal/office/retention` (settings store, `Sweeper`,
`Scheduler`, `CensusTracker`, PostgreSQL advisory lock with SQLite
equivalent, `Handler` for `GET`/`PUT /api/v1/system/retention`) plus the
`RetentionSettingsCard` on Settings > System > Data & Logs.
- Full backend gauntlet green: `go build ./...`, `go vet`, `gofmt -l`,
`go test -race ./internal/office/retention/...` (SQLite), the
PostgreSQL-gated suite against a real scratch instance (125 tests, 0
skipped, all PASS, covering the advisory lock, the EvalPlanQual
concurrent-resurrection TOCTOU regression, and chunked batch deletes),
`golangci-lint run ./...` (0 issues).
- Frontend: `pnpm run typecheck`, the retention card and System route-copy
Vitest suites, and a scoped Playwright run (`--grep "System retention
settings"`, 2 passed).
- Delivered as PR [#3566](https://github.com/kdlbs/kandev/pull/3566)
("feat(office): bound run history growth with a scheduled retention
sweep"), through four Build rounds and four Review rounds; remaining
non-blocking test-rigor gaps and CI-wiring follow-ups are tracked on the
linked follow-up card rather than blocking this PR.
docs/public/operations.md
+28
View File
@@ -348,6 +348,34 @@ the Kandev service first provides the clearest maintenance boundary.
</details>
## Office run history retention
Open **Settings > System > Data & Logs** to manage automatic deletion of old
Office run history. Deletion is enabled by default. The first sweep starts five
minutes after the backend starts or after you enable deletion.
The first sweep for each history table is a preview. It reports the rows that
would be deleted and removes no rows. A later sweep can delete eligible rows.
Deletion is permanent. Back up the database before you enable deletion if you
need to keep old history outside the configured window.
The retention window controls the age of rows that can be deleted. The minimum
kept per owner control keeps the newest rows for each routine or agent, even
when those rows are older than the window. A floor of zero removes this extra
protection. Run event, route attempt, and skill rows are deleted with their
parent run.
The page shows the current policy, retained row counts, preview results, the
last sweep, and any backlog or errors. Counts continue to update when deletion
is disabled, so you can monitor growth before you enable it again.
To disable automatic deletion:
1. Open **Settings > System > Data & Logs**.
2. Clear **Delete eligible run history**.
3. Select **Save changes**.
4. Check the retention status. It must show that deletion is disabled.
## Database operation
> **Single-owner rule:** SQLite uses one writer connection in WAL mode; only one Kandev backend should own the file. **Factory reset** is destructive and removes managed data after creating a pre-reset backup.
docs/specs/office/requirements/run-history-retention-operations.md
+214
View File
@@ -0,0 +1,214 @@
---
status: draft
system: office
created: 2026-09-09
owners:
- kandev
---
# Office Run History Retention Operations Requirements
## Overview
[Run history retention](run-history-retention.md) defines which Office run
history rows may be deleted and how the sweep that deletes them behaves. This
document defines the other half of that contract: what an operator is told
before the first row is removed, what they are told while the tables grow, what
they can configure, and what they can read about the last sweep.
The split follows the ownership boundary the retention contract already draws.
Office owns the rows and the deletion policy because they are Office primitives.
The System pages own the operator surface those decisions are reported through,
and that surface is a separate contract with a separate consumer: an operator
reading a settings page and a health card, rather than a scheduler deleting
rows. Splitting here keeps each document inside its size limit without cutting
either contract.
The requirement IDs continue the retention capability's sequence rather than
starting a new one, because these are the same capability's requirements viewed
from the operator's side.
## Terminology
Terms are defined once, in
[run history retention](run-history-retention.md#terminology), and used here
with the same meaning. The ones this document leans on most:
- **Retention sweep**, **preview**, and **retained count**.
- **Swept tables** (`office_routine_runs`, `runs`), **reported tables** (those
two plus the three run satellites), and **thresholded tables**
(`office_routine_runs`, `runs`, `run_events`). "Per table" always names one of
these three sets, never an unqualified "table".
## Requirements
### REQ-OFFICE-RUN-HISTORY-RETENTION-003: Warn before deleting, and warn while growing
**Intent:** An operator learns what retention is about to remove before it
removes anything, and learns that history is growing past a threshold while
there is still time to widen the window or fix the routine.
As an operator upgrading an install with a year of run history, I want to be
told what the first sweep would delete before it deletes it, so that I can widen
the retention window first if that history matters to me.
#### Acceptance criteria
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.1:** The first evaluation of each swept table
on a database shall be a preview for that table: it evaluates the policy,
reports the number of rows it would delete from that table, and deletes
nothing from it. A table that has not completed a preview never deletes,
regardless of what any other table has done.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.2:** When a preview sweep would delete
at least one row, the system shall emit an operator-visible warning naming
each table, its would-delete count, and the configured retention window, and
stating that deletion begins at the next scheduled sweep.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.3:** When a preview sweep would delete
no rows, the system shall not emit a warning.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.4:** The system shall run a preview at most
once per swept table per database. A table's preview shall be recorded only
when that table's preview evaluation completed successfully; a table that
failed during a sweep in which it was being previewed shall be previewed again
on the next sweep rather than deleting. Disabling and re-enabling retention,
restarting the backend, or changing any retention setting shall not produce a
second preview for a table that has already completed one.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.5:** When a thresholded table's retained
count exceeds that table's configured warning threshold, the system shall emit
an operator-visible warning naming the table, its retained count, and the
threshold. For `office_routine_runs`, the warning shall also name the routine
holding the largest share of retained rows and that share, where the share is
that routine's retained rows as a proportion of the table's retained count.
When two routines hold an equal largest share, the lower routine identifier
shall be named.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.6:** When a table reports remaining
backlog after a sweep, the system shall emit an operator-visible warning
naming the table, stating that retention is behind, and reporting the number
of rows deleted in that sweep. A table that was previewed in that sweep shall
not produce this warning however many rows were eligible, because a preview
deletes nothing by design and retention is therefore not behind; the preview
warning of AC-OFFICE-RUN-HISTORY-RETENTION-003.2 reports its eligible count
instead.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.7:** When retention is disabled and a
table's retained count exceeds its warning threshold, the system shall emit
the same threshold warning and shall additionally state that retention is
disabled, so a silent unbounded table is distinguishable from one being
actively managed.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.8:** Every warning in this requirement
shall be emitted through a channel available in a production build, and shall
not be observable only through the debug metrics endpoint.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.9:** A preview sweep's would-delete
counts shall be the full eligible count per table, not capped by the batch
limit, so an operator is told the real size of what is about to be removed.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.10:** When a swept table's recorded preview
state cannot be read or parsed, the system shall treat that table as not yet
previewed and shall emit an operator-visible warning naming the table.
- **AC-OFFICE-RUN-HISTORY-RETENTION-003.11:** Retained counts for the thresholded
tables shall be produced by a count evaluation that does not require a sweep
to have run, so the counts and the threshold warnings required by AC-OFFICE-RUN-HISTORY-RETENTION-003.7 and
AC-OFFICE-RUN-HISTORY-RETENTION-004.8 are available on a fresh install and while retention is disabled. The
counts shall be refreshed on the sweep interval and reused between refreshes
rather than recomputed for each operator page load or health poll. The first
count evaluation shall run at startup, before the delay of
AC-OFFICE-RUN-HISTORY-RETENTION-002.10 arms the first sweep, so an operator is
not shown countless tables for a sweep interval after every restart. When a
count evaluation fails, the system shall keep the counts from the last
successful evaluation, report them as stale together with the time they were
produced, and emit an operator-visible warning. A failed count shall not
suppress the threshold warnings derived from the last successful counts, and
shall not fail the sweep. Until the first evaluation has completed, and when it
fails with no earlier successful evaluation to fall back on, the system shall
report the counts as not yet computed and shall not render them as zero, on the
same grounds as AC-OFFICE-RUN-HISTORY-RETENTION-004.7: a count nobody has taken
must not read as a table that is empty. Success and failure shall be tracked
per thresholded table, so one table's failed count neither discards nor marks
stale another table's successful one.
### REQ-OFFICE-RUN-HISTORY-RETENTION-004: Operator configuration and sweep visibility
**Intent:** Retention is configurable, its values are validated, and the result
of the most recent sweep is readable by an operator without reading logs.
#### Acceptance criteria
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.1:** An operator can read and change
whether retention is enabled, the retention window per in-scope table, the
sweep interval, the retention floor per in-scope table, the batch limit, and
the warning threshold per in-scope table.
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.2:** Retention shall be enabled by
default. Default values shall be: retention window 30 days for both
`office_routine_runs` and `runs`; sweep interval 6 hours; retention floor 50
rows per owner for both tables; batch limit 5,000 rows per table per sweep;
warning threshold 25,000 rows for `office_routine_runs`, 25,000 for `runs`,
and 250,000 for `run_events`.
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.3:** The system shall reject a setting
outside its permitted range with an error naming the field, and shall leave
the stored settings unchanged. Permitted ranges are: retention window 1 to
3,650 days; sweep interval 1 to 168 hours; retention floor 0 to 10,000; batch
limit 100 to 100,000; warning threshold 0 or greater, where 0 disables that
table's threshold warning.
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.4:** When stored retention settings
cannot be read or parsed, the system shall use the documented defaults, emit
an operator-visible warning, and continue. Unreadable settings shall not
disable retention silently and shall not fail startup.
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.5:** A settings change shall take effect
without a backend restart, and shall apply from the next scheduled sweep
rather than interrupting a sweep in progress. Two concurrent writes resolve
last-writer-wins; repeating an identical write changes nothing and returns the
same normalized document. A sweep shall read the stored settings at its start
rather than relying on a value cached when this process last observed a change,
so that where several backend processes share one database a process that did
not serve the write still sweeps under the new settings rather than the
replaced ones. When that read fails or the stored settings cannot be parsed,
the sweep shall be skipped and recorded as skipped rather than run against the
documented defaults, because a default window is shorter than a window an
operator has widened and sweeping under it would delete the history they
configured the system to keep. This does not change
AC-OFFICE-RUN-HISTORY-RETENTION-004.4, which governs reading settings for
reporting and startup, where using the defaults deletes nothing.
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.6:** An operator can read, on the System
pages, the outcome of the most recent completed sweep: when it ran, which
swept tables it previewed, the rows deleted per reported table, the rows it
would have deleted per previewed table, the retained count per thresholded
table, whether any table has remaining backlog, and any table that failed.
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.7:** When no sweep has run since the
backend started, the surface in AC-OFFICE-RUN-HISTORY-RETENTION-004.6 shall
say so explicitly rather than render an empty or zeroed result that reads as a
sweep that deleted nothing.
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.8:** The sweep-result surface shall be
readable while retention is disabled, reporting retained counts and the
disabled state.
- **AC-OFFICE-RUN-HISTORY-RETENTION-004.9:** A settings write shall replace the whole
settings document. A field the caller omits shall take its documented default
rather than its previously stored value, so the same request always produces
the same stored document. An unrecognized field, a numeric field whose
value is not a whole number, a field present with a JSON `null`, and a field
whose value is not of its documented type, shall each be rejected with an error
naming the field, leaving the stored settings unchanged. `null` shall be
rejected rather than treated as omission: omission means the documented
default, so reading `null` the same way would let a client silently replace a
configured retention window with a shorter default and destroy history the
operator meant to keep.
## Out of scope
- **A persisted history of retention sweeps.** The last sweep's result is held
in memory and reported alongside the durable warnings. A growing table
recording the work of the job that stops tables growing is the same defect in
a new place.
- **Per-workspace or per-routine retention overrides.** Settings are
instance-wide. The retention floor is already per owner, which covers what a
per-routine override would most often be used for.
- **Archival or export before deletion.** Deleted history is gone. An install
needing it kept sets a longer window or takes a backup.
- **Anything the deletion policy owns.** Eligibility, ordering, batching,
atomicity, and engine parity are specified in
[run history retention](run-history-retention.md) and are not restated here.
## Prior art
The receipts for both prior-art legs are recorded once, in
[run history retention](run-history-retention.md#prior-art). The finding that
bears on this document specifically: neither surveyed product previews before a
policy's first deletion, nor warns ahead of the window. Those two behaviors are
where this capability goes further, and they are the reason this document
exists rather than being a settings page bolted onto a sweep.
docs/specs/office/requirements/run-history-retention.md
+324
View File
@@ -0,0 +1,324 @@
---
status: draft
system: office
created: 2026-09-09
owners:
- kandev
---
# Office Run History Retention Requirements
## Overview
Office writes two families of history rows that nothing removes on a schedule.
`office_routine_runs` records one row per routine firing. `run_events` records
the timeline of every Office run, append-only, alongside the `runs` queue row
and its per-run satellites. The only deletions today are manual and structural:
deleting a routine removes its runs, and deleting a workspace removes everything
belonging to it. An install that never deletes a routine or a workspace grows
forever.
The growth is driven by a clock, not by usage. A routine on a `*/5 * * * *`
schedule fires 105,120 times a year, and each firing writes a routine-run row
whether or not it does anything. On the reference install, one routine had
accumulated 323 consecutive `coalesced` rows over 28 days while producing no
work at all: that is the curve with the loop broken, and a working loop also
writes the run row, its `run_events` timeline, and its route-attempt rows.
This document bounds those tables. It defines what is history and may be
deleted, what is live state and must never be deleted on age, when the deletion
runs, and that it behaves identically on both database engines. What an operator
is told before the first row is removed, what they are warned about while the
tables grow, and what they can configure and read is the paired contract in
[run history retention operations](run-history-retention-operations.md).
Office owns this contract because the rows are defined by Office primitives: the
routine dispatch ledger, the run queue, and the run event timeline. The System
pages own the operator surface it reports through. The filesystem and container
cleanup owned by [storage
maintenance](../../system-page/requirements/storage-maintenance.md) is a
separate capability that never touches database rows.
## Terminology
- **Retention sweep** (or **sweep**): one pass that evaluates every in-scope
table against the configured policy and deletes the eligible rows.
- **History row**: a row recording something that already happened, which no
live decision reads. History rows are eligible for deletion.
- **Live-state row**: a row a live decision still reads, regardless of age.
Never eligible for age-based deletion.
- **Recovery-protected run**: a failed `runs` row named by an active
`office_agent_pause_recoveries.failed_run_id`; retention keeps it until the
recovery row is consumed or discarded.
- **Run satellite row**: a row keyed by a `runs` row's identifier and owned by
it: a `run_events`, `office_run_route_attempts`, or `office_run_skills` entry.
- **Retention window**: the age past which a history row becomes eligible,
measured from the row's own completion time.
- **Completion time**: `COALESCE(completed_at, created_at)` for a routine-run
row and `COALESCE(finished_at, requested_at)` for a `runs` row. Both fallback
columns are `NOT NULL`, so completion time is never null.
- **Retention floor**: most-recent history rows kept per owner regardless of
age, so a rarely-firing routine or rarely-woken agent keeps visible history.
- **Batch limit**: the maximum rows one sweep deletes from one table.
- **Preview**: a swept table's first evaluation on a database; it counts what it
would delete from that table and deletes nothing. Tracked per swept table.
- **Retained count**: the number of rows a table currently holds — a property of
the table, not of a sweep, defined whether or not a sweep has ever run and
whether or not retention is enabled.
- **Swept tables**: the two tables retention selects rows from by policy,
`office_routine_runs` and `runs`.
- **Reported tables**: the five tables a sweep can delete rows from and reports
deleted counts for: the two swept tables plus `run_events`,
`office_run_route_attempts`, and `office_run_skills`.
- **Thresholded tables**: the three tables carrying a warning threshold,
`office_routine_runs`, `runs`, and `run_events`.
## Requirements
### REQ-OFFICE-RUN-HISTORY-RETENTION-001: History is bounded and live state is not
**Intent:** Bound `office_routine_runs`, `runs`, and the run satellite tables by
age and by an owner-scoped floor, while guaranteeing that no row another
decision still reads is removed because it is old.
As an operator running Office continuously, I want old run history removed
automatically, so a scheduled routine does not grow the database without limit.
#### Acceptance criteria
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.1:** A routine-run row is a history row
only when its status is one of `skipped`, `coalesced`, `failed`, `done`, or
`cancelled`. When a routine-run row's status is `received` or `task_created`,
the system shall treat it as a live-state row and shall not delete it on age,
at any age.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.2:** A `runs` row is a history row when its
status is `finished`, `failed`, or `cancelled`. `cancelled` is terminal: its
only writer moves a row there from `queued` or `claimed` and stamps the
completion timestamp in the same statement. When a `runs` row's status is
`queued` or `claimed`, the system shall treat it as a live-state row and shall
not delete it on age, at any age, including a run parked for a future routing
retry. A failed run referenced by an active
`office_agent_pause_recoveries.failed_run_id` is also live state for
retention and shall remain until that recovery row is consumed or discarded.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.3:** When a history row's completion time is
older than that table's retention window, the system shall make it eligible
for deletion. Completion time is defined in Terminology. A row's
classification as history shall depend on its status alone and shall not
additionally require `completed_at` or `finished_at` to be set; a terminal row
with an unset stamp is history, dated by its fallback column, and ages out
rather than being retained forever.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.4:** The system shall retain the newest
history rows of each owner up to that table's retention floor even when they
are older than the retention window. The owner is the routine for
`office_routine_runs` and the agent profile for `runs`. Newest is completion
time descending, with the row identifier descending as the tiebreak.
Completion time is never null, so this ordering is total on both engines and
does not depend on either engine's default placement of nulls. The identifier
gives a stable order for equal timestamps and is not claimed to be
chronological.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.5:** When the system deletes a `runs`
row, it shall delete that run's satellite rows in the same database
transaction, and no satellite row shall remain that references a `runs` row
the system has deleted.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.6:** The system shall not delete
`run_events` rows for a run it is not deleting in the same transaction, at any
age.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.7:** A routine's status shall not exempt
its history from retention. When a routine is `paused`, its history rows are
evaluated by the same policy as an `active` routine's.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.8:** The system shall not delete, alter,
archive, or cancel a task, a session, a task checkout, or a workspace as part
of a retention sweep. A routine-run row naming a task in `linked_task_id` may
be deleted while that task continues to exist.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.9:** A retained `coalesced` routine-run
row may name a `coalesced_into_run_id` whose row has already been deleted.
`coalesced_into_run_id` records provenance and is not a referential
constraint; the system shall not delete a coalesced row because its target was
deleted, and shall not retain a target because a coalesced row names it.
- **AC-OFFICE-RUN-HISTORY-RETENTION-001.10:** When a row in a swept table holds a
status belonging to neither that table's history set nor its live-state set,
the system shall treat it as a live-state row, shall not delete it at any age,
and shall emit an operator-visible warning naming the table and the
unrecognized status, through the channel required by AC-OFFICE-RUN-HISTORY-RETENTION-003.8 in [run history
retention operations](run-history-retention-operations.md). The warning shall be
produced by the count evaluation required by
AC-OFFICE-RUN-HISTORY-RETENTION-003.11 rather than by the deletion path, which
cannot observe a status it does not select; it shall therefore be emitted on an
install where retention is disabled and no sweep runs. When one table holds more
than one unrecognized status, the system shall emit a single warning for that
table naming every unrecognized status in ascending lexicographic order with its
row count.
### REQ-OFFICE-RUN-HISTORY-RETENTION-002: The retention sweep
**Intent:** Run retention on its own schedule, off the run-claiming hot path, in
bounded batches, with one sweep at a time and no dependence on database cascade
behavior.
#### Acceptance criteria
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.1:** The system shall run retention on a
dedicated schedule whose interval is configurable in hours, and shall not
perform retention work on the Office run-processing tick.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.2:** When a scheduled sweep is due while a
previous sweep is still running, the system shall skip the due sweep rather
than run two concurrently or queue it, and shall record that it was skipped.
Recording a skip shall not replace the reported result of the last completed
sweep.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.3:** A sweep shall delete at most the batch
limit of rows from `office_routine_runs` and at most that many from `runs`. A
preview evaluation shall never be reported as having remaining backlog: it
deletes nothing by design, so "retention is behind" is not true of it however
many rows were eligible.
The limit counts those rows only; every satellite row of a deleted run is
removed regardless of the limit. When more rows were eligible than the limit
allowed, the system shall complete the sweep, report that table as having
remaining backlog, and continue on the next scheduled sweep. Within a table
the sweep shall select its batch in completion-time ascending order, with the
row identifier ascending as the tiebreak, so the oldest eligible rows are
removed first and both engines select the same batch.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.4:** The system shall re-assert every
eligibility condition in the deletion itself, not only when selecting
candidates. A run that returns to `queued` between selection and deletion,
which a scheduled retry does by clearing `finished_at`, shall not be deleted
by that sweep. The re-asserted conditions shall include the retention floor as
well as status and age.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.5:** Each batch shall be atomic: after a
sweep is interrupted by shutdown or error, every batch that was applied is
complete, including its satellite rows, and no batch is partially applied.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.6:** Once a table holds no eligible row,
a further sweep against unchanged settings shall delete nothing from that table
and report zero deletions for it. This criterion is scoped to that drained
state and does not contradict AC-OFFICE-RUN-HISTORY-RETENTION-002.3: while a
table still reports remaining backlog, the next sweep is required to delete its
next batch, so "deletes nothing further" is not claimed of a backlogged table.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.7:** When a table's sweep fails with a
database error, the system shall record the failure for that table, continue
with the remaining in-scope tables, and retry the failed table on the next
scheduled sweep. A retention failure shall not fail backend startup, stop the
Office scheduler, or abort the remainder of the sweep. A batch abandoned
because its deletion could not be applied consistently shall be recorded as a
failure for that table rather than as backlog. Every table whose rows that
batch's transaction addressed shall report zero rows deleted for that sweep, so
no table reports rows that the rollback restored.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.8:** When retention is disabled, the
system shall run no sweep and delete no row, and shall still report retained
counts and threshold warnings.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.9:** When every in-scope table is empty
or holds no eligible row, the sweep shall complete reporting zero deletions,
without error and without a warning.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.10:** The first sweep after the backend
starts shall run after a short fixed delay rather than after a full sweep
interval, so that an install restarted more often than the interval still runs
retention. Later sweeps use the configured interval.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.11:** A sweep shall compute one cutoff
instant at its start and evaluate every table against that instant, so two
tables in one sweep cannot disagree about what "older than the window" means.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.12:** On PostgreSQL, where several backend
processes can share one database, sweep exclusivity shall hold across
processes and not only within one: a backend that cannot acquire the retention
lock shall skip its due sweep exactly as it would for a sweep already running
in its own process. On SQLite one backend process owns the database file, so
the in-process guard is sufficient.
- **AC-OFFICE-RUN-HISTORY-RETENTION-002.13:** Sweeps shall be scheduled fixed-delay:
the next sweep is armed when the previous one finishes, so a sweep running
longer than the interval delays its successor rather than causing an immediate
second one. A settings change re-arms the delay from the moment of the change.
Enabling retention that was disabled arms the next sweep at the same short
delay as AC-OFFICE-RUN-HISTORY-RETENTION-002.10 rather than at a full interval.
Each backend shall periodically reread the shared settings record, including
while retention is disabled, so a backend that did not serve a settings write
still adopts enablement and interval changes.
### REQ-OFFICE-RUN-HISTORY-RETENTION-005: Integrity and database engine parity
**Intent:** Retention leaves the database consistent and behaves identically on
both supported engines, including where the schema differs between them.
#### Acceptance criteria
- **AC-OFFICE-RUN-HISTORY-RETENTION-005.1:** The system shall produce the same
observable retention outcome on SQLite and on PostgreSQL for the same settings
and the same starting rows: the same rows deleted, the same rows retained, the
same reported counts, and the same warnings. This shall hold on the backlog
path as well, because batch selection order is fixed by named columns in
AC-OFFICE-RUN-HISTORY-RETENTION-002.3 rather than left to the engine.
- **AC-OFFICE-RUN-HISTORY-RETENTION-005.2:** The system shall delete satellite
rows explicitly and shall not depend on a foreign-key cascade to remove them.
Neither engine declares a foreign key from a satellite table to `runs`.
- **AC-OFFICE-RUN-HISTORY-RETENTION-005.3:** After any sequence of sweeps, the
concurrency gate that finds a routine's active run by dispatch fingerprint
shall return exactly what it would have returned had no sweep run.
- **AC-OFFICE-RUN-HISTORY-RETENTION-005.4:** After any sequence of sweeps, the
lookup that closes out a routine run when its linked task reaches a terminal
step shall not resolve a deleted row to a different routine's run. A deleted
row resolves to nothing.
- **AC-OFFICE-RUN-HISTORY-RETENTION-005.5:** Retention shall not change the
behavior of deleting a routine or deleting a workspace. Those paths continue
to remove every row they remove today, including rows retention has not yet
reached.
## Out of scope
Each exclusion below is a decision, not an oversight.
- **Automation run history (`automation_runs` and its tables).** Owned by the
automation system, and its published contract is that history is removed only
by an explicit per-run or delete-all action. It has the same unbounded-growth
gap and needs its own requirement; changing it here would silently break a
documented promise.
- **`office_activity_log`, inbox dismissals, and approval rows.** The
[inbox requirement](inbox.md) already states these accumulate indefinitely and
excludes activity-log retention. Reopening it belongs to that contract.
- **`office_cost_events`.** Deleting cost rows changes reported spend and budget
enforcement: a decision about financial records, not about storage.
- **Detecting a stuck routine.** The 323-row reference case came from a routine
that fired correctly and did nothing useful for 28 days. Bounding its history
does not detect it; a detector for that state is a scheduler or stall
visibility concern.
- **Reclaiming file bytes after deletion.** Deleting rows does not shrink a
SQLite file. `VACUUM` is already an operator action on the System pages and a
sweep does not trigger it.
- **Archival or export before deletion.** Deleted history is gone. An install
needing it kept sets a longer window or takes a backup.
- **Per-workspace or per-routine retention overrides.** Settings are
instance-wide here. The retention floor is already per owner, which covers what
a per-routine override would most often be used for.
- **A persisted history of retention sweeps.** The last sweep's result is held
in memory and reported alongside the durable warnings. A growing table
recording the work of the job that stops tables growing is the same defect in
a new place.
- **Tables outside Office.** Nothing here changes task, session, workflow,
plugin, or auth storage.
## Prior art
Receipts for both legs. The design document carries what each finding changed.
**Our own wiki: not consulted, tool unavailable.** The `@henry` pin resolved
`~/.obsidian-wiki/config` to `config.henry`, giving
`OBSIDIAN_VAULT_PATH=/Users/henry/Documents/henry/wiki` and
`QMD_WIKI_COLLECTION=wiki`. Neither retrieval path ran: `qmd` and
`obsidian-wiki` are absent from `PATH`, no QMD MCP tool is registered in this
session, and the vault directory returns `Operation not permitted` both
sandboxed and unsandboxed, a macOS file-access restriction on this process
rather than a missing vault. The grep fallback is blocked the same way, so this
leg is a tooling gap, not evidence the wiki is silent on retention.
**What other products shipped: consulted.** Queried the `saas-kb` server
(`search_fsm_docs`, `category: "ai_sdlc"`) three times: run-history retention and
database growth; session history retention and automatic deletion; and
scheduled-automation run-history limits. Relevance was low across all three,
itself a finding about corpus coverage. Two useful hits: **GitLab Duo** deletes
sessions 30 days after last activity, which anchors the default window here; and
**the Claude apps gateway** documents four tables with per-table windows
enforced by one hourly sweep, marking one table "until deleted via the API"
instead of giving it a window, which is the split this document draws between
history rows and live-state rows.
Neither previewed before a policy's first deletion, nor warned ahead of the
window. Those are where this capability goes further, because an upgrade that
silently deletes a year of history on its first sweep is the failure mode a
shipped default carries.
docs/specs/office/system-design/run-history-retention-operations.md
+379
View File
@@ -0,0 +1,379 @@
---
status: current
system: office
requirements:
- REQ-OFFICE-RUN-HISTORY-RETENTION-003
- REQ-OFFICE-RUN-HISTORY-RETENTION-004
---
# Office Run History Retention Operations System Design
## Purpose and boundaries
This design covers the operator half of run history retention: the settings
record, the per-table preview marker, the reporting value, the health check, and
the read/write System page surface. The sweep itself, the eligibility
predicates, the batching, and the engine-parity guarantees are designed in [run
history retention](run-history-retention.md).
The two documents share one component, `internal/office/retention`, and one
scheduler goroutine. They are split because they are two contracts with two
consumers: a scheduler deleting rows, and an operator reading a page. The split
also keeps each document inside the specification size limit without cutting
either contract.
Adjacent contracts read and constrained but not owned:
- `internal/health` — `Checker`, `Issue`, and the `/api/v1/system/health`
response the System page's health card renders.
- `internal/system/settings.Store` — the key/value settings table, already used
by storage maintenance under one JSON key with normalization on read.
## Component: the operator surface
### Settings
One `settings` key, `office_run_retention`, holding a JSON document,
read through a `Get`/`Save` pair with `Normalize` on both, matching
`internal/system/storage/settings.go`. Unparseable content returns the defaults
plus a sentinel error the caller turns into a health issue, never a boot failure
(AC-OFFICE-RUN-HISTORY-RETENTION-004.4).
```json
{
"enabled": true,
"sweep_interval_hours": 6,
"batch_limit": 5000,
"routine_runs": { "window_days": 30, "floor_per_owner": 50, "warn_rows": 25000 },
"runs": { "window_days": 30, "floor_per_owner": 50, "warn_rows": 25000 },
"run_events": { "warn_rows": 250000 }
}
```
`run_events` carries only a threshold: its lifetime is its run's, so it has no
window and no floor of its own. Ranges and rejection behavior are
AC-OFFICE-RUN-HISTORY-RETENTION-004.2 and .3; validation returns a field-named
error and writes nothing, as `validateRange` does for storage maintenance.
Writes are last-writer-wins through `Store.Save`; `CompareAndSwap` is not used
because these are operator-scale settings edited from one page, and an identical
repeated write is indistinguishable from no write
(AC-OFFICE-RUN-HISTORY-RETENTION-004.5).
**Each sweep re-reads the settings from the store at its start, and skips if that
read fails.** The buffered
wake channel that re-arms the interval is process-local, so on a deployment where
several backends share one PostgreSQL database — the deployment
AC-OFFICE-RUN-HISTORY-RETENTION-002.12 exists for — it only ever reaches the
process that served the `PUT`. A backend relying on a cached effective value
could win the sweep lock still holding the policy the operator has just replaced
and delete rows the new policy retains. Re-reading costs one indexed key lookup
per sweep interval, against the risk of deleting history under a window the
operator already widened. The wake channel keeps its job — re-arming the timer
promptly in the process that saw the change — and stops being the only path by
which a change reaches a sweep.
A failed or unparseable re-read **skips the sweep** rather than falling back to
the documented defaults. Everywhere else in this design an unreadable settings
document yields the defaults and a health issue
(AC-OFFICE-RUN-HISTORY-RETENTION-004.4), which is right for reporting and for
startup because neither deletes anything. It is wrong here: the default window is
30 days, an operator who widened theirs to 3,650 has by definition configured
something longer, and falling back would delete the decade of history they
configured the system to keep. Reading settings to *show* them fails open;
reading them to *delete by* fails closed.
### The preview marker
A second `settings` key, `office_run_retention_preview_completed`,
holding a JSON object keyed by swept table name whose values are the timestamp
at which that table's preview completed:
`{"office_routine_runs": "...", "runs": "..."}`.
The marker is **per table, not per database**. A single global flag is unsafe
against AC-OFFICE-RUN-HISTORY-RETENTION-002.7, which lets one table fail while
the sweep as a whole still completes: if `runs` errored during the preview sweep
and `office_routine_runs` succeeded, a global flag would be written anyway and
`runs` would delete for real on the next sweep having never shown an operator a
would-delete count — the exact failure this capability exists to prevent. With a
per-table marker, `runs` is simply previewed again next sweep
(AC-OFFICE-RUN-HISTORY-RETENTION-003.4).
A table's entry is written only when that table's preview evaluation completed
successfully, is never cleared by a settings change or a restart, and is not
written by a deleting sweep. A table whose preview finds nothing still gets its
entry, so its next sweep deletes normally.
If the key is present but unparseable, every swept table is treated as **not yet
previewed** and `office_retention_preview_unreadable` is raised
(AC-OFFICE-RUN-HISTORY-RETENTION-003.10). The asymmetry is deliberate: a
spurious re-preview deletes nothing and costs one sweep, whereas assuming a
preview had completed permits a first deletion no operator ever saw. The safe
direction is the one that cannot delete.
The preview's per-table counts are uncapped by the batch limit
(AC-OFFICE-RUN-HISTORY-RETENTION-003.9): capping them would report 5,000 to an
operator holding 200,000 eligible rows, which is exactly the number the warning
exists to convey.
### Reporting
An in-memory `LastSweep` value replaced wholesale at the end of each sweep that
actually ran: start and finish times, and per **reported** table the deleted
count, a backlog flag, and an error string. Deleted counts are counted from
**committed** transactions only: when a `runs` batch is abandoned and rolled back
(AC-OFFICE-RUN-HISTORY-RETENTION-002.7), the three satellite tables whose rows
that transaction addressed report **zero** deleted for that sweep, not the counts
their statements returned before the rollback. The failure is recorded against
`runs`, but the satellites must not show rows the rollback restored — the one
surface an operator has for "what actually happened" would otherwise be wrong
precisely in the failure case it exists for. Per **swept** table it also carries
whether that table was previewed in this sweep and, when it was, its
`WouldDelete` count — without that field the preview's headline numbers, which
AC-OFFICE-RUN-HISTORY-RETENTION-003.2 and
AC-OFFICE-RUN-HISTORY-RETENTION-003.9 require an operator to see, would have
nowhere to be read from. Because the preview marker is per table, `preview` is a
per-table flag rather than one flag for the sweep. Skips are held separately and
never overwrite this value. Nothing is persisted
(AC-OFFICE-RUN-HISTORY-RETENTION-004.6, and the "no persisted sweep history"
exclusion: a growing table recording the work of the job that stops tables
growing is the same defect in a new place). Before the first sweep the value is
absent, and the surface says so rather than rendering zeros
(AC-OFFICE-RUN-HISTORY-RETENTION-004.7).
**Retained counts do not come from `LastSweep`.** They are a property of the
table, not of a sweep, and three ACs need them when no sweep has run at all:
AC-OFFICE-RUN-HISTORY-RETENTION-003.7 (threshold warning while retention is
disabled), AC-OFFICE-RUN-HISTORY-RETENTION-004.8 (surface readable while
disabled), and AC-OFFICE-RUN-HISTORY-RETENTION-002.8 (disabled means no sweep,
yet counts are still reported). A separate `RetainedCounts` value therefore
holds one count per thresholded table — produced by the census described next —
plus, for `office_routine_runs`, the top routine by retained rows for
AC-OFFICE-RUN-HISTORY-RETENTION-003.5's attribution.
**The count evaluation is a status census, not a bare `COUNT(*)`.** For each
swept table it issues one
`SELECT status, COUNT(*) FROM <table> GROUP BY status`. The retained count is the
sum of those rows, so the census costs one scan rather than two, and the same
result set is what detects a status in neither the history nor the live-state set
and raises `office_retention_unknown_status:<table>`
(AC-OFFICE-RUN-HISTORY-RETENTION-001.10). That detector has to live here rather
than in the sweep: the sweep's predicate selects `status IN (<history statuses>)`
and therefore structurally cannot observe a status it does not select.
`run_events` is thresholded but not swept and has no status column, so it keeps a
plain `COUNT(*)`.
It is refreshed on the sweep interval by the same scheduler goroutine — on that
schedule whether or not retention is enabled, since a disabled install is
precisely the one whose tables grow unattended, and is also the only install
where an unrecognized status would otherwise never be noticed — and served from
memory in between.
**The first evaluation runs at `Start`, not at the first sweep.** The first sweep
is deliberately delayed five minutes (AC-OFFICE-RUN-HISTORY-RETENTION-002.10);
hanging the first count off that tick would leave every restart with five minutes
of absent counts, and a fresh install with none at all until it had swept once —
while AC-OFFICE-RUN-HISTORY-RETENTION-003.7 and -004.8 require the threshold
warning and the surface to work on exactly those installs.
It runs **on the scheduler goroutine, not on the caller of `Start`**. The census
is three unbounded scans of the largest tables in the database, and the install
that most needs them is the one where they take longest; blocking `Start` on them
would make bounding these tables a cause of slow boots. `Start` returns
immediately, the tables read *not yet computed* until the first census lands
seconds later, and that state is already required and already renders honestly.
`RetainedCounts` is therefore **tri-state per thresholded table**, not a number
that defaults to zero: *not yet computed*, *fresh as of T*, or *stale as of T*.
Zero is a real measurement and must not be how "nobody has counted yet" renders —
the same argument AC-OFFICE-RUN-HISTORY-RETENTION-004.7 makes for `LastSweep`,
and the reason AC-OFFICE-RUN-HISTORY-RETENTION-003.11 states it. The three states
are tracked **per table**, so one table's failing query neither discards nor
staleness-marks another's successful one; a failure with no prior success leaves
that table at *not yet computed* rather than fabricating a zero.
The evaluation is explicitly **not** computed per health poll or per page load:
`/api/v1/system/health` is polled by every open browser tab, and issuing three
unbounded `COUNT(*)`s against the largest tables in the database on each poll
would make this feature a cause of the load it exists to prevent
(AC-OFFICE-RUN-HISTORY-RETENTION-003.11).
Three surfaces, in descending durability:
1. **Structured logs.** One `info` line per sweep. One `warn` line per condition
in REQ-OFFICE-RUN-HISTORY-RETENTION-003, carrying the table, the counts, and
the threshold. Always on, in every profile.
2. **Health issues.** The package implements `health.Checker` with
`Name() = "Office run retention"` and `Category() = "office"`, returning a
`health.Issue` per active condition with
`FixURL = "/settings/system/data-storage"`, which is the live route
registered in `apps/web/src/settings-routes.tsx` — note the suffix, as
`/settings/system/data` is not a registered path and `/settings/system/database`
is only a redirect to it. `internal/health/checks_test.go` pins fix URLs to
live routes ("expectedGitHubFixURL is the live route in ..."), and this one
is pinned the same way. This is the production-visible surface
required by AC-OFFICE-RUN-HISTORY-RETENTION-003.8 and is why the debug
metrics endpoint is not it: `/debug/vars` is gated on the dev profile.
Issue ids are stable and one per condition:
`office_retention_preview_pending`, `office_retention_backlog:<table>`,
`office_retention_threshold:<table>`, `office_retention_disabled:<table>`,
`office_retention_settings_invalid`, `office_retention_failed:<table>`,
`office_retention_preview_unreadable`, and
`office_retention_unknown_status:<table>`.
`Check` returns issues sorted by id, matching `storage.Runtime.Check`, so the
health card's order is stable across polls.
For `office_routine_runs`, the threshold issue's message names the routine
holding the largest share of retained rows and that share
(AC-OFFICE-RUN-HISTORY-RETENTION-003.5) — attribution computed from the same
retained-count query, not a second detector.
3. **expvar.** Counters under `office_retention_*` following
`office/scheduler/metrics_vars.go`. Development convenience only; nothing in
REQ-OFFICE-RUN-HISTORY-RETENTION-003 depends on it.
### HTTP and frontend
`GET /api/v1/system/retention` returns effective settings, `LastSweep`, the
separately-held skip record, and `RetainedCounts`.
`PUT /api/v1/system/retention` **replaces the whole document**; it is not a
merge patch. A field the caller omits takes its documented default rather than
its stored value, so the same request body always yields the same stored
document and a repeated identical write is genuinely a no-op — which is what
AC-OFFICE-RUN-HISTORY-RETENTION-004.5's "identical repeated write changes
nothing and returns the same normalized document" requires. A merge would break
that: under a merge, whether a request is a no-op depends on what was stored
before it. An unrecognized field, a numeric field whose value is not a whole
number, a field present with a JSON `null`, and a field whose value is of the
wrong type, are each rejected with a field-named 400 and nothing is written
(AC-OFFICE-RUN-HISTORY-RETENTION-004.9); rejecting rather than ignoring an
unknown field means a client that misspells `window_days` is told so instead of
silently getting the default. `null` needs saying because this endpoint is a full
replace: since an omitted field deliberately means "take the default", the
tempting reading of `null` is the same one, and that reading turns
`{"runs": {"window_days": null}}` into a silent 3,650-to-30-day reduction that
destroys a decade of history on the next sweep. Decode into pointer fields and
reject an explicit null, rather than into value fields where `null` and omission
are indistinguishable. Successful writes return the normalized document.
Both routes are admin-scoped like the other System routes and are readable while
retention is disabled (AC-OFFICE-RUN-HISTORY-RETENTION-004.8).
One card on **Settings > System > Data & Logs**, the page served at
`/settings/system/data-storage` and rendered by
`apps/web/components/settings/system/data-logs-settings.tsx`, beside
`database-stats-card.tsx`: the enable toggle, the numeric fields, and the
last-sweep readout. It follows the storage-maintenance cards' shape. All new
copy goes through `t()` and must ship in `pt-pt`, `zh-cn`, `zh-hk`, and `zh-tw`;
`pnpm run i18n:check` and the new-code ratchet gate the build. Health issue
titles and messages stay English, matching every other backend-produced
`health.Issue`.
## Ordering, concurrency, and failure
| Question | Answer | AC |
|---|---|---|
| Concurrent settings writes | Last-writer-wins; an identical repeat changes nothing | 004.5 |
| Settings unreadable | Defaults used, health issue raised, sweep proceeds | 004.4 |
| Settings out of range | Rejected with the field named, stored settings unchanged | 004.3 |
| Retention disabled | No sweep, no deletion; counts and threshold warnings still reported | 002.8, 003.7, 004.8 |
| Preview scope | Per swept table, not per database; a table that failed its preview is previewed again | 003.4 |
| Preview marker unreadable | Treated as not previewed; re-preview deletes nothing | 003.10 |
| Retained counts | Table property from a `GROUP BY status` census; first evaluation at `Start`, then on the sweep interval; served from memory; computed even while disabled | 003.11 |
| Counts before the first evaluation, or first evaluation fails | Reported *not yet computed* per table, never as zero; state tracked per thresholded table | 003.11, 004.7 |
| Unknown status detected | By the census, not the sweep predicate; one issue per table listing every unrecognized status in ascending order | 001.10 |
| Preview on a table with more eligible rows than the batch limit | Reports the full eligible count; never reported as backlog | 003.6, 003.9 |
| Settings changed on another backend | Each sweep re-reads settings at its start, so a process that did not serve the write still uses the new policy | 004.5 |
| Settings unreadable at sweep start | Sweep skipped and recorded as skipped; never run under the shorter default window | 004.5, 004.4 |
| Batch abandoned and rolled back | Satellite tables report zero deleted, not their pre-rollback statement counts | 002.7, 004.6 |
| Settings write shape | Full replace; omitted field takes its default; unknown field 400 | 004.9 |
## Testing
Unit tests in `internal/office/retention`, plus the frontend checks below.
- The first sweep on a seeded database deletes nothing and reports a
would-delete count; the second deletes (003.1, 003.4).
- A preview on a database with more eligible rows than the batch limit reports
the full eligible count (003.9).
- A table that errors during the sweep in which it was being previewed is
previewed again on the next sweep instead of deleting, while a sibling table
that succeeded proceeds to delete (003.4).
- A preview marker that is present but unparseable causes a re-preview, not a
deletion, and raises `office_retention_preview_unreadable` (003.10).
- Retained counts and the threshold warning are produced on a fresh install
before any sweep, and while retention is disabled (003.7, 003.11, 004.8), and
a health poll does not issue a table count.
- Counts are available immediately after `Start`, without advancing any clock to
the first sweep's delay (003.11). A test that waits out the delay would pass
against an implementation that hangs the first count off the sweep tick, which
is the defect this asserts against.
- Before the first evaluation, and when the first evaluation fails with no
earlier success, the surface reports the table as *not yet computed* and not as
zero (003.11, 004.7).
- With one thresholded table's count query failing and the other two succeeding,
the two keep fresh counts and only the failing one is marked, and the threshold
warnings derived from the successful counts still fire (003.11).
- A swept table holding a status in neither status set raises
`office_retention_unknown_status:<table>` from the census while retention is
**disabled** and no sweep has ever run (001.10, 003.11).
- A preview on a table with more eligible rows than the batch limit reports the
full eligible count and raises no backlog warning (003.6, 003.9).
- A settings change written through one store handle is used by a sweep driven
from a second handle that never saw the change notification (004.5).
- A backend census refresh adopts a settings change written by another backend,
including when retention was disabled, and arms the appropriate sweep timer
(002.13, 004.5).
- A sweep whose settings read fails is skipped and recorded as skipped, and
deletes nothing under the default window (004.5).
- A `runs` batch abandoned after its retry reports zero deleted for the three
satellite tables rather than their pre-rollback counts (002.7, 004.6).
- A recorded skip leaves the previous `LastSweep` readable and unchanged (002.2,
004.7).
- A settings write omitting a field stores that field's default; a write with an
unknown field or a fractional number is rejected 400 naming the field and
stores nothing; the same write applied twice is a no-op (004.9).
Commands:
```
cd apps/backend
go test ./internal/office/... -race -count=1
KANDEV_TEST_POSTGRES_DSN=<dsn> go test -race ./internal/office/... -count=1
cd apps/web && pnpm run typecheck && pnpm run i18n:check
```
## Rejected alternatives
- **One global "preview completed" flag.** Simpler, and unsafe: because
AC-OFFICE-RUN-HISTORY-RETENTION-002.7 lets one table fail while the sweep
completes, a global flag is written even when a table never got its preview,
and that table then deletes for real having shown the operator nothing. The
per-table marker costs one JSON object.
- **Compute retained counts inside the health check.** Direct, and it puts three
unbounded `COUNT(*)`s on a route every open browser tab polls. Refreshing on
the sweep interval and serving from memory gives the same number without
making the bound-the-tables feature a source of load on those tables.
- **A merge-patch settings write.** Under a merge, whether a request is a no-op
depends on what was stored before it, which
AC-OFFICE-RUN-HISTORY-RETENTION-004.5 forbids.
- **Deletion off by default.** Safe, and it means the gap stays open on every
install that never visits the settings page. The per-table preview gives the
same protection without that outcome.
- **Report warnings only through `/debug/vars`.** That endpoint is gated on the
dev profile, so on a production build the warnings would not exist
(AC-OFFICE-RUN-HISTORY-RETENTION-003.8).
## Prior art, applied
`internal/system/storage` supplies the settings storage and normalization
pattern, the hours-based interval with min and max bounds, and the
`health.Checker` route to a production-visible warning; `storage.Runtime.Check`
also sorts its issues by id, which this check matches so the health card's order
is stable across polls.
Neither GitLab Duo nor the Claude apps gateway previews before a policy's first
deletion or warns ahead of the window. Those are this capability's additions,
and they are the whole reason this document is separate from the sweep's.
docs/specs/office/system-design/run-history-retention.md
+571
View File
@@ -0,0 +1,571 @@
---
status: current
system: office
requirements:
- REQ-OFFICE-RUN-HISTORY-RETENTION-001
- REQ-OFFICE-RUN-HISTORY-RETENTION-002
- REQ-OFFICE-RUN-HISTORY-RETENTION-005
---
# Office Run History Retention System Design
## Purpose and boundaries
This design adds one scheduled sweep that deletes aged Office run history from
five tables. It changes no run lifecycle, no routine dispatch decision, and no
task, session, or checkout.
Office owns the sweep because the eligibility rules are defined by Office
primitives. The settings record, the health check, the reporting value, and the
System page surface are the operator half of the same capability and are
designed in [run history retention
operations](run-history-retention-operations.md), which is where every
requirement in REQ-OFFICE-RUN-HISTORY-RETENTION-003 and -004 is satisfied.
Adjacent contracts read and constrained but not owned:
- `internal/health` — `Checker`, `Issue`, and the `/api/v1/system/health`
response the System page's health card renders.
- `internal/system/settings.Store` — the key/value settings table, already used
by storage maintenance under one JSON key with normalization on read.
- `internal/runs/repository/sqlite` — the `runs` queue and `run_events` access
methods.
- `internal/office/repository/sqlite` — the schema owner for `runs`,
`run_events`, `office_run_route_attempts`, `office_run_skills`, and
`office_routine_runs`.
## Measured starting state
Read from the reference install's SQLite database on 2026-09-09. These numbers
set the defaults and are the baseline any regression test can be written
against.
| Table | Rows | Notes |
|---|---|---|
| `office_routine_runs` | 326 | 323 `coalesced`, 3 `task_created` |
| `runs` | 53 | |
| `run_events` | 340 | across 53 runs, mean 6.4 per run |
| `office_run_route_attempts` | 55 | |
| `office_run_skills` | 382 | |
The 323 `coalesced` rows span 2026-08-03 to 2026-08-31 and belong to a single
routine that is now `paused`. Orphan `run_events` today: zero, because nothing
has ever deleted a `runs` row.
## Schema facts this design depends on
Verified by reading the schema owners, not assumed.
- `office_routine_runs` (`office/repository/sqlite/base.go`) has
`FOREIGN KEY (routine_id) REFERENCES office_routines(id) ON DELETE CASCADE`
on both engines, and SQLite opens with `_foreign_keys=on`
(`internal/db/sqlite.go`). Its two indexes are partial and serve the dispatch
gate, not an age scan: `idx_office_routine_runs_active_fingerprint`
(`WHERE status = 'task_created'`) and `idx_office_routine_runs_linked_task`
(`WHERE linked_task_id != ''`).
- `run_events`, `office_run_route_attempts`, and `office_run_skills` declare
**no foreign key to `runs`** on either engine. Confirmed in
`office/repository/sqlite/base.go` and in the PostgreSQL conformance snapshot
`internal/persistence/storeconformance/testdata/upgrades/v0.93.0/postgres.sql`,
where `run_events` has only a primary key and one index. A cascade would
therefore delete nothing; satellite deletion must be explicit
(AC-OFFICE-RUN-HISTORY-RETENTION-005.2).
- `run_events.seq` is assigned by `AppendRunEvent` as
`COALESCE(MAX(seq) + 1, 0)` scoped to the run. Deleting a live run's whole
timeline restarts its sequence at zero, and the run detail view's incremental
tail reads `WHERE seq > afterSeq`. This is why
AC-OFFICE-RUN-HISTORY-RETENTION-001.6 forbids event deletion outside the
transaction that deletes the run.
- `runs` indexes are `idx_run_status_requested (status, requested_at)` and the
partial unique `idx_run_idempotency`. Nothing indexes `finished_at`.
- `office_agent_pause_recoveries.failed_run_id` identifies failed runs that an
active pause recovery still needs. It has no foreign key, so retention uses a
correlated `NOT EXISTS` predicate and the supporting
`idx_office_agent_pause_recoveries_failed_run` index.
- `ScheduleRetry` (`runs/repository/sqlite/runs.go`) sets
`status = 'queued', finished_at = NULL` on an existing run, keyed by id with
**no status guard in its `WHERE` clause**. Any terminal run can therefore
become live again at any moment, which is the race
AC-OFFICE-RUN-HISTORY-RETENTION-002.4 closes. Because the guard is absent, a
`cancelled` row is resurrectible on exactly the same terms as a `failed` one,
so admitting `cancelled` to the history set adds no new race — it is covered
by the same re-assertion.
- `CancelRunsWhere` (`runs/repository/sqlite/cancel.go`) is documented as "the
single writer of the terminal cancel state on the runs table". It sets
`status = 'cancelled', cancel_reason = ?, finished_at = ?` and is guarded by
`status IN ('queued', 'claimed')`, so a cancelled row is always terminal and
always carries a completion timestamp. It is production-reachable through
`CancelRunsForTasks` from `office/service/tree_controls.go` and
`office/repository/sqlite/participants.go`. It writes the literal string, so
`office/models/enums.go`'s four-value `RunStatus` block does not enumerate it:
the database has five `runs` statuses, not four. This is why
AC-OFFICE-RUN-HISTORY-RETENTION-001.2 classifies `cancelled` as history and
AC-OFFICE-RUN-HISTORY-RETENTION-001.10 makes any sixth value fail safe.
- Every production writer of a terminal `runs` status stamps the completion
timestamp: `FinishRun` sets `finished_at = now`, `MarkRunFailed` sets
`finished_at = COALESCE(finished_at, now)`, and `CancelRunsWhere` sets it
outright. Only the test helper `SetRunStatusForTest` can leave a terminal row
with a null `finished_at`. The `COALESCE(finished_at, requested_at)` fallback
in AC-OFFICE-RUN-HISTORY-RETENTION-001.3 is therefore defensive rather than a
routine path — but it is also what makes the ordering key non-null, which is
what keeps the floor deterministic across engines (see below).
- `runs.requested_at` and `office_routine_runs.created_at` are both
`TIMESTAMP NOT NULL`, so the completion-time expression is total. This matters
for parity, not just tidiness: SQLite and PostgreSQL differ in where they sort
nulls by default, so an `ORDER BY` over a nullable timestamp would rank the
floor differently on the two engines from identical data.
- `CleanExpired` (`runs/repository/sqlite/runs.go`) already deletes terminal
`runs` rows older than a cutoff and **has no production caller** — only
tests reach it. It deletes only `runs`, so wiring it as-is would orphan every
satellite row. It is superseded by the batched, satellite-aware delete below
rather than reused.
## Component: `internal/office/retention`
One package holding policy, settings, the sweep, and the health check.
### Ownership and lifecycle
A single goroutine owner modelled on `internal/system/storage.Scheduler`: a
`Start(ctx)` that is a no-op when already running, a `Stop()` that cancels and
joins, and a buffered wake channel so a settings change re-arms the interval
without interrupting a sweep in progress
(AC-OFFICE-RUN-HISTORY-RETENTION-004.5). `Stop` is joined from the same place
that stops the Office scheduler.
The loop is deliberately not the Office run-processing tick
(`office/service/scheduler_integration.go`, `DefaultTickInterval = 5s`). That
tick already carries `RecoverStale` and `ReapStaleCheckouts` unthrottled and is
the run-claim hot path; on SQLite a bulk delete there contends with the single
writer that claims runs, 17,280 times a day, for an input that changes on a
scale of days (AC-OFFICE-RUN-HISTORY-RETENTION-002.1).
A `sweeping bool` guarded by the same mutex makes a due sweep a skip rather than
a second goroutine (AC-OFFICE-RUN-HISTORY-RETENTION-002.2). That guard is
process-local, which is sufficient on SQLite, where the database file is owned
by one backend process. On PostgreSQL, where several backends can share one
database, it is not: two backends would each see `sweeping == false` and sweep
the same rows concurrently.
The sweep therefore also takes a PostgreSQL advisory lock, and it must be a
**session-scoped, non-blocking** one, which is a different shape from every
existing advisory lock in this repository:
```
conn := db.Conn(ctx) // one dedicated connection
SELECT pg_try_advisory_lock(:retention_key) // boolean, returns immediately
... whole sweep, every table, every batch, on the pool ...
SELECT pg_advisory_unlock(:retention_key) // on that same connection
conn.Close() // deferred
```
Both properties are load-bearing and neither is optional:
- **Session-scoped, not transaction-scoped.** The sweep is multi-transaction by
construction: AC-OFFICE-RUN-HISTORY-RETENTION-002.5 makes each batch its own
transaction, and AC-OFFICE-RUN-HISTORY-RETENTION-002.7 requires one table's
failure to leave another table's committed deletions intact, which forbids
wrapping the sweep in a single transaction. A `pg_advisory_xact_lock` is
released when its transaction ends, so it would protect one batch and then let
a second backend in between tables — the interleaving
AC-OFFICE-RUN-HISTORY-RETENTION-002.12 exists to prevent. The lock is held on a
connection checked out for the sweep and released in a `defer`; the batches
themselves continue to use the pool.
- **`try`, not the blocking form.** AC-OFFICE-RUN-HISTORY-RETENTION-002.12 says a
backend that cannot acquire the lock *skips*. `pg_advisory_xact_lock` and
`pg_advisory_lock` wait instead of failing, which would convert a concurrent
sweep into a queued one and eventually stall the scheduler behind a long sweep.
`pg_try_advisory_lock` returns `false` immediately; on `false` the backend
records a skip and returns, exactly as for a local concurrent sweep.
This deliberately departs from the established repo pattern
`SELECT pg_advisory_xact_lock(hashtextextended(?, 0))`
(`office/repository/sqlite/participants.go`, `internal/secrets/sqlite_store.go`,
`internal/workflow/repository/phase2_sqlite.go`), and the departure is the point:
each of those call sites performs its entire unit of work inside the one
transaction that holds the lock, and each *wants* to wait rather than skip. The
sweep can do neither. The key is derived the same way, `hashtextextended` over a
constant distinct from every key those sites use, so retention never contends
with participant-seat, secret-transfer, or workflow-phase locking.
**If the lock connection drops mid-sweep**, PostgreSQL releases the session's
advisory locks as part of ending the session, so a crashed or partitioned backend
cannot wedge retention permanently — that self-healing is the reason for a
session lock rather than a lease row in the `settings` table, which would need its
own expiry and its own stale-holder rule. The cost is that the surviving sweep no
longer holds exclusivity without knowing it, so the sweep **verifies the lock
connection is still alive between tables** and, if it is not, stops before the
next table, records that table as skipped rather than failed, and returns. It
does not attempt to re-acquire mid-sweep: a re-acquisition after another backend
has taken the lock would produce exactly the concurrent sweep this protects
against. Batches already committed stay committed, which
AC-OFFICE-RUN-HISTORY-RETENTION-002.5 permits.
A skip is recorded as its own value — a skip counter and a last-skip timestamp —
and does **not** overwrite `LastSweep`. `LastSweep` holds the last sweep that
actually ran, so a burst of skips cannot blank the operator's view of the last
real result (AC-OFFICE-RUN-HISTORY-RETENTION-002.2, and
AC-OFFICE-RUN-HISTORY-RETENTION-004.7, which forbids rendering a
never-swept-looking surface when a sweep has in fact run).
The first sweep after `Start` is armed at a fixed 5-minute delay rather than a
full interval (AC-OFFICE-RUN-HISTORY-RETENTION-002.10). Arming at the interval,
as the storage scheduler does with its 24-hour default, means an install
restarted more often than the interval never sweeps at all; 5 minutes keeps
startup clear of schema init and run recovery without depending on uptime. Each
sweep computes `time.Now().UTC()` once and passes that instant to every table,
so two tables in one sweep cannot disagree about the cutoff
(AC-OFFICE-RUN-HISTORY-RETENTION-002.11).
Scheduling is **fixed-delay, not fixed-rate**: the next sweep is armed when the
previous one returns, so a sweep that overruns its interval delays its successor
instead of causing an immediate second one (which the concurrency guard would
only skip anyway, turning a slow sweep into a stream of skips). A settings
change re-arms the delay from the moment of the change rather than from the last
sweep, and enabling retention that was disabled arms at the same 5-minute delay
as a fresh start rather than at a full interval — otherwise an operator who
enables retention on a 168-hour interval waits a week to see whether it works
(AC-OFFICE-RUN-HISTORY-RETENTION-002.13).
The census timer also re-reads shared settings. It runs while retention is
disabled, so other backends discover enablement and interval changes and re-arm
their timers.
### Eligibility, expressed once
Two predicates, each defined in exactly one place and reused by the count, the
preview, and the delete.
**Routine runs.** History statuses are `skipped`, `coalesced`, `failed`, `done`,
`cancelled`. `received` and `task_created` are absent by construction, which is
what makes AC-OFFICE-RUN-HISTORY-RETENTION-005.3 hold: the dispatch gate
`GetActiveRunForFingerprint` reads only `status = 'task_created'`, so no sweep
can change its answer.
```
DELETE FROM office_routine_runs
WHERE id IN (
SELECT id FROM (
SELECT id,
ROW_NUMBER() OVER (
PARTITION BY routine_id
ORDER BY COALESCE(completed_at, created_at) DESC, id DESC
) AS rn
FROM office_routine_runs
WHERE status IN (<history statuses>)
) ranked
WHERE rn > :floor
AND completion_time < :cutoff
ORDER BY completion_time ASC, id ASC
LIMIT :batch
)
AND status IN (<history statuses>)
AND COALESCE(completed_at, created_at) < :cutoff
```
where the ranked subquery also projects
`COALESCE(completed_at, created_at) AS completion_time`.
Two orderings appear here and they are not the same ordering; conflating them is
the defect this section exists to prevent.
- The **`ORDER BY` inside `ROW_NUMBER()`** ranks rows *within* a partition so the
floor keeps the newest per owner: `completion_time DESC, id DESC`
(AC-OFFICE-RUN-HISTORY-RETENTION-001.4).
- The **`ORDER BY` on the outer select** decides *which* eligible rows a
batch-limited sweep takes: `completion_time ASC, id ASC`, oldest first
(AC-OFFICE-RUN-HISTORY-RETENTION-002.3). The window function's ordering does
not reach the outer `LIMIT`, so without this clause the engine is free to
return any subset and the two engines may drain a backlog differently from
identical data — which would contradict
AC-OFFICE-RUN-HISTORY-RETENTION-005.1 and make the parity test below flake for
a reason unrelated to a genuine engine difference.
`id` is a UUID and so is not chronological; it is used only as a total, stable
tiebreak for equal timestamps, in both orderings.
The trailing `AND` clauses are the re-assertion required by
AC-OFFICE-RUN-HISTORY-RETENTION-002.4. For this table the whole ranked subquery
is re-evaluated inside the `DELETE`, so the floor is re-asserted atomically along
with status and age; the two-phase `runs` path below has to do that explicitly.
Window functions are available on both engines: SQLite 3.54.0 through
`github.com/mattn/go-sqlite3 v1.14.33`, verified by running this exact
`ROW_NUMBER() OVER (PARTITION BY routine_id ...)` against the reference
database.
**Runs.** History is `status IN ('finished','failed','cancelled')`, partitioned
by `agent_profile_id`, ranked `COALESCE(finished_at, requested_at) DESC, id DESC`
and batch-ordered `COALESCE(finished_at, requested_at) ASC, id ASC`. A failed
run named by an active `office_agent_pause_recoveries.failed_run_id` is
protected by a `NOT EXISTS` clause in this predicate. Count, selection, and
delete use the same clause, so a preview cannot promise deletion of a run that
the recovery flow still needs. There is no
`finished_at IS NOT NULL` conjunct: requiring one would make a terminal row with
an unset stamp immortal and unobservable, which
AC-OFFICE-RUN-HISTORY-RETENTION-001.3 forbids. `queued` and `claimed` are
absent, which covers a routing-parked run whose `earliest_retry_at` is far in
the future (AC-OFFICE-RUN-HISTORY-RETENTION-001.2).
A status in neither set is treated as live state and raises
`office_retention_unknown_status:<table>`
(AC-OFFICE-RUN-HISTORY-RETENTION-001.10). Both sets are closed and asserted
against `office/models/enums.go` plus the literal-SQL writers in a test, so a
sixth status cannot enter the database without failing that test — the failure
mode that hid `cancelled` in the first place.
**That warning needs a producer, and the eligibility predicate cannot be it.**
The predicate selects `status IN (<history statuses>)`, so a row holding an
unrecognized status is never selected, never counted, and never seen: a fail-safe
whose only detector is a CI test fires on the developer's machine and stays
silent on the install that actually has the row. The producer is instead the
**status census** — one
```
SELECT status, COUNT(*) FROM <swept table> GROUP BY status
```
per swept table, run by the retained-count evaluation described in [run history
retention operations](run-history-retention-operations.md#reporting). Three
consequences follow from siting it there rather than in the sweep, and each one
closes a hole:
- The census **subsumes the retained count** rather than adding a second scan:
the retained count is the sum of the census rows, so one `GROUP BY` yields
both. (`run_events` is thresholded but not swept and has no status column, so
it keeps a plain `COUNT(*)`.)
- It runs **whether or not retention is enabled**, because the count evaluation
does (AC-OFFICE-RUN-HISTORY-RETENTION-003.11). A disabled install is precisely
where an unrecognized status would otherwise never be noticed, since
AC-OFFICE-RUN-HISTORY-RETENTION-002.8 means no sweep runs there at all.
- It sees a status **because the row exists**, not because the row was
selectable, which is the property the eligibility predicate structurally
cannot have.
One issue per table, not one per status: a table with several unrecognized
statuses raises the single id `office_retention_unknown_status:<table>` whose
message lists every unrecognized status **in ascending lexicographic order** with
its row count. Ordering is named because the message is compared across health
polls; an unordered list would make a stable condition look like a changing one.
### Deleting a run
Per batch, one transaction, satellites first, run last:
1. Select up to `batch_limit` eligible run ids, excluding runs named by an
active `office_agent_pause_recoveries.failed_run_id`.
2. `DELETE FROM run_events WHERE run_id IN (...)`
3. `DELETE FROM office_run_route_attempts WHERE run_id IN (...)`
4. `DELETE FROM office_run_skills WHERE run_id IN (...)`
5. `DELETE FROM runs WHERE id IN (:selected_ids) AND id IN (<ranked eligible
subquery>)` — the re-assertion is the **whole ranked subquery**, not just the
status and cutoff conjuncts, so the retention floor is re-evaluated at delete
time along with them. Unlike the single-statement `office_routine_runs`
delete, this path selected its ids in a separate earlier statement, so a
`ScheduleRetry` in between can re-rank a partition and push a row that was
`rn > floor` at selection to `rn <= floor` now. Re-asserting only status and
age would delete a row that has since become floor-protected
(AC-OFFICE-RUN-HISTORY-RETENTION-002.4).
Step 5 can delete fewer rows than steps 2 to 4 addressed, when a `ScheduleRetry`
resurrected a run between selection and delete. That is the correct outcome for
AC-OFFICE-RUN-HISTORY-RETENTION-002.4 only if the whole batch rolls back rather
than leaving a live run without its timeline. **The transaction is therefore
rolled back and retried once with a fresh selection when the step 5 row count
does not match the selected id count**; a second mismatch rolls back again and
abandons the batch.
An abandoned batch is recorded as a **failure** for that table, raising
`office_retention_failed:<table>`, not as backlog
(AC-OFFICE-RUN-HISTORY-RETENTION-002.7). The distinction is not cosmetic:
backlog means work correctly deferred by the batch limit and is expected on a
large install, so routing this case there would file the one genuinely dangerous
outcome under the one routine one. The table is retried on the next scheduled
sweep either way, but only the failure classification tells an operator that a
deletion could not be applied consistently.
Step 5 can only ever delete *fewer* rows than steps 2 to 4 addressed, never
more, because it is bounded by the same selected id set. This is the one place
where the naive implementation silently corrupts a live run, and it is the
reason satellite deletion is not a separate statement outside the transaction.
`run_events` is never addressed by any predicate other than membership in this
id set (AC-OFFICE-RUN-HISTORY-RETENTION-001.6). There is no age-based delete on
`run_events`.
### Indexes to add
Retention adds indexes that serve the sweep.
- `idx_office_routine_runs_retention ON office_routine_runs(routine_id, status, (COALESCE(completed_at, created_at)) DESC, id DESC)`
- `idx_runs_retention ON runs(agent_profile_id, status, (COALESCE(finished_at, requested_at)) DESC, id DESC)`
- `idx_office_agent_pause_recoveries_failed_run ON office_agent_pause_recoveries(failed_run_id)`
Added through the existing `office/repository/sqlite` schema path so both the
fresh-install `CREATE` and the upgrade path get them, and recorded in the
conformance fixtures.
## Ordering, concurrency, and failure
| Question | Answer | AC |
|---|---|---|
| Sweep ordering across tables | `office_routine_runs`, then `runs` with its satellites. Independent sets; the order is fixed only so results and logs are reproducible. | 002.6 |
| Floor ordering | `COALESCE(completed_at, created_at) DESC, id DESC` / `COALESCE(finished_at, requested_at) DESC, id DESC` | 001.4 |
| Two sweeps due at once | Second is skipped, not queued, and recorded as skipped | 002.2 |
| Sweep vs. live writer | Selection is a snapshot; status, age **and floor** are re-asserted in the delete; a batch whose step 5 count disagrees is rolled back | 002.4, 002.5 |
| Sweep interrupted | Each batch is one transaction; applied batches are whole, the interrupted one is not applied | 002.5 |
| First sweep after startup | Armed at a fixed 5-minute delay, not a full interval | 002.10 |
| Cutoff instant | Computed once per sweep, shared by every table | 002.11 |
| Re-run once no eligible rows remain | Deletes nothing further, reports zero. While backlog remains, the next sweep deletes the next batch by 002.3 — the two are not in tension because 002.6 is scoped to the drained state | 002.6, 002.3 |
| One table errors | That table is recorded failed; remaining tables continue; retried next sweep; startup unaffected | 002.7 |
| Empty tables | Zero deletions, no error, no warning | 002.9 |
| Deleted coalesce target | Allowed; `coalesced_into_run_id` is provenance, read by nothing | 001.9 |
| Routine paused | Irrelevant to eligibility | 001.7 |
| Cancelled run | History, like `finished`/`failed`; its writer stamps the completion timestamp | 001.2 |
| Terminal row, null completion stamp | Still history; dated by `created_at` / `requested_at` | 001.3 |
| Status in neither set | Treated as live state, never deleted, warned | 001.10 |
| Which rows a batch-limited sweep takes | `completion_time ASC, id ASC` — oldest first, named columns | 002.3, 005.1 |
| Batch abandoned after retry | Recorded as that table's failure, not as backlog | 002.7 |
| Preview finds more eligible rows than the batch limit | Not backlog. A preview deletes nothing by design, so "retention is behind" would be false; it reports the full eligible count instead | 002.3, 003.6, 003.9 |
| Sweep skipped | Recorded separately; never overwrites the last real `LastSweep` | 002.2, 004.7 |
| Two backends, one PostgreSQL | Session-scoped `pg_try_advisory_lock` held on a dedicated connection for the whole sweep; the loser skips without waiting | 002.12 |
| Lock connection drops mid-sweep | PostgreSQL releases the lock with the session; the sweep stops before the next table and records it skipped, and does not re-acquire | 002.12 |
| Scheduling model | Fixed-delay from the end of the previous sweep; a settings change re-arms from the change | 002.13 |
## Testing
Unit and repository tests in `internal/office/retention` and
`internal/office/repository/sqlite`, plus the persistence gates.
Behaviors that must have a test, because each is a way the naive implementation
is wrong:
- A `task_created` routine run older than any window survives, and
`GetActiveRunForFingerprint` still finds it (001.1, 005.3).
- A `queued` run with `finished_at` cleared by `ScheduleRetry` survives (001.2).
- A run resurrected between selection and delete leaves both the run and its
full `run_events` timeline intact (002.4, and the rollback above).
- After a sweep, no `run_events`, `office_run_route_attempts`, or
`office_run_skills` row references a missing `runs` row (001.5, 005.2).
- No `run_events` row of a surviving run is ever deleted (001.6).
- A routine with 3 history rows all older than the window keeps all 3 under a
floor of 50; a routine with 200 keeps exactly 50 (001.4).
- Two sweeps triggered concurrently produce one sweep and one recorded skip
(002.2).
- A sweep hitting the batch limit reports backlog and the next sweep continues,
and its satellite rows are deleted in full rather than capped (002.3, 003.6).
- A terminal row whose completion timestamp is unset is dated by `created_at`
or `requested_at` and ages out rather than being retained forever (001.3).
This test is only satisfiable because AC-OFFICE-RUN-HISTORY-RETENTION-001.2
classifies history by status alone; an implementation that also required
`finished_at IS NOT NULL` would retain the row forever and fail here.
- A `cancelled` run older than the window is deleted together with its satellite
rows, and a `cancelled` run inside the floor is retained (001.2). Seed it
through the real cancel path, not by writing the status directly, so the test
fails if that path stops stamping the completion timestamp.
- A `runs` row holding a status in neither the history set nor the live-state set
survives every sweep at any age and raises
`office_retention_unknown_status:runs` (001.10). Three companions: the same row
raises the same issue with retention **disabled**, where no sweep runs at all;
a table holding two unrecognized statuses raises **one** issue listing both in
ascending order with their counts; and a test asserts the two status sets
together cover every value in `office/models/enums.go` *and* every status
literal written by SQL in `internal/runs/repository/sqlite`, so a future sixth
value cannot be added silently — the exact gap through which `cancelled` was
missed.
- With more eligible rows than the batch limit, the sweep deletes the oldest
eligible rows first, and the same seed data yields the identical deleted set on
SQLite and PostgreSQL (002.3, 005.1). Without the outer `ORDER BY` this test is
the one that fails.
- A row that becomes floor-protected between selection and delete is not deleted
by the `runs` path (002.4).
- A batch abandoned after its retry is reported as that table's failure and not
as backlog (002.7).
- Two backends against one PostgreSQL run one sweep, and the loser records a
skip (002.12). The seed must give the winner **more than one table** to sweep,
so that a transaction-scoped lock — which would release between tables and let
the loser in — fails this test rather than passing it. A companion test drops
the winner's lock connection mid-sweep and asserts it stops before the next
table and does not re-acquire.
- A `runs` batch abandoned after its retry reports zero rows deleted for
`run_events`, `office_run_route_attempts` and `office_run_skills`, not the
counts the rolled-back statements addressed (002.7, 004.6).
Commands:
```
cd apps/backend
go test ./internal/office/... ./internal/runs/... -race -count=1
go run ./cmd/sqlguard ./internal
go test -race ./internal/persistence/storeconformance -count=1
KANDEV_TEST_POSTGRES_DSN=<dsn> go test -race ./internal/office/... \
./internal/persistence/storeconformance -count=1
cd apps/web && pnpm run typecheck && pnpm run i18n:check
```
The PostgreSQL line is not optional. The repository's dialect-sensitive suites
self-skip when `KANDEV_TEST_POSTGRES_DSN` is unset, so a green local run without
it proves nothing about AC-OFFICE-RUN-HISTORY-RETENTION-005.1. The engine-parity
test asserts identical deleted sets, retained sets, and counts from identical
seed data on both engines.
## Rejected alternatives
- **Wire the existing `CleanExpired` and stop.** One line, and it orphans every
`run_events` row it passes, permanently and invisibly, because no foreign key
on either engine would clean up after it.
- **Add `ON DELETE CASCADE` to the satellite tables instead.** A schema change
to three tables on two engines, requiring a table rebuild on SQLite, to avoid
three `DELETE` statements. It also hides the deletion from the code that has
to count it for the sweep report.
- **Age-prune `run_events` directly.** The obvious implementation, and the one
that breaks the run detail view's incremental tail and can restart a live
run's sequence at zero. Forbidden by 001.6.
- **Run the sweep on the 5s Office tick.** Rejected in 002.1.
- **Count-only retention, keep newest N per owner with no window.** Bounds the
table but makes "how long is my history kept" unanswerable for an install
with mixed routine frequencies. The floor covers the case count-only is good
at; the window covers the case it is bad at.
- **Deletion off by default.** Safe, and it means the gap stays open on every
install that never visits the settings page. The per-table preview, designed
in [run history retention
operations](run-history-retention-operations.md), gives the same protection
without that outcome.
- **Persist sweep history.** Named in the operations requirement's exclusions.
- **Rely on the window function's `ORDER BY` to order the batch.** It ranks rows
inside each partition; it does not order the rows the outer `LIMIT` draws
from. Leaving the outer select unordered makes a backlog sweep's deleted set
engine-dependent and quietly breaks
AC-OFFICE-RUN-HISTORY-RETENTION-005.1 only on installs large enough to exceed
the batch limit — the installs that need this feature most.
## Prior art, applied
**Wiki: unavailable** — receipt in the requirement document. Nothing here should
be read as departing from a wiki position, because none could be consulted.
**`internal/automation/run_retention.go`** is the closest in-repo precedent, and it
prunes worktrees rather than rows. Three things are taken from it: retention scoped
per owner rather than globally, so one noisy owner cannot evict a quiet one's
only record; a bounded sweep window so a backlog drains across sweeps instead
of walking the whole table each time; and re-checking liveness immediately
before the destructive act, which appears here as the re-asserted predicate and
the batch rollback. One thing is deliberately not taken: it hangs its sweep off
a finalization hook, which ties cleanup frequency to firing frequency and leaves
a stopped routine's history untouched forever. This design uses a clock.
**`internal/system/storage`** supplies the scheduler shape, the settings
storage and normalization pattern, the hours-based interval with min and max
bounds, and the `health.Checker` route to a production-visible warning.
**GitLab Duo** and **the Claude apps gateway** are surveyed in the requirement
document's Prior art. What this design takes from them: the 30-day default
window, and the history-vs-live-state split (their per-table windows with one
table marked "until deleted via the API" rather than given a window). Neither
previews before the first deletion; that addition is ours, and it exists because
this ships enabled by default onto installs that already hold history.