4.5 KiB
4.5 KiB
spec, created, status
| spec | created | status |
|---|---|---|
| docs/specs/tasks/system-design/runtime-cleanup.md | 2026-07-29 | implemented |
Implementation Plan: Task Cleanup Idempotency and Retry Bounds
Overview
GitHub issue #2027 is reproducible through the SQLite-backed cleanup worker:
missing worktrees and missing task-environment rows are returned as cleanup
errors, so cascade_delete jobs enter retry_wait forever. The repair first
makes those two deletion boundaries explicitly idempotent, then adds a bounded
backoff state machine so any other permanent error reaches a durable terminal
state.
Root Cause
teardownEnvironmentResourcesappendsworktree.ErrWorktreeNotFoundto its aggregate error instead of recognizing that the desired deletion state is already true.- SQLite task-environment deletion returns an unclassifiable formatted error
when the row is absent, and
cleanupTaskEnvironmentretries that error. retryTaskResourceCleanupJobalways writesretry_waitwith a fixed one-minute deadline.Attemptsis used as a claim generation but never as a retry limit.
Backend
Missing-resource contracts
- Add
ErrTaskEnvironmentNotFoundtoapps/backend/internal/task/repository/repoerrors/errors.go, re-export it fromrepository/interface.goandrepository/sqlite/errors.go, and wrap task-environment not-found results inrepository/sqlite/task_environment.go. - Update
teardownEnvironmentResourcesinservice/service_task_environments.goto treat onlyworktree.ErrWorktreeNotFoundas a successful worktree teardown. - Update
cleanupTaskEnvironmentinservice/service_tasks.goto treat onlyrepository.ErrTaskEnvironmentNotFoundfrom the row deletion as complete. Other joined teardown and persistence errors remain retryable.
Bounded cleanup retries
- Add terminal
TaskResourceCleanupStateFailedintask/models/resource_cleanup.go. - Replace the fixed reschedule in
task/service/resource_cleanup_jobs.gowith the spec's seven-step retry schedule and an eight-attempt ceiling. Attempts 1-7 writeretry_wait; attempt 8 writesfailed, preserves the final error, clearsnext_attempt_at, and emits no further automatic work. - Update
task/repository/sqlite/resource_cleanup.gosofailedcompletions receivecompleted_at; due-job queries continue selecting onlypendingandretry_wait.
Tests
- What: deleting a missing task-environment row returns a classifiable
sentinel.
File:
apps/backend/internal/task/repository/sqlite/task_environment_test.goHow: SQLite repository test assertingerrors.Is. - What: both missing-resource variants complete a
cascade_deletecleanup job successfully. File:apps/backend/internal/task/service/resource_cleanup_jobs_test.goHow: table-driven SQLite-backed service-worker test using the real job claim/completion path and a worktree destroyer fake only at the external worktree boundary. - What: non-not-found teardown failures remain retryable.
File:
apps/backend/internal/task/service/service_task_environments_test.goHow: focused service test with a generic worktree error. - What: retry delays follow the documented schedule and the eighth failure
becomes terminal.
File:
apps/backend/internal/task/service/resource_cleanup_jobs_test.goHow: table-driven delay unit test plus SQLite-backed claim-completion tests for attempts 1, 7, and 8. - What: terminal failed jobs have a completion timestamp and are never due.
File:
apps/backend/internal/task/repository/sqlite/resource_cleanup_test.goHow: SQLite repository test that completes a claimed job asfailedand queries the due inventory.
Implementation Waves And Parallel Candidates
Execution is sequential because both tasks touch the cleanup service tests and the second task builds on the error classification established by the first.
Documentation Impact
The durable task-runtime and storage-maintenance specs are amended in this design package. No public CLI, configuration, API, or UI documentation changes are required.
Risks
- Treating only typed sentinels as success is deliberate: matching error strings could hide unrelated repository or Git failures.
- Terminal failure stops automatic cleanup, so the final error and completion metadata must remain queryable. A user-facing replay action and failure UI are explicitly out of scope.