fix: keep large database startup alive (#3624)

This commit is contained in:
Carlos Florêncio
2026-09-12 16:29:38 +01:00
committed by GitHub
parent 3d3209aa1a
commit b6571a9854
58 changed files with 2781 additions and 439 deletions
+4
View File
@@ -35,6 +35,8 @@ retains ownership of its values, validation, authority, and persistence.
### Requirements
- [Startup lifecycle](requirements/startup-lifecycle.md)
- [Agent-accessible Kandev settings](requirements/agent-settings-parity.md)
@@ -80,6 +82,8 @@ retains ownership of its values, validation, authority, and persistence.
### System design
- [Startup lifecycle](system-design/startup-lifecycle.md)
- [Agent-accessible Kandev settings](system-design/agent-settings-parity.md)
- [Core settings domain adoption](system-design/agent-settings-domains.md)
@@ -65,11 +65,11 @@ resolved by explicit decision — see [Decisions](#decisions).
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.1:** `GET /health` SHALL include the running Kandev version in its JSON response body.
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.2:** The field SHALL be named **`version`**, matching `GET /api/v1/system/info`.
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.3:** The field SHALL be present in **both** the ready (200) and not-ready (503) responses, so an operator can identify the build of a backend that is stuck starting.
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.3:** The field SHALL be present in both `/health` and bootstrap `/ready` responses, so an operator can identify the build of a backend that is stuck starting.
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.4:** The field value SHALL be the same string `GET /api/v1/system/info` reports as `version` for the same running process.
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.5:** The field SHALL be served to unauthenticated callers, exactly as the rest of the `/health` payload already is. This is a deliberate, accepted disclosure — see [Security](#security-and-permissions).
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.6:** Every existing field (`status`, `service`, `mode`) SHALL retain its current name, value, and semantics. This is a purely additive change.
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.7:** The existing HTTP status semantics (200 when ready, 503 while starting) SHALL be unchanged.
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.7:** `/health` SHALL return 200 once bound, including during initialization. `/ready` SHALL return 503 until application initialization finishes.
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.8:** The desktop health-token response header SHALL continue to be set on the 200 path only, unchanged.
## System design
@@ -0,0 +1,39 @@
---
status: active
system: platform
created: 2026-09-11
owners:
- kandev
---
# Startup lifecycle requirements
## Overview
Platform owns process startup, liveness, readiness, and shutdown across native launch modes and desktop.
Large databases must complete initialization without a listener-health deadline interrupting valid work.
## Requirements
### REQ-PLATFORM-STARTUP-LIFECYCLE-001: Observable initialization
**Intent:** Separate process liveness from application readiness while preserving upgrade safety.
#### Acceptance criteria
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.1:** During database opening, backup, migration, and required recovery, `/health` shall answer successfully with existing version and identity semantics.
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.2:** Until required initialization and router installation finish, `/ready` and application endpoints shall remain unsuccessful.
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.3:** After liveness succeeds, the launcher shall wait for readiness without an initialization deadline. Explicit listener-timeout overrides shall remain effective.
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.4:** Startup shall report phase transitions and elapsed time, with bounded periodic status during long phases and no estimated percentages.
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.5:** Failed initialization shall report its phase and underlying error, release resources, and exit unsuccessfully. The launcher shall detect child exit.
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.6:** Cancellation shall close startup listeners and stop further initialization admission at safe schema and store boundaries. Signal handling shall remain available throughout startup and shutdown. Restore worker quiescing shall not cancel the process and HTTP lifetime before the restore job can publish its result.
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.7:** A required upgrade backup shall complete before application migrations mutate the database. Backup failure shall prevent migrations.
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.8:** A listener bind failure shall stop startup promptly and identify the bind error.
## Exclusions
Backup reuse, retention redesign, migration-registry conversion, and background lifecycle-sweep scheduling are outside this change.
## System design
See [startup lifecycle](../system-design/startup-lifecycle.md).
@@ -0,0 +1,105 @@
---
status: current
system: platform
requirements:
- REQ-PLATFORM-STARTUP-LIFECYCLE-001
---
# Startup lifecycle design
## Ownership and flow
Platform owns this shared backend lifecycle. Configuration, runtime-state ownership,
logging, and startup security validation precede the bootstrap listener.
The listener precedes event-bus and database initialization. One handler switch,
HTTP server, and listener set survive through application-router installation.
Only the bootstrap handler serves traffic during initialization.
`backendapp.run` owns the listener and cancellation lifetime. Startup state travels
through context to persistence and router composition. Initialization remains
sequential; cleanup must not race constructors or database mutation.
Signal handling begins before database work. The first signal cancels startup;
the existing second-signal forced-exit behavior remains available.
The bootstrap owns separate process and worker cancellation contexts. A startup
signal cancels the process context and closes the bootstrap listener while the
initializer drains. Restore quiescing cancels only worker and runtime contexts;
the process context, HTTP listener, and job tracker remain alive until restore
has installed the staged database and published `restart_required`, after
which an explicit shutdown can close the process.
## Readiness and status
`/health` retains its existing body and desktop token header. Bootstrap `/ready`
returns 503 with a `startup` object containing a fixed phase name, total elapsed
milliseconds, and phase elapsed milliseconds. No paths, SQL, credentials, or raw
errors enter this unauthenticated payload. Application routes remain 503.
Phases cover database opening, backup, migrations, service initialization,
required session recovery, and ready. Structured logs record transitions and
phase duration. Launcher readiness polling reads JSON, reports transitions and
periodic elapsed status, and remembers the last phase for child-exit diagnostics.
Underlying errors remain in backend diagnostic output. Readiness waits retain
cancellation and child-exit checks without a separate deadline.
Router publication occurs only after required stores, recovery gates, final
persistence checks, and router construction succeed. The existing background
lifecycle-token sweep remains outside readiness: its watcher/scheduler ordering
and deadline are unchanged.
## Timeout policy
Keep release listener-health timeout at 45000 ms, development/E2E at 600000 ms.
Keep environment and YAML overrides. A 300000 ms default gives no database-work
benefit after early binding; it delays genuine pre-listener failure detection.
Desktop retains its existing independent 60-second liveness budget and unlimited
readiness wait. No timeout default changes are required.
Initialization checks cancellation before and after each repository schema step
and before each required store admission. The task repository passes the same
context to its migration SQL, including data backfills and transactional table
rebuilds. Other legacy constructors can finish an already admitted statement
because they do not expose a context API; the next admission barrier prevents
later stores from opening. A statement already executing in a context-aware
driver is also subject to that driver's cancellation behavior, so cancellation
does not promise an immediate interrupt at every SQLite virtual-machine or
user-function instruction. Cleanup waits until the synchronous initializer
returns, avoiding a pool-close race with in-flight SQL.
## Persistence safety
Keep `VACUUM INTO`, version-based backup eligibility, and two-snapshot retention.
Context cancellation applies to the backup SQL. Repository initialization failure
must release previously constructed resources; a completed backup remains intact.
The successful version marker stays behind the final persistence/router gates.
Pending migrations cannot be inferred from `MigrateLogger`: owners also execute
DDL, table rebuilds, data repairs, and seed changes directly, including stores
constructed after `provideRepositories`. A future registry must account for all
required stores, late constructors, dialect-specific DDL, data backfills, and
seed migrations, with an immutable applied ledger and completeness tests.
Only then can backup eligibility safely depend on pending work.
Retries before version recording take another backup. After partial migration,
that snapshot is not necessarily a pre-upgrade image. Two successful retries can
prune the original snapshot. Filename timestamps have one-second resolution and
can also collide on rapid retries. This existing risk requires a separate recovery
design: database identity, durable attempt state, original-snapshot pinning,
crash-safe retention, and restore tests. Never reuse by filename or target version.
## Requirement mapping
| Criteria | Mechanism |
| --- | --- |
| .1, .2, .3, .8 | Early bootstrap binding, atomic router handoff, existing launcher waits |
| .4, .5 | Typed phase snapshot, structured timing, child-exit diagnostics |
| .6, .7 | Separate process/worker cancellation ownership, schema/store admission barriers, sequential initialization, backup gate |
## Decisions
This extends the existing bootstrap architecture rather than creating a new
persistence contract. See [database upgrade safety](../../../decisions/0008-db-upgrade-safety.md).
## Implementation plans
- [Large database startup](../../../plans/large-database-startup/plan.md)
@@ -1,165 +1,9 @@
---
status: draft
created: 2026-08-22
revised: 2026-09-05
type: repair
---
# Startup listener before recovery
# The Backend Must Bind and Answer Liveness Before Startup Recovery Runs
The authoritative lifecycle contract is now [startup lifecycle requirements](../platform/requirements/startup-lifecycle.md)
and [startup lifecycle design](../platform/system-design/startup-lifecycle.md).
## Problem
The backend does not bind its HTTP socket until orchestrator startup
reconciliation has finished, and that reconciliation can wait on agent
processes for hours. Under the launcher's health window the process is killed
before it ever listens, restarted, and the cycle repeats without converging.
`backendapp.startGatewayAndServe` calls `orchestrator.Service.Start` at
`main.go:877` and reaches the first `net.Listen` only via `main.go:1064` ->
`httpserver.go:76`. `/health` is a route on the server built after that point
(`helpers.go:716`); no earlier path serves it. So while `Start` runs there is
no listener at all, which is exactly what `lsof` showed.
`Start` runs `reconcileTaskLifecycleTokens` synchronously
(`service.go:2255`), fanning four workers over every task with a pending
lifecycle token and waiting on a `sync.WaitGroup`. The per-task work reaches
`autoStartStepPrompt` -> `PromptTask` -> `ensureSessionRunning` ->
`attemptColdResume`.
**The per-task cost is far larger than the prompt-ready timeout suggests.**
`attemptColdResume` first waits for session readiness via `waitForSessionReady`
(`task_operations.go:2498`), whose budget is `constants.AgentLaunchTimeout` =
**15 minutes** (`timeouts.go:44`, 10m setup script + 5m allowance). Only after
that does the 30s `waitForAgentPromptReady` apply, and the attempt is retried
once. Worst case is roughly 31 minutes per task, four at a time.
Measured on the affected instance: 424 tasks, 32 carrying
`manual_move_lifecycle` metadata, on a 1.0 GB database. A boot given a full
10-minute health window still never bound, and produced only 2 successful ACP
session creations in those 10 minutes. That is the signature of workers parked
in the 15-minute session-ready wait, not of workers cycling through tasks.
Three consequences, in severity order:
1. **Socket liveness is coupled to agent readiness.** The process cannot answer
a health probe, serve the UI, or expose the API until agents that may never
become ready are waited out.
2. **The failure is self-sustaining.** `healthTimeoutReleaseMS` is 45s
(`internal/launcher/constants.go`). The launcher declares a backend health
failure and stops the process; a supervisor restarts it; the same tokens are
still pending because none were cleared, and the runtimes orphaned by the
kill are added to the next boot's work. Ten consecutive boots were observed,
none of which bound a socket.
3. **The wait cannot be bounded from outside.** `attemptColdResume` and
`reapAndReloadSession` deliberately use `context.WithoutCancel`
(`task_operations.go:2308`, `task_operations.go:2255`) so a resume survives
its caller's cancellation. A deadline handed to the sweep therefore does not
stop an in-flight resume. Any fix that relies on cancelling the sweep to
protect startup is ineffective.
This is startup ordering, not a defect in the recovery logic. The same frames
are reached whenever a lifecycle token is pending.
## Correction (2026-09-05)
This spec's original Desired-behaviour and Out-of-scope sections asserted
that the sweep's position *before* `watcher.Start` was load-bearing and that
moving it would introduce races with live traffic. Runtime evidence gathered
after this spec's socket-bind fix shipped disproves that: with the socket-bind
fix in place, boots with pending lifecycle tokens still never reached `API
configured` — the sweep itself was still holding the readiness path open
indefinitely, just behind the new bootstrap 503 handler instead of behind a
closed socket.
The reason is a **self-inflicted event blackout**, not merely a slow
individual resume. `Watcher.Start` is what registers
`{events.AgentBootReady, w.handlers.OnAgentBootReady}` (the *only* handler
that flips a launching session to `WAITING_FOR_INPUT` on the resume path).
The event bus used here has no replay, so any `agent.boot_ready` published
while the sweep runs *before* `watcher.Start` is dropped on the floor.
`waitForSessionReady`'s poll can then never observe the state change it is
waiting for, and burns its full `constants.AgentLaunchTimeout` (15 minutes)
on every recovered task, regardless of whether the underlying agent process
is actually healthy. Four workers wide, retried up to three times, this is
indistinguishable from a hang and gates the entire router.
The fix inverts the ordering this spec forbade: the sweep now runs in a
background goroutine *after* `watcher.Start` and `scheduler.Start` succeed,
so the events its own resume path depends on are deliverable from the
moment recovery starts. Readiness is not gated on the sweep at all — the
socket-bind / bootstrap-503 behavior this spec introduced is unaffected and
still the mechanism that makes an in-progress sweep observable as "starting"
rather than a hang. The Desired-behaviour and Out-of-scope sections below
are amended accordingly; the regression scenarios still hold, since a bound
socket does not depend on this ordering.
## Broken behavior
- No socket is bound, and `/health` cannot be answered, until orchestrator
startup reconciliation returns.
- A backend whose recovery exceeds the health window is killed and restarted
with at least as much recovery work as before, indefinitely.
## Desired behavior
- The backend binds its socket and answers a liveness probe within the release
health window, regardless of how much recovery work is pending.
- Startup recovery runs in the background *after* the watcher and scheduler
start (amended 2026-09-05 — see Correction above): its own resume path
depends on events only the watcher's subscriptions deliver, so running it
earlier does not protect anything and instead guarantees every recovered
task times out. Running concurrently with live watcher/scheduler traffic is
safe because the sweep's operations are already built for concurrent
delivery — per-task locks (`queuedMoveLifecycleLocks`) and one-shot tokens
claimed by an atomic metadata-key removal (`claimTaskEventMetadata`) —
confirmed by
`TestLifecycleSweepConcurrentWithLiveTaskMovedRunsManualMoveSideEffectsOnce`,
which runs the sweep's recovery path concurrently with a live `task.moved`
redelivery of the same admitted manual move and asserts the step exit/enter
side effects run exactly once.
- Requests that need a started application are answered with an explicit
"starting" response rather than by a hung connection or a refused socket.
- Readiness is observable and distinct from liveness: an operator can tell
"bound, still starting" from "fully started".
## Regression scenarios
- **GIVEN** a backend with pending lifecycle tokens whose agent sessions never
reach ready, **WHEN** it starts, **THEN** the socket is bound and the
liveness probe returns success well inside `healthTimeoutReleaseMS`, and the
launcher does not shut the process down.
- **GIVEN** the same backend while startup reconciliation is still running,
**WHEN** an API route is requested, **THEN** it returns an explicit starting
status rather than hanging, and readiness reports not-ready while liveness
reports healthy.
- **GIVEN** startup reconciliation that has completed, **WHEN** readiness is
requested, **THEN** it reports ready and all routes serve normally.
## Out of scope
- Changing WIP promotion, feeder pull, or manual-move lifecycle semantics.
- Changing `AgentLaunchTimeout`, `agentPromptReadyTimeout`, or the cold-resume
retry shape (`context.WithoutCancel`). See the follow-up note below — a
caller-side deadline on the sweep bounds admission and reporting, not an
in-flight resume, precisely because of this contract.
- Raising `healthTimeoutReleaseMS`, which hides the coupling instead of fixing
it. Empirically a 10-minute window did not help.
## Known follow-up, addressed 2026-09-05
The note below originally described work deliberately deferred past this
spec's socket-bind fix. It has since been addressed in the same change that
corrected the ordering above (see Correction): the sweep now runs off the
readiness path entirely (background goroutine, not gating `API configured`),
carries its own bounded overall deadline separate from the interactive
`AgentLaunchTimeout`, and no longer accumulates already-completed
`manual_move_lifecycle_completed` tokens as permanent per-boot work. The
`context.WithoutCancel` resume contract itself is unchanged, so an
individual in-flight resume still cannot be aborted early — the sweep's
deadline bounds its own dispatch loop returning, not that resume.
> Fixing the ordering stops the crash loop but does not make a large board
> start quickly: a backend with many pending tokens still spends a long time
> in "starting" because bulk recovery inherits the interactive 15-minute
> `AgentLaunchTimeout` per task. A bulk startup sweep needs its own, much
> smaller per-task budget, and a resume path that can actually be abandoned.
The background lifecycle-token sweep still starts after watcher and scheduler
subscriptions. It does not gate readiness. Its per-task locks and atomic metadata
claims prevent duplicate effects during live event delivery. This change does not
alter its bounded dispatch deadline or in-flight resume behavior.