fix: keep large database startup alive (#3624)
This commit is contained in:
Carlos Florêncio
@@ -0,0 +1,84 @@
|
||||
---
|
||||
created: 2026-09-11
|
||||
status: complete
|
||||
requirements:
|
||||
- REQ-PLATFORM-STARTUP-LIFECYCLE-001
|
||||
system_design:
|
||||
- ../../specs/platform/system-design/startup-lifecycle.md
|
||||
legacy_specs: []
|
||||
---
|
||||
|
||||
# Large database startup implementation plan
|
||||
|
||||
## Root cause
|
||||
|
||||
`provideRepositories` performs the upgrade backup and migrations before
|
||||
`startGatewayAndServe` binds the bootstrap listener. The reported 42.5-second
|
||||
backup consumes almost all of the 45-second listener budget.
|
||||
|
||||
## Scope and approach
|
||||
|
||||
Move shared bootstrap ownership before initialization. Add typed phase reporting
|
||||
and cancellation-aware backup execution. Preserve version gates, retention,
|
||||
health identity, binding semantics, and background recovery ordering.
|
||||
|
||||
## Work orders
|
||||
|
||||
- [x] [01: Startup lifecycle](task-01-lifecycle.md)
|
||||
- [x] [02: Startup measurements and documentation](task-02-evidence.md)
|
||||
|
||||
## Verification
|
||||
|
||||
Backend commands run from `apps/backend`; script and diff commands run from root.
|
||||
|
||||
- `rtk go test -race ./internal/backendapp ./internal/persistence ./internal/launcher ./internal/common/config ./internal/db`
|
||||
- `rtk go test ./internal/task/repository/sqlite -run 'Test(StartupMigrationCosts|StartupMigrationCostsPopulated|NewWithDBContextStopsActiveMigration|BackfillPromptSeqStopsOnCanceledContext)' -v`
|
||||
- `rtk proxy python3 scripts/lint-spec-files.py --all`
|
||||
- `rtk proxy node --test scripts/validate-public-docs.test.mjs`
|
||||
- `rtk proxy node scripts/validate-public-docs.mjs`
|
||||
- `rtk git diff --check`
|
||||
|
||||
## End-to-end evidence
|
||||
|
||||
Use real HTTP listeners with controlled initialization and short launcher health
|
||||
deadlines. Cover success, failure, route isolation, cancellation, and bind failure.
|
||||
No browser UI changes or live database access are needed.
|
||||
|
||||
## Risks
|
||||
|
||||
Legacy constructors include non-context SQL. Shutdown must not race cleanup with
|
||||
those constructors. Retry retention can discard the original snapshot; no reuse
|
||||
or retention changes are authorized in this package.
|
||||
|
||||
## Results
|
||||
|
||||
Implemented early bootstrap ownership, typed startup phases, cancellation-aware
|
||||
SQLite backup and task migration execution, worker-only restore quiescing,
|
||||
launcher phase diagnostics, populated startup measurements, and public
|
||||
troubleshooting documentation.
|
||||
|
||||
Validation passed:
|
||||
|
||||
- `env -u KANDEV_INTERNAL_CONFIG_FILE -u KANDEV_INTERNAL_CONFIG_HOME_FILE -u KANDEV_INTERNAL_AGENTCTL_STARTUP_CONFIG -u KANDEV_HEALTH_TIMEOUT_MS go test -race ./internal/backendapp ./internal/persistence ./internal/launcher ./internal/common/config ./internal/db -count=1`
|
||||
- `go test -race ./internal/startup -count=1`
|
||||
- `go test -race ./internal/persistence/storeconformance -count=1`
|
||||
- `go test ./internal/task/repository/sqlite -run 'Test(StartupMigrationCosts|StartupMigrationCostsPopulated|NewWithDBContextStopsActiveMigration|BackfillPromptSeqStopsOnCanceledContext)$' -v -count=1`
|
||||
- `make -C apps/backend build`
|
||||
- `make -C apps/backend lint` (0 issues)
|
||||
- `go run ./cmd/sqlguard ./internal`
|
||||
- `python3 scripts/lint-spec-files.py --all`
|
||||
- `node --test scripts/validate-public-docs.test.mjs` (62 passed)
|
||||
- `node scripts/validate-public-docs.mjs` (46 published pages)
|
||||
- `git diff --check`
|
||||
|
||||
The populated isolated SQLite evidence is recorded in
|
||||
[task-02-evidence](task-02-evidence.md). It covers a 929,792-byte database,
|
||||
6,000 messages, a 2,252,800-byte `VACUUM INTO` snapshot, 285.06 ms replay,
|
||||
and separate recurring backfill timings on Linux amd64 with Go 1.26.0. The
|
||||
measurement remains synthetic and does not predict production startup time.
|
||||
|
||||
Deferred risks remain outside this package: backup retry retention and the
|
||||
backup-registry redesign are unchanged. Constructors for non-task stores still
|
||||
have bounded admission barriers rather than a new shared context API; an
|
||||
already admitted statement can finish according to its driver's cancellation
|
||||
behavior before the next store is rejected.
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
id: "01-lifecycle"
|
||||
title: "Startup lifecycle"
|
||||
status: done
|
||||
wave: 1
|
||||
depends_on: []
|
||||
plan: plan.md
|
||||
requirements:
|
||||
- REQ-PLATFORM-STARTUP-LIFECYCLE-001
|
||||
acceptance_criteria:
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.1
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.2
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.3
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.4
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.5
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.6
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.7
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.8
|
||||
system_design:
|
||||
- ../../specs/platform/system-design/startup-lifecycle.md
|
||||
---
|
||||
|
||||
# Startup lifecycle
|
||||
|
||||
## Scope
|
||||
|
||||
Early bootstrap ownership, phase reporting, cancellation, backup ordering, and launcher diagnostics.
|
||||
|
||||
## Exclusions
|
||||
|
||||
No backup mechanism replacement, registry conversion, retention redesign, or publication.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Meet the referenced lifecycle criteria with isolated regression evidence.
|
||||
- Preserve health identity, timeout overrides, and required persistence gates.
|
||||
- Record exact verification results and measurement limitations.
|
||||
|
||||
## Files likely touched
|
||||
|
||||
`apps/backend/internal/backendapp`, `internal/persistence`, `internal/launcher`,
|
||||
`internal/db`, task repository measurement tests, and owning/public documentation.
|
||||
|
||||
## Verification
|
||||
|
||||
Use the corresponding commands in [plan](plan.md#verification), from their stated directories.
|
||||
|
||||
## Dependencies
|
||||
|
||||
None.
|
||||
|
||||
## Parallelism
|
||||
|
||||
Sequential. No delegation.
|
||||
|
||||
## Risks
|
||||
|
||||
See plan and system design for cancellation and retry retention constraints.
|
||||
|
||||
## Results
|
||||
|
||||
Implemented and tested the lifecycle changes. The bootstrap listener now binds
|
||||
before database opening, backup, migrations, service initialization, and
|
||||
session recovery. `/health` is live during initialization, `/ready` reports a
|
||||
typed phase with elapsed measurements, cancellation closes listeners while
|
||||
initialization drains, and the launcher reports the last phase on early exit.
|
||||
|
||||
The backend lifecycle and launcher race suites passed. Backup failure remains
|
||||
before migrations, canceled persistence does not start backup work, and
|
||||
restore quiescing cancels workers without ending the process lifetime before
|
||||
the restore result is delivered. Repository startup checks cancellation at
|
||||
schema and store admission boundaries; task migrations pass cancellation into
|
||||
their data statements and table rebuilds. An in-flight database statement can
|
||||
still finish according to the driver's cancellation semantics, and cleanup
|
||||
waits for the synchronous initializer to return.
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
id: "02-evidence"
|
||||
title: "Startup evidence"
|
||||
status: done
|
||||
wave: 2
|
||||
depends_on: ["01-lifecycle"]
|
||||
plan: plan.md
|
||||
requirements:
|
||||
- REQ-PLATFORM-STARTUP-LIFECYCLE-001
|
||||
acceptance_criteria:
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.1
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.2
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.3
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.4
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.5
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.6
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.7
|
||||
- AC-PLATFORM-STARTUP-LIFECYCLE-001.8
|
||||
system_design:
|
||||
- ../../specs/platform/system-design/startup-lifecycle.md
|
||||
---
|
||||
|
||||
# Startup evidence
|
||||
|
||||
## Scope
|
||||
|
||||
Isolated populated SQLite backup and migration timing, deferred registry and
|
||||
retention findings, public troubleshooting documentation.
|
||||
|
||||
## Exclusions
|
||||
|
||||
No backup mechanism replacement, registry conversion, retention redesign, or publication.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Meet the referenced lifecycle criteria with isolated regression evidence.
|
||||
- Preserve health identity, timeout overrides, and required persistence gates.
|
||||
- Record exact verification results and measurement limitations.
|
||||
|
||||
## Files likely touched
|
||||
|
||||
`apps/backend/internal/backendapp`, `internal/persistence`, `internal/launcher`,
|
||||
`internal/db`, task repository measurement tests, and owning/public documentation.
|
||||
|
||||
## Verification
|
||||
|
||||
Use the corresponding commands in [plan](plan.md#verification), from their stated directories.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Task 01.
|
||||
|
||||
## Parallelism
|
||||
|
||||
Sequential. No delegation.
|
||||
|
||||
## Risks
|
||||
|
||||
See plan and system design for cancellation and retry retention constraints.
|
||||
|
||||
## Results
|
||||
|
||||
`TestStartupMigrationCostsPopulated` creates a fresh schema, then rebuilds
|
||||
only the message table into the legacy shape needed by the recurring startup
|
||||
backfills. It inserts 24 task sessions and 6,000 messages (one third user
|
||||
messages), leaves `updated_at` nullable and empty, and omits `prompt_seq` so a
|
||||
replay exercises both migration paths. The test also runs the same
|
||||
`VACUUM INTO` snapshot used by production and measures the two backfills
|
||||
separately after replay. The fixture is isolated under `t.TempDir()` and can be
|
||||
reproduced with:
|
||||
|
||||
```text
|
||||
go test ./internal/task/repository/sqlite -run '^TestStartupMigrationCostsPopulated$' -v -count=1
|
||||
```
|
||||
|
||||
One recorded run on Linux amd64, Go 1.26.0, 11 CPUs, reported:
|
||||
|
||||
| Measurement | Result |
|
||||
| --- | ---: |
|
||||
| Populated database file before replay | 929,792 bytes |
|
||||
| `VACUUM INTO` backup | 2,252,800 bytes in 18.99 ms |
|
||||
| Fresh schema initialization | 48.75 ms |
|
||||
| Populated replay | 285.06 ms |
|
||||
| `task_session_messages.updated_at` backfill | 19.25 ms |
|
||||
| Prompt sequence update plus counter aggregation | 41.52 ms |
|
||||
|
||||
The values are observations from one local run, not startup budgets. The
|
||||
fixture does not model a production database's row distribution, WAL state,
|
||||
filesystem, CPU contention, provider recovery, or full backup directory. It
|
||||
does distinguish the liveness and readiness contract: the bootstrap tests keep
|
||||
`/health` at 200 while `/ready` and application routes remain 503 until the
|
||||
initializer publishes the router, and the launcher health deadline is not
|
||||
used as the readiness deadline. The evidence does not change `VACUUM INTO`,
|
||||
backup retry behavior, or backup-registry and retention design.
|
||||
|
||||
Lifecycle regression coverage now includes delayed success past a short health
|
||||
deadline, active SQL cancellation through `NewWithDBContext`, worker-only
|
||||
restore quiescing with `restart_required`, explicit post-readiness signal
|
||||
handling, and initializer joining after cancellation.
|
||||
|
||||
PR review follow-up also verifies the retry boundaries: pre-migration snapshots
|
||||
are created in a private staging directory, validated, set to mode `0600`, and
|
||||
installed only after validation; a failed `VACUUM INTO` destination is removed
|
||||
when the path did not exist before the attempt. The authenticated system-job
|
||||
notification path remains on the process lifetime while restore workers are
|
||||
quiesced. These changes do not alter backup retention or the deferred backup
|
||||
registry redesign.
|
||||
@@ -132,6 +132,11 @@ On a normal start, the launcher:
|
||||
7. waits for `/ready` (the backend has finished startup recovery and can serve real requests); this wait is unbounded by design, since recovery can legitimately take much longer than 45 seconds; and
|
||||
8. opens the reachable access URL in the default browser.
|
||||
|
||||
While `/ready` returns 503, it reports the current startup phase and elapsed
|
||||
time. The launcher prints phase changes and periodic status, so a long backup,
|
||||
migration, or recovery can be distinguished from an unreachable backend. These
|
||||
values describe elapsed work and do not estimate completion.
|
||||
|
||||
The launcher remains in the foreground. Press `Ctrl+C` or terminate it to stop the backend and its managed children cleanly. A force-kill can leave worktree processes or containers running; inspect them before deleting data.
|
||||
|
||||
Use headless mode for SSH sessions, containers, or an external reverse proxy:
|
||||
|
||||
@@ -263,7 +263,7 @@ Remote Docker profiles are not a workaround: that executor runtime is currently
|
||||
|
||||
## Health and observability
|
||||
|
||||
`GET /health` returns 200 as soon as the listener is accepting connections, even mid-startup; `GET /ready` returns 503 during startup and 200 after routes are registered:
|
||||
`GET /health` returns 200 as soon as the listener is accepting connections, even mid-startup. `GET /ready` returns 503 during startup and includes the current startup phase and elapsed time; it returns 200 after routes are registered:
|
||||
|
||||
```bash
|
||||
curl --fail http://localhost:38429/ready
|
||||
|
||||
+1
-1
@@ -489,7 +489,7 @@ The universal image already runs as `kandev`; use `kubectl exec -it deployment/k
|
||||
|
||||
The example requests 250 millicores and 512 MiB, with limits of 2 CPU and 2 GiB. Those are placeholders, not capacity recommendations. Local/Worktree agents share the pod limit with the control plane and can exceed it during builds. Measure workload memory, CPU, ephemeral storage, PVC growth, and process counts; then set requests/limits accordingly.
|
||||
|
||||
The example liveness probe calls `/health`; the example readiness probe calls `/ready`. `/health` returns 200 as soon as the TCP listener accepts connections, before startup finishes: it confirms the process is alive, not that it can serve real traffic, so gating liveness on it never restarts a pod that is merely still starting up. `/ready` returns 503 until routes are wired, the agent registry is seeded, and (in e2e builds) the mock-harness routes are mounted, then 200. Neither is a deep check of database, repository, Docker, provider, or agent health.
|
||||
The example liveness probe calls `/health`; the example readiness probe calls `/ready`. `/health` returns 200 as soon as the TCP listener accepts connections, before startup finishes: it confirms the process is alive, not that it can serve real traffic, so gating liveness on it never restarts a pod that is merely still starting up. `/ready` returns 503 until routes are wired, the agent registry is seeded, and (in e2e builds) the mock-harness routes are mounted, then 200. While it returns 503, its `startup` object identifies the current database, migration, service, or recovery phase and reports elapsed time without estimating completion. Neither is a deep check of database, repository, Docker, provider, or agent health.
|
||||
|
||||
Long migrations or slow storage may need a startup probe to prevent premature liveness restarts:
|
||||
|
||||
|
||||
@@ -83,7 +83,25 @@ It returns HTTP 200 as soon as the listener accepts connections, with:
|
||||
curl -fsS http://127.0.0.1:38429/ready
|
||||
```
|
||||
|
||||
It returns HTTP 503 with `status: "starting"` (plus the same `version`) until routes, the agent registry, and the listener are ready, then HTTP 200. The supplied Kubernetes liveness probe uses `/health`; the readiness probe uses `/ready`. Both are unauthenticated even when auth is enabled, so they're also a credential-free way for monitoring to read the running version; there is no need to authenticate to **System > About** just to check what build is deployed.
|
||||
It returns HTTP 503 with `status: "starting"` (plus the same `version`) until routes, the agent registry, and the listener are ready, then HTTP 200. During startup, the response also includes a bounded `startup` object:
|
||||
|
||||
```json
|
||||
{
|
||||
"phase": "backing_up_database",
|
||||
"elapsed_ms": 42000,
|
||||
"phase_elapsed_ms": 18000
|
||||
}
|
||||
```
|
||||
|
||||
The phase is a label such as `opening_database`, `backing_up_database`,
|
||||
`applying_migrations`, `initializing_services`, or `recovering_sessions`.
|
||||
Elapsed values are status measurements, not progress estimates. The launcher
|
||||
prints phase changes and periodic elapsed status, and names the last phase if
|
||||
the backend exits before readiness. The supplied Kubernetes liveness probe uses
|
||||
`/health`; the readiness probe uses `/ready`. Both are unauthenticated even
|
||||
when auth is enabled, so they're also a credential-free way for monitoring to
|
||||
read the running version; there is no need to authenticate to **System > About**
|
||||
just to check what build is deployed.
|
||||
|
||||
For application diagnostics, open **Settings > System > Status** or request:
|
||||
|
||||
@@ -422,7 +440,7 @@ The UI flow applies to the configured SQLite path:
|
||||
|
||||
1. Stop or finish active agent sessions and preserve unpushed work.
|
||||
2. Open **Settings > System > Backups**, choose **Restore**, type `RESTORE`, and confirm.
|
||||
3. Kandev stops scheduling, active executions, and database-backed workers. It copies the selected snapshot to `<configured-database-path>.new`, validates the SQLite checkpoint result, and closes the pool. It then quarantines the configured database and its `-wal`/`-shm` sidecars before installing the staged file. If installation fails, Kandev restores the quarantined files.
|
||||
3. Kandev stops scheduling, active executions, and database-backed workers. It copies the selected snapshot to `<configured-database-path>.new`, validates the SQLite checkpoint result, and closes the pool. It then quarantines the configured database and its `-wal`/`-shm` sidecars before installing the staged file. If installation fails, Kandev restores the quarantined files. The restore job remains available long enough to report `restart_required` before process shutdown.
|
||||
4. Click **Restart Kandev** in the success dialog. If automatic restart is unavailable, quit and relaunch Kandev manually. The backend must restart before database-backed work resumes.
|
||||
5. Check `/ready`, **System > Status**, database schema version, secrets, and representative tasks.
|
||||
|
||||
|
||||
@@ -58,7 +58,7 @@ The installer writes the managed unit or plist and starts the process. It does *
|
||||
curl --fail http://127.0.0.1:43127/ready
|
||||
```
|
||||
|
||||
`/ready` reports backend readiness after routes and the agent registry are initialized and the HTTP listener is accepting connections. It is not a deep health check of the database, message bus, executors, or remote providers.
|
||||
`/ready` reports backend readiness after routes and the agent registry are initialized and the HTTP listener is accepting connections. While it returns 503, its `startup` object reports the current phase and elapsed time for database opening, backup, migrations, service initialization, or session recovery. It is not a deep health check of the database, message bus, executors, or remote providers.
|
||||
|
||||
### System service
|
||||
|
||||
|
||||
@@ -35,6 +35,8 @@ retains ownership of its values, validation, authority, and persistence.
|
||||
|
||||
### Requirements
|
||||
|
||||
- [Startup lifecycle](requirements/startup-lifecycle.md)
|
||||
|
||||
- [Agent-accessible Kandev settings](requirements/agent-settings-parity.md)
|
||||
|
||||
|
||||
@@ -80,6 +82,8 @@ retains ownership of its values, validation, authority, and persistence.
|
||||
|
||||
### System design
|
||||
|
||||
- [Startup lifecycle](system-design/startup-lifecycle.md)
|
||||
|
||||
- [Agent-accessible Kandev settings](system-design/agent-settings-parity.md)
|
||||
- [Core settings domain adoption](system-design/agent-settings-domains.md)
|
||||
|
||||
|
||||
@@ -65,11 +65,11 @@ resolved by explicit decision — see [Decisions](#decisions).
|
||||
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.1:** `GET /health` SHALL include the running Kandev version in its JSON response body.
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.2:** The field SHALL be named **`version`**, matching `GET /api/v1/system/info`.
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.3:** The field SHALL be present in **both** the ready (200) and not-ready (503) responses, so an operator can identify the build of a backend that is stuck starting.
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.3:** The field SHALL be present in both `/health` and bootstrap `/ready` responses, so an operator can identify the build of a backend that is stuck starting.
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.4:** The field value SHALL be the same string `GET /api/v1/system/info` reports as `version` for the same running process.
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.5:** The field SHALL be served to unauthenticated callers, exactly as the rest of the `/health` payload already is. This is a deliberate, accepted disclosure — see [Security](#security-and-permissions).
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.6:** Every existing field (`status`, `service`, `mode`) SHALL retain its current name, value, and semantics. This is a purely additive change.
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.7:** The existing HTTP status semantics (200 when ready, 503 while starting) SHALL be unchanged.
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.7:** `/health` SHALL return 200 once bound, including during initialization. `/ready` SHALL return 503 until application initialization finishes.
|
||||
- **AC-PLATFORM-HEALTH-ENDPOINT-VERSION-001.8:** The desktop health-token response header SHALL continue to be set on the 200 path only, unchanged.
|
||||
|
||||
## System design
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
status: active
|
||||
system: platform
|
||||
created: 2026-09-11
|
||||
owners:
|
||||
- kandev
|
||||
---
|
||||
|
||||
# Startup lifecycle requirements
|
||||
|
||||
## Overview
|
||||
|
||||
Platform owns process startup, liveness, readiness, and shutdown across native launch modes and desktop.
|
||||
Large databases must complete initialization without a listener-health deadline interrupting valid work.
|
||||
|
||||
## Requirements
|
||||
|
||||
### REQ-PLATFORM-STARTUP-LIFECYCLE-001: Observable initialization
|
||||
|
||||
**Intent:** Separate process liveness from application readiness while preserving upgrade safety.
|
||||
|
||||
#### Acceptance criteria
|
||||
|
||||
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.1:** During database opening, backup, migration, and required recovery, `/health` shall answer successfully with existing version and identity semantics.
|
||||
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.2:** Until required initialization and router installation finish, `/ready` and application endpoints shall remain unsuccessful.
|
||||
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.3:** After liveness succeeds, the launcher shall wait for readiness without an initialization deadline. Explicit listener-timeout overrides shall remain effective.
|
||||
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.4:** Startup shall report phase transitions and elapsed time, with bounded periodic status during long phases and no estimated percentages.
|
||||
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.5:** Failed initialization shall report its phase and underlying error, release resources, and exit unsuccessfully. The launcher shall detect child exit.
|
||||
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.6:** Cancellation shall close startup listeners and stop further initialization admission at safe schema and store boundaries. Signal handling shall remain available throughout startup and shutdown. Restore worker quiescing shall not cancel the process and HTTP lifetime before the restore job can publish its result.
|
||||
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.7:** A required upgrade backup shall complete before application migrations mutate the database. Backup failure shall prevent migrations.
|
||||
- **AC-PLATFORM-STARTUP-LIFECYCLE-001.8:** A listener bind failure shall stop startup promptly and identify the bind error.
|
||||
|
||||
## Exclusions
|
||||
|
||||
Backup reuse, retention redesign, migration-registry conversion, and background lifecycle-sweep scheduling are outside this change.
|
||||
|
||||
## System design
|
||||
|
||||
See [startup lifecycle](../system-design/startup-lifecycle.md).
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
status: current
|
||||
system: platform
|
||||
requirements:
|
||||
- REQ-PLATFORM-STARTUP-LIFECYCLE-001
|
||||
---
|
||||
|
||||
# Startup lifecycle design
|
||||
|
||||
## Ownership and flow
|
||||
|
||||
Platform owns this shared backend lifecycle. Configuration, runtime-state ownership,
|
||||
logging, and startup security validation precede the bootstrap listener.
|
||||
The listener precedes event-bus and database initialization. One handler switch,
|
||||
HTTP server, and listener set survive through application-router installation.
|
||||
Only the bootstrap handler serves traffic during initialization.
|
||||
|
||||
`backendapp.run` owns the listener and cancellation lifetime. Startup state travels
|
||||
through context to persistence and router composition. Initialization remains
|
||||
sequential; cleanup must not race constructors or database mutation.
|
||||
Signal handling begins before database work. The first signal cancels startup;
|
||||
the existing second-signal forced-exit behavior remains available.
|
||||
|
||||
The bootstrap owns separate process and worker cancellation contexts. A startup
|
||||
signal cancels the process context and closes the bootstrap listener while the
|
||||
initializer drains. Restore quiescing cancels only worker and runtime contexts;
|
||||
the process context, HTTP listener, and job tracker remain alive until restore
|
||||
has installed the staged database and published `restart_required`, after
|
||||
which an explicit shutdown can close the process.
|
||||
|
||||
## Readiness and status
|
||||
|
||||
`/health` retains its existing body and desktop token header. Bootstrap `/ready`
|
||||
returns 503 with a `startup` object containing a fixed phase name, total elapsed
|
||||
milliseconds, and phase elapsed milliseconds. No paths, SQL, credentials, or raw
|
||||
errors enter this unauthenticated payload. Application routes remain 503.
|
||||
|
||||
Phases cover database opening, backup, migrations, service initialization,
|
||||
required session recovery, and ready. Structured logs record transitions and
|
||||
phase duration. Launcher readiness polling reads JSON, reports transitions and
|
||||
periodic elapsed status, and remembers the last phase for child-exit diagnostics.
|
||||
Underlying errors remain in backend diagnostic output. Readiness waits retain
|
||||
cancellation and child-exit checks without a separate deadline.
|
||||
|
||||
Router publication occurs only after required stores, recovery gates, final
|
||||
persistence checks, and router construction succeed. The existing background
|
||||
lifecycle-token sweep remains outside readiness: its watcher/scheduler ordering
|
||||
and deadline are unchanged.
|
||||
|
||||
## Timeout policy
|
||||
|
||||
Keep release listener-health timeout at 45000 ms, development/E2E at 600000 ms.
|
||||
Keep environment and YAML overrides. A 300000 ms default gives no database-work
|
||||
benefit after early binding; it delays genuine pre-listener failure detection.
|
||||
Desktop retains its existing independent 60-second liveness budget and unlimited
|
||||
readiness wait. No timeout default changes are required.
|
||||
|
||||
Initialization checks cancellation before and after each repository schema step
|
||||
and before each required store admission. The task repository passes the same
|
||||
context to its migration SQL, including data backfills and transactional table
|
||||
rebuilds. Other legacy constructors can finish an already admitted statement
|
||||
because they do not expose a context API; the next admission barrier prevents
|
||||
later stores from opening. A statement already executing in a context-aware
|
||||
driver is also subject to that driver's cancellation behavior, so cancellation
|
||||
does not promise an immediate interrupt at every SQLite virtual-machine or
|
||||
user-function instruction. Cleanup waits until the synchronous initializer
|
||||
returns, avoiding a pool-close race with in-flight SQL.
|
||||
|
||||
## Persistence safety
|
||||
|
||||
Keep `VACUUM INTO`, version-based backup eligibility, and two-snapshot retention.
|
||||
Context cancellation applies to the backup SQL. Repository initialization failure
|
||||
must release previously constructed resources; a completed backup remains intact.
|
||||
The successful version marker stays behind the final persistence/router gates.
|
||||
|
||||
Pending migrations cannot be inferred from `MigrateLogger`: owners also execute
|
||||
DDL, table rebuilds, data repairs, and seed changes directly, including stores
|
||||
constructed after `provideRepositories`. A future registry must account for all
|
||||
required stores, late constructors, dialect-specific DDL, data backfills, and
|
||||
seed migrations, with an immutable applied ledger and completeness tests.
|
||||
Only then can backup eligibility safely depend on pending work.
|
||||
|
||||
Retries before version recording take another backup. After partial migration,
|
||||
that snapshot is not necessarily a pre-upgrade image. Two successful retries can
|
||||
prune the original snapshot. Filename timestamps have one-second resolution and
|
||||
can also collide on rapid retries. This existing risk requires a separate recovery
|
||||
design: database identity, durable attempt state, original-snapshot pinning,
|
||||
crash-safe retention, and restore tests. Never reuse by filename or target version.
|
||||
|
||||
## Requirement mapping
|
||||
|
||||
| Criteria | Mechanism |
|
||||
| --- | --- |
|
||||
| .1, .2, .3, .8 | Early bootstrap binding, atomic router handoff, existing launcher waits |
|
||||
| .4, .5 | Typed phase snapshot, structured timing, child-exit diagnostics |
|
||||
| .6, .7 | Separate process/worker cancellation ownership, schema/store admission barriers, sequential initialization, backup gate |
|
||||
|
||||
## Decisions
|
||||
|
||||
This extends the existing bootstrap architecture rather than creating a new
|
||||
persistence contract. See [database upgrade safety](../../../decisions/0008-db-upgrade-safety.md).
|
||||
|
||||
## Implementation plans
|
||||
|
||||
- [Large database startup](../../../plans/large-database-startup/plan.md)
|
||||
@@ -1,165 +1,9 @@
|
||||
---
|
||||
status: draft
|
||||
created: 2026-08-22
|
||||
revised: 2026-09-05
|
||||
type: repair
|
||||
---
|
||||
# Startup listener before recovery
|
||||
|
||||
# The Backend Must Bind and Answer Liveness Before Startup Recovery Runs
|
||||
The authoritative lifecycle contract is now [startup lifecycle requirements](../platform/requirements/startup-lifecycle.md)
|
||||
and [startup lifecycle design](../platform/system-design/startup-lifecycle.md).
|
||||
|
||||
## Problem
|
||||
|
||||
The backend does not bind its HTTP socket until orchestrator startup
|
||||
reconciliation has finished, and that reconciliation can wait on agent
|
||||
processes for hours. Under the launcher's health window the process is killed
|
||||
before it ever listens, restarted, and the cycle repeats without converging.
|
||||
|
||||
`backendapp.startGatewayAndServe` calls `orchestrator.Service.Start` at
|
||||
`main.go:877` and reaches the first `net.Listen` only via `main.go:1064` ->
|
||||
`httpserver.go:76`. `/health` is a route on the server built after that point
|
||||
(`helpers.go:716`); no earlier path serves it. So while `Start` runs there is
|
||||
no listener at all, which is exactly what `lsof` showed.
|
||||
|
||||
`Start` runs `reconcileTaskLifecycleTokens` synchronously
|
||||
(`service.go:2255`), fanning four workers over every task with a pending
|
||||
lifecycle token and waiting on a `sync.WaitGroup`. The per-task work reaches
|
||||
`autoStartStepPrompt` -> `PromptTask` -> `ensureSessionRunning` ->
|
||||
`attemptColdResume`.
|
||||
|
||||
**The per-task cost is far larger than the prompt-ready timeout suggests.**
|
||||
`attemptColdResume` first waits for session readiness via `waitForSessionReady`
|
||||
(`task_operations.go:2498`), whose budget is `constants.AgentLaunchTimeout` =
|
||||
**15 minutes** (`timeouts.go:44`, 10m setup script + 5m allowance). Only after
|
||||
that does the 30s `waitForAgentPromptReady` apply, and the attempt is retried
|
||||
once. Worst case is roughly 31 minutes per task, four at a time.
|
||||
|
||||
Measured on the affected instance: 424 tasks, 32 carrying
|
||||
`manual_move_lifecycle` metadata, on a 1.0 GB database. A boot given a full
|
||||
10-minute health window still never bound, and produced only 2 successful ACP
|
||||
session creations in those 10 minutes. That is the signature of workers parked
|
||||
in the 15-minute session-ready wait, not of workers cycling through tasks.
|
||||
|
||||
Three consequences, in severity order:
|
||||
|
||||
1. **Socket liveness is coupled to agent readiness.** The process cannot answer
|
||||
a health probe, serve the UI, or expose the API until agents that may never
|
||||
become ready are waited out.
|
||||
2. **The failure is self-sustaining.** `healthTimeoutReleaseMS` is 45s
|
||||
(`internal/launcher/constants.go`). The launcher declares a backend health
|
||||
failure and stops the process; a supervisor restarts it; the same tokens are
|
||||
still pending because none were cleared, and the runtimes orphaned by the
|
||||
kill are added to the next boot's work. Ten consecutive boots were observed,
|
||||
none of which bound a socket.
|
||||
3. **The wait cannot be bounded from outside.** `attemptColdResume` and
|
||||
`reapAndReloadSession` deliberately use `context.WithoutCancel`
|
||||
(`task_operations.go:2308`, `task_operations.go:2255`) so a resume survives
|
||||
its caller's cancellation. A deadline handed to the sweep therefore does not
|
||||
stop an in-flight resume. Any fix that relies on cancelling the sweep to
|
||||
protect startup is ineffective.
|
||||
|
||||
This is startup ordering, not a defect in the recovery logic. The same frames
|
||||
are reached whenever a lifecycle token is pending.
|
||||
|
||||
## Correction (2026-09-05)
|
||||
|
||||
This spec's original Desired-behaviour and Out-of-scope sections asserted
|
||||
that the sweep's position *before* `watcher.Start` was load-bearing and that
|
||||
moving it would introduce races with live traffic. Runtime evidence gathered
|
||||
after this spec's socket-bind fix shipped disproves that: with the socket-bind
|
||||
fix in place, boots with pending lifecycle tokens still never reached `API
|
||||
configured` — the sweep itself was still holding the readiness path open
|
||||
indefinitely, just behind the new bootstrap 503 handler instead of behind a
|
||||
closed socket.
|
||||
|
||||
The reason is a **self-inflicted event blackout**, not merely a slow
|
||||
individual resume. `Watcher.Start` is what registers
|
||||
`{events.AgentBootReady, w.handlers.OnAgentBootReady}` (the *only* handler
|
||||
that flips a launching session to `WAITING_FOR_INPUT` on the resume path).
|
||||
The event bus used here has no replay, so any `agent.boot_ready` published
|
||||
while the sweep runs *before* `watcher.Start` is dropped on the floor.
|
||||
`waitForSessionReady`'s poll can then never observe the state change it is
|
||||
waiting for, and burns its full `constants.AgentLaunchTimeout` (15 minutes)
|
||||
on every recovered task, regardless of whether the underlying agent process
|
||||
is actually healthy. Four workers wide, retried up to three times, this is
|
||||
indistinguishable from a hang and gates the entire router.
|
||||
|
||||
The fix inverts the ordering this spec forbade: the sweep now runs in a
|
||||
background goroutine *after* `watcher.Start` and `scheduler.Start` succeed,
|
||||
so the events its own resume path depends on are deliverable from the
|
||||
moment recovery starts. Readiness is not gated on the sweep at all — the
|
||||
socket-bind / bootstrap-503 behavior this spec introduced is unaffected and
|
||||
still the mechanism that makes an in-progress sweep observable as "starting"
|
||||
rather than a hang. The Desired-behaviour and Out-of-scope sections below
|
||||
are amended accordingly; the regression scenarios still hold, since a bound
|
||||
socket does not depend on this ordering.
|
||||
|
||||
## Broken behavior
|
||||
|
||||
- No socket is bound, and `/health` cannot be answered, until orchestrator
|
||||
startup reconciliation returns.
|
||||
- A backend whose recovery exceeds the health window is killed and restarted
|
||||
with at least as much recovery work as before, indefinitely.
|
||||
|
||||
## Desired behavior
|
||||
|
||||
- The backend binds its socket and answers a liveness probe within the release
|
||||
health window, regardless of how much recovery work is pending.
|
||||
- Startup recovery runs in the background *after* the watcher and scheduler
|
||||
start (amended 2026-09-05 — see Correction above): its own resume path
|
||||
depends on events only the watcher's subscriptions deliver, so running it
|
||||
earlier does not protect anything and instead guarantees every recovered
|
||||
task times out. Running concurrently with live watcher/scheduler traffic is
|
||||
safe because the sweep's operations are already built for concurrent
|
||||
delivery — per-task locks (`queuedMoveLifecycleLocks`) and one-shot tokens
|
||||
claimed by an atomic metadata-key removal (`claimTaskEventMetadata`) —
|
||||
confirmed by
|
||||
`TestLifecycleSweepConcurrentWithLiveTaskMovedRunsManualMoveSideEffectsOnce`,
|
||||
which runs the sweep's recovery path concurrently with a live `task.moved`
|
||||
redelivery of the same admitted manual move and asserts the step exit/enter
|
||||
side effects run exactly once.
|
||||
- Requests that need a started application are answered with an explicit
|
||||
"starting" response rather than by a hung connection or a refused socket.
|
||||
- Readiness is observable and distinct from liveness: an operator can tell
|
||||
"bound, still starting" from "fully started".
|
||||
|
||||
## Regression scenarios
|
||||
|
||||
- **GIVEN** a backend with pending lifecycle tokens whose agent sessions never
|
||||
reach ready, **WHEN** it starts, **THEN** the socket is bound and the
|
||||
liveness probe returns success well inside `healthTimeoutReleaseMS`, and the
|
||||
launcher does not shut the process down.
|
||||
- **GIVEN** the same backend while startup reconciliation is still running,
|
||||
**WHEN** an API route is requested, **THEN** it returns an explicit starting
|
||||
status rather than hanging, and readiness reports not-ready while liveness
|
||||
reports healthy.
|
||||
- **GIVEN** startup reconciliation that has completed, **WHEN** readiness is
|
||||
requested, **THEN** it reports ready and all routes serve normally.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Changing WIP promotion, feeder pull, or manual-move lifecycle semantics.
|
||||
- Changing `AgentLaunchTimeout`, `agentPromptReadyTimeout`, or the cold-resume
|
||||
retry shape (`context.WithoutCancel`). See the follow-up note below — a
|
||||
caller-side deadline on the sweep bounds admission and reporting, not an
|
||||
in-flight resume, precisely because of this contract.
|
||||
- Raising `healthTimeoutReleaseMS`, which hides the coupling instead of fixing
|
||||
it. Empirically a 10-minute window did not help.
|
||||
|
||||
## Known follow-up, addressed 2026-09-05
|
||||
|
||||
The note below originally described work deliberately deferred past this
|
||||
spec's socket-bind fix. It has since been addressed in the same change that
|
||||
corrected the ordering above (see Correction): the sweep now runs off the
|
||||
readiness path entirely (background goroutine, not gating `API configured`),
|
||||
carries its own bounded overall deadline separate from the interactive
|
||||
`AgentLaunchTimeout`, and no longer accumulates already-completed
|
||||
`manual_move_lifecycle_completed` tokens as permanent per-boot work. The
|
||||
`context.WithoutCancel` resume contract itself is unchanged, so an
|
||||
individual in-flight resume still cannot be aborted early — the sweep's
|
||||
deadline bounds its own dispatch loop returning, not that resume.
|
||||
|
||||
> Fixing the ordering stops the crash loop but does not make a large board
|
||||
> start quickly: a backend with many pending tokens still spends a long time
|
||||
> in "starting" because bulk recovery inherits the interactive 15-minute
|
||||
> `AgentLaunchTimeout` per task. A bulk startup sweep needs its own, much
|
||||
> smaller per-task budget, and a resume path that can actually be abandoned.
|
||||
The background lifecycle-token sweep still starts after watcher and scheduler
|
||||
subscriptions. It does not gate readiness. Its per-task locks and atomic metadata
|
||||
claims prevent duplicate effects during live event delivery. This change does not
|
||||
alter its bounded dispatch deadline or in-flight resume behavior.
|
||||
|
||||
Reference in New Issue
Block a user