fix: align SQLite backups with database path (#2686)
This commit is contained in:
Carlos Florêncio
@@ -0,0 +1,135 @@
|
||||
---
|
||||
spec: docs/specs/system-page/spec.md
|
||||
created: 2026-08-15
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Implementation Plan: SQLite backups follow the database path
|
||||
|
||||
## Overview
|
||||
|
||||
Resolve the live SQLite path once in the System composer. Pass that exact path to the Database and Backups services. Both services derive one sibling `backups/` directory from the path.
|
||||
|
||||
This change aligns manual, pre-reset, and pre-migration snapshots. It also makes restore replace the configured file, including a custom filename, after quiescing the SQLite pool and removing stale WAL sidecars.
|
||||
|
||||
## Confirmed root cause
|
||||
|
||||
`persistence.Provide` opens `cfg.Database.Path` and writes pre-migration snapshots under `filepath.Dir(dbPath)/backups`. `system.Provide` instead passes `cfg.ResolvedDataDir()` to the Database and Backups services.
|
||||
|
||||
Both System services then reconstruct `<data-dir>/kandev.db`. They do not receive the live path. This split causes list, create, download, restore, database statistics, and pre-reset snapshots to use the wrong location.
|
||||
|
||||
Manual create still snapshots the live writer pool. Its contents are current, but its output location is wrong. Restore replaces a different file and cannot affect the live custom database.
|
||||
|
||||
A focused test configured `<custom>/custom-name.db` and placed `manual-1.db` under `<custom>/backups/`. `system.Provide(...).Backups.List()` returned an empty list. The command failed at the expected assertion:
|
||||
|
||||
```bash
|
||||
cd apps/backend && go test ./internal/system -run '^TestReproIssue2679BackupsFollowDatabasePath$' -count=1 -v
|
||||
```
|
||||
|
||||
The throwaway test was removed after reproduction.
|
||||
|
||||
## Backend
|
||||
|
||||
### System composition
|
||||
|
||||
Update `apps/backend/internal/system/system.go` to select the SQLite database path once. Use `cfg.Database.Path` when it is set. Otherwise, use `<ResolvedDataDir>/kandev.db`.
|
||||
|
||||
Pass the exact file path to `database.NewService` and `backups.NewService`. Keep home-owned reset directories unchanged.
|
||||
|
||||
### Backups service
|
||||
|
||||
Update `apps/backend/internal/system/backups/store.go` so `Service` owns `databasePath`. Derive the backup directory with `filepath.Join(filepath.Dir(databasePath), "backups")`.
|
||||
|
||||
Use `databasePath` directly for restore staging and replacement. Do not infer the filename `kandev.db`.
|
||||
|
||||
Before replacement, quiesce scheduling, active executions, and database-backed workers. Validate the `wal_checkpoint(TRUNCATE)` result, close the shared SQLite pool, quarantine the main file and its `-wal`/`-shm` sidecars, and restore the quarantine if staged installation fails. Reject PostgreSQL restore before staging or shutdown. The frontend requires a restart after restore, so the closed pool prevents stale WAL frames from replaying onto the restored file.
|
||||
|
||||
### Database service
|
||||
|
||||
Update `apps/backend/internal/system/database/stats.go` so `NewService` accepts the exact database path. Derive its backup directory from the path.
|
||||
|
||||
Use the exact path for the Database page, WAL statistics, last-backup time, and pre-reset snapshots. Update `apps/backend/internal/system/database/reset.go` to use the derived backup directory.
|
||||
|
||||
## Frontend
|
||||
|
||||
Update the restore success dialog to state that the pool is closed and database-backed work is unavailable until restart. Use the existing restart action, with quit-and-relaunch guidance when automatic restart is unavailable.
|
||||
|
||||
## Public documentation
|
||||
|
||||
Remove the warning that System Database and Backups ignore `KANDEV_DATABASE_PATH`. Document the sibling backup directory and exact restore target.
|
||||
|
||||
Update the CLI reference, configuration reference, operations guide, feature-status table, and introductory storage text. State that old misrouted snapshots do not move automatically.
|
||||
|
||||
Update `apps/backend/AGENTS.md` so the backend backup convention names the configured SQLite path. Do not change the historical ADR.
|
||||
|
||||
## Tests
|
||||
|
||||
- **What:** the composed backup handler lists snapshots beside a custom database path.
|
||||
**File:** `apps/backend/internal/system/system_database_path_test.go`.
|
||||
**How:** construct `system.Provide` with a custom filename, register the routes, request the backup list, and assert the sibling snapshot appears.
|
||||
- **What:** restore stages and replaces the exact custom database filename.
|
||||
**File:** `apps/backend/internal/system/backups/path_test.go`.
|
||||
**How:** restore known bytes through the service and assert the configured file changes, the quiescence hook runs, the pool closes, the replacement sidecars are gone, and the default-home file remains absent. Add regressions for PostgreSQL rejection, busy checkpoint preservation, and rollback after replacement failure.
|
||||
- **What:** restore quiescence stops every database writer before pool closure.
|
||||
**File:** `apps/backend/internal/backendapp/restore_quiesce_test.go`.
|
||||
**How:** assert scheduling, orchestrator, agents, and registered workers are all attempted and all stop errors are returned.
|
||||
- **What:** the restore success dialog directs the user to restart.
|
||||
**File:** `apps/web/components/settings/system/system-confirmation-dialogs.test.tsx`.
|
||||
**How:** render a succeeded restore job and assert the restart action is present while the normal close action is absent.
|
||||
- **What:** database statistics and pre-reset snapshots use the custom path and its sibling backup directory.
|
||||
**File:** `apps/backend/internal/system/database/path_test.go`.
|
||||
**How:** assert the reported path, WAL lookup, last-backup time, and pre-reset snapshot parent.
|
||||
- **What:** default-path behavior remains unchanged.
|
||||
**Files:** existing Backups and Database package tests.
|
||||
**How:** update constructor inputs to exact database paths and run all `internal/system/...` tests.
|
||||
|
||||
## E2E Tests
|
||||
|
||||
No new browser test is planned. The dialog keeps the existing shared responsive composition; this remediation changes localized success copy and invokes the existing restart flow. The focused component test covers the changed rendered action.
|
||||
|
||||
## Verification Results
|
||||
|
||||
- `cd apps/backend && go test -race ./internal/system/backups ./internal/system ./internal/backendapp -run 'TestRestore|TestConfiguredDatabasePath|TestProvideWiresRestoreQuiesce|TestQuiesceForRestore' -count=1`
|
||||
passed 11 restore-safety tests across 3 packages.
|
||||
- `cd apps/backend && go test ./internal/system/... ./internal/backendapp -count=1`
|
||||
passed 844 tests across 20 packages.
|
||||
- `cd apps/backend && make lint` passed with 0 issues.
|
||||
- `cd apps && pnpm --filter @kandev/web exec vitest run components/settings/system/system-confirmation-dialogs.test.tsx`
|
||||
passed 5 UI tests.
|
||||
- `cd apps && pnpm --filter @kandev/web run i18n:check`, `i18n:ratchet`, and `typecheck`
|
||||
passed.
|
||||
- `rg -n 'KANDEV_DATABASE_PATH|data/backups|database path|backup caveat' docs/public apps/backend/AGENTS.md`
|
||||
confirmed the corrected terminology and intentional default-path references.
|
||||
- `node --test scripts/validate-public-docs.test.mjs` passed 61 tests.
|
||||
- `node scripts/validate-public-docs.mjs` validated 41 published docs pages.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Restore safety remediation
|
||||
|
||||
- Red phase: PostgreSQL restore was accepted, a busy checkpoint was treated as success, and sidecars were deleted before a replacement failure could roll back.
|
||||
- Green phase: focused backend restore, system wiring, and quiescence tests pass after rejecting PostgreSQL, validating checkpoint rows, and using rollback-capable quarantine replacement.
|
||||
|
||||
## Implementation Waves And Parallel Candidates
|
||||
|
||||
Wave 1 (sequential):
|
||||
|
||||
- [x] [task-01-backend-database-path](task-01-backend-database-path.md)
|
||||
|
||||
Wave 2 (sequential):
|
||||
|
||||
- [x] [task-02-document-database-path](task-02-document-database-path.md)
|
||||
|
||||
The documentation task depends on the final backend behavior. No task is parallel-safe.
|
||||
|
||||
## Risks
|
||||
|
||||
- Existing manual snapshots can remain under the old default directory. Automatic movement is unsafe because Kandev cannot prove which database produced each file.
|
||||
- A custom database filename must remain exact. Joining `kandev.db` to its parent preserves part of the bug.
|
||||
- The System Status disk-usage service remains home-based. It does not count sibling backups outside the resolved home.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Moving or merging old snapshot directories.
|
||||
- Changing snapshot filename families or retention rules.
|
||||
- Changing PostgreSQL backup behavior.
|
||||
- Changing the System Status disk-usage path model.
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
id: "01-backend-database-path"
|
||||
title: "Route System backups to the SQLite path"
|
||||
status: done
|
||||
wave: 1
|
||||
depends_on: []
|
||||
plan: "plan.md"
|
||||
spec: "../../specs/system-page/spec.md"
|
||||
---
|
||||
|
||||
# Task 01: Route System backups to the SQLite path
|
||||
|
||||
## Intent
|
||||
|
||||
Make the System Database and Backups services use the live SQLite file path. Keep all snapshot families in one sibling `backups/` directory.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A custom SQLite filename is the reported database path and the restore destination.
|
||||
- Manual, pre-reset, and pre-migration snapshots use the `backups/` directory beside that file.
|
||||
- Restore quiesces scheduling, active executions, and database-backed workers, validates the checkpoint result, closes the SQLite pool, and uses rollback-capable quarantine replacement. PostgreSQL restore is rejected before staging or shutdown. A restart is required before database-backed work resumes.
|
||||
- The default `<home>/data/kandev.db` behavior and existing backup safety rules remain unchanged.
|
||||
|
||||
## Files likely touched
|
||||
|
||||
- `apps/backend/internal/system/system.go`
|
||||
- `apps/backend/internal/system/system_database_path_test.go`
|
||||
- `apps/backend/internal/system/backups/store.go`
|
||||
- `apps/backend/internal/system/backups/store_test.go`
|
||||
- `apps/backend/internal/system/backups/path_test.go`
|
||||
- `apps/backend/internal/system/database/stats.go`
|
||||
- `apps/backend/internal/system/database/stats_test.go`
|
||||
- `apps/backend/internal/system/database/reset.go`
|
||||
- `apps/backend/internal/system/database/path_test.go`
|
||||
- `apps/backend/internal/backendapp/restore_quiesce_test.go`
|
||||
|
||||
## Dependencies
|
||||
|
||||
None.
|
||||
|
||||
## Parallelism
|
||||
|
||||
Sequential. The constructors and the System composer share one path contract.
|
||||
|
||||
## Inputs
|
||||
|
||||
- Spec: `docs/specs/system-page/spec.md`, Database, Backups, scenarios, failure modes, and persistence guarantees.
|
||||
- Plan: `plan.md`, confirmed root cause and backend sections.
|
||||
- Existing pre-migration rule: `apps/backend/internal/persistence/provider.go` derives `backups/` from the live database path.
|
||||
- Issue: `https://github.com/kdlbs/kandev/issues/2679`.
|
||||
|
||||
## Verification
|
||||
|
||||
Write the custom-path regression cases first. Run them before production changes and make sure that they fail for the path split:
|
||||
|
||||
```bash
|
||||
cd apps/backend && go test ./internal/system/... -run 'Test.*(Configured|Custom)DatabasePath' -count=1 -v
|
||||
```
|
||||
|
||||
After the fix, run the custom-path regressions and all affected System packages:
|
||||
|
||||
```bash
|
||||
cd apps/backend && go test ./internal/system/... -run 'Test.*(Configured|Custom)DatabasePath' -count=1 -v
|
||||
cd apps/backend && go test ./internal/system/... -count=1
|
||||
git diff --check
|
||||
```
|
||||
|
||||
## Output contract
|
||||
|
||||
Report the exact changed files and test counts. Include the expected red failure, green results, and cleanup evidence. Update this task and `plan.md` in the same conversation.
|
||||
|
||||
## Results
|
||||
|
||||
- Red phase: the four custom-path regressions failed on the old path split. The
|
||||
composer listed the old default backup, restore could not reach the custom
|
||||
filename, stats missed the custom WAL and backup, and reset used the wrong
|
||||
parent directory.
|
||||
- Green phase: `cd apps/backend && go test -race ./internal/system/backups
|
||||
./internal/system ./internal/backendapp -run 'TestRestore|TestConfiguredDatabasePath|TestProvideWiresRestoreQuiesce|TestQuiesceForRestore' -count=1`
|
||||
passed 11 restore-safety tests across 3 packages, including restore quiescing.
|
||||
- Full affected suite: `cd apps/backend && go test ./internal/system/... ./internal/backendapp -count=1`
|
||||
passed 844 tests across 20 packages.
|
||||
- `cd apps/backend && make lint` passed with 0 issues.
|
||||
- Changed files: `system.go`, `system_database_path_test.go`,
|
||||
`backups/store.go`, `backups/store_test.go`, `backups/path_test.go`,
|
||||
`database/stats.go`, `database/stats_test.go`, `database/reset.go`,
|
||||
`database/maintenance_test.go`, and `database/path_test.go`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Restore safety remediation
|
||||
|
||||
- Red phase: the PostgreSQL path closed the pool, busy checkpoint rows were ignored, and replacement failures deleted live sidecars.
|
||||
- Green phase: restore safety tests pass for PostgreSQL rejection, busy checkpoint preservation, rollback of the main file and sidecars, and the complete quiescence callback.
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
id: "02-document-database-path"
|
||||
title: "Document the SQLite backup location"
|
||||
status: done
|
||||
wave: 2
|
||||
depends_on: ["01-backend-database-path"]
|
||||
plan: "plan.md"
|
||||
spec: "../../specs/system-page/spec.md"
|
||||
---
|
||||
|
||||
# Task 02: Document the SQLite backup location
|
||||
|
||||
## Intent
|
||||
|
||||
Replace the documented custom-path limitation with the corrected System behavior. Keep the legacy-snapshot and home-storage boundaries explicit.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Public references state that SQLite snapshots use `backups/` beside the configured database file.
|
||||
- Restore documentation names `<configured-database-path>.new` and the exact configured destination.
|
||||
- Restore documentation explains that scheduling, active executions, and database-backed workers stop, the SQLite pool closes, replacement is rollback-safe, PostgreSQL restore is rejected, and an immediate restart is required.
|
||||
- Public and agent guidance states that old misrouted snapshots do not move automatically.
|
||||
|
||||
## Files likely touched
|
||||
|
||||
- `docs/public/cli.md`
|
||||
- `docs/public/configuration.md`
|
||||
- `docs/public/operations.md`
|
||||
- `docs/public/feature-status.md`
|
||||
- `docs/public/use-kandev.md`
|
||||
- `apps/backend/AGENTS.md`
|
||||
- `apps/web/components/settings/system/restore-dialog.tsx`
|
||||
- `apps/web/src/locales/*/system.json`
|
||||
|
||||
## Dependencies
|
||||
|
||||
Task 01 must establish the final backend behavior.
|
||||
|
||||
## Parallelism
|
||||
|
||||
Sequential. These pages document the result of Task 01.
|
||||
|
||||
## Inputs
|
||||
|
||||
- Spec: `docs/specs/system-page/spec.md`, configured database path scenarios and persistence guarantees.
|
||||
- Plan: `plan.md`, public documentation and risk sections.
|
||||
- Docs type: `configuration.md` and `cli.md` are references. `operations.md` is a how-to guide. `feature-status.md` is a reference. `use-kandev.md` is a tutorial.
|
||||
|
||||
## Verification
|
||||
|
||||
Run the focused terminology search and public-doc validators:
|
||||
|
||||
```bash
|
||||
rg -n 'KANDEV_DATABASE_PATH|data/backups|database path|backup caveat' docs/public apps/backend/AGENTS.md
|
||||
node --test scripts/validate-public-docs.test.mjs
|
||||
node scripts/validate-public-docs.mjs
|
||||
git diff --check
|
||||
```
|
||||
|
||||
## Output contract
|
||||
|
||||
Report each changed public page and its primary documentation type. Report validator results and remaining intentional `data/backups` references. Update this task and `plan.md` in the same conversation.
|
||||
|
||||
## Restore safety remediation
|
||||
|
||||
- Updated the public and backend guidance for checkpoint validation, quarantine rollback, PostgreSQL rejection, and the closed-pool restart requirement.
|
||||
- Updated the localized restore success dialog to invoke the existing restart flow and to provide manual quit-and-relaunch guidance.
|
||||
|
||||
## Results
|
||||
|
||||
- Public docs updated:
|
||||
- `docs/public/cli.md` (reference)
|
||||
- `docs/public/configuration.md` (reference)
|
||||
- `docs/public/operations.md` (how-to and recovery guide)
|
||||
- `docs/public/feature-status.md` (reference)
|
||||
- `docs/public/use-kandev.md` (tutorial)
|
||||
- Backend guidance updated: `apps/backend/AGENTS.md` now names the configured
|
||||
SQLite path and its sibling backup directory.
|
||||
- `rg -n 'KANDEV_DATABASE_PATH|data/backups|database path|backup caveat'
|
||||
docs/public apps/backend/AGENTS.md` passed. Remaining `data/backups`
|
||||
references describe the default path, development safety snapshots, or the
|
||||
home-based System Status walk.
|
||||
- `node --test scripts/validate-public-docs.test.mjs` passed 61 tests.
|
||||
- `node scripts/validate-public-docs.mjs` validated 41 published docs pages.
|
||||
- `git diff --check` passed.
|
||||
Reference in New Issue
Block a user