fix: recover failed task launches safely (#3437)

This commit is contained in:
Carlos Florêncio
2026-09-07 11:19:09 +01:00
committed by GitHub
parent 3c6af41063
commit 3fba828e2f
69 changed files with 2280 additions and 277 deletions
@@ -0,0 +1,159 @@
---
created: 2026-09-05
status: completed
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
- REQ-TASKS-REMOTE-CONTRIBUTION-TASKS-001
system_design:
- ../../specs/tasks/system-design/task-launch-failure-recovery.md
- ../../specs/tasks/system-design/remote-contribution-tasks.md
legacy_specs: []
---
# Implementation Plan: Task Launch Error Consolidation
## Overview
The change first isolates pull-request refs from reused local branch names.
Then it makes recovery actions category-specific. The final work order gives
one error card ownership of the desktop and phone presentation.
## Confirmed root cause
PR #3408 uses `feature/unit-7-reject-leg-wa-waj` at
`a76c5d8de780b0b857af8195eab4772bb1a5839b`. The shared clone still has the same
local branch from PR #3294 at `f825122f586978ed5e33fe48d61e401a610318e2`.
Neither commit contains the other. The required fetch tried to update the local
branch directly. Git rejected the update as non-fast-forward, and preparation
stopped before the agent started.
The backend stored one typed launch error. The frontend also derived a previous
agent error, a failed preparation row, an empty-turn warning, a generic failed
status, and a stopped-session recovery banner from the same failed start.
## Scope
### In scope
- Preserve a reused local branch and prepare the exact pull-request head.
- Classify checkout preparation separately from base-branch errors.
- Offer only category-valid recovery actions.
- Render one matching launch-error card on desktop and phone.
- Keep bounded technical details in one disclosure.
- Prove the error and recovery flows with unit and Playwright tests.
### Out of scope
- Reconcile provider and local history after an agent starts.
- Change generic runtime-error recovery after successful agent startup.
- Add new provider-specific branch pickers.
- Change pull, push, merge, rebase, or force-push policy.
## Technical approach
### Pull-request ref isolation
Update `apps/backend/internal/worktree/manager_lifecycle.go` to fetch a
pull-request head into the Kandev-owned `refs/kandev/pull/<N>/head` ref. Verify
that full ref before worktree creation. If a same-named local branch has
unrelated history, create a unique task branch from the verified ref using a
task-owned suffix and bounded collision retries. Set a source-branch upstream
only when its commit matches the verified PR start point.
Keep the named local branch unchanged. Continue to use the existing Git
admission, timeout, and non-interactive environment.
### Failure category and recovery
Add `workspace_checkout_failed` and `retry_launch` to
`apps/backend/internal/task/models/launch_errors.go`. Preserve unknown-value
filtering in the task-status projection.
Update `apps/backend/internal/orchestrator/executor/launch_failure.go` so the
failure category selects the action list. A repository row must not add branch
actions to a generic error.
Extend `task.launch.recover` in
`apps/backend/internal/orchestrator/task_launch_recovery.go` and its WebSocket
handler. `retry_launch` must validate the current stamp and relaunch without a
base-branch write.
### Single frontend owner
Derive one ownership flag in `apps/web/components/task/task-chat-panel.tsx` from
the active error, session identity, and stamp. Pass this flag to the transcript
and composer boundaries.
Update `TaskLaunchErrorEntry` with category-specific titles, cause text, a
no-change statement, and a technical-details disclosure. Keep action state and
WebSocket requests in the existing component.
When the flag is active, suppress the matching last-agent-error notice,
preparation row, empty-turn notice, failed-agent status, stopped-session banner,
and focused-task toast. Do not suppress unrelated historical or runtime errors.
### Mobile contract
Desktop and phone use the task Chat surface. The phone card stacks full-width
actions. All touch actions have at least a 44-pixel hit area.
The task transcript owns scrolling. The details disclosure wraps its bounded
content without an inner scroll region. Base-branch selection keeps the existing
`MobilePickerSheet` behavior.
## Tests
- `apps/backend/internal/worktree/manager_pr_head_collision_test.go` covers two
pull requests that reuse one branch name with divergent histories.
- `apps/backend/internal/orchestrator/executor/executor_launch_failure_classification_test.go`
covers category-specific actions.
- `apps/backend/internal/orchestrator/task_launch_recovery_test.go` covers
authorized, stale, and failed `retry_launch` requests.
- Frontend tests cover one-owner arbitration, the details disclosure, action
payloads, launch-error ownership in the composer, recovery-state reset, and
preservation of unrelated runtime errors.
## E2E tests
- `apps/web/e2e/tests/task/launch-failure-recovery.spec.ts` proves one desktop
error card, one retry action, and no matching duplicate surfaces.
- `apps/web/e2e/tests/task/mobile-launch-failure-recovery.spec.ts` proves the
same result on `mobile-chrome`, 44-pixel actions, and no horizontal overflow.
These tests map to
`AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.2`,
`AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.7`, and
`AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.8`.
## Work orders
- [x] [Task 01: Isolate reused pull-request refs](task-01-isolate-reused-pr-refs.md)
- [x] [Task 02: Classify launch recovery actions](task-02-classify-launch-recovery-actions.md)
- [x] [Task 03: Consolidate launch-error presentation](task-03-consolidate-launch-error-presentation.md)
## Verification results
- Backend focused race coverage passed: 1,212 tests across worktree, task
models, status summary, and executor packages, plus 11 targeted task-launch
recovery tests.
- Frontend focused coverage passed: 10 Vitest files and 152 tests.
- Review-remediation frontend coverage passed: 7 Vitest files and 75 tests,
including task-wide launch-error visibility, composer ownership, and
recovery-state reset.
- `pnpm run typecheck`, `pnpm run lint`, `pnpm run i18n:check`, and
`pnpm run i18n:ratchet` passed.
- Managed Playwright coverage passed: desktop launch-failure recovery 3/3 and
mobile launch-failure recovery 2/2.
- The broader orchestrator package run was attempted but exceeded its timeout
in an existing SQLite/concurrency test; the focused race and recovery suites
passed independently.
## Risks
- A ref-selection error can discard local-only commits. Tests must prove that
the existing branch remains unchanged.
- Error events can arrive in a different order from task-status updates. The
owner logic must converge after either event order.
- Old persisted generic errors can contain branch actions. Projection and UI
normalization must not show actions that conflict with the current category.
@@ -0,0 +1,85 @@
---
id: "01-isolate-reused-pr-refs"
title: "Isolate reused pull-request refs"
status: completed
wave: 1
depends_on: []
plan: "plan.md"
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
- REQ-TASKS-REMOTE-CONTRIBUTION-TASKS-001
acceptance_criteria:
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.2
- AC-TASKS-REMOTE-CONTRIBUTION-TASKS-001.2
system_design:
- ../../specs/tasks/system-design/task-launch-failure-recovery.md
- ../../specs/tasks/system-design/remote-contribution-tasks.md
---
# Task 01: Isolate reused pull-request refs
## Summary
Prepare each pull-request head through a PR-scoped internal ref. Preserve any
same-named local branch that contains different history.
## In scope
- Fetch and verify a pull-request head without updating the named local branch.
- Create a unique task branch when the named branch has unrelated history.
- Preserve Git admission, timeout, credential, and cancellation behavior.
## Out of scope
- Provider-history reconciliation after launch.
- Remote branch mutation.
- Changes to non-pull-request local branch refresh.
## Acceptance
- Two pull requests can reuse one head-branch name without a fetch conflict.
- The new worktree starts at the requested pull-request head.
- The prior local branch and its commit remain unchanged.
## Verification
```bash
cd apps/backend && go test ./internal/worktree -race
```
## Files likely touched
- `apps/backend/internal/worktree/manager_lifecycle.go`
- `apps/backend/internal/worktree/manager_pr_head_collision_test.go`
## Dependencies
None.
## Risks
- Ref cleanup must not remove a user-owned ref.
- A local-ahead branch must keep its local-only commits.
## Parallelism
`sequential`
## Inputs
- Task launch failure recovery design, pull-request checkout isolation.
- Remote contribution task requirement `AC-TASKS-REMOTE-CONTRIBUTION-TASKS-001.2`.
- ADR `2026-08-10-remote-contribution-head-drift`.
## Results
- Added a Kandev-owned `refs/kandev/pull/<N>/head` fetch ref with full-ref
verification and typed `ErrWorkspaceCheckoutFailed` failures.
- PR worktrees now create a unique local branch when the requested source name
already exists, without resetting or updating that branch. The fallback
uses a task-owned suffix and bounded retries for branch-name collisions.
- PR worktrees only track the source branch when its remote-tracking commit
matches the verified PR head. Ordinary `fetch --all --prune` cannot remove
or replace the internal snapshot.
- Recreate restores deleted PR branches from the isolated ref.
- Verification passed: `go test ./internal/worktree` (368 passed).
@@ -0,0 +1,94 @@
---
id: "02-classify-launch-recovery-actions"
title: "Classify launch recovery actions"
status: completed
wave: 2
depends_on:
- "01-isolate-reused-pr-refs"
plan: "plan.md"
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
acceptance_criteria:
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.2
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.4
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.8
system_design:
- ../../specs/tasks/system-design/task-launch-failure-recovery.md
---
# Task 02: Classify launch recovery actions
## Summary
Give checkout preparation a stable error category. Derive recovery actions from
the category and add a stamp-checked launch retry.
## In scope
- Add `workspace_checkout_failed` and `retry_launch` wire values.
- Map checkout preparation errors to the new category.
- Make every action list category-specific.
- Relaunch without a base-branch write for `retry_launch`.
- Keep authorization and current-stamp checks before relaunch.
## Out of scope
- New persistence tables.
- Automatic retries.
- Changes to `session.recover`.
## Acceptance
- A checkout error offers only `retry_launch`.
- A generic error does not offer base-branch actions because a repository row exists.
- A stale or foreign retry request makes no change and starts no session.
## Verification
```bash
cd apps/backend && go test ./internal/task/models ./internal/orchestrator/executor ./internal/orchestrator/... -race
```
## Files likely touched
- `apps/backend/internal/worktree/errors.go`
- `apps/backend/internal/task/models/launch_errors.go`
- `apps/backend/internal/task/models/launch_errors_test.go`
- `apps/backend/internal/orchestrator/executor/launch_failure.go`
- `apps/backend/internal/orchestrator/executor/executor_launch_failure_classification_test.go`
- `apps/backend/internal/orchestrator/task_launch_recovery.go`
- `apps/backend/internal/orchestrator/task_launch_recovery_test.go`
- `apps/backend/internal/orchestrator/handlers/handlers.go`
## Dependencies
Task 01.
## Risks
- Old persisted action arrays can conflict with the new category matrix.
- A retry must not clear the current error before launch succeeds.
## Parallelism
`sequential`
## Inputs
- Task launch failure recovery design, error model and recovery contract.
- Existing task-status projection normalization tests.
## Results
- Added the `workspace_checkout_failed` category and `retry_launch` action.
- Checkout failures now use category-specific recovery actions, including
retry without changing a task repository base branch.
- Task-scoped recovery validates the current error stamp and repository
identity before relaunch or mutation.
- Persisted and projected errors normalize action lists by category and keep
bounded technical details.
- Focused race coverage passed: 1,212 tests across worktree, task models,
status summary, and executor packages. The 11 targeted task-launch recovery
tests also passed.
- The broader orchestrator package run was attempted but exceeded its timeout
in an existing SQLite/concurrency test.
@@ -0,0 +1,108 @@
---
id: "03-consolidate-launch-error-presentation"
title: "Consolidate launch-error presentation"
status: completed
wave: 3
depends_on:
- "02-classify-launch-recovery-actions"
plan: "plan.md"
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
acceptance_criteria:
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.2
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.7
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.8
system_design:
- ../../specs/tasks/system-design/task-launch-failure-recovery.md
---
# Task 03: Consolidate launch-error presentation
## Summary
Make the task launch card the only primary presentation for its matching error.
Give desktop and phone users the same cause, details, and valid recovery.
## In scope
- Derive separate launch-card visibility and session-surface ownership from
session identity and error stamp.
- Add clear category text, a no-change statement, and a details disclosure.
- Suppress matching duplicate notices, rows, banners, empty-turn copy, and toasts.
- Preserve unrelated historical and runtime errors, including stale
launch-only transcript rows.
- Add all required locale values.
- Add desktop and mobile Playwright coverage.
## Out of scope
- Redesign completed-session UI.
- Change non-launch runtime-error recovery.
- Change the mobile branch picker.
## Acceptance
- One failed start renders one error card after all events converge.
- Recovery failure updates the same card without another error surface.
- Phone actions remain reachable, at least 44 pixels high, and free of horizontal overflow.
## Verification
```bash
cd apps && pnpm install --frozen-lockfile
cd apps/web && pnpm exec vitest run components/task/simple/components/task-launch-error-entry.test.tsx components/task/chat/chat-input-container.test.tsx components/task/chat/message-list-footer.test.tsx components/task/chat/session-stopped-banner.test.tsx components/task/chat/stopped-banner-props.test.ts components/task/chat/types.test.ts lib/ws/handlers/empty-turn-notice.test.ts
cd apps/web && pnpm run typecheck && pnpm run i18n:check && pnpm run i18n:ratchet
cd apps/web && pnpm e2e:run tests/task/launch-failure-recovery.spec.ts
cd apps/web && pnpm e2e:run --project mobile-chrome tests/task/mobile-launch-failure-recovery.spec.ts
```
## Files likely touched
- `apps/web/components/task/task-chat-panel.tsx`
- `apps/web/components/task/task-launch-error-context.tsx`
- `apps/web/components/task/simple/components/task-launch-error-entry.tsx`
- `apps/web/components/task/chat/message-list-shared.tsx`
- `apps/web/components/task/chat/message-list-footer.tsx`
- `apps/web/components/task/chat/chat-input-container.tsx`
- `apps/web/lib/ws/handlers/empty-turn-notice.ts`
- Focused tests beside these files.
- `apps/web/src/locales/*/task.json`
- `apps/web/e2e/tests/task/launch-failure-recovery.spec.ts`
- `apps/web/e2e/tests/task/mobile-launch-failure-recovery.spec.ts`
## Dependencies
Task 02.
## Risks
- Event order can briefly show a derived error before the typed record arrives.
- Broad suppression can hide a later agent runtime error.
## Parallelism
`sequential`
## Inputs
- Task launch failure recovery design, presentation ownership and responsive sections.
- Mobile exemplar `apps/web/components/task/mobile/mobile-picker-sheet.tsx`.
- Existing desktop and mobile launch-failure tests.
## Results
- The typed task launch card is the single owner for its matching session and
error stamp. Matching preparation, empty-turn, agent-error, failed-status,
stopped-session, and toast surfaces are suppressed without hiding unrelated
runtime errors.
- The card now shows category-specific copy, bounded technical details, a
no-change statement, inline recovery failures, and mobile-sized stacked
actions.
- Added complete locale values and generated the pseudo-locale catalog.
- Focused frontend coverage passed: 10 Vitest files and 152 tests.
- `pnpm run typecheck`, `pnpm run lint`, `pnpm run i18n:check`, and
`pnpm run i18n:ratchet` passed.
- Managed Playwright coverage passed: desktop launch-failure recovery 3/3 and
mobile launch-failure recovery 2/2.
- Review-remediation coverage passed: 7 focused Vitest files and 75 tests,
including composer ownership and launch-error recovery-state reset.