fix: recover failed task launches safely (#3437)

This commit is contained in:
Carlos Florêncio
2026-09-07 11:19:09 +01:00
committed by GitHub
parent 3c6af41063
commit 3fba828e2f
69 changed files with 2280 additions and 277 deletions
@@ -0,0 +1,159 @@
---
created: 2026-09-05
status: completed
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
- REQ-TASKS-REMOTE-CONTRIBUTION-TASKS-001
system_design:
- ../../specs/tasks/system-design/task-launch-failure-recovery.md
- ../../specs/tasks/system-design/remote-contribution-tasks.md
legacy_specs: []
---
# Implementation Plan: Task Launch Error Consolidation
## Overview
The change first isolates pull-request refs from reused local branch names.
Then it makes recovery actions category-specific. The final work order gives
one error card ownership of the desktop and phone presentation.
## Confirmed root cause
PR #3408 uses `feature/unit-7-reject-leg-wa-waj` at
`a76c5d8de780b0b857af8195eab4772bb1a5839b`. The shared clone still has the same
local branch from PR #3294 at `f825122f586978ed5e33fe48d61e401a610318e2`.
Neither commit contains the other. The required fetch tried to update the local
branch directly. Git rejected the update as non-fast-forward, and preparation
stopped before the agent started.
The backend stored one typed launch error. The frontend also derived a previous
agent error, a failed preparation row, an empty-turn warning, a generic failed
status, and a stopped-session recovery banner from the same failed start.
## Scope
### In scope
- Preserve a reused local branch and prepare the exact pull-request head.
- Classify checkout preparation separately from base-branch errors.
- Offer only category-valid recovery actions.
- Render one matching launch-error card on desktop and phone.
- Keep bounded technical details in one disclosure.
- Prove the error and recovery flows with unit and Playwright tests.
### Out of scope
- Reconcile provider and local history after an agent starts.
- Change generic runtime-error recovery after successful agent startup.
- Add new provider-specific branch pickers.
- Change pull, push, merge, rebase, or force-push policy.
## Technical approach
### Pull-request ref isolation
Update `apps/backend/internal/worktree/manager_lifecycle.go` to fetch a
pull-request head into the Kandev-owned `refs/kandev/pull/<N>/head` ref. Verify
that full ref before worktree creation. If a same-named local branch has
unrelated history, create a unique task branch from the verified ref using a
task-owned suffix and bounded collision retries. Set a source-branch upstream
only when its commit matches the verified PR start point.
Keep the named local branch unchanged. Continue to use the existing Git
admission, timeout, and non-interactive environment.
### Failure category and recovery
Add `workspace_checkout_failed` and `retry_launch` to
`apps/backend/internal/task/models/launch_errors.go`. Preserve unknown-value
filtering in the task-status projection.
Update `apps/backend/internal/orchestrator/executor/launch_failure.go` so the
failure category selects the action list. A repository row must not add branch
actions to a generic error.
Extend `task.launch.recover` in
`apps/backend/internal/orchestrator/task_launch_recovery.go` and its WebSocket
handler. `retry_launch` must validate the current stamp and relaunch without a
base-branch write.
### Single frontend owner
Derive one ownership flag in `apps/web/components/task/task-chat-panel.tsx` from
the active error, session identity, and stamp. Pass this flag to the transcript
and composer boundaries.
Update `TaskLaunchErrorEntry` with category-specific titles, cause text, a
no-change statement, and a technical-details disclosure. Keep action state and
WebSocket requests in the existing component.
When the flag is active, suppress the matching last-agent-error notice,
preparation row, empty-turn notice, failed-agent status, stopped-session banner,
and focused-task toast. Do not suppress unrelated historical or runtime errors.
### Mobile contract
Desktop and phone use the task Chat surface. The phone card stacks full-width
actions. All touch actions have at least a 44-pixel hit area.
The task transcript owns scrolling. The details disclosure wraps its bounded
content without an inner scroll region. Base-branch selection keeps the existing
`MobilePickerSheet` behavior.
## Tests
- `apps/backend/internal/worktree/manager_pr_head_collision_test.go` covers two
pull requests that reuse one branch name with divergent histories.
- `apps/backend/internal/orchestrator/executor/executor_launch_failure_classification_test.go`
covers category-specific actions.
- `apps/backend/internal/orchestrator/task_launch_recovery_test.go` covers
authorized, stale, and failed `retry_launch` requests.
- Frontend tests cover one-owner arbitration, the details disclosure, action
payloads, launch-error ownership in the composer, recovery-state reset, and
preservation of unrelated runtime errors.
## E2E tests
- `apps/web/e2e/tests/task/launch-failure-recovery.spec.ts` proves one desktop
error card, one retry action, and no matching duplicate surfaces.
- `apps/web/e2e/tests/task/mobile-launch-failure-recovery.spec.ts` proves the
same result on `mobile-chrome`, 44-pixel actions, and no horizontal overflow.
These tests map to
`AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.2`,
`AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.7`, and
`AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.8`.
## Work orders
- [x] [Task 01: Isolate reused pull-request refs](task-01-isolate-reused-pr-refs.md)
- [x] [Task 02: Classify launch recovery actions](task-02-classify-launch-recovery-actions.md)
- [x] [Task 03: Consolidate launch-error presentation](task-03-consolidate-launch-error-presentation.md)
## Verification results
- Backend focused race coverage passed: 1,212 tests across worktree, task
models, status summary, and executor packages, plus 11 targeted task-launch
recovery tests.
- Frontend focused coverage passed: 10 Vitest files and 152 tests.
- Review-remediation frontend coverage passed: 7 Vitest files and 75 tests,
including task-wide launch-error visibility, composer ownership, and
recovery-state reset.
- `pnpm run typecheck`, `pnpm run lint`, `pnpm run i18n:check`, and
`pnpm run i18n:ratchet` passed.
- Managed Playwright coverage passed: desktop launch-failure recovery 3/3 and
mobile launch-failure recovery 2/2.
- The broader orchestrator package run was attempted but exceeded its timeout
in an existing SQLite/concurrency test; the focused race and recovery suites
passed independently.
## Risks
- A ref-selection error can discard local-only commits. Tests must prove that
the existing branch remains unchanged.
- Error events can arrive in a different order from task-status updates. The
owner logic must converge after either event order.
- Old persisted generic errors can contain branch actions. Projection and UI
normalization must not show actions that conflict with the current category.
@@ -0,0 +1,85 @@
---
id: "01-isolate-reused-pr-refs"
title: "Isolate reused pull-request refs"
status: completed
wave: 1
depends_on: []
plan: "plan.md"
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
- REQ-TASKS-REMOTE-CONTRIBUTION-TASKS-001
acceptance_criteria:
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.2
- AC-TASKS-REMOTE-CONTRIBUTION-TASKS-001.2
system_design:
- ../../specs/tasks/system-design/task-launch-failure-recovery.md
- ../../specs/tasks/system-design/remote-contribution-tasks.md
---
# Task 01: Isolate reused pull-request refs
## Summary
Prepare each pull-request head through a PR-scoped internal ref. Preserve any
same-named local branch that contains different history.
## In scope
- Fetch and verify a pull-request head without updating the named local branch.
- Create a unique task branch when the named branch has unrelated history.
- Preserve Git admission, timeout, credential, and cancellation behavior.
## Out of scope
- Provider-history reconciliation after launch.
- Remote branch mutation.
- Changes to non-pull-request local branch refresh.
## Acceptance
- Two pull requests can reuse one head-branch name without a fetch conflict.
- The new worktree starts at the requested pull-request head.
- The prior local branch and its commit remain unchanged.
## Verification
```bash
cd apps/backend && go test ./internal/worktree -race
```
## Files likely touched
- `apps/backend/internal/worktree/manager_lifecycle.go`
- `apps/backend/internal/worktree/manager_pr_head_collision_test.go`
## Dependencies
None.
## Risks
- Ref cleanup must not remove a user-owned ref.
- A local-ahead branch must keep its local-only commits.
## Parallelism
`sequential`
## Inputs
- Task launch failure recovery design, pull-request checkout isolation.
- Remote contribution task requirement `AC-TASKS-REMOTE-CONTRIBUTION-TASKS-001.2`.
- ADR `2026-08-10-remote-contribution-head-drift`.
## Results
- Added a Kandev-owned `refs/kandev/pull/<N>/head` fetch ref with full-ref
verification and typed `ErrWorkspaceCheckoutFailed` failures.
- PR worktrees now create a unique local branch when the requested source name
already exists, without resetting or updating that branch. The fallback
uses a task-owned suffix and bounded retries for branch-name collisions.
- PR worktrees only track the source branch when its remote-tracking commit
matches the verified PR head. Ordinary `fetch --all --prune` cannot remove
or replace the internal snapshot.
- Recreate restores deleted PR branches from the isolated ref.
- Verification passed: `go test ./internal/worktree` (368 passed).
@@ -0,0 +1,94 @@
---
id: "02-classify-launch-recovery-actions"
title: "Classify launch recovery actions"
status: completed
wave: 2
depends_on:
- "01-isolate-reused-pr-refs"
plan: "plan.md"
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
acceptance_criteria:
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.2
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.4
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.8
system_design:
- ../../specs/tasks/system-design/task-launch-failure-recovery.md
---
# Task 02: Classify launch recovery actions
## Summary
Give checkout preparation a stable error category. Derive recovery actions from
the category and add a stamp-checked launch retry.
## In scope
- Add `workspace_checkout_failed` and `retry_launch` wire values.
- Map checkout preparation errors to the new category.
- Make every action list category-specific.
- Relaunch without a base-branch write for `retry_launch`.
- Keep authorization and current-stamp checks before relaunch.
## Out of scope
- New persistence tables.
- Automatic retries.
- Changes to `session.recover`.
## Acceptance
- A checkout error offers only `retry_launch`.
- A generic error does not offer base-branch actions because a repository row exists.
- A stale or foreign retry request makes no change and starts no session.
## Verification
```bash
cd apps/backend && go test ./internal/task/models ./internal/orchestrator/executor ./internal/orchestrator/... -race
```
## Files likely touched
- `apps/backend/internal/worktree/errors.go`
- `apps/backend/internal/task/models/launch_errors.go`
- `apps/backend/internal/task/models/launch_errors_test.go`
- `apps/backend/internal/orchestrator/executor/launch_failure.go`
- `apps/backend/internal/orchestrator/executor/executor_launch_failure_classification_test.go`
- `apps/backend/internal/orchestrator/task_launch_recovery.go`
- `apps/backend/internal/orchestrator/task_launch_recovery_test.go`
- `apps/backend/internal/orchestrator/handlers/handlers.go`
## Dependencies
Task 01.
## Risks
- Old persisted action arrays can conflict with the new category matrix.
- A retry must not clear the current error before launch succeeds.
## Parallelism
`sequential`
## Inputs
- Task launch failure recovery design, error model and recovery contract.
- Existing task-status projection normalization tests.
## Results
- Added the `workspace_checkout_failed` category and `retry_launch` action.
- Checkout failures now use category-specific recovery actions, including
retry without changing a task repository base branch.
- Task-scoped recovery validates the current error stamp and repository
identity before relaunch or mutation.
- Persisted and projected errors normalize action lists by category and keep
bounded technical details.
- Focused race coverage passed: 1,212 tests across worktree, task models,
status summary, and executor packages. The 11 targeted task-launch recovery
tests also passed.
- The broader orchestrator package run was attempted but exceeded its timeout
in an existing SQLite/concurrency test.
@@ -0,0 +1,108 @@
---
id: "03-consolidate-launch-error-presentation"
title: "Consolidate launch-error presentation"
status: completed
wave: 3
depends_on:
- "02-classify-launch-recovery-actions"
plan: "plan.md"
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
acceptance_criteria:
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.2
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.7
- AC-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001.8
system_design:
- ../../specs/tasks/system-design/task-launch-failure-recovery.md
---
# Task 03: Consolidate launch-error presentation
## Summary
Make the task launch card the only primary presentation for its matching error.
Give desktop and phone users the same cause, details, and valid recovery.
## In scope
- Derive separate launch-card visibility and session-surface ownership from
session identity and error stamp.
- Add clear category text, a no-change statement, and a details disclosure.
- Suppress matching duplicate notices, rows, banners, empty-turn copy, and toasts.
- Preserve unrelated historical and runtime errors, including stale
launch-only transcript rows.
- Add all required locale values.
- Add desktop and mobile Playwright coverage.
## Out of scope
- Redesign completed-session UI.
- Change non-launch runtime-error recovery.
- Change the mobile branch picker.
## Acceptance
- One failed start renders one error card after all events converge.
- Recovery failure updates the same card without another error surface.
- Phone actions remain reachable, at least 44 pixels high, and free of horizontal overflow.
## Verification
```bash
cd apps && pnpm install --frozen-lockfile
cd apps/web && pnpm exec vitest run components/task/simple/components/task-launch-error-entry.test.tsx components/task/chat/chat-input-container.test.tsx components/task/chat/message-list-footer.test.tsx components/task/chat/session-stopped-banner.test.tsx components/task/chat/stopped-banner-props.test.ts components/task/chat/types.test.ts lib/ws/handlers/empty-turn-notice.test.ts
cd apps/web && pnpm run typecheck && pnpm run i18n:check && pnpm run i18n:ratchet
cd apps/web && pnpm e2e:run tests/task/launch-failure-recovery.spec.ts
cd apps/web && pnpm e2e:run --project mobile-chrome tests/task/mobile-launch-failure-recovery.spec.ts
```
## Files likely touched
- `apps/web/components/task/task-chat-panel.tsx`
- `apps/web/components/task/task-launch-error-context.tsx`
- `apps/web/components/task/simple/components/task-launch-error-entry.tsx`
- `apps/web/components/task/chat/message-list-shared.tsx`
- `apps/web/components/task/chat/message-list-footer.tsx`
- `apps/web/components/task/chat/chat-input-container.tsx`
- `apps/web/lib/ws/handlers/empty-turn-notice.ts`
- Focused tests beside these files.
- `apps/web/src/locales/*/task.json`
- `apps/web/e2e/tests/task/launch-failure-recovery.spec.ts`
- `apps/web/e2e/tests/task/mobile-launch-failure-recovery.spec.ts`
## Dependencies
Task 02.
## Risks
- Event order can briefly show a derived error before the typed record arrives.
- Broad suppression can hide a later agent runtime error.
## Parallelism
`sequential`
## Inputs
- Task launch failure recovery design, presentation ownership and responsive sections.
- Mobile exemplar `apps/web/components/task/mobile/mobile-picker-sheet.tsx`.
- Existing desktop and mobile launch-failure tests.
## Results
- The typed task launch card is the single owner for its matching session and
error stamp. Matching preparation, empty-turn, agent-error, failed-status,
stopped-session, and toast surfaces are suppressed without hiding unrelated
runtime errors.
- The card now shows category-specific copy, bounded technical details, a
no-change statement, inline recovery failures, and mobile-sized stacked
actions.
- Added complete locale values and generated the pseudo-locale catalog.
- Focused frontend coverage passed: 10 Vitest files and 152 tests.
- `pnpm run typecheck`, `pnpm run lint`, `pnpm run i18n:check`, and
`pnpm run i18n:ratchet` passed.
- Managed Playwright coverage passed: desktop launch-failure recovery 3/3 and
mobile launch-failure recovery 2/2.
- Review-remediation coverage passed: 7 focused Vitest files and 75 tests,
including composer ownership and launch-error recovery-state reset.
@@ -4,7 +4,7 @@ system: tasks
requirements:
- REQ-TASKS-TASK-LAUNCH-FAILURE-RECOVERY-001
created: 2026-08-24
updated: 2026-09-03
updated: 2026-09-06
owners:
- cfl12
---
@@ -45,8 +45,20 @@ actions, exact task-repository identity when applicable, and an idempotency
stamp.
The supported categories are `base_branch_missing`, `pr_already_closed`,
`default_branch_unresolved`, and `generic_launch_failure`. The supported
actions are `retry_default`, `pick_base_branch`, and `mark_review_done`.
`default_branch_unresolved`, `workspace_checkout_failed`, and
`generic_launch_failure`. The supported actions are `retry_launch`,
`retry_default`, `pick_base_branch`, and `mark_review_done`.
The failure category controls the available actions. A repository identity
alone does not make a base-branch action valid.
| Category | Valid actions |
| --- | --- |
| `base_branch_missing` | `retry_default`, `pick_base_branch` |
| `default_branch_unresolved` | `pick_base_branch` |
| `workspace_checkout_failed` | `retry_launch` |
| `pr_already_closed` | `mark_review_done` when the workflow permits it |
| `generic_launch_failure` | `retry_launch` |
The `TaskStatusSummary.active_error` projection selects the newest active
record, limits strings and actions, removes duplicate actions without changing
@@ -64,6 +76,8 @@ the current error stamp; stale stamps fail without mutation.
relaunches.
- `pick_base_branch` validates and persists one selected branch before
relaunch.
- `retry_launch` keeps the task-repository settings unchanged and repeats the
failed launch. It still requires the current error stamp.
- `mark_review_done` is allowed only for a valid terminal workflow step and
when every relevant PR is terminal; it uses the normal task-move service.
@@ -106,6 +120,32 @@ but `retry_default` and `pick_base_branch` must write the resolved base to
the exact `task_repositories` row and that write must succeed before relaunch.
No new table is required.
## Pull-request checkout isolation
A pull-request head uses a Kandev-owned ref that includes the pull-request
number, specifically `refs/kandev/pull/<N>/head`. The fetch can force-update
this internal ref because users do not own it. Ordinary remote refresh and
pruning do not manage this namespace. The fetch never writes directly to a
user-named local branch.
The worktree manager verifies the fetched ref before it selects a start point.
If the named local branch has unrelated history, the manager preserves that
branch. It creates a unique task branch from the verified pull-request ref.
This rule also applies when two pull requests reuse the same source-branch
name. A local branch from the first pull request cannot block the second pull
request. A fallback branch uses a task-owned deterministic suffix and a
bounded retry sequence, so an existing fallback branch cannot make launch fail
because of one random-name collision.
The manager sets `origin/<source-branch>` as upstream only when that ref points
to the verified pull-request start point. A remote branch with different
history is never attached as the worktree upstream.
If Kandev cannot fetch or verify the pull-request ref, preparation fails with a
typed `workspace_checkout_failed` error. The error retains the existing
credential and path redaction rules.
## Failure and security
PR lookup failures launch normally. Remote-default timeout, authentication,
@@ -117,12 +157,49 @@ Initial-prompt errors use a safe generic durable message and the same
stale-event checks as other agent failures. Raw attachment paths and provider
details remain in backend diagnostics and do not reach the durable projection.
## Launch-error presentation ownership
The active `TaskStatusSummary.active_error` record owns the primary launch
error for its matching session and stamp. A task-wide record remains visible
while a prior session is selected, but it owns session surfaces only when no
session is selected. A session-owned record owns surfaces only for its exact
session and stamp. The task Chat surface renders one persistent card from this
record.
The matching launch error suppresses these secondary presentations:
- The previous-agent-error notice.
- The standalone failed preparation row.
- The empty-turn warning.
- The generic failed-agent status row.
- The stopped-session recovery banner.
- A launch-error toast while the user views the affected task.
Only synthetic rows generated by the active launch failure are suppressed;
stale stamped or historical launch-only rows remain visible. An unrelated
historical error keeps its normal transcript presentation. A runtime error that
occurs after agent startup keeps the existing session
recovery surface.
The card shows a category title, a short cause, a no-change statement, and only
valid recovery actions. One disclosure shows bounded technical details. The
disclosure does not show raw Git output, credentials, or local file paths.
A failed recovery updates the same card. It does not add another banner or
toast. A successful recovery removes the card after launch succeeds.
## Responsive presentation
The task Chat surface renders one persistent error card from the shared
projection. Desktop actions are inline; mobile actions wrap and branch
selection uses the existing mobile picker. Both surfaces use the same action
authorization and remain free of horizontal overflow.
The task Chat surface is the entry point on desktop and phone. The card remains
inline because the error and its actions belong to the current task.
Desktop actions use one compact row. Phone actions use a vertical layout with
44-pixel touch targets. The existing mobile picker remains the branch-selection
surface for `base_branch_missing` and `default_branch_unresolved`.
The transcript owns vertical scrolling. Expanded technical details wrap inside
the card and do not create a second scroll region. Both layouts prevent
document-level horizontal overflow.
`useEnsureTaskSession` keeps one request latch after an error. Store updates and
loader identity changes do not start another request. The Retry control starts