fix: restore orphaned sessions on task open (#3840)

This commit is contained in:
Carlos Florêncio
2026-09-21 14:07:49 +01:00
committed by GitHub
parent 3ebd7381bb
commit ac397c5486
48 changed files with 4032 additions and 568 deletions
+1 -1
View File
@@ -174,7 +174,7 @@ history and remains immutable.
- ADR 0015's step-completion-signal telemetry lives under `workflow_*`: `workflow_step_completion_signal_received_total` (`internal/workflow/signalmetrics/`), labelled by `source` and `agent_type`, counts accepted `step_complete_kandev` signals. Its separate `workflow_step_completion_signal_fallback_used_total` counter is labelled by `agent_type` and counts manual fallback uses. The fallback button does not have a production increment site yet, so the counter remains zero until that UI ships.
- Office stall detection (REQ-OFFICE-STALL-VISIBILITY) lives under `office_stall_*`: `office_stall_stranded_signal_total` (labelled by `gate`, which names which of the two watchdog gate sites saw it), `office_stall_decision_waiting_total`, and `office_stall_detector_skipped_total` (labelled by `reason`, so a detector that fails closed is visible rather than silent). Detection only: these counters never accompany a transition, a synthesized decision, or a queued run, because Office tasks are surfaced and never reclaimed. Also emitted as structured zap logs.
- Office session-termination suppression (REQ-OFFICE-SESSION-TERM) lives under `office_session_term_suppressed_total`, labelled by `reason` (one of `task_reassigned`, `participant_removed`, `participant_seat_claimed`) and `outcome` (`runner` or `seat` when the agent still holds that capacity, `read_failed` when the capacity read itself failed). It counts guarded terminations that left a session live, so a suppression is distinguishable from a path that never ran. Both dimensions are closed sets; no task, agent, step or session identifier is ever a label. Also emitted as structured zap logs.
- Task-level stall detection (issue #3712) runs in the session reconciliation sweep (`internal/task/service/`, one-minute tick shared with the archived-session pass): an unarchived task holding an active session with no live execution behind it and no events/messages past `tasks.stallDetectionThreshold` (default 2h) emits a `task.stalled` event plus a `stalled_task` WARN, once per stall episode. At or beyond twice the threshold the sweep cancels the orphaned sessions (`error_message` = "orphaned session") through the archived pass's `finalizeCancelledSessions` transition scoped to the classified session IDs; the heal runs only when every active session of the task is orphaned and past the grace window (a live or merely-stalled sibling cancels nothing), and liveness is re-checked at the cancellation boundary. A stall past the grace window can therefore emit `task.stalled` and a `session.state_changed` in the same sweep; the cancel never depends on the detection event.
- Task-level stall detection (issue #3712) runs in the session reconciliation sweep (`internal/task/service/`, one-minute tick shared with the archived-session pass): an unarchived task holding a non-idle active session with no live execution behind it and no events/messages past `tasks.stallDetectionThreshold` (default 2h) emits a `task.stalled` event plus a `stalled_task` WARN, once per stall episode. At or beyond twice the threshold the sweep conditionally settles classified interrupted sessions to `WAITING_FOR_INPUT`, preserving the same conversation and abandoning only the observed turn without a completion event; the recovery runs only when every active session of the task is execution-less and past the grace window (an idle waiting sibling blocks it), and liveness is re-checked at the recovery boundary. A stall past the grace window can therefore emit `task.stalled` and a `session.state_changed` in the same sweep; recovery never depends on the detection event. The separate restart pass applies the same guarded settlement per stale `STARTING`/`RUNNING` session, including rows without an executor record.
### GitHub Operations
@@ -170,8 +170,9 @@ func provideOrchestrator(
// Runtime-aware liveness lets durable cleanup treat a not-found stop for a
// confirmed-dead local runtime as already stopped instead of retrying forever.
taskSvc.SetRowLivenessProber(agentManagerClient)
// The orphan-session sweep terminalizes stale STARTING/RUNNING sessions no
// live in-memory execution backs (backend-restart residue, #3711).
// The orphan-session sweep preserves stale STARTING/RUNNING sessions when no
// live in-memory execution backs them, so the conversation can recover on
// task focus after a backend restart.
taskSvc.SetExecutionLivenessChecker(agentManagerClient)
// The session reconciliation sweep's active-task pass (stall detection and
// orphaned-session healing) verifies "no live execution" against the agent
@@ -567,7 +567,16 @@ func (s *Service) handleAgentBootReady(ctx context.Context, data watcher.AgentEv
zap.String("session_state", string(session.State)))
return
}
recoveryResolvedAt := s.markRecoveryResolved(ctx, data.SessionID, session)
markerSnapshot, markerSnapshotKnown := s.interruptedMarkerSnapshotForResumeAttempt(data.SessionID, data.AttemptID)
// Every boot-ready callback is a recovery callback for marker purposes. A
// missing/finished attempt, empty attempt ID, or failed task snapshot leaves
// the marker snapshot unavailable and therefore fails closed in
// clearTaskInterruptedMarker.
// Ordinary sessions without a marker take the same no-op path; only an
// immutable marker captured for this attempt can clear the warning.
recoveryResolvedAt := s.markRecoveryResolved(
ctx, data.SessionID, session, markerSnapshot, markerSnapshotKnown,
)
// Idempotent: if the session is already WAITING_FOR_INPUT (e.g. revived
// from a previously launched session and the boot signal arrived faster
@@ -3215,7 +3224,13 @@ func (s *Service) dismissRecoveredAgentError(
// The boot transcript is useful observability, but its writes are best effort.
// Session metadata is the authoritative recovery result used by the frontend
// after a reload when the transcript row is missing or incomplete.
func (s *Service) markRecoveryResolved(ctx context.Context, sessionID string, session *models.TaskSession) *time.Time {
func (s *Service) markRecoveryResolved(
ctx context.Context,
sessionID string,
session *models.TaskSession,
markerSnapshot interruptedMarkerSnapshot,
markerSnapshotKnown bool,
) *time.Time {
resolvedAt := time.Now().UTC()
resolvedAtValue := resolvedAt.Format(time.RFC3339Nano)
if err := s.repo.SetSessionMetadataKey(
@@ -3235,10 +3250,51 @@ func (s *Service) markRecoveryResolved(ctx context.Context, sessionID string, se
session.Metadata[models.SessionMetaKeyRecoveryResolvedAt] = resolvedAtValue
if session.TaskID != "" {
s.dismissRecoveredAgentError(ctx, session.TaskID, session, resolvedAt)
// Boot-ready is the provider-confirmed recovery boundary. Keeping this
// out of the STARTING/RUNNING state funnel leaves the durable warning in
// place when a launch later fails or is cancelled.
if markerSnapshotKnown && markerSnapshot.captured {
s.clearTaskInterruptedMarker(ctx, session.TaskID, markerSnapshot.value)
}
}
// A guarded boot callback may clear recovery ownership only when it carries
// a valid immutable marker snapshot. A known recovery attempt whose task
// read failed keeps its durable settlement so a later sweep can retry it.
if !markerSnapshotKnown || markerSnapshot.captured {
s.clearRecoveryMetadataAfterBoot(ctx, sessionID, session)
}
return &resolvedAt
}
func (s *Service) clearRecoveryMetadataAfterBoot(
ctx context.Context,
sessionID string,
session *models.TaskSession,
) {
remover, supported := s.repo.(interface {
RemoveSessionMetadataKeyIfJSONValue(context.Context, string, string, interface{}) (bool, error)
})
if !supported || session == nil {
return
}
if pending, ok := session.Metadata[models.SessionMetaKeyInterruptedRecoveryPending].(string); ok && pending != "" {
if _, err := remover.RemoveSessionMetadataKeyIfJSONValue(
ctx, sessionID, models.SessionMetaKeyInterruptedRecoveryPending, pending,
); err != nil {
s.logger.Warn("failed to clear interrupted recovery marker",
zap.String("session_id", sessionID), zap.Error(err))
}
}
if settlement, ok := models.LoadInterruptedRecoverySettlement(session.Metadata); ok {
if _, err := remover.RemoveSessionMetadataKeyIfJSONValue(
ctx, sessionID, models.SessionMetaKeyRecoverySettlementPending, settlement,
); err != nil {
s.logger.Warn("failed to clear recovery settlement marker after boot",
zap.String("session_id", sessionID), zap.Error(err))
}
}
}
// providerRemediationURL returns the adapter-validated remediation URL from the
// normalized provider diagnostic, or "" when the failure carried none. The URL
// is only ever set by the adapter's allowlist validator; the orchestrator does
@@ -1203,11 +1203,10 @@ func (s *Service) updateTaskSessionStateWithHook(
zap.Error(err))
}
}
// Work has resumed: a session entering STARTING/RUNNING clears the
// startup interruption marker and republishes the task so open clients
// drop the red interruption icon. No-op when the marker is absent.
// Entering STARTING/RUNNING only records a recovery attempt. The durable
// interruption marker is cleared after the provider confirms boot/readiness,
// so failed or cancelled attempts remain visible to the user.
if nextState == models.TaskSessionStateStarting || nextState == models.TaskSessionStateRunning {
s.clearTaskInterruptedMarker(ctx, taskID)
s.clearTaskAutoStartFailedMarker(ctx, taskID)
}
if authoritativeUpdatedAt == nil {
@@ -2174,9 +2173,8 @@ func (s *Service) setSessionStartingWithOptions(
}
// The launch path moves a session to STARTING without going through
// updateTaskSessionStateWithHook, so clear the interruption marker here
// too (no-op when absent).
s.clearTaskInterruptedMarker(ctx, taskID)
// updateTaskSessionStateWithHook. It records an attempt only; the
// interruption marker is cleared after confirmed provider readiness.
s.clearTaskAutoStartFailedMarker(ctx, taskID)
if publishSession != nil {
@@ -2187,14 +2185,39 @@ func (s *Service) setSessionStartingWithOptions(
// clearTaskInterruptedMarker removes the startup interruption marker from a
// task and republishes task.updated when it was actually present, so open
// clients drop the red interruption icon. Called from the session-start
// funnel when a session enters STARTING/RUNNING — work has resumed, so the
// task is no longer interrupted. No-op when the marker is absent.
func (s *Service) clearTaskInterruptedMarker(ctx context.Context, taskID string) {
// clients drop the warning icon. Callers invoke it only after provider
// readiness confirms that recovery succeeded. No-op when the marker is absent.
func (s *Service) clearTaskInterruptedMarker(
ctx context.Context,
taskID string,
expectedMarker string,
) {
if taskID == "" {
return
}
removed, err := s.repo.RemoveTaskMetadataKey(ctx, taskID, models.MetaKeyInterruptedAt)
var (
removed bool
err error
)
if strings.TrimSpace(expectedMarker) == "" {
// A recovery callback without a valid immutable marker snapshot fails
// closed. There is no safe unconditional removal path.
return
}
if remover, ok := s.repo.(interface {
RemoveTaskMetadataKeyIfValue(context.Context, string, string, string) (bool, error)
}); ok {
removed, err = remover.RemoveTaskMetadataKeyIfValue(
ctx, taskID, models.MetaKeyInterruptedAt, expectedMarker,
)
} else {
// A read followed by an unconditional legacy removal is not a
// compare-and-set. Refuse to clear when the adapter cannot provide the
// guarded primitive so a delayed callback cannot erase a newer marker.
s.logger.Warn("skipping interrupted-marker clear without compare-and-set support",
zap.String("task_id", taskID))
return
}
if err != nil {
s.logger.Warn("failed to clear interrupted marker",
zap.String("task_id", taskID),
@@ -2216,9 +2239,9 @@ func (s *Service) clearTaskInterruptedMarker(ctx context.Context, taskID string)
// clearTaskAutoStartFailedMarker removes the auto-start-failure marker from a
// task and republishes task.updated when it was actually present, so open
// clients drop the failure badge. Called from the same session-start funnel
// as clearTaskInterruptedMarker: a session entering STARTING/RUNNING means an
// agent did launch, so any earlier auto-start failure no longer applies.
// clients drop the failure badge. It shares the session-start funnel with the
// interruption marker, but remains clear at launch admission because it means
// the auto-start request itself was accepted.
// No-op when the marker is absent.
func (s *Service) clearTaskAutoStartFailedMarker(ctx context.Context, taskID string) {
if taskID == "" {
@@ -2699,7 +2722,6 @@ func (s *Service) setQueuedSessionRunningForIdentity(
session.UpdatedAt = updatedAt
if oldState != models.TaskSessionStateRunning {
s.reconcileRunningTaskStateLocked(ctx, identity.TaskID, identity.SessionID)
s.clearTaskInterruptedMarker(ctx, identity.TaskID)
s.clearTaskAutoStartFailedMarker(ctx, identity.TaskID)
s.publishTaskSessionStateChanged(
ctx,
@@ -2758,10 +2758,10 @@ func seedInterruptedMarker(t *testing.T, repo *sqliterepo.Repository, taskID str
}
}
func TestSessionStartClearsInterruptedMarker(t *testing.T) {
func TestSessionStartKeepsInterruptedMarkerUntilRecovery(t *testing.T) {
ctx := context.Background()
t.Run("state hook transition to STARTING clears and republishes", func(t *testing.T) {
t.Run("state hook transition to STARTING keeps the marker", func(t *testing.T) {
repo := setupTestRepo(t)
seedSession(t, repo, "t1", "s1", "step1")
seedInterruptedMarker(t, repo, "t1")
@@ -2779,12 +2779,12 @@ func TestSessionStartClearsInterruptedMarker(t *testing.T) {
task, err := repo.GetTask(ctx, "t1")
require.NoError(t, err)
_, marked := task.Metadata[models.MetaKeyInterruptedAt]
require.False(t, marked, "marker must be cleared when the session enters STARTING")
require.Equal(t, []string{"t1"}, publisher.updatedTaskIDs,
"task.updated must be republished after clearing the marker")
require.True(t, marked, "marker must survive the STARTING transition")
require.Empty(t, publisher.updatedTaskIDs,
"STARTING must not publish a marker-clearing task.updated")
})
t.Run("launch path via setSessionStarting clears and republishes", func(t *testing.T) {
t.Run("launch path via setSessionStarting keeps the marker", func(t *testing.T) {
repo := setupTestRepo(t)
seedSession(t, repo, "t1", "s1", "step1")
seedInterruptedMarker(t, repo, "t1")
@@ -2804,9 +2804,151 @@ func TestSessionStartClearsInterruptedMarker(t *testing.T) {
task, err := repo.GetTask(ctx, "t1")
require.NoError(t, err)
_, marked := task.Metadata[models.MetaKeyInterruptedAt]
require.False(t, marked, "marker must be cleared on the launch path")
require.True(t, marked, "the launch attempt must not clear the marker")
require.Empty(t, publisher.updatedTaskIDs,
"the launch attempt must not publish a marker-clearing task.updated")
})
t.Run("confirmed boot recovery clears and republishes", func(t *testing.T) {
repo := setupTestRepo(t)
seedSession(t, repo, "t1", "s1", "step1")
seedInterruptedMarker(t, repo, "t1")
publisher := &recordingTaskUpdatedPublisher{}
svc := createTestService(repo, newMockStepGetter(), newMockTaskRepo())
svc.SetTaskEventPublisher(publisher)
session, err := repo.GetTaskSession(ctx, "s1")
require.NoError(t, err)
task, err := repo.GetTask(ctx, "t1")
require.NoError(t, err)
marker, ok := task.Metadata[models.MetaKeyInterruptedAt].(string)
require.True(t, ok)
require.NotNil(t, svc.markRecoveryResolved(ctx, "s1", session, interruptedMarkerSnapshot{value: marker, captured: true}, true))
task, err = repo.GetTask(ctx, "t1")
require.NoError(t, err)
_, marked := task.Metadata[models.MetaKeyInterruptedAt]
require.False(t, marked, "successful boot recovery must clear the marker")
require.Equal(t, []string{"t1"}, publisher.updatedTaskIDs,
"task.updated must be republished after clearing the marker")
"successful boot recovery must publish the marker-clearing task.updated")
})
t.Run("stale recovery preserves a newer interruption marker", func(t *testing.T) {
repo := setupTestRepo(t)
seedSession(t, repo, "t1", "s1", "step1")
require.NoError(t, repo.SetTaskMetadataKey(ctx, "t1", models.MetaKeyInterruptedAt, "old-marker"))
publisher := &recordingTaskUpdatedPublisher{}
svc := createTestService(repo, newMockStepGetter(), newMockTaskRepo())
svc.SetTaskEventPublisher(publisher)
session, err := repo.GetTaskSession(ctx, "s1")
require.NoError(t, err)
require.NoError(t, repo.SetTaskMetadataKey(ctx, "t1", models.MetaKeyInterruptedAt, "new-marker"))
require.NotNil(t, svc.markRecoveryResolved(ctx, "s1", session, interruptedMarkerSnapshot{value: "old-marker", captured: true}, true))
task, err := repo.GetTask(ctx, "t1")
require.NoError(t, err)
require.Equal(t, "new-marker", task.Metadata[models.MetaKeyInterruptedAt])
require.Empty(t, publisher.updatedTaskIDs,
"a stale recovery callback must not publish a marker-clearing update")
})
t.Run("recovery callback with no marker snapshot fails closed", func(t *testing.T) {
repo := setupTestRepo(t)
seedSession(t, repo, "t1", "s1", "step1")
require.NoError(t, repo.SetTaskMetadataKey(ctx, "t1", models.MetaKeyInterruptedAt, "new-marker"))
publisher := &recordingTaskUpdatedPublisher{}
svc := createTestService(repo, newMockStepGetter(), newMockTaskRepo())
svc.SetTaskEventPublisher(publisher)
session, err := repo.GetTaskSession(ctx, "s1")
require.NoError(t, err)
// A recovery attempt can finish after its task snapshot failed to load.
// An uncaptured snapshot must remain a guarded no-op.
require.NotNil(t, svc.markRecoveryResolved(ctx, "s1", session, interruptedMarkerSnapshot{}, true))
task, err := repo.GetTask(ctx, "t1")
require.NoError(t, err)
require.Equal(t, "new-marker", task.Metadata[models.MetaKeyInterruptedAt])
require.Empty(t, publisher.updatedTaskIDs)
})
t.Run("finished recovery attempt with absent marker preserves a newer marker", func(t *testing.T) {
repo := setupTestRepo(t)
seedSession(t, repo, "t1", "s1", "step1")
svc := createTestService(repo, newMockStepGetter(), newMockTaskRepo())
registry := newResumeAttemptRegistry()
svc.resumeAttemptsMu.Lock()
svc.resumeAttempts = registry
svc.resumeAttemptsMu.Unlock()
attempt, owner := registry.begin(ctx, "t1", "s1")
require.True(t, owner)
attempt.setInterruptedMarkerSnapshot("", true)
attempt.finish(registry)
require.NoError(t, repo.SetTaskMetadataKey(ctx, "t1", models.MetaKeyInterruptedAt, "new-marker"))
session, err := repo.GetTaskSession(ctx, "s1")
require.NoError(t, err)
snapshot, known := svc.interruptedMarkerSnapshotForResumeAttempt("s1", attempt.identity())
require.True(t, known)
require.True(t, snapshot.captured)
require.Empty(t, snapshot.value)
require.NotNil(t, svc.markRecoveryResolved(ctx, "s1", session, snapshot, known))
task, err := repo.GetTask(ctx, "t1")
require.NoError(t, err)
require.Equal(t, "new-marker", task.Metadata[models.MetaKeyInterruptedAt])
})
t.Run("finished recovery attempt with failed marker read preserves a newer marker", func(t *testing.T) {
repo := setupTestRepo(t)
seedSession(t, repo, "t1", "s1", "step1")
svc := createTestService(repo, newMockStepGetter(), newMockTaskRepo())
registry := newResumeAttemptRegistry()
svc.resumeAttemptsMu.Lock()
svc.resumeAttempts = registry
svc.resumeAttemptsMu.Unlock()
attempt, owner := registry.begin(ctx, "t1", "s1")
require.True(t, owner)
// Leave the snapshot uncaptured to model a failed GetTask read.
attempt.finish(registry)
require.NoError(t, repo.SetTaskMetadataKey(ctx, "t1", models.MetaKeyInterruptedAt, "new-marker"))
session, err := repo.GetTaskSession(ctx, "s1")
require.NoError(t, err)
snapshot, known := svc.interruptedMarkerSnapshotForResumeAttempt("s1", attempt.identity())
require.True(t, known)
require.False(t, snapshot.captured)
require.NotNil(t, svc.markRecoveryResolved(ctx, "s1", session, snapshot, known))
task, err := repo.GetTask(ctx, "t1")
require.NoError(t, err)
require.Equal(t, "new-marker", task.Metadata[models.MetaKeyInterruptedAt])
})
t.Run("finished recovery attempt retains its immutable marker snapshot", func(t *testing.T) {
repo := setupTestRepo(t)
seedSession(t, repo, "t1", "s1", "step1")
require.NoError(t, repo.SetTaskMetadataKey(ctx, "t1", models.MetaKeyInterruptedAt, "old-marker"))
svc := createTestService(repo, newMockStepGetter(), newMockTaskRepo())
registry := newResumeAttemptRegistry()
svc.resumeAttemptsMu.Lock()
svc.resumeAttempts = registry
svc.resumeAttemptsMu.Unlock()
attempt, owner := registry.begin(ctx, "t1", "s1")
require.True(t, owner)
attempt.setInterruptedMarkerSnapshot("old-marker", true)
attempt.finish(registry)
require.NoError(t, repo.SetTaskMetadataKey(ctx, "t1", models.MetaKeyInterruptedAt, "new-marker"))
session, err := repo.GetTaskSession(ctx, "s1")
require.NoError(t, err)
marker := svc.interruptedMarkerForResumeAttempt("s1", attempt.identity())
require.Equal(t, "old-marker", marker)
require.NotNil(t, svc.markRecoveryResolved(ctx, "s1", session, interruptedMarkerSnapshot{value: marker, captured: true}, true))
task, err := repo.GetTask(ctx, "t1")
require.NoError(t, err)
require.Equal(t, "new-marker", task.Metadata[models.MetaKeyInterruptedAt])
})
t.Run("no marker means no republish", func(t *testing.T) {
@@ -2992,7 +3134,7 @@ func TestClearRecoveredAgentErrorOnTurnCompletion(t *testing.T) {
},
))
require.NotNil(t, svc.markRecoveryResolved(ctx, "s1", snapshot))
require.NotNil(t, svc.markRecoveryResolved(ctx, "s1", snapshot, interruptedMarkerSnapshot{}, false))
stored, err := repo.GetTaskSession(ctx, "s1")
require.NoError(t, err)
@@ -3074,7 +3216,7 @@ func TestTransitionBootstrapFailurePersistsSessionHistory(t *testing.T) {
reloaded, err := repo.GetTaskSession(ctx, "bootstrap-history-session")
require.NoError(t, err)
resolvedAt := svc.markRecoveryResolved(ctx, reloaded.ID, reloaded)
resolvedAt := svc.markRecoveryResolved(ctx, reloaded.ID, reloaded, interruptedMarkerSnapshot{}, false)
require.NotNil(t, resolvedAt)
afterRecovery, err := repo.GetTaskSession(ctx, "bootstrap-history-session")
@@ -1679,16 +1679,15 @@ func (e *Executor) applyExecutorConfigToResumeRequest(ctx context.Context, req *
return execConfig
}
// isArchiveCancelledResumeSession reports whether session was cancelled by an
// archive (Service.ArchiveTask's single-task path or HandoffService's cascade
// archive) rather than an explicit user/coordinator stop. Mirrors
// orchestrator.isArchiveCancelledSession — kept local to this package since
// the two live on opposite sides of the executor/orchestrator boundary and
// the check is two lines over already-exported models helpers.
func isArchiveCancelledResumeSession(session *models.TaskSession) bool {
// isRecoverableCancelledResumeSession reports whether session was cancelled by
// a system reconciliation path rather than an explicit user/coordinator stop.
// Mirrors the orchestrator's eligibility checks while keeping the executor's
// prompt-free launch behavior aligned for legacy archive and orphan rows.
func isRecoverableCancelledResumeSession(session *models.TaskSession) bool {
return session != nil &&
session.State == models.TaskSessionStateCancelled &&
models.IsArchiveCancelReason(session.ErrorMessage)
(models.IsArchiveCancelReason(session.ErrorMessage) ||
models.IsOrphanCancelReason(session.ErrorMessage))
}
// applyRunningRecordToResumeRequest loads the ExecutorRunning record and applies
@@ -1702,12 +1701,12 @@ func (e *Executor) applyRunningRecordToResumeRequest(
) *models.ExecutorRunning {
if running == nil {
// Archive cleanup tears down the executors_running row entirely, so an
// archive-cancelled session reaches this point with running == nil. The
// system-cancelled session reaches this point with running == nil. The
// session metadata mirrors the provider conversation identity so an
// explicit completed follow-up can still restore the same conversation
// after runtime cleanup removed the operational row.
noAutoPromptState := session.State == models.TaskSessionStateWaitingForInput ||
isArchiveCancelledResumeSession(session) ||
isRecoverableCancelledResumeSession(session) ||
session.State == models.TaskSessionStateCompleted
if startAgent && noAutoPromptState {
if token := persistedSessionResumeToken(session); token != "" {
@@ -1755,9 +1754,9 @@ func (e *Executor) applyRunningRecordToResumeRequest(
zap.String("session_id", session.ID),
zap.Bool("has_resume_token", running.ResumeToken != ""))
} else if startAgent && (session.State == models.TaskSessionStateWaitingForInput ||
isArchiveCancelledResumeSession(session) || session.State == models.TaskSessionStateCompleted) {
isRecoverableCancelledResumeSession(session) || session.State == models.TaskSessionStateCompleted) {
// Fresh-start resume (no resume token): don't auto-prompt with the task
// description. Also covers completed and archive-cancelled sessions whose
// description. Also covers completed and system-cancelled sessions whose
// running record survived cleanup but carries no token — the same
// auto-resume shape as the running==nil branch above.
req.TaskDescription = ""
@@ -1011,6 +1011,65 @@ func TestResumeSession_ArchiveCancelledWithoutRunningRow_ClearsTaskDescription(t
}
}
// TestResumeSession_OrphanCancelledWithoutRunningRow_ClearsTaskDescription
// preserves the prompt-free recovery contract for legacy rows that the
// reconciliation sweep cancelled after losing their runtime inventory.
func TestResumeSession_OrphanCancelledWithoutRunningRow_ClearsTaskDescription(t *testing.T) {
repo := newMockRepository()
now := time.Now().UTC()
repo.tasks["task-1"] = &models.Task{
ID: "task-1",
WorkspaceID: "workspace-1",
Description: "do the original thing",
CreatedAt: now,
UpdatedAt: now,
}
repo.sessions["sess-1"] = &models.TaskSession{
ID: "sess-1",
TaskID: "task-1",
AgentProfileID: "profile-1",
State: models.TaskSessionStateCancelled,
ErrorMessage: models.SessionOrphanedCancelReason,
}
var capturedReq *LaunchAgentRequest
agentMgr := &mockAgentManager{
launchAgentFunc: func(_ context.Context, req *LaunchAgentRequest) (*LaunchAgentResponse, error) {
capturedReq = req
return &LaunchAgentResponse{AgentExecutionID: "exec-new", Status: v1.AgentStatusStarting}, nil
},
}
exec := newTestExecutor(t, agentMgr, repo)
callerSession := *repo.sessions["sess-1"]
if _, err := exec.ResumeSession(context.Background(), &callerSession, true); err != nil {
t.Fatalf("ResumeSession: %v", err)
}
if capturedReq == nil {
t.Fatal("LaunchAgent was not called")
}
if capturedReq.TaskDescription != "" {
t.Errorf("TaskDescription = %q, want empty — auto-resuming an orphan-cancelled "+
"session without a running row must not replay the original prompt", capturedReq.TaskDescription)
}
}
func TestApplyRunningRecordToResumeRequest_OrphanCancelledTokenlessRowClearsTaskDescription(t *testing.T) {
repo := newMockRepository()
exec := newTestExecutor(t, &mockAgentManager{}, repo)
req := &LaunchAgentRequest{TaskDescription: "do the original thing"}
task := &v1.Task{ID: "task-1"}
session := &models.TaskSession{
ID: "sess-1", State: models.TaskSessionStateCancelled,
ErrorMessage: models.SessionOrphanedCancelReason,
}
exec.applyRunningRecordToResumeRequest(req, task, session, true, &models.ExecutorRunning{})
if req.TaskDescription != "" {
t.Fatalf("TaskDescription = %q, want empty for tokenless orphan recovery", req.TaskDescription)
}
}
func TestApplyRunningRecordToResumeRequest_FailedSessionKeepsTaskDescription(t *testing.T) {
repo := newMockRepository()
exec := newTestExecutor(t, &mockAgentManager{}, repo)
@@ -144,6 +144,8 @@ func TestReconcileSessionsOnStartupAppliesFullReconciliationWhenNotRetracked(t *
turns := &countingTurnService{}
svc := createTestServiceWithAgent(repo, newMockStepGetter(), taskRepo, &mockAgentManager{})
svc.turnService = turns
publisher := &recordingTaskUpdatedPublisher{}
svc.SetTaskEventPublisher(publisher)
svc.SetRetrackedSessionChecker(func(sessionID string) bool { return false })
svc.reconcileSessionsOnStartup(ctx)
@@ -171,6 +173,9 @@ func TestReconcileSessionsOnStartupAppliesFullReconciliationWhenNotRetracked(t *
if got := turns.calls(); got != 1 {
t.Fatalf("AbandonOpenTurns called %d times, want 1", got)
}
if len(publisher.updatedTaskIDs) != 1 || publisher.updatedTaskIDs[0] != "task1" {
t.Fatalf("task.updated publishes = %v, want [task1] after startup marker commit", publisher.updatedTaskIDs)
}
}
// TestReconcileSessionsOnStartupUnwiredCheckerTreatsEverySessionAsNotRetracked
@@ -11,6 +11,7 @@ import (
"go.uber.org/zap"
"github.com/kandev/kandev/internal/orchestrator/executor"
"github.com/kandev/kandev/internal/task/models"
)
// ErrResumeAttemptCancelled is returned when a startup continuation no longer
@@ -33,6 +34,12 @@ type resumeAttempt struct {
finishOnce sync.Once
executionMu sync.Mutex
executionID string
// interruptedMarker is the marker value observed when this recovery attempt
// was admitted. A delayed boot callback may clear only that exact marker;
// a later restart must remain visible until its own attempt succeeds.
interruptedMarkerMu sync.RWMutex
interruptedMarker string
interruptedMarkerCaptured bool
// retained is protected by resumeAttemptRegistry.mu. A cancelled attempt
// can be retained when a replacement is admitted and retained again when
// its owner eventually returns; both paths must describe one tombstone.
@@ -66,10 +73,17 @@ const maxResumeAttemptTombstones = 16
const resumeAttemptIdentityPrefix = "resume-"
type resumeAttemptTombstone struct {
id uint64
executionID string
cancelled bool
accepted bool
id uint64
executionID string
cancelled bool
accepted bool
interruptedMarker string
interruptedMarkerCaptured bool
}
type interruptedMarkerSnapshot struct {
value string
captured bool
}
func newResumeAttemptRegistry() *resumeAttemptRegistry {
@@ -231,6 +245,8 @@ func (r *resumeAttemptRegistry) retainLocked(attempt *resumeAttempt) {
}
if entries[index].id == attempt.id {
entries[index].accepted = attempt.accepted
entries[index].interruptedMarker = attempt.interruptedMarkerValue()
entries[index].interruptedMarkerCaptured = attempt.interruptedMarkerSnapshotCaptured()
break
}
}
@@ -240,10 +256,12 @@ func (r *resumeAttemptRegistry) retainLocked(attempt *resumeAttempt) {
attempt.retained = true
entries := r.tombstones[attempt.sessionID]
entries = append(entries, resumeAttemptTombstone{
id: attempt.id,
executionID: executionID,
cancelled: attempt.ctx.Err() != nil,
accepted: attempt.accepted,
id: attempt.id,
executionID: executionID,
cancelled: attempt.ctx.Err() != nil,
accepted: attempt.accepted,
interruptedMarker: attempt.interruptedMarkerValue(),
interruptedMarkerCaptured: attempt.interruptedMarkerSnapshotCaptured(),
})
if len(entries) > maxResumeAttemptTombstones {
entries = entries[len(entries)-maxResumeAttemptTombstones:]
@@ -424,6 +442,34 @@ func (attempt *resumeAttempt) setExecutionID(executionID string) {
attempt.executionMu.Unlock()
}
func (attempt *resumeAttempt) setInterruptedMarkerSnapshot(marker string, captured bool) {
if attempt == nil {
return
}
attempt.interruptedMarkerMu.Lock()
attempt.interruptedMarker = marker
attempt.interruptedMarkerCaptured = captured
attempt.interruptedMarkerMu.Unlock()
}
func (attempt *resumeAttempt) interruptedMarkerValue() string {
if attempt == nil {
return ""
}
attempt.interruptedMarkerMu.RLock()
defer attempt.interruptedMarkerMu.RUnlock()
return attempt.interruptedMarker
}
func (attempt *resumeAttempt) interruptedMarkerSnapshotCaptured() bool {
if attempt == nil {
return false
}
attempt.interruptedMarkerMu.RLock()
defer attempt.interruptedMarkerMu.RUnlock()
return attempt.interruptedMarkerCaptured
}
func (attempt *resumeAttempt) execution() string {
if attempt == nil {
return ""
@@ -494,6 +540,9 @@ func (s *Service) beginResumeAttempt(
attempt, owner := s.resumeAttemptStore().begin(ctx, taskID, sessionID)
lock.Unlock()
release()
if owner {
s.captureInterruptedMarkerForResumeAttempt(ctx, attempt)
}
return attempt, owner, nil
}
lock.Unlock()
@@ -508,6 +557,61 @@ func (s *Service) beginResumeAttempt(
}
}
func (s *Service) captureInterruptedMarkerForResumeAttempt(ctx context.Context, attempt *resumeAttempt) {
if s == nil || s.repo == nil || attempt == nil || attempt.taskID == "" {
return
}
task, err := s.repo.GetTask(ctx, attempt.taskID)
if err != nil || task == nil {
return
}
if task.Metadata == nil {
attempt.setInterruptedMarkerSnapshot("", true)
return
}
marker, ok := task.Metadata[models.MetaKeyInterruptedAt].(string)
// A successful task read with no marker is still an immutable snapshot.
// Preserve that fact through tombstones, while the empty value remains a
// fail-closed generation for the recovery callback.
attempt.setInterruptedMarkerSnapshot(marker, ok)
}
func (s *Service) interruptedMarkerSnapshotForResumeAttempt(
sessionID, attemptID string,
) (interruptedMarkerSnapshot, bool) {
if s == nil || sessionID == "" || attemptID == "" {
return interruptedMarkerSnapshot{}, false
}
id, isRecovery := parseResumeAttemptIdentity(attemptID)
if !isRecovery {
return interruptedMarkerSnapshot{}, false
}
registry := s.resumeAttemptStore()
registry.mu.Lock()
defer registry.mu.Unlock()
if attempt := registry.attempts[sessionID]; attempt != nil && attempt.id == id {
return interruptedMarkerSnapshot{
value: attempt.interruptedMarkerValue(),
captured: attempt.interruptedMarkerSnapshotCaptured(),
}, true
}
if tombstone, ok := registry.tombstoneLocked(sessionID, id); ok {
return interruptedMarkerSnapshot{
value: tombstone.interruptedMarker,
captured: tombstone.interruptedMarkerCaptured,
}, true
}
return interruptedMarkerSnapshot{}, false
}
func (s *Service) interruptedMarkerForResumeAttempt(sessionID, attemptID string) string {
snapshot, known := s.interruptedMarkerSnapshotForResumeAttempt(sessionID, attemptID)
if !known || !snapshot.captured {
return ""
}
return snapshot.value
}
func (s *Service) validateResumeAttempt(attempt *resumeAttempt) error {
if attempt == nil {
return nil
+19 -5
View File
@@ -3671,20 +3671,34 @@ func (s *Service) reconcileActiveSessionOnStartup(
}
// Mark the task interrupted when its session was mid-turn when the backend
// died (STARTING/RUNNING) so task-list surfaces can show the red
// interruption icon. WAITING_FOR_INPUT sessions were idle, not interrupted.
// died (STARTING/RUNNING) so task-list surfaces can show the warning
// indicator. WAITING_FOR_INPUT sessions were idle, not interrupted.
// A re-tracked session's task was never actually interrupted -- its agent
// kept running across the restart -- so this is skipped for it per
// AC-EXECUTORS-SURVIVAL-003.2.
// The write is archive-atomic (SetTaskMetadataKeyIfNotArchived), so an
// archive that commits after this check cannot leave a stale marker on an
// archived task. The marker is cleared when a session of the task next
// enters STARTING/RUNNING (see updateTaskSessionStateWithHook).
// archived task. The marker is cleared after the provider confirms recovery
// (see markRecoveryResolved).
if !retracked && running.TaskID != "" && (previousState == models.TaskSessionStateStarting || previousState == models.TaskSessionStateRunning) {
if _, setErr := s.repo.SetTaskMetadataKeyIfNotArchived(ctx, running.TaskID, models.MetaKeyInterruptedAt, time.Now().UTC().Format(time.RFC3339)); setErr != nil {
changed, setErr := s.repo.SetTaskMetadataKeyIfNotArchived(ctx, running.TaskID, models.MetaKeyInterruptedAt, time.Now().UTC().Format(time.RFC3339))
if setErr != nil {
s.logger.Warn("failed to mark task interrupted on startup",
zap.String("task_id", running.TaskID),
zap.Error(setErr))
} else if changed {
// Startup reconciliation writes the marker directly because it runs
// before the task service's periodic sweep. Publish the committed
// task projection so already-connected clients show the warning
// without waiting for a reload or unrelated task update.
task, taskErr := s.repo.GetTask(ctx, running.TaskID)
if taskErr != nil || task == nil {
s.logger.Warn("failed to load task after startup interruption marker",
zap.String("task_id", running.TaskID),
zap.Error(taskErr))
} else {
s.publishTaskUpdated(ctx, task)
}
}
}
@@ -77,6 +77,18 @@ const resumeReasonFailedSessionResumable = "failed_session_resumable"
// instead of staying stuck on read-only workspace restore.
const resumeReasonArchiveCancelledResumable = "archive_cancelled_resumable"
// resumeReasonOrphanCancelledResumable is the resume reason returned when the
// session reconciliation sweep cancelled an execution-less session. Existing
// rows with this exact system reason remain eligible for same-session recovery;
// explicit user/coordinator stops continue through the terminal-state path.
const resumeReasonOrphanCancelledResumable = "orphan_cancelled_resumable"
// resumeReasonInterruptedSessionResumable is returned for a session that the
// restart/stall sweep settled to WAITING_FOR_INPUT after execution loss. The
// durable session marker keeps this shape resumable even when the executor row
// was removed before the sweep ran.
const resumeReasonInterruptedSessionResumable = "interrupted_session_resumable"
// resumeReasonTaskArchived tells clients that recovery is unavailable until
// the owning task is unarchived. It is intentionally distinct from a runtime
// failure so archived history never enters recovery UI.
@@ -2975,6 +2987,7 @@ func (s *Service) resumeTaskSessionWithContinuation(
if (err != nil || running == nil) &&
session.State != models.TaskSessionStateCancelled &&
session.State != models.TaskSessionStateFailed &&
!models.HasInterruptedRecoveryPending(session.Metadata) &&
!allowCompletedResume {
// Executor record is required for non-terminal sessions. For cancelled/failed sessions
// the record may already have been cleaned up before the user clicked Resume — allow it.
@@ -4152,6 +4165,8 @@ func (s *Service) GetTaskSessionStatus(ctx context.Context, taskID, sessionID st
resp.AgentProfileID = session.AgentProfileID
resp.AutoResumeAllowed, resp.AutoResumeBlockedReason = s.autoResumeEligibility(ctx, task, session)
if task.ArchivedAt != nil {
resp.AutoResumeAllowed = false
resp.AutoResumeBlockedReason = resumeReasonTaskArchived
resp.ResumeReason = resumeReasonTaskArchived
return resp, nil
}
@@ -4231,6 +4246,21 @@ func (s *Service) GetTaskSessionStatus(ctx context.Context, taskID, sessionID st
// (needs_resume && is_resumable) can never satisfy.
return evaluateFreshStartResume(session, running, runErr, resp), nil
}
// The active-session reconciliation sweep used the exact orphan reason
// when it cancelled execution-less sessions. Treat those legacy rows like
// archive cancellations so opening the task restores the same conversation
// without replaying its prior task prompt. Other CANCELLED rows remain
// explicit user/coordinator stops and stay on workspace restore.
if isOrphanCancelledSession(session) {
if running != nil && running.Resumable {
out := s.validateResumeEligibility(session, resp)
if out.NeedsResume {
out.ResumeReason = resumeReasonOrphanCancelledResumable
}
return out, nil
}
return evaluateFreshStartResume(session, running, runErr, resp), nil
}
// Don't auto-resume other terminal sessions (CANCELLED stays stopped, COMPLETED is done).
if !isActiveSessionState(session.State) {
resp.IsAgentRunning = false
@@ -4343,6 +4373,16 @@ func evaluateFreshStartResume(session *models.TaskSession, running *models.Execu
resp.ResumeReason = "agent_not_running_fresh_start"
return resp
}
if running == nil &&
(session.State == models.TaskSessionStateWaitingForInput || session.State == models.TaskSessionStateStarting || session.State == models.TaskSessionStateRunning) &&
(runErr == nil || errors.Is(runErr, models.ErrExecutorRunningNotFound)) &&
models.HasInterruptedRecoveryPending(session.Metadata) {
resp.IsAgentRunning = false
resp.IsResumable = true
resp.NeedsResume = true
resp.ResumeReason = resumeReasonInterruptedSessionResumable
return resp
}
// The archive cleanup (Service.ArchiveTask / HandoffService cascade) tears
// down the ExecutorRunning row entirely, so an archive-cancelled session
// reaches this function with running == nil — exactly the shape an
@@ -4355,6 +4395,13 @@ func evaluateFreshStartResume(session *models.TaskSession, running *models.Execu
resp.ResumeReason = resumeReasonArchiveCancelledResumable
return resp
}
if isOrphanCancelledSession(session) {
resp.IsAgentRunning = false
resp.IsResumable = true
resp.NeedsResume = true
resp.ResumeReason = resumeReasonOrphanCancelledResumable
return resp
}
resp.IsAgentRunning = false
resp.IsResumable = false
resp.NeedsResume = false
@@ -4492,6 +4539,15 @@ func isArchiveCancelledSession(session *models.TaskSession) bool {
models.IsArchiveCancelReason(session.ErrorMessage)
}
// isOrphanCancelledSession reports the exact legacy cancellation marker used
// by session reconciliation. Matching the reason exactly keeps explicit user
// stops and unrelated terminal states out of automatic recovery.
func isOrphanCancelledSession(session *models.TaskSession) bool {
return session != nil &&
session.State == models.TaskSessionStateCancelled &&
models.IsOrphanCancelReason(session.ErrorMessage)
}
func shouldHealStuckStartingSession(session *models.TaskSession, running *models.ExecutorRunning) bool {
if session == nil || running == nil {
return false
@@ -0,0 +1,219 @@
package orchestrator
import (
"context"
"testing"
"time"
"github.com/kandev/kandev/internal/orchestrator/dto"
"github.com/kandev/kandev/internal/orchestrator/executor"
"github.com/kandev/kandev/internal/task/models"
sqliterepo "github.com/kandev/kandev/internal/task/repository/sqlite"
)
type orphanStatusCase struct {
name string
profileID string
reason string
archive bool
deferred bool
live bool
running bool
resumable bool
resumeToken string
wantNeedsResume bool
wantResumable bool
wantAgentRunning bool
wantAutoResumeAllowed bool
wantBlockReason string
wantResumeReason string
wantError string
}
func TestGetTaskSessionStatus_OrphanCancelledRecoveryGuards(t *testing.T) {
for _, tc := range orphanStatusCases() {
t.Run(tc.name, func(t *testing.T) { runOrphanStatusCase(t, tc) })
}
}
func orphanStatusCases() []orphanStatusCase {
return []orphanStatusCase{
{
name: "missing runtime row remains recoverable", profileID: "profile1",
reason: models.SessionOrphanedCancelReason, wantNeedsResume: true,
wantResumable: true, wantAutoResumeAllowed: true,
wantResumeReason: resumeReasonOrphanCancelledResumable,
},
{
name: "non-resumable runtime falls back to fresh start", profileID: "profile1",
reason: models.SessionOrphanedCancelReason, running: true,
resumeToken: "acp-session-123", wantNeedsResume: true, wantResumable: true,
wantAutoResumeAllowed: true, wantResumeReason: resumeReasonOrphanCancelledResumable,
},
{
name: "missing profile reports actionable error", reason: models.SessionOrphanedCancelReason,
running: true, resumable: true, resumeToken: "acp-session-123",
wantAutoResumeAllowed: true, wantError: "session missing agent profile",
},
{
name: "explicit stop stays terminal", profileID: "profile1",
reason: "user stopped the session", running: true, resumable: true,
resumeToken: "acp-session-123", wantAutoResumeAllowed: true,
},
{
name: "archived task blocks recovery", profileID: "profile1",
reason: models.SessionOrphanedCancelReason, archive: true, running: true,
resumable: true, resumeToken: "acp-session-123",
wantBlockReason: resumeReasonTaskArchived, wantResumeReason: resumeReasonTaskArchived,
},
{
name: "deferred owner blocks passive recovery", profileID: "profile1",
reason: models.SessionOrphanedCancelReason, deferred: true, running: true,
resumable: true, resumeToken: "acp-session-123", wantNeedsResume: true,
wantResumable: true, wantBlockReason: autoResumeBlockedLaunchQueued,
wantResumeReason: resumeReasonOrphanCancelledResumable,
},
{
name: "live execution wins over recovery", profileID: "profile1",
reason: models.SessionOrphanedCancelReason, live: true, running: true,
resumable: true, resumeToken: "acp-session-123", wantResumable: true,
wantAgentRunning: true, wantAutoResumeAllowed: true,
},
}
}
func runOrphanStatusCase(t *testing.T, tc orphanStatusCase) {
t.Helper()
ctx := context.Background()
repo := setupTestRepo(t)
seedTaskAndSession(t, repo, "task1", "session1", models.TaskSessionStateCancelled)
setOrphanStatusSession(t, repo, tc)
if tc.deferred {
setOrphanStatusDeferral(t, repo)
}
if tc.running {
setOrphanStatusRuntime(t, repo, tc)
}
agentMgr := &mockAgentManager{repoForExecutionLookup: repo}
if tc.live {
agentMgr.isAgentRunningFn = func(context.Context, string) bool { return true }
}
svc := createTestServiceWithAgent(repo, newMockStepGetter(), newMockTaskRepo(), agentMgr)
svc.executor = executor.NewExecutor(agentMgr, repo, testLogger(), executor.ExecutorConfig{})
resp, err := svc.GetTaskSessionStatus(ctx, "task1", "session1")
if err != nil {
t.Fatalf("GetTaskSessionStatus: %v", err)
}
assertOrphanStatus(t, resp, tc)
}
func setOrphanStatusSession(t *testing.T, repo *sqliterepo.Repository, tc orphanStatusCase) {
t.Helper()
ctx := context.Background()
session, err := repo.GetTaskSession(ctx, "session1")
if err != nil {
t.Fatalf("load session: %v", err)
}
session.AgentProfileID = tc.profileID
session.ErrorMessage = tc.reason
if err := repo.UpdateTaskSession(ctx, session); err != nil {
t.Fatalf("update session: %v", err)
}
if !tc.archive {
return
}
if err := repo.ArchiveTask(ctx, "task1"); err != nil {
t.Fatalf("archive task: %v", err)
}
session, err = repo.GetTaskSession(ctx, "session1")
if err != nil {
t.Fatalf("reload archived session: %v", err)
}
session.AgentProfileID = tc.profileID
session.ErrorMessage = tc.reason
if err := repo.UpdateTaskSession(ctx, session); err != nil {
t.Fatalf("restore archived session marker: %v", err)
}
}
func setOrphanStatusDeferral(t *testing.T, repo *sqliterepo.Repository) {
t.Helper()
deferred := models.CeilingRecordKeys(models.CeilingDeferral{
Kind: models.CeilingLaunchStartCreated,
Payload: map[string]interface{}{
metaKeySessionID: "session1",
metaKeyAgentProfileID: "profile1",
},
})
if err := repo.SetTaskMetadataKey(context.Background(), "task1", models.MetaKeyDeferredLaunch, deferred); err != nil {
t.Fatalf("set deferred launch: %v", err)
}
}
func setOrphanStatusRuntime(t *testing.T, repo *sqliterepo.Repository, tc orphanStatusCase) {
t.Helper()
now := time.Now().UTC()
if err := repo.UpsertExecutorRunning(context.Background(), &models.ExecutorRunning{
ID: "running1", SessionID: "session1", TaskID: "task1", Status: "ready",
Resumable: tc.resumable, ResumeToken: tc.resumeToken, CreatedAt: now, UpdatedAt: now,
}); err != nil {
t.Fatalf("upsert executor running: %v", err)
}
}
func assertOrphanStatus(t *testing.T, resp dto.TaskSessionStatusResponse, tc orphanStatusCase) {
t.Helper()
if resp.NeedsResume != tc.wantNeedsResume || resp.IsResumable != tc.wantResumable {
t.Fatalf("recovery flags = needs:%t resumable:%t, want needs:%t resumable:%t; response=%+v",
resp.NeedsResume, resp.IsResumable, tc.wantNeedsResume, tc.wantResumable, resp)
}
if resp.IsAgentRunning != tc.wantAgentRunning || resp.AutoResumeAllowed != tc.wantAutoResumeAllowed {
t.Fatalf("runtime flags = running:%t auto:%t, want running:%t auto:%t; response=%+v",
resp.IsAgentRunning, resp.AutoResumeAllowed, tc.wantAgentRunning, tc.wantAutoResumeAllowed, resp)
}
if resp.AutoResumeBlockedReason != tc.wantBlockReason || resp.ResumeReason != tc.wantResumeReason {
t.Fatalf("recovery reasons = blocked:%q resume:%q, want blocked:%q resume:%q; response=%+v",
resp.AutoResumeBlockedReason, resp.ResumeReason, tc.wantBlockReason, tc.wantResumeReason, resp)
}
if resp.Error != tc.wantError {
t.Fatalf("Error = %q, want %q; response=%+v", resp.Error, tc.wantError, resp)
}
}
func TestGetTaskSessionStatus_OrphanCancelledStatusIsIdempotent(t *testing.T) {
ctx := context.Background()
repo := setupTestRepo(t)
seedTaskAndSession(t, repo, "task1", "session1", models.TaskSessionStateCancelled)
session, err := repo.GetTaskSession(ctx, "session1")
if err != nil {
t.Fatalf("load session: %v", err)
}
session.AgentProfileID = "profile1"
session.ErrorMessage = models.SessionOrphanedCancelReason
if err := repo.UpdateTaskSession(ctx, session); err != nil {
t.Fatalf("update session: %v", err)
}
now := time.Now().UTC()
if err := repo.UpsertExecutorRunning(ctx, &models.ExecutorRunning{
ID: "running1", SessionID: "session1", TaskID: "task1", Status: "ready",
Resumable: true, ResumeToken: "acp-session-123", CreatedAt: now, UpdatedAt: now,
}); err != nil {
t.Fatalf("upsert executor running: %v", err)
}
agentMgr := &mockAgentManager{repoForExecutionLookup: repo}
svc := createTestServiceWithAgent(repo, newMockStepGetter(), newMockTaskRepo(), agentMgr)
svc.executor = executor.NewExecutor(agentMgr, repo, testLogger(), executor.ExecutorConfig{})
first, err := svc.GetTaskSessionStatus(ctx, "task1", "session1")
if err != nil {
t.Fatalf("first GetTaskSessionStatus: %v", err)
}
second, err := svc.GetTaskSessionStatus(ctx, "task1", "session1")
if err != nil {
t.Fatalf("second GetTaskSessionStatus: %v", err)
}
if first != second {
t.Fatalf("repeated status changed response: first=%+v second=%+v", first, second)
}
}
@@ -111,6 +111,34 @@ func TestGetTaskSessionStatus_AutoResumesNormalWaitingSession(t *testing.T) {
}
}
func TestGetTaskSessionStatus_RecoversSweptSessionWithoutExecutorRow(t *testing.T) {
ctx := context.Background()
repo := setupTestRepo(t)
seedTaskAndSession(t, repo, "task1", "session1", models.TaskSessionStateWaitingForInput)
if err := repo.SetSessionMetadataKey(ctx, "session1", models.SessionMetaKeyInterruptedRecoveryPending, "recovery-token"); err != nil {
t.Fatalf("set interrupted recovery marker: %v", err)
}
session, err := repo.GetTaskSession(ctx, "session1")
if err != nil {
t.Fatalf("load session: %v", err)
}
session.AgentProfileID = "profile1"
if err := repo.UpdateTaskSession(ctx, session); err != nil {
t.Fatalf("update session: %v", err)
}
agentMgr := &mockAgentManager{repoForExecutionLookup: repo}
svc := createTestServiceWithAgent(repo, newMockStepGetter(), newMockTaskRepo(), agentMgr)
svc.executor = executor.NewExecutor(agentMgr, repo, testLogger(), executor.ExecutorConfig{})
resp, err := svc.GetTaskSessionStatus(ctx, "task1", "session1")
if err != nil {
t.Fatalf("GetTaskSessionStatus: %v", err)
}
if !resp.NeedsResume || !resp.IsResumable {
t.Fatalf("status = %+v, want resumable lazy recovery", resp)
}
}
func TestAutoResumeEligibilityPreservesDeferredLaunchOwnership(t *testing.T) {
queuedRecord := models.CeilingRecordKeys(models.CeilingDeferral{
Kind: models.CeilingLaunchStartCreated,
@@ -473,6 +501,63 @@ func TestGetTaskSessionStatus_ArchiveCancelledSessionWithTokenAutoResumes(t *tes
}
}
// TestGetTaskSessionStatus_OrphanCancelledSessionAutoResumes verifies that a
// session cancelled by the orphan reconciliation sweep remains recoverable in
// the same conversation. The cancellation reason is exact so an explicit user
// stop cannot accidentally enter this path.
func TestGetTaskSessionStatus_OrphanCancelledSessionAutoResumes(t *testing.T) {
ctx := context.Background()
repo := setupTestRepo(t)
seedTaskAndSession(t, repo, "task1", "session1", models.TaskSessionStateCancelled)
session, err := repo.GetTaskSession(ctx, "session1")
if err != nil {
t.Fatalf("failed to load session: %v", err)
}
session.AgentProfileID = "profile1"
session.ErrorMessage = models.SessionOrphanedCancelReason
if err := repo.UpdateTaskSession(ctx, session); err != nil {
t.Fatalf("failed to update session: %v", err)
}
now := time.Now().UTC()
if err := repo.UpsertExecutorRunning(ctx, &models.ExecutorRunning{
ID: "er1",
SessionID: "session1",
TaskID: "task1",
Status: "ready",
Resumable: true,
ResumeToken: "acp-session-123",
CreatedAt: now,
UpdatedAt: now,
}); err != nil {
t.Fatalf("failed to upsert executor running: %v", err)
}
taskRepo := newMockTaskRepo()
taskRepo.tasks["task1"] = &v1.Task{ID: "task1", State: v1.TaskStateReview}
agentMgr := &mockAgentManager{repoForExecutionLookup: repo}
svc := createTestServiceWithAgent(repo, newMockStepGetter(), taskRepo, agentMgr)
svc.executor = executor.NewExecutor(agentMgr, repo, testLogger(), executor.ExecutorConfig{})
resp, err := svc.GetTaskSessionStatus(ctx, "task1", "session1")
if err != nil {
t.Fatalf("GetTaskSessionStatus returned error: %v", err)
}
if !resp.NeedsResume {
t.Fatal("expected NeedsResume=true for orphan-cancelled session")
}
if !resp.IsResumable {
t.Fatal("expected IsResumable=true for orphan-cancelled session")
}
if resp.NeedsWorkspaceRestore {
t.Fatal("expected NeedsWorkspaceRestore=false when auto-resuming")
}
if resp.ResumeReason != resumeReasonOrphanCancelledResumable {
t.Fatalf("expected ResumeReason=%q, got %q", resumeReasonOrphanCancelledResumable, resp.ResumeReason)
}
}
// TestGetTaskSessionStatus_ArchiveCancelledSessionWithNonResumableTokenFreshStarts
// covers the CodeRabbit-flagged gap: an archive-cancelled session can carry a
// resume token whose ExecutorRunning row has Resumable=false (the runtime
@@ -773,6 +858,98 @@ func TestResumeTaskSession_WaitsForPromptReady(t *testing.T) {
}
}
func TestResumeTaskSession_RecoversSweptSessionWithoutExecutorRow(t *testing.T) {
ctx := context.Background()
repo := setupTestRepo(t)
seedTaskAndSession(t, repo, "task1", "session1", models.TaskSessionStateRunning)
session, err := repo.GetTaskSession(ctx, "session1")
if err != nil {
t.Fatalf("load session: %v", err)
}
session.AgentProfileID = "profile1"
if err := repo.UpdateTaskSession(ctx, session); err != nil {
t.Fatalf("update session: %v", err)
}
// Use the same guarded transition as the active/restart sweep. This
// intentionally has no executors_running row.
recovered, err := repo.RecoverTaskSessionByCandidate(ctx, models.ActiveSessionRecoveryCandidate{
TaskID: session.TaskID, SessionID: session.ID, ExpectedState: session.State,
ExpectedUpdatedAt: session.UpdatedAt, ExpectedLastEventAt: session.UpdatedAt,
}, time.Time{})
if err != nil || recovered == nil {
t.Fatalf("recover swept session = %+v, err=%v", recovered, err)
}
ready := make(chan struct{})
checked := make(chan struct{}, 1)
pollCtx, cancelPoll := context.WithTimeout(context.Background(), 5*time.Second)
t.Cleanup(cancelPoll)
agentMgr := &mockAgentManager{
isAgentRunning: false,
repoForExecutionLookup: repo,
isAgentReadyFn: func(_ context.Context, _ string) bool {
select {
case checked <- struct{}{}:
default:
}
select {
case <-ready:
return true
default:
return false
}
},
launchAgentFunc: func(_ context.Context, req *executor.LaunchAgentRequest) (*executor.LaunchAgentResponse, error) {
go func(sessionID string) {
for {
select {
case <-pollCtx.Done():
return
default:
}
sess, getErr := repo.GetTaskSession(context.Background(), sessionID)
if getErr == nil && sess != nil && sess.State == models.TaskSessionStateStarting {
sess.State = models.TaskSessionStateWaitingForInput
sess.UpdatedAt = time.Now().UTC()
_ = repo.UpdateTaskSession(context.Background(), sess)
return
}
time.Sleep(time.Millisecond)
}
}(req.SessionID)
return &executor.LaunchAgentResponse{AgentExecutionID: "exec-swept-resume"}, nil
},
}
taskRepo := newMockTaskRepo()
taskRepo.tasks["task1"] = &v1.Task{ID: "task1", State: v1.TaskStateInProgress}
svc := createTestServiceWithAgent(repo, newMockStepGetter(), taskRepo, agentMgr)
svc.executor = executor.NewExecutor(agentMgr, repo, testLogger(), executor.ExecutorConfig{})
status, err := svc.GetTaskSessionStatus(ctx, "task1", "session1")
if err != nil || !status.NeedsResume || !status.IsResumable {
t.Fatalf("status after sweep = %+v, err=%v, want focus recovery", status, err)
}
done := make(chan error, 1)
go func() {
_, resumeErr := svc.ResumeTaskSession(ctx, "task1", "session1")
done <- resumeErr
}()
select {
case <-checked:
case <-time.After(3 * time.Second):
t.Fatal("ResumeTaskSession did not reach prompt readiness")
}
close(ready)
select {
case err := <-done:
if err != nil {
t.Fatalf("ResumeTaskSession after sweep: %v", err)
}
case <-time.After(3 * time.Second):
t.Fatal("ResumeTaskSession did not complete")
}
}
func TestResumeTaskSession_ReapsPromptDeadExecutionAndRelaunches(t *testing.T) {
oldReadyTimeout := agentPromptReadyTimeout
oldReadyInterval := agentPromptReadyInterval
+96 -5
View File
@@ -178,11 +178,10 @@ const (
// because the winner will (or already did) handle it.
// Absent on ordinary (non-watcher) auto-start tasks, which launch normally.
MetaKeyAutoStartClaimed = "auto_start_claimed"
// MetaKeyInterruptedAt is set by startup reconciliation when a task's
// session was mid-turn (STARTING/RUNNING) when the backend died. Its
// presence makes the task DTO report `interrupted: true` so task-list
// surfaces show the red interruption icon; the orchestrator removes the
// key when a session of the task next enters STARTING/RUNNING.
// MetaKeyInterruptedAt is set by reconciliation when a task's session lost
// an active turn. Its presence makes the task DTO report `interrupted: true`
// so task-list surfaces show the warning indicator. The orchestrator removes
// the key only after the provider confirms recovery.
MetaKeyInterruptedAt = "interrupted_at"
// MetaKeyAutoStartFailed is set when a workflow step's auto_start_agent
// on_enter action fails to launch a run (kanban StartTask error, or an
@@ -400,6 +399,14 @@ const (
// latest successful agent boot. Recovery cards compare this timestamp with
// their own creation time, so the result survives transcript write failures.
SessionMetaKeyRecoveryResolvedAt = "recovery_resolved_at"
// SessionMetaKeyInterruptedRecoveryPending is a durable token written by
// session reconciliation when execution loss returns a conversation to
// WAITING_FOR_INPUT. It distinguishes an interrupted waiting session from
// an ordinary idle conversation even when the executor row was removed.
SessionMetaKeyInterruptedRecoveryPending = "interrupted_recovery_pending"
// SessionMetaKeyRecoverySettlementPending carries the immutable recovery
// snapshot needed to retry post-commit effects after a partial failure.
SessionMetaKeyRecoverySettlementPending = "recovery_settlement_pending"
// SessionMetaKeyInitialCreatePromptPassthrough stores the durable
// execution and turn evidence for a creation prompt admitted before a
// passthrough agent.running event. It prevents recovery from evaluating
@@ -407,6 +414,69 @@ const (
SessionMetaKeyInitialCreatePromptPassthrough = "initial_create_prompt_passthrough"
)
// InterruptedRecoverySettlement is the durable recovery snapshot used when a
// session transition committed but one of its cross-repository effects did
// not. The executor identity and turn ID are compare-and-set inputs; they
// must never be refreshed from a successor execution during retry.
type InterruptedRecoverySettlement struct {
Token string `json:"token"`
ExpectedState TaskSessionState `json:"expected_state"`
// RecoveredUpdatedAt is the session generation written by the recovery
// transition. Retries must compare against this immutable timestamp rather
// than treating a later session read as the recovery generation.
RecoveredUpdatedAt time.Time `json:"recovered_updated_at"`
ExpectedTurnID string `json:"expected_turn_id,omitempty"`
ExpectedExecutorID string `json:"expected_executor_id,omitempty"`
ExpectedExecutorAgentExecutionID string `json:"expected_executor_agent_execution_id,omitempty"`
ExpectedExecutorUpdatedAt time.Time `json:"expected_executor_updated_at,omitempty"`
}
// HasInterruptedRecoveryPending reports whether metadata carries a valid
// interrupted-session token. Older in-process callers used a boolean marker,
// so explicit true remains accepted during the transition to tokens.
func HasInterruptedRecoveryPending(metadata map[string]interface{}) bool {
if metadata == nil {
return false
}
value, ok := metadata[SessionMetaKeyInterruptedRecoveryPending]
if !ok || value == nil {
return false
}
switch typed := value.(type) {
case bool:
return typed
case string:
return strings.TrimSpace(typed) != ""
default:
return true
}
}
// LoadInterruptedRecoverySettlement decodes the immutable post-commit
// snapshot. Invalid or incomplete values fail closed so a retry cannot repair
// a successor executor or abandon an unrelated turn.
func LoadInterruptedRecoverySettlement(metadata map[string]interface{}) (InterruptedRecoverySettlement, bool) {
if metadata == nil {
return InterruptedRecoverySettlement{}, false
}
value, ok := metadata[SessionMetaKeyRecoverySettlementPending]
if !ok || value == nil {
return InterruptedRecoverySettlement{}, false
}
payload, err := json.Marshal(value)
if err != nil {
return InterruptedRecoverySettlement{}, false
}
var settlement InterruptedRecoverySettlement
if err := json.Unmarshal(payload, &settlement); err != nil {
return InterruptedRecoverySettlement{}, false
}
if strings.TrimSpace(settlement.Token) == "" || settlement.ExpectedState == "" || settlement.RecoveredUpdatedAt.IsZero() {
return InterruptedRecoverySettlement{}, false
}
return settlement, true
}
// IsCompletionFollowUpSession reports whether a session was explicitly
// reopened only to continue its completed conversation. The marker remains
// meaningful after the session settles back to WAITING_FOR_INPUT.
@@ -1903,6 +1973,27 @@ type ActiveSessionCancellationCandidate struct {
ExpectedTurnID string
}
// ActiveSessionRecoveryCandidate is the compare-and-set snapshot used by
// session reconciliation when execution loss should return a conversation to
// WAITING_FOR_INPUT. The observed state, activity clock, and turn identity
// must still match at the write boundary; a newer launch, message, or turn
// therefore owns the session and prevents stale recovery.
type ActiveSessionRecoveryCandidate struct {
TaskID string
SessionID string
ExpectedState TaskSessionState
ExpectedUpdatedAt time.Time
// RecoveredUpdatedAt is populated from the durable settlement on retry.
// The initial candidate leaves it zero because the repository assigns the
// committed generation when it performs the state transition.
RecoveredUpdatedAt time.Time
ExpectedLastEventAt time.Time
ExpectedTurnID string
ExpectedExecutorID string
ExpectedExecutorAgentExecutionID string
ExpectedExecutorUpdatedAt time.Time
}
// ToAPI converts internal TaskSession to API type
// TODO: Add v1.TaskSession type to pkg/api/v1/
func (s *TaskSession) ToAPI() map[string]interface{} {
@@ -79,15 +79,23 @@ const (
SessionArchiveTreeCancelReason = "task tree archived"
)
// SessionOrphanedCancelReason is the TaskSession.ErrorMessage value written
// when the session reconciliation sweep cancels an active session that no
// longer has a live execution behind it (e.g. after a backend restart) and
// has been event-silent beyond the sweep's grace window. It is deliberately
// not an archive reason: unarchived tasks must treat the cancelled session
// like an explicit stop, not like archive cancellation awaiting unarchive.
// SessionOrphanedCancelReason is the exact TaskSession.ErrorMessage value
// written when the session reconciliation sweep cancels an active session that
// no longer has a live execution behind it (for example, after a backend
// restart) and has been event-silent beyond the sweep's grace window. It is
// deliberately distinct from archive and explicit-stop reasons: legacy rows
// with this marker remain eligible for same-session recovery when the user
// opens the task.
const SessionOrphanedCancelReason = "orphaned session"
// IsArchiveCancelReason reports whether reason came from an archive path.
func IsArchiveCancelReason(reason string) bool {
return reason == SessionArchiveCancelReason || reason == SessionArchiveTreeCancelReason
}
// IsOrphanCancelReason reports whether reason is the exact marker emitted by
// session reconciliation for an execution-less interruption. Callers must not
// broaden this match: explicit user stops retain their terminal recovery rules.
func IsOrphanCancelReason(reason string) bool {
return reason == SessionOrphanedCancelReason
}
@@ -54,3 +54,20 @@ func TestRowMustBePreserved(t *testing.T) {
})
}
}
func TestIsOrphanCancelReasonRequiresExactSystemMarker(t *testing.T) {
if !IsOrphanCancelReason(SessionOrphanedCancelReason) {
t.Fatal("the reconciliation marker must be recognized")
}
for _, reason := range []string{
"",
"orphaned session ",
"ORPHANED SESSION",
SessionArchiveCancelReason,
"user stopped",
} {
if IsOrphanCancelReason(reason) {
t.Fatalf("reason %q must not be treated as an orphan cancellation", reason)
}
}
}
@@ -2240,6 +2240,163 @@ func (r *Repository) CancelRunningTaskSessionByID(ctx context.Context, sessionID
return session, rows.Err()
}
// RecoverTaskSessionByCandidate returns one execution-less session to
// WAITING_FOR_INPUT when its state, activity clock, and current turn still
// match the reconciliation snapshot. The optional staleBefore cutoff is used
// by the restart pass to preserve its launch grace window; a zero cutoff is
// used by the active-task stall pass.
//
// The write is session-scoped and compare-and-set guarded. A newer message,
// state transition, successor turn, or refreshed launch causes the UPDATE to
// match no row, leaving the current owner untouched for the next sweep.
//
//nolint:cyclop,funlen // The SQL predicate mirrors the complete recovery CAS contract.
func (r *Repository) RecoverTaskSessionByCandidate(
ctx context.Context,
candidate models.ActiveSessionRecoveryCandidate,
staleBefore time.Time,
) (*models.TaskSession, error) {
if candidate.SessionID == "" || candidate.TaskID == "" {
return nil, nil
}
switch candidate.ExpectedState {
case models.TaskSessionStateCreated,
models.TaskSessionStateStarting,
models.TaskSessionStateRunning,
models.TaskSessionStateWaitingForInput:
default:
return nil, nil
}
// PostgreSQL timestamp columns retain microsecond precision. Use the same
// precision in the JSON settlement snapshot so a retry compares the
// stored generation exactly on both database dialects.
now := time.Now().UTC().Truncate(time.Microsecond)
writeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
// Preserve a durable recovery token and immutable effect snapshot in the
// same compare-and-set write as the WAITING_FOR_INPUT transition. A later
// focus request can therefore distinguish this recovered conversation from
// an ordinary idle session even when executors_running has no row, and a
// partial post-commit settlement can retry without inspecting a successor.
settlementPending := candidate.ExpectedTurnID != "" ||
candidate.ExpectedState == models.TaskSessionStateCreated ||
candidate.ExpectedState == models.TaskSessionStateStarting ||
candidate.ExpectedState == models.TaskSessionStateRunning
setClause := "state = ?, error_message = ?, completed_at = ?, updated_at = ?"
query := `
UPDATE task_sessions
SET `
args := make([]interface{}, 0, 16)
if settlementPending {
token := interruptedRecoveryToken(candidate)
settlement := models.InterruptedRecoverySettlement{
Token: token,
ExpectedState: candidate.ExpectedState,
RecoveredUpdatedAt: now,
ExpectedTurnID: candidate.ExpectedTurnID,
ExpectedExecutorID: candidate.ExpectedExecutorID,
ExpectedExecutorAgentExecutionID: candidate.ExpectedExecutorAgentExecutionID,
ExpectedExecutorUpdatedAt: candidate.ExpectedExecutorUpdatedAt,
}
pendingJSON, err := json.Marshal(token)
if err != nil {
return nil, err
}
settlementJSON, err := json.Marshal(settlement)
if err != nil {
return nil, err
}
if dialect.IsPostgres(r.db.DriverName()) {
base := postgresMetadataObject
setClause = "metadata = jsonb_set(jsonb_set(" + base + ", ARRAY[?]::text[], ?::jsonb, true), ARRAY[?]::text[], ?::jsonb, true)::text, " + setClause
args = append(args,
models.SessionMetaKeyInterruptedRecoveryPending, string(pendingJSON),
models.SessionMetaKeyRecoverySettlementPending, string(settlementJSON),
)
} else {
base := sqliteMetadataObject
setClause = "metadata = json_set(json_set(" + base + ", ?, json(?)), ?, json(?)), " + setClause
args = append(args,
jsonPath(models.SessionMetaKeyInterruptedRecoveryPending), string(pendingJSON),
jsonPath(models.SessionMetaKeyRecoverySettlementPending), string(settlementJSON),
)
}
}
query += setClause + `
WHERE id = ?
AND task_id = ?
AND EXISTS (
SELECT 1 FROM tasks recoverable_task
WHERE recoverable_task.id = task_sessions.task_id
AND recoverable_task.archived_at IS NULL
)
AND state = ?
AND updated_at = ?
`
args = append(args,
string(models.TaskSessionStateWaitingForInput), "", nil, now,
candidate.SessionID, candidate.TaskID, string(candidate.ExpectedState), candidate.ExpectedUpdatedAt,
)
if !staleBefore.IsZero() {
query += " AND updated_at < ?\n"
args = append(args, staleBefore)
}
if !candidate.ExpectedLastEventAt.IsZero() {
query += `
AND NOT EXISTS (
SELECT 1 FROM task_session_messages newer_message
WHERE newer_message.task_session_id = task_sessions.id
AND newer_message.updated_at > ?
)
`
args = append(args, candidate.ExpectedLastEventAt)
}
if candidate.ExpectedTurnID == "" {
query += `
AND NOT EXISTS (
SELECT 1 FROM task_session_turns active_turn
WHERE active_turn.task_session_id = task_sessions.id
AND active_turn.completed_at IS NULL
)
`
} else {
query += `
AND EXISTS (
SELECT 1 FROM task_session_turns active_turn
WHERE active_turn.id = ?
AND active_turn.task_session_id = task_sessions.id
AND active_turn.completed_at IS NULL
)
`
args = append(args, candidate.ExpectedTurnID)
}
query += `
RETURNING id, agent_profile_id, agent_profile_snapshot, is_passthrough, name,
review_status, metadata, task_environment_id, state, updated_at, is_primary
`
rows, err := r.db.QueryContext(writeCtx, r.db.Rebind(query), args...)
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
if !rows.Next() {
return nil, rows.Err()
}
session, err := scanCancelledTaskSessionRow(rows, candidate.TaskID)
if err != nil {
return nil, err
}
return session, rows.Err()
}
func interruptedRecoveryToken(candidate models.ActiveSessionRecoveryCandidate) string {
return fmt.Sprintf("%s:%s:%s:%s", candidate.SessionID,
candidate.ExpectedState,
candidate.ExpectedUpdatedAt.UTC().Format(time.RFC3339Nano),
candidate.ExpectedTurnID)
}
// CancelActiveTaskSessionsByIDs is documented on the SessionRepository
// interface. It shares CancelActiveTaskSessionsByTaskID's atomic
// UPDATE ... RETURNING shape and detached-but-bounded write context, so the
@@ -2846,6 +3003,52 @@ func (r *Repository) RemoveSessionMetadataKeyIfStamp(
return rows > 0, err
}
// RemoveSessionMetadataKeyIfJSONValue removes one metadata key only when its
// complete JSON value still equals expectedValue. Recovery settlement uses
// this compare-and-set boundary so a delayed effect cannot erase a newer
// recovery snapshot written after a successor launch.
func (r *Repository) RemoveSessionMetadataKeyIfJSONValue(
ctx context.Context,
sessionID, key string,
expectedValue interface{},
) (bool, error) {
payload, err := json.Marshal(expectedValue)
if err != nil {
return false, fmt.Errorf("failed to serialize expected session metadata: %w", err)
}
now := time.Now().UTC()
driver := r.db.DriverName()
var query string
var args []interface{}
if dialect.IsPostgres(driver) {
query = `
UPDATE task_sessions
SET metadata = (
CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}'::jsonb ELSE metadata::jsonb END
#- ARRAY[?]::text[]
)::text, updated_at = ?
WHERE id = ?
AND (CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}'::jsonb ELSE metadata::jsonb END -> ?) = ?::jsonb
`
args = []interface{}{key, now, sessionID, key, string(payload)}
} else {
path := jsonPath(key)
query = `
UPDATE task_sessions
SET metadata = json_remove(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}' ELSE metadata END, ?), updated_at = ?
WHERE id = ?
AND json_extract(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}' ELSE metadata END, ?) = json(?)
`
args = []interface{}{path, now, sessionID, path, string(payload)}
}
result, err := r.db.ExecContext(ctx, r.db.Rebind(query), args...)
if err != nil {
return false, err
}
rows, err := result.RowsAffected()
return rows > 0, err
}
func setSessionMetadataKeyIfAbsentQuery(driver string) string {
if dialect.IsPostgres(driver) {
return `
@@ -2286,6 +2286,99 @@ func TestCancelActiveTaskSessionsByCandidatesRejectsNewActivityAndTurn(t *testin
}
}
func TestRecoverTaskSessionByCandidatePreservesConversationAndGuardsTurn(t *testing.T) {
repo := newRepoForSessionTests(t)
ctx := context.Background()
seedRepoLink(t, repo, "ws-recover", "repo-recover", "task-recover", "session-recover", "RUNNING")
seedRepoLink(t, repo, "ws-recover", "repo-recover-2", "task-recover-2", "session-recover-2", "RUNNING")
seedRepoLink(t, repo, "ws-recover", "repo-recover-created", "task-recover-created", "session-recover-created", "CREATED")
created, err := repo.GetTaskSession(ctx, "session-recover-created")
if err != nil {
t.Fatalf("GetTaskSession created candidate: %v", err)
}
recovered, err := repo.RecoverTaskSessionByCandidate(ctx, models.ActiveSessionRecoveryCandidate{
TaskID: created.TaskID,
SessionID: created.ID,
ExpectedState: created.State,
ExpectedUpdatedAt: created.UpdatedAt,
ExpectedLastEventAt: created.UpdatedAt,
}, time.Now().UTC().Add(time.Minute))
if err != nil {
t.Fatalf("RecoverTaskSessionByCandidate created session: %v", err)
}
if recovered == nil {
t.Fatal("RecoverTaskSessionByCandidate created session returned nil")
}
if !models.HasInterruptedRecoveryPending(recovered.Metadata) {
t.Fatalf("created recovered metadata = %#v, want durable recovery marker", recovered.Metadata)
}
if _, ok := models.LoadInterruptedRecoverySettlement(recovered.Metadata); !ok {
t.Fatalf("created recovered metadata = %#v, want settlement snapshot", recovered.Metadata)
}
idleCandidate, err := repo.GetTaskSession(ctx, "session-recover")
if err != nil {
t.Fatalf("GetTaskSession idle candidate: %v", err)
}
// A cutoff in the future admits this stale snapshot. The absence of an
// active turn is part of the candidate predicate for a session whose actor
// died before a turn row was persisted.
recovered, err = repo.RecoverTaskSessionByCandidate(ctx, models.ActiveSessionRecoveryCandidate{
TaskID: idleCandidate.TaskID,
SessionID: idleCandidate.ID,
ExpectedState: idleCandidate.State,
ExpectedUpdatedAt: idleCandidate.UpdatedAt,
ExpectedLastEventAt: idleCandidate.UpdatedAt,
}, time.Now().UTC().Add(time.Minute))
if err != nil {
t.Fatalf("RecoverTaskSessionByCandidate without turn: %v", err)
}
if recovered == nil {
t.Fatal("RecoverTaskSessionByCandidate without turn returned nil")
}
if recovered.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("recovered state = %q, want WAITING_FOR_INPUT", recovered.State)
}
if recovered.TaskID != idleCandidate.TaskID {
t.Fatalf("recovered task id = %q, want %q", recovered.TaskID, idleCandidate.TaskID)
}
turn := &models.Turn{ID: "turn-recover", TaskSessionID: "session-recover-2", TaskID: "task-recover-2"}
if err := repo.CreateTurn(ctx, turn); err != nil {
t.Fatalf("CreateTurn: %v", err)
}
turnCandidate, err := repo.GetTaskSession(ctx, "session-recover-2")
if err != nil {
t.Fatalf("GetTaskSession turn candidate: %v", err)
}
guarded := models.ActiveSessionRecoveryCandidate{
TaskID: turnCandidate.TaskID,
SessionID: turnCandidate.ID,
ExpectedState: turnCandidate.State,
ExpectedUpdatedAt: turnCandidate.UpdatedAt,
ExpectedLastEventAt: turnCandidate.UpdatedAt,
ExpectedTurnID: turn.ID,
}
// A successor turn refreshes the session and invalidates the observed
// identity before the recovery write.
if err := repo.CreateTurn(ctx, &models.Turn{
ID: "turn-recover-successor", TaskSessionID: "session-recover-2", TaskID: "task-recover-2",
}); err != nil {
t.Fatalf("Create successor turn: %v", err)
}
recovered, err = repo.RecoverTaskSessionByCandidate(ctx, guarded, time.Time{})
if err != nil {
t.Fatalf("RecoverTaskSessionByCandidate stale turn: %v", err)
}
if recovered != nil {
t.Fatalf("stale turn candidate recovered session = %+v, want nil", recovered)
}
if got := sessionState(t, repo, "session-recover-2"); got != "RUNNING" {
t.Fatalf("session with successor turn = %q, want RUNNING", got)
}
}
// sameStringSet reports whether got and want contain the same strings,
// ignoring order and duplicates count-for-count — used to compare the set of
// cancelled session IDs against an expected set regardless of return order.
@@ -1806,6 +1806,43 @@ func (r *Repository) RemoveTaskMetadataKey(ctx context.Context, taskID, key stri
return r.removeTaskMetadataKeyWithExecutor(ctx, r.db, taskID, key)
}
// RemoveTaskMetadataKeyIfValue removes one metadata key only when its scalar
// JSON value still equals expectedValue. Recovery callbacks use this compare
// and set boundary so a delayed callback cannot erase a newer interruption
// marker written by a later restart.
func (r *Repository) RemoveTaskMetadataKeyIfValue(
ctx context.Context,
taskID, key, expectedValue string,
) (bool, error) {
if strings.TrimSpace(expectedValue) == "" {
return false, nil
}
var query string
var args []interface{}
if dialect.IsPostgres(r.db.DriverName()) {
query = `
UPDATE tasks
SET metadata = (CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}'::jsonb ELSE metadata::jsonb END #- ARRAY[?]::text[])::text, updated_at = ?
WHERE id = ? AND jsonb_extract_path_text(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}'::jsonb ELSE metadata::jsonb END, ?) = ?
`
args = []interface{}{key, time.Now().UTC(), taskID, key, expectedValue}
} else {
path := jsonPath(key)
query = `
UPDATE tasks
SET metadata = json_remove(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}' ELSE metadata END, ?), updated_at = ?
WHERE id = ? AND json_extract(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}' ELSE metadata END, ?) = ?
`
args = []interface{}{path, time.Now().UTC(), taskID, path, expectedValue}
}
result, err := r.db.ExecContext(ctx, r.db.Rebind(query), args...)
if err != nil {
return false, err
}
rows, err := result.RowsAffected()
return rows > 0, err
}
// ClearManualMoveLifecycleMarkersIfCompleted atomically removes the pending
// and completed markers for a manual move only while the completed marker and
// task update generation still match the recovery snapshot. A new move clears
@@ -2190,6 +2227,115 @@ func (r *Repository) SetTaskMetadataKeyIfNotArchived(ctx context.Context, taskID
return rows > 0, err
}
// SetTaskMetadataKeyIfAbsentNotArchived writes one metadata key only when the
// task is live and the key is absent. Recovery marker creation uses this
// compare-and-set boundary so an older settlement cannot overwrite a newer
// interruption generation.
func (r *Repository) SetTaskMetadataKeyIfAbsentNotArchived(ctx context.Context, taskID, key string, value interface{}) (bool, error) {
payload, err := json.Marshal(value)
if err != nil {
return false, err
}
var query string
if dialect.IsPostgres(r.db.DriverName()) {
query = `UPDATE tasks
SET metadata = jsonb_set(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}'::jsonb ELSE metadata::jsonb END, ARRAY[?]::text[], ?::jsonb, true)::text, updated_at = ?
WHERE id = ? AND archived_at IS NULL
AND jsonb_extract_path(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}'::jsonb ELSE metadata::jsonb END, ?) IS NULL`
} else {
query = `UPDATE tasks
SET metadata = json_set(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}' ELSE metadata END, ?, json(?)), updated_at = ?
WHERE id = ? AND archived_at IS NULL
AND json_type(CASE WHEN metadata IS NULL OR metadata = 'null' OR metadata = '' THEN '{}' ELSE metadata END, ?) IS NULL`
}
path := key
if !dialect.IsPostgres(r.db.DriverName()) {
path = jsonPath(key)
}
result, err := r.db.ExecContext(ctx, r.db.Rebind(query), path, string(payload), time.Now().UTC(), taskID, path)
if err != nil {
return false, err
}
rows, err := result.RowsAffected()
return rows > 0, err
}
// SetTaskMetadataKeyIfRecoveryCurrent writes a marker only while the session
// that owns the recovery settlement is still waiting at the same generation.
// The settlement token check is deliberately in the same UPDATE as the task
// metadata write: a successor that changes the session state or consumes the
// token cannot be followed by a delayed recovery callback that re-adds the
// interruption warning.
func (r *Repository) SetTaskMetadataKeyIfRecoveryCurrent(
ctx context.Context,
taskID, sessionID string,
expectedSessionUpdatedAt time.Time,
expectedRecoveryToken, key string,
value interface{},
) (bool, error) {
if taskID == "" || sessionID == "" || expectedRecoveryToken == "" {
return false, nil
}
payload, err := json.Marshal(value)
if err != nil {
return false, err
}
now := time.Now().UTC()
var query string
var args []interface{}
if dialect.IsPostgres(r.db.DriverName()) {
base := postgresMetadataObject
query = `UPDATE tasks
SET metadata = jsonb_set(` + base + `, ARRAY[?]::text[], ?::jsonb, true)::text, updated_at = ?
WHERE id = ? AND archived_at IS NULL
AND jsonb_extract_path(` + base + `, ?) IS NULL
AND EXISTS (
SELECT 1 FROM task_sessions recovery_session
WHERE recovery_session.id = ?
AND recovery_session.task_id = tasks.id
AND recovery_session.state = ?
AND recovery_session.updated_at = ?
AND jsonb_extract_path_text(
CASE WHEN recovery_session.metadata IS NULL OR recovery_session.metadata = 'null' OR recovery_session.metadata = '' THEN '{}'::jsonb ELSE recovery_session.metadata::jsonb END,
?, 'token'
) = ?
)`
args = []interface{}{
key, string(payload), now, taskID,
key, sessionID, string(models.TaskSessionStateWaitingForInput), expectedSessionUpdatedAt,
models.SessionMetaKeyRecoverySettlementPending, expectedRecoveryToken,
}
} else {
path := jsonPath(key)
markerPath := jsonPath(key)
settlementTokenPath := jsonPath(models.SessionMetaKeyRecoverySettlementPending + ".token")
base := sqliteMetadataObject
query = `UPDATE tasks
SET metadata = json_set(` + base + `, ?, json(?)), updated_at = ?
WHERE id = ? AND archived_at IS NULL
AND json_type(` + base + `, ?) IS NULL
AND EXISTS (
SELECT 1 FROM task_sessions recovery_session
WHERE recovery_session.id = ?
AND recovery_session.task_id = tasks.id
AND recovery_session.state = ?
AND recovery_session.updated_at = ?
AND json_extract(CASE WHEN recovery_session.metadata IS NULL OR recovery_session.metadata = 'null' OR recovery_session.metadata = '' THEN '{}' ELSE recovery_session.metadata END, ?) = ?
)`
args = []interface{}{
path, string(payload), now, taskID,
markerPath, sessionID, string(models.TaskSessionStateWaitingForInput), expectedSessionUpdatedAt,
settlementTokenPath, expectedRecoveryToken,
}
}
result, err := r.db.ExecContext(ctx, r.db.Rebind(query), args...)
if err != nil {
return false, err
}
rows, err := result.RowsAffected()
return rows > 0, err
}
// SetTaskMetadataKeyIfPresent rewrites one metadata key only while that key is
// still present, and reports whether the write landed.
//
@@ -144,6 +144,82 @@ func runMetadataNoActiveSessionContract(t *testing.T, repo *Repository) {
}
}
func runInterruptedMarkerCASContract(t *testing.T, repo *Repository) {
t.Helper()
ctx := context.Background()
if err := repo.SetTaskMetadataKey(ctx, casTaskID, models.MetaKeyInterruptedAt, "old-marker"); err != nil {
t.Fatalf("seed interrupted marker: %v", err)
}
removed, err := repo.RemoveTaskMetadataKeyIfValue(ctx, casTaskID, models.MetaKeyInterruptedAt, "new-marker")
if err != nil {
t.Fatalf("remove with stale marker: %v", err)
}
if removed {
t.Fatal("stale marker comparison must not remove the current value")
}
if value, ok := metadataValue(t, repo, models.MetaKeyInterruptedAt); !ok || value != "old-marker" {
t.Fatalf("marker after stale removal = %v (present=%v), want old-marker", value, ok)
}
removed, err = repo.RemoveTaskMetadataKeyIfValue(ctx, casTaskID, models.MetaKeyInterruptedAt, "old-marker")
if err != nil {
t.Fatalf("remove with current marker: %v", err)
}
if !removed {
t.Fatal("current marker comparison must remove the marker")
}
}
func runRecoveryMarkerCASContract(t *testing.T, repo *Repository) {
t.Helper()
ctx := context.Background()
now := time.Now().UTC()
settlement := models.InterruptedRecoverySettlement{
Token: "recovery-token",
ExpectedState: models.TaskSessionStateRunning,
RecoveredUpdatedAt: now,
}
if err := repo.CreateTaskSession(ctx, &models.TaskSession{
ID: "session-recovery-cas", TaskID: casTaskID,
State: models.TaskSessionStateWaitingForInput, UpdatedAt: now,
Metadata: map[string]interface{}{
models.SessionMetaKeyRecoverySettlementPending: settlement,
},
}); err != nil {
t.Fatalf("create recovery session: %v", err)
}
session, err := repo.GetTaskSession(ctx, "session-recovery-cas")
if err != nil {
t.Fatalf("load recovery session: %v", err)
}
written, err := repo.SetTaskMetadataKeyIfRecoveryCurrent(
ctx, casTaskID, session.ID, session.UpdatedAt, settlement.Token,
models.MetaKeyInterruptedAt, "recovery-marker",
)
if err != nil {
t.Fatalf("write recovery marker: %v", err)
}
if !written {
t.Fatal("current recovery generation must write its marker")
}
if value, ok := metadataValue(t, repo, models.MetaKeyInterruptedAt); !ok || value != "recovery-marker" {
t.Fatalf("recovery marker = %v (present=%v), want recovery-marker", value, ok)
}
if err := repo.UpdateTaskSessionState(ctx, session.ID, models.TaskSessionStateStarting, ""); err != nil {
t.Fatalf("advance successor session: %v", err)
}
written, err = repo.SetTaskMetadataKeyIfRecoveryCurrent(
ctx, casTaskID, session.ID, session.UpdatedAt, settlement.Token,
"later_marker", "stale-marker",
)
if err != nil {
t.Fatalf("write stale recovery marker: %v", err)
}
if written {
t.Fatal("a successor session state must block a stale recovery marker")
}
}
func runManualMoveLifecycleMarkerContract(t *testing.T, repo *Repository) {
t.Helper()
ctx := context.Background()
@@ -210,6 +286,8 @@ func TestSetTaskMetadataKeyIfPresentSQLite(t *testing.T) {
})
runMetadataCASContract(t, repo)
runMetadataNoActiveSessionContract(t, repo)
runInterruptedMarkerCASContract(t, repo)
runRecoveryMarkerCASContract(t, repo)
runManualMoveLifecycleMarkerContract(t, repo)
}
@@ -227,6 +305,8 @@ func TestPostgresSetTaskMetadataKeyIfPresent(t *testing.T) {
})
runMetadataCASContract(t, repo)
runMetadataNoActiveSessionContract(t, repo)
runInterruptedMarkerCASContract(t, repo)
runRecoveryMarkerCASContract(t, repo)
runManualMoveLifecycleMarkerContract(t, repo)
}
@@ -17,10 +17,10 @@ const (
// an execution-less active session is classified as stalled
// (tasks.stallDetectionThreshold defaults to 2h).
defaultStallDetectionThreshold = 2 * time.Hour
// stallHealGraceMultiplier derives the orphaned-session healing grace
// window from the stall threshold. Healing waits twice as long as
// stallHealGraceMultiplier derives the interrupted-session recovery grace
// window from the stall threshold. Recovery waits twice as long as
// detection so an operator has a full threshold window to act on the
// task.stalled event before the sweep cancels the orphaned session, and
// task.stalled event before the sweep recovers the orphaned session, and
// so a live in-flight launch that has not yet registered its execution
// never races the grace window.
stallHealGraceMultiplier = 2
@@ -46,11 +46,10 @@ func (s *Service) stallThreshold() time.Duration {
//
// - emits one task.stalled event per stall episode once the session has
// been event-silent beyond the stall threshold (detection only), and
// - cancels the session via the same finalizeCancelledSessions transition
// the archived pass uses once the silence exceeds the grace window
// (twice the threshold), making the DB state truthful, delivering the
// session.state_changed event clients key off, and unblocking the task's
// step lifecycle.
// - returns the session to WAITING_FOR_INPUT once the silence exceeds the
// grace window (twice the threshold), preserving the same conversation
// for lazy recovery and delivering the session.state_changed event clients
// key off.
//
// The execution check is fail-closed: without the registry the pass cannot
// prove "no live execution", so it skips rather than healing or alerting on
@@ -117,6 +116,18 @@ func (s *Service) sweepTaskSessions(
return
}
s.retryPendingRecoverySettlements(ctx, task, activeSessions, liveSessions)
orphaned, filterErr := s.excludeIdleWaitingSessions(ctx, orphaned)
if filterErr != nil {
// A turn read failure leaves the session's intent unknown. Do not
// report or recover any part of this task from an incomplete snapshot.
return
}
if len(orphaned) == 0 {
s.clearStallNotifications(task.ID)
return
}
stalled, healable, lastEventBySession, classErr := s.classifyOrphanedSessions(
ctx, orphaned, now, threshold, grace,
)
@@ -132,6 +143,40 @@ func (s *Service) sweepTaskSessions(
s.healOrphanedSessions(ctx, task, activeSessions, orphaned, healable, lastEventBySession)
}
// excludeIdleWaitingSessions removes ordinary waiting conversations from the
// execution-loss candidate set. WAITING_FOR_INPUT with no active turn is a
// healthy idle state and must neither emit task.stalled nor participate in
// task-wide healing. A turn read error fails closed for the whole task.
func (s *Service) excludeIdleWaitingSessions(
ctx context.Context,
candidates []*models.TaskSession,
) ([]*models.TaskSession, error) {
filtered := make([]*models.TaskSession, 0, len(candidates))
for _, session := range candidates {
if session == nil || session.State != models.TaskSessionStateWaitingForInput {
filtered = append(filtered, session)
continue
}
if _, pending := models.LoadInterruptedRecoverySettlement(session.Metadata); pending {
// A prior recovery already owns this waiting row. The retry pass
// above handles its effects; it must not be reclassified as a new
// stall episode while the settlement marker remains.
continue
}
turn, err := s.GetActiveTurn(ctx, session.ID)
if err != nil {
s.logger.Warn("active-session sweep: failed to inspect waiting session turn; skipping task",
zap.String("session_id", session.ID),
zap.Error(err))
return nil, err
}
if turn != nil {
filtered = append(filtered, session)
}
}
return filtered, nil
}
// classifyOrphanedSessions splits execution-less sessions into stalled
// (event-silent beyond threshold) and healable (silent beyond the grace
// window), and reports each stalled session's resolved event time for the
@@ -170,7 +215,9 @@ func (s *Service) classifyOrphanedSessions(
func sessionIDs(sessions []*models.TaskSession) []string {
ids := make([]string, 0, len(sessions))
for _, session := range sessions {
ids = append(ids, session.ID)
if session != nil && session.ID != "" {
ids = append(ids, session.ID)
}
}
return ids
}
@@ -271,20 +318,18 @@ func (s *Service) notifyStalledSessions(
}
}
// healOrphanedSessions cancels execution-less sessions that have been silent
// beyond the grace window, reusing the archived pass's
// finalizeCancelledSessions transition (cancellation with the orphaned
// reason, clarification expiry, parked-projection cleanup, ceiling release,
// and the session.state_changed event) so clients keying off that event see
// the session stop. Repeat passes are no-ops: the underlying UPDATE only
// matches rows still in an active state.
// healOrphanedSessions returns execution-less sessions that have been silent
// beyond the grace window to WAITING_FOR_INPUT. It preserves the conversation
// and settles only the turn observed by the compare-and-set snapshot. No
// cancellation effects run: questions, queued work, and workflow ownership
// remain available for lazy recovery.
//
// Healing is task-scoped at the eligibility boundary, so it only runs when
// every active session of the task is execution-less and past the grace
// window. The write itself is candidate-scoped, so a same-session resume or a
// newer turn cannot be cancelled by a stale sweep snapshot.
// newer turn cannot be changed by a stale sweep snapshot.
//
// The guard is re-evaluated at the cancellation boundary itself: the
// The guard is re-evaluated at the recovery boundary itself: the
// liveness snapshot was taken before the silence reads, and an execution
// that registered (or a session that started) in between must abort the
// heal. Re-checking LiveSessionIDsForTask immediately before the write is
@@ -308,7 +353,13 @@ func (s *Service) healOrphanedSessions(
zap.Int("live_sessions", len(live)))
return
}
candidates := make([]models.ActiveSessionCancellationCandidate, 0, len(healable))
repo, ok := s.sessions.(interface {
RecoverTaskSessionByCandidate(context.Context, models.ActiveSessionRecoveryCandidate, time.Time) (*models.TaskSession, error)
})
if !ok {
return
}
candidates := make([]models.ActiveSessionRecoveryCandidate, 0, len(healable))
for _, session := range healable {
turn, err := s.GetActiveTurn(ctx, session.ID)
if err != nil {
@@ -322,32 +373,98 @@ func (s *Service) healOrphanedSessions(
if lastEvent.IsZero() {
lastEvent = session.UpdatedAt
}
candidate := models.ActiveSessionCancellationCandidate{
candidate := models.ActiveSessionRecoveryCandidate{
TaskID: task.ID,
SessionID: session.ID,
ExpectedState: session.State,
ExpectedUpdatedAt: session.UpdatedAt,
ExpectedLastEventAt: lastEvent,
}
if turn != nil {
candidate.ExpectedTurnID = turn.ID
}
if err := s.captureRecoveryExecutorSnapshot(ctx, &candidate); err != nil {
s.logger.Warn("active-session sweep: failed to capture executor reservation; skipping heal",
zap.String("task_id", task.ID),
zap.String("session_id", session.ID),
zap.Error(err))
return
}
candidates = append(candidates, candidate)
}
s.logger.Info("active-session sweep: healing orphaned sessions",
s.logger.Info("active-session sweep: preserving interrupted sessions for lazy recovery",
zap.String("task_id", task.ID),
zap.Strings("session_ids", cancellationCandidateIDs(candidates)))
zap.Strings("session_ids", recoveryCandidateIDs(candidates)))
deadline := archivecascade.ArchiveDeadline(ctx)
healCtx, cancel := archivecascade.ContinuationContextUntil(ctx, deadline)
defer cancel()
// Candidates scope the cancellation to exactly the sessions this pass
// classified as orphaned and compare their activity/turn identity in the
// UPDATE. A same-session resume or successor turn therefore survives the
// write and is retried by the next sweep.
s.finalizeCancelledSessionCandidates(
healCtx, task.ID, activeSessions, candidates, deadline, models.SessionOrphanedCancelReason,
)
for _, candidate := range candidates {
if !time.Now().Before(deadline) {
return
}
if _, live := s.liveSessionSet(task.ID)[candidate.SessionID]; live {
continue
}
recovered, err := repo.RecoverTaskSessionByCandidate(healCtx, candidate, time.Time{})
if err != nil {
s.logger.Warn("active-session sweep: failed to recover interrupted session",
zap.String("task_id", task.ID),
zap.String("session_id", candidate.SessionID),
zap.Error(err))
continue
}
if recovered == nil {
continue
}
completionCtx, cancelCompletion := context.WithTimeout(
context.WithoutCancel(ctx), taskPublicationTimeout,
)
s.settleRecoveredSession(completionCtx, candidate, recovered)
cancelCompletion()
}
}
func cancellationCandidateIDs(candidates []models.ActiveSessionCancellationCandidate) []string {
// retryPendingRecoverySettlements retries cross-repository effects that did
// not finish during an earlier recovery write. The session metadata carries
// the original turn and executor snapshot, so this pass never reads a
// successor identity merely to repair it.
func (s *Service) retryPendingRecoverySettlements(
ctx context.Context,
task *models.Task,
activeSessions []*models.TaskSession,
liveSessions map[string]struct{},
) {
if task == nil {
return
}
for _, session := range activeSessions {
if session == nil || session.ID == "" {
continue
}
if _, live := liveSessions[session.ID]; live {
continue
}
settlement, ok := models.LoadInterruptedRecoverySettlement(session.Metadata)
if !ok {
continue
}
candidate := models.ActiveSessionRecoveryCandidate{
TaskID: task.ID,
SessionID: session.ID,
ExpectedState: settlement.ExpectedState,
RecoveredUpdatedAt: settlement.RecoveredUpdatedAt,
ExpectedTurnID: settlement.ExpectedTurnID,
ExpectedExecutorID: settlement.ExpectedExecutorID,
ExpectedExecutorAgentExecutionID: settlement.ExpectedExecutorAgentExecutionID,
ExpectedExecutorUpdatedAt: settlement.ExpectedExecutorUpdatedAt,
}
completionCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), taskPublicationTimeout)
s.settleRecoveredSession(completionCtx, candidate, session)
cancel()
}
}
func recoveryCandidateIDs(candidates []models.ActiveSessionRecoveryCandidate) []string {
ids := make([]string, 0, len(candidates))
for _, candidate := range candidates {
ids = append(ids, candidate.SessionID)
@@ -9,6 +9,7 @@ import (
"github.com/kandev/kandev/internal/events"
"github.com/kandev/kandev/internal/events/bus"
"github.com/kandev/kandev/internal/task/models"
"github.com/kandev/kandev/internal/task/repository"
sqliterepo "github.com/kandev/kandev/internal/task/repository/sqlite"
)
@@ -166,6 +167,120 @@ func TestService_ActiveSessionSweepSkipsRecentSessions(t *testing.T) {
}
}
// TestService_ActiveSessionSweepPreservesIdleWaitingSession proves that a
// waiting conversation with no unfinished turn is ordinary idle state. It must
// not be reported as stalled or changed by the execution-loss sweep, even
// after the full healing grace window.
func TestService_ActiveSessionSweepPreservesIdleWaitingSession(t *testing.T) {
fixture := newSweepFixture(t, 5*time.Hour)
ctx := context.Background()
if _, err := fixture.rawRepo.DB().ExecContext(ctx, `
UPDATE task_sessions
SET state = ?, updated_at = ?
WHERE id = ?
`, string(models.TaskSessionStateWaitingForInput), fixture.now.Add(-5*time.Hour), "session-1"); err != nil {
t.Fatalf("make session idle waiting: %v", err)
}
fixture.svc.runActiveSessionSweep(ctx, fixture.now)
session, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
if session.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("session state = %q, want WAITING_FOR_INPUT", session.State)
}
if stalled := findTaskStalledEvents(fixture.eventBus); len(stalled) != 0 {
t.Fatalf("task.stalled events = %d, want 0 for idle waiting session", len(stalled))
}
}
// TestService_ActiveSessionSweepRecoversInterruptedConversation proves that
// a stale execution-less turn returns to the same waiting conversation. The
// turn is settled without a successful completion event so workflow ownership
// and the transcript remain available for a later user message.
func TestService_ActiveSessionSweepRecoversInterruptedConversation(t *testing.T) {
fixture := newSweepFixture(t, 5*time.Hour)
ctx := context.Background()
turn, err := fixture.svc.StartTurn(ctx, "session-1")
if err != nil {
t.Fatalf("StartTurn: %v", err)
}
old := fixture.now.Add(-5 * time.Hour)
if _, err := fixture.rawRepo.DB().ExecContext(ctx,
`UPDATE task_session_turns SET started_at = ?, created_at = ?, updated_at = ? WHERE id = ?`,
old, old, old, turn.ID); err != nil {
t.Fatalf("age turn: %v", err)
}
if _, err := fixture.rawRepo.DB().ExecContext(ctx,
`UPDATE task_sessions SET updated_at = ? WHERE id = ?`, old, "session-1"); err != nil {
t.Fatalf("age session: %v", err)
}
fixture.svc.runActiveSessionSweep(ctx, fixture.now)
session, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
if session.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("session state = %q, want WAITING_FOR_INPUT", session.State)
}
storedTurn, err := fixture.svc.turns.GetTurn(ctx, turn.ID)
if err != nil {
t.Fatalf("GetTurn: %v", err)
}
if storedTurn.CompletedAt == nil || !storedTurn.CompletedAt.Equal(storedTurn.StartedAt) {
t.Fatalf("turn completion = %v, want zero-duration completion at %v", storedTurn.CompletedAt, storedTurn.StartedAt)
}
for _, event := range fixture.eventBus.GetPublishedEvents() {
if event.Type != events.TurnCompleted {
continue
}
if data, ok := event.Data.(map[string]interface{}); ok && data["id"] == turn.ID {
t.Fatal("system recovery published turn.completed and may advance workflow")
}
}
}
func TestService_ActiveSessionSweepRecoversWaitingSessionWithUnfinishedTurn(t *testing.T) {
fixture := newSweepFixture(t, 5*time.Hour)
ctx := context.Background()
turn, err := fixture.svc.StartTurn(ctx, "session-1")
if err != nil {
t.Fatalf("StartTurn: %v", err)
}
old := fixture.now.Add(-5 * time.Hour)
if _, err := fixture.rawRepo.DB().ExecContext(ctx, `
UPDATE task_sessions SET state = ?, updated_at = ? WHERE id = ?
`, string(models.TaskSessionStateWaitingForInput), old, "session-1"); err != nil {
t.Fatalf("age waiting session: %v", err)
}
if _, err := fixture.rawRepo.DB().ExecContext(ctx,
`UPDATE task_session_turns SET started_at = ?, created_at = ?, updated_at = ? WHERE id = ?`,
old, old, old, turn.ID); err != nil {
t.Fatalf("age turn: %v", err)
}
fixture.svc.runActiveSessionSweep(ctx, fixture.now)
session, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
if session.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("session state = %q, want WAITING_FOR_INPUT", session.State)
}
storedTurn, err := fixture.svc.turns.GetTurn(ctx, turn.ID)
if err != nil {
t.Fatalf("GetTurn: %v", err)
}
if storedTurn.CompletedAt == nil || !storedTurn.CompletedAt.Equal(storedTurn.StartedAt) {
t.Fatalf("turn completion = %v, want zero-duration completion at %v", storedTurn.CompletedAt, storedTurn.StartedAt)
}
}
// TestService_ActiveSessionSweepStallEventEmittedOncePerEpisode proves the
// episode dedupe: a sweep ticking every minute must not re-emit task.stalled
// for the same stalled session, but must report it again after the episode
@@ -191,11 +306,10 @@ func TestService_ActiveSessionSweepStallEventEmittedOncePerEpisode(t *testing.T)
}
}
// TestService_ActiveSessionSweepHealsOrphanedSessions is the #3711 half of
// TestService_ActiveSessionSweepHealsOrphanedSessions is the recovery half of
// the sweep: once an execution-less session has been silent beyond the grace
// window (twice the threshold), the sweep cancels it through the same
// finalizeCancelledSessions transition the archived pass uses, so the DB
// state becomes truthful and the session.state_changed event fires.
// window (twice the threshold), the sweep returns it to WAITING_FOR_INPUT so
// the same conversation remains available for lazy recovery.
func TestService_ActiveSessionSweepHealsOrphanedSessions(t *testing.T) {
// 5h of silence: past the 4h grace window (2 x 2h threshold).
fixture := newSweepFixture(t, 5*time.Hour)
@@ -206,11 +320,11 @@ func TestService_ActiveSessionSweepHealsOrphanedSessions(t *testing.T) {
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
if session.State != models.TaskSessionStateCancelled {
t.Fatalf("session state after healing = %q, want CANCELLED", session.State)
if session.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("session state after healing = %q, want WAITING_FOR_INPUT", session.State)
}
if session.ErrorMessage != models.SessionOrphanedCancelReason {
t.Fatalf("session error_message = %q, want %q", session.ErrorMessage, models.SessionOrphanedCancelReason)
if session.ErrorMessage != "" {
t.Fatalf("session error_message = %q, want empty", session.ErrorMessage)
}
var stateChanged *bus.Event
@@ -226,8 +340,259 @@ func TestService_ActiveSessionSweepHealsOrphanedSessions(t *testing.T) {
t.Fatal("expected a session.state_changed event for session-1 from the healing pass, got none")
}
data := stateChanged.Data.(map[string]interface{})
if got := data["new_state"]; got != string(models.TaskSessionStateCancelled) {
t.Errorf("new_state = %v, want CANCELLED", got)
if got := data["new_state"]; got != string(models.TaskSessionStateWaitingForInput) {
t.Errorf("new_state = %v, want WAITING_FOR_INPUT", got)
}
}
// The active sweep must keep a durable recovery marker on the session when it
// has no executors_running row. The orchestrator uses that marker to offer the
// same conversation for lazy focus recovery after a restart.
func TestService_ActiveSessionSweepMarksMissingExecutorRecovery(t *testing.T) {
fixture := newSweepFixture(t, 5*time.Hour)
ctx := context.Background()
fixture.svc.runActiveSessionSweep(ctx, fixture.now)
session, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
if !models.HasInterruptedRecoveryPending(session.Metadata) {
t.Fatalf("session metadata = %#v, want an interrupted recovery marker", session.Metadata)
}
task, err := fixture.rawRepo.GetTask(ctx, "task-1")
if err != nil {
t.Fatalf("GetTask: %v", err)
}
if _, ok := task.Metadata[models.MetaKeyInterruptedAt]; !ok {
t.Fatalf("task metadata = %#v, want interrupted marker", task.Metadata)
}
var taskUpdated bool
for _, event := range fixture.eventBus.GetPublishedEvents() {
if event.Type == events.TaskUpdated {
if data, ok := event.Data.(map[string]interface{}); ok && data["task_id"] == "task-1" {
taskUpdated = true
if interrupted, ok := data["interrupted"].(bool); !ok || !interrupted {
t.Fatalf("task.updated interrupted = %#v, want true", data["interrupted"])
}
}
}
}
if !taskUpdated {
t.Fatal("expected task.updated after setting the interrupted marker")
}
}
func TestService_ActiveSessionSweepUsesFreshEffectsContextAfterDelayedCommit(t *testing.T) {
delayed := &delayedOrphanSessionRepository{
entered: make(chan struct{}),
release: make(chan struct{}),
}
svc, eventBus, repo := createTestServiceWithSessionsRepo(t, func(base *sqliterepo.Repository) repository.SessionRepository {
delayed.Repository = base
return delayed
})
svc.SetSessionExecutionRegistry(&stubExecutionRegistry{})
svc.SetStallDetectionThreshold(2 * time.Hour)
ctx := context.Background()
seedOrphanSweepFixtures(t, repo)
seedOrphanSweepTask(t, repo, "task-active-delay", false)
stale := time.Now().UTC().Add(-5 * time.Hour)
if err := repo.CreateTaskSession(ctx, &models.TaskSession{
ID: "session-active-delay", TaskID: "task-active-delay", State: models.TaskSessionStateRunning,
AgentProfileID: "agent-1", IsPrimary: true, UpdatedAt: stale, StartedAt: stale,
}); err != nil {
t.Fatalf("CreateTaskSession: %v", err)
}
passCtx, cancel := context.WithTimeout(ctx, 100*time.Millisecond)
defer cancel()
done := make(chan struct{})
go func() {
svc.runActiveSessionSweep(passCtx, time.Now().UTC())
close(done)
}()
select {
case <-delayed.entered:
case <-time.After(time.Second):
t.Fatal("active sweep did not start the delayed recovery")
}
select {
case <-passCtx.Done():
case <-time.After(time.Second):
t.Fatal("active sweep did not reach its deadline")
}
close(delayed.release)
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("active sweep did not finish after delayed commit")
}
if !sessionRecoveredEventPublished(eventBus, "session-active-delay") {
t.Fatal("expected recovery event after delayed active-sweep commit")
}
}
type failOnceToolSettlementTurns struct {
repository.TurnRepository
fail bool
}
func (r *failOnceToolSettlementTurns) CompletePendingToolCallsForTurn(ctx context.Context, turnID string) (int64, error) {
if r.fail {
r.fail = false
return 0, errors.New("tool settlement unavailable")
}
return r.TurnRepository.CompletePendingToolCallsForTurn(ctx, turnID)
}
func TestService_ActiveSessionSweepRetriesPartialSettlement(t *testing.T) {
fixture := newSweepFixture(t, 5*time.Hour)
ctx := context.Background()
turn, err := fixture.svc.StartTurn(ctx, "session-1")
if err != nil {
t.Fatalf("StartTurn: %v", err)
}
old := fixture.now.Add(-5 * time.Hour)
if err := fixture.svc.messages.CreateMessage(ctx, &models.Message{
ID: "retry-pending-tool-message",
TaskSessionID: "session-1",
TaskID: "task-1",
TurnID: turn.ID,
AuthorType: models.MessageAuthorAgent,
Type: models.MessageTypeToolExecute,
Content: "tool output",
Metadata: map[string]interface{}{"tool_call_id": "retry-call-1", "status": "pending"},
CreatedAt: old,
UpdatedAt: old,
}); err != nil {
t.Fatalf("CreateMessage: %v", err)
}
if _, err := fixture.rawRepo.DB().ExecContext(ctx,
`UPDATE task_session_turns SET started_at = ?, created_at = ?, updated_at = ? WHERE id = ?`,
old, old, old, turn.ID); err != nil {
t.Fatalf("age turn: %v", err)
}
if _, err := fixture.rawRepo.DB().ExecContext(ctx,
`UPDATE task_sessions SET updated_at = ? WHERE id = ?`, old, "session-1"); err != nil {
t.Fatalf("age session: %v", err)
}
if err := fixture.rawRepo.UpsertExecutorRunning(ctx, &models.ExecutorRunning{
ID: "stale-recovery-row", SessionID: "session-1", TaskID: "task-1",
Status: models.ExecutorRunningStatusRunning, AgentExecutionID: "stale-recovery-execution",
LocalPID: 42, CreatedAt: old, UpdatedAt: old,
}); err != nil {
t.Fatalf("seed stale executor row: %v", err)
}
fixture.svc.turns = &failOnceToolSettlementTurns{TurnRepository: fixture.svc.turns, fail: true}
fixture.svc.runActiveSessionSweep(ctx, fixture.now)
failed, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession after partial settlement: %v", err)
}
if _, pending := models.LoadInterruptedRecoverySettlement(failed.Metadata); !pending {
t.Fatalf("session metadata after partial settlement = %#v, want durable retry marker", failed.Metadata)
}
stopped, err := fixture.rawRepo.GetExecutorRunningBySessionID(ctx, "session-1")
if err != nil {
t.Fatalf("GetExecutorRunningBySessionID after partial settlement: %v", err)
}
if stopped.Status != models.ExecutorRunningStatusStopped || stopped.LocalPID != 0 {
t.Fatalf("executor after partial settlement = %+v, want repaired stopped row", stopped)
}
fixture.svc.runActiveSessionSweep(ctx, fixture.now.Add(time.Minute))
retried, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession after retry: %v", err)
}
if _, pending := models.LoadInterruptedRecoverySettlement(retried.Metadata); pending {
t.Fatalf("session metadata after retry = %#v, want settlement marker cleared", retried.Metadata)
}
stopped, err = fixture.rawRepo.GetExecutorRunningBySessionID(ctx, "session-1")
if err != nil {
t.Fatalf("GetExecutorRunningBySessionID after retry: %v", err)
}
if stopped.Status != models.ExecutorRunningStatusStopped || stopped.LocalPID != 0 {
t.Fatalf("executor after retry = %+v, want repaired stopped row", stopped)
}
message, err := fixture.svc.messages.GetMessage(ctx, "retry-pending-tool-message")
if err != nil {
t.Fatalf("GetMessage after retry: %v", err)
}
if got := message.Metadata["status"]; got != "complete" {
t.Fatalf("pending tool status after retry = %v, want complete", got)
}
}
func TestService_ActiveSessionSweepSkipsRecoveryWhenExecutorSnapshotFails(t *testing.T) {
fixture := newSweepFixture(t, 5*time.Hour)
fixture.svc.executors = failGetExecutorRunningRepository{
ExecutorRepository: fixture.rawRepo,
err: errors.New("executor inventory unavailable"),
}
fixture.svc.runActiveSessionSweep(context.Background(), fixture.now)
session, err := fixture.repo.GetTaskSession(context.Background(), "session-1")
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
if session.State != models.TaskSessionStateRunning {
t.Fatalf("session state = %q, want RUNNING when executor snapshot fails", session.State)
}
if models.HasInterruptedRecoveryPending(session.Metadata) {
t.Fatalf("session metadata = %#v, want no recovery marker", session.Metadata)
}
}
func TestService_ActiveSessionSweepRejectsRetryAfterSessionGenerationChanges(t *testing.T) {
fixture := newSweepFixture(t, 5*time.Hour)
ctx := context.Background()
turn, err := fixture.svc.StartTurn(ctx, "session-1")
if err != nil {
t.Fatalf("StartTurn: %v", err)
}
old := fixture.now.Add(-5 * time.Hour)
if _, err := fixture.rawRepo.DB().ExecContext(ctx,
`UPDATE task_session_turns SET started_at = ?, created_at = ?, updated_at = ? WHERE id = ?`,
old, old, old, turn.ID); err != nil {
t.Fatalf("age turn: %v", err)
}
if _, err := fixture.rawRepo.DB().ExecContext(ctx,
`UPDATE task_sessions SET updated_at = ? WHERE id = ?`, old, "session-1"); err != nil {
t.Fatalf("age session: %v", err)
}
if err := fixture.rawRepo.UpsertExecutorRunning(ctx, &models.ExecutorRunning{
ID: "stale-retry-generation-row", SessionID: "session-1", TaskID: "task-1",
Status: models.ExecutorRunningStatusRunning, AgentExecutionID: "stale-retry-generation-execution",
LocalPID: 42, CreatedAt: old, UpdatedAt: old,
}); err != nil {
t.Fatalf("seed stale executor row: %v", err)
}
fixture.svc.turns = &failOnceToolSettlementTurns{TurnRepository: fixture.svc.turns, fail: true}
fixture.svc.runActiveSessionSweep(ctx, fixture.now)
first, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession after first sweep: %v", err)
}
if _, pending := models.LoadInterruptedRecoverySettlement(first.Metadata); !pending {
t.Fatalf("session metadata after first sweep = %#v, want durable retry marker", first.Metadata)
}
if _, err := fixture.rawRepo.DB().ExecContext(ctx,
`UPDATE task_sessions SET updated_at = ? WHERE id = ?`, first.UpdatedAt.Add(time.Second), "session-1"); err != nil {
t.Fatalf("advance session generation: %v", err)
}
fixture.svc.runActiveSessionSweep(ctx, fixture.now.Add(time.Minute))
second, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession after stale retry: %v", err)
}
if _, pending := models.LoadInterruptedRecoverySettlement(second.Metadata); !pending {
t.Fatalf("session metadata after stale retry = %#v, want settlement marker preserved", second.Metadata)
}
}
@@ -264,6 +629,42 @@ func TestService_ActiveSessionSweepDoesNotHealAroundLiveSibling(t *testing.T) {
}
}
func TestService_ActiveSessionSweepIdleWaitingSiblingBlocksRecovery(t *testing.T) {
fixture := newSweepFixture(t, 5*time.Hour)
ctx := context.Background()
if err := fixture.repo.CreateTaskSession(ctx, &models.TaskSession{
ID: "session-idle-sibling", TaskID: "task-1", State: models.TaskSessionStateWaitingForInput,
AgentProfileID: "agent-1", UpdatedAt: fixture.now.Add(-5 * time.Hour),
}); err != nil {
t.Fatalf("CreateTaskSession: %v", err)
}
fixture.svc.runActiveSessionSweep(ctx, fixture.now)
running, err := fixture.repo.GetTaskSession(ctx, "session-1")
if err != nil {
t.Fatalf("GetTaskSession(session-1): %v", err)
}
if running.State != models.TaskSessionStateRunning {
t.Fatalf("running sibling state = %q, want RUNNING while idle sibling blocks recovery", running.State)
}
idle, err := fixture.repo.GetTaskSession(ctx, "session-idle-sibling")
if err != nil {
t.Fatalf("GetTaskSession(session-idle-sibling): %v", err)
}
if idle.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("idle sibling state = %q, want WAITING_FOR_INPUT", idle.State)
}
stalled := findTaskStalledEvents(fixture.eventBus)
if len(stalled) != 1 {
t.Fatalf("task.stalled events = %d, want 1", len(stalled))
}
ids, _ := stalled[0].Data.(map[string]interface{})["session_ids"].([]string)
if len(ids) != 1 || ids[0] != "session-1" {
t.Fatalf("stalled session_ids = %v, want [session-1]", ids)
}
}
// TestService_ActiveSessionSweepSkipsArchivedTasks proves the pass boundary:
// archived tasks with active sessions belong to the archived pass, not the
// active-task pass, and must not produce task.stalled events.
@@ -331,8 +732,8 @@ func TestService_ActiveSessionSweepHealAbortedWhenExecutionRegistersMidSweep(t *
}
// TestService_ActiveSessionSweepHealCancelsOnlyClassifiedSessions proves the
// ID-scoped cancellation: when the task genuinely qualifies for healing, the
// cancellation only touches the sessions the sweep classified. A session
// ID-scoped recovery: when the task genuinely qualifies for healing, the
// transition only touches the sessions the sweep classified. A session
// that became active after classification — outside the classified set —
// keeps running.
func TestService_ActiveSessionSweepHealCancelsOnlyClassifiedSessions(t *testing.T) {
@@ -347,8 +748,8 @@ func TestService_ActiveSessionSweepHealCancelsOnlyClassifiedSessions(t *testing.
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
if session.State != models.TaskSessionStateCancelled {
t.Fatalf("session-1 state = %q, want CANCELLED", session.State)
if session.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("session-1 state = %q, want WAITING_FOR_INPUT", session.State)
}
}
@@ -2,6 +2,8 @@ package service
import (
"context"
"errors"
"fmt"
"time"
"github.com/kandev/kandev/internal/common/constants"
@@ -37,7 +39,8 @@ import (
// session (the turn-complete path dies with the process), leaving
// task_sessions stuck in STARTING/RUNNING forever on tasks that were never
// archived. Every tick, stale STARTING/RUNNING sessions with no live
// in-memory execution backing them are terminalized.
// in-memory execution backing them return to WAITING_FOR_INPUT for lazy
// recovery.
//
// Both passes follow the exact periodic-sweep shape StartAutoArchiveLoop
// already uses and re-invoke the same finalize transitions; as long as the
@@ -123,19 +126,16 @@ func orphanedSessionGracePeriod() time.Duration {
// repository is the only production implementer.
type orphanedSessionRepository interface {
ListStaleRunningSessionsOnUnarchivedTasks(ctx context.Context, staleBefore time.Time) ([]*models.TaskSession, error)
CancelRunningTaskSessionByID(ctx context.Context, sessionID, reason string, staleBefore time.Time) (*models.TaskSession, error)
RecoverTaskSessionByCandidate(ctx context.Context, candidate models.ActiveSessionRecoveryCandidate, staleBefore time.Time) (*models.TaskSession, error)
}
// runOrphanedSessionReconciliation is pass 2 of the reconciliation loop: it
// terminalizes stale STARTING/RUNNING sessions of unarchived tasks that no
// live in-memory execution backs. Without it, a backend restart mid-turn
// leaves those sessions RUNNING forever (#3711): the actor that would have
// transitioned them died with the process, and no other path ever retries.
// Sessions backed by a live execution — including survivors re-tracked by
// startup recovery — are never touched, and the per-session cancel keeps
// healthy sibling sessions of the same task (e.g. WAITING_FOR_INPUT) intact.
// A session whose cancellation write fails is simply retried on the next
// tick; the sweep never gives up for as long as the process runs.
// returns stale STARTING/RUNNING sessions of unarchived tasks to
// WAITING_FOR_INPUT when no live in-memory execution backs them. An
// interrupted turn is settled without a completion event, so the same
// conversation remains available for lazy recovery. Sessions backed by a
// live execution — including survivors re-tracked by startup recovery — are
// never touched. A failed recovery write is retried on the next tick.
func (s *Service) runOrphanedSessionReconciliation(ctx context.Context) {
if s.executionLivenessChecker == nil {
// Absence-from-store is the sweep's only dead signal; without the
@@ -166,10 +166,9 @@ func (s *Service) reconcileOrphanedSessions(ctx context.Context, candidates []*m
}
// reconcileOrphanedSessionsUntil applies one orphan sweep within the supplied
// admission deadline. A cancellation write may finish after that deadline
// because the repository owns a detached write context; its post-commit
// effects therefore receive a fresh bounded context instead of the expired
// sweep context.
// admission deadline. A recovery write may finish after that deadline because
// the repository owns a detached write context; the state event and turn
// settlement therefore use the bounded context supplied to the write.
func (s *Service) reconcileOrphanedSessionsUntil(
ctx context.Context,
candidates []*models.TaskSession,
@@ -180,6 +179,12 @@ func (s *Service) reconcileOrphanedSessionsUntil(
if !ok {
return
}
lastMessageBySession, err := s.messages.GetLastMessageTimeBySessionIDs(ctx, sessionIDs(candidates))
if err != nil {
s.logger.Warn("orphaned-session reconciliation: failed to load last message times; skipping pass",
zap.Error(err))
return
}
for _, session := range candidates {
if session == nil || session.ID == "" {
continue
@@ -191,36 +196,381 @@ func (s *Service) reconcileOrphanedSessionsUntil(
}
// Re-check liveness at the moment of the write, not just at the
// candidate read: a launch that raced the grace window since the read
// must not be reaped. The staleBefore predicate inside the cancel
// UPDATE is the second half of that guard — it re-asserts the
// staleness cutoff so a row refreshed by an in-flight launch between
// this check and the write no longer matches.
// must not be recovered. The staleBefore predicate inside the recovery
// UPDATE is the second half of that guard — it re-asserts the staleness
// cutoff so a row refreshed by an in-flight launch between this check
// and the write no longer matches.
if s.executionLivenessChecker.HasLiveExecution(session.ID) {
continue
}
cancelled, err := repo.CancelRunningTaskSessionByID(ctx, session.ID, models.SessionOrphanedCancelReason, staleBefore)
turn, err := s.GetActiveTurn(ctx, session.ID)
if err != nil {
s.logger.Warn("orphaned-session reconciliation: failed to cancel session",
s.logger.Warn("orphaned-session reconciliation: failed to inspect current turn; skipping session",
zap.String("session_id", session.ID),
zap.Error(err))
continue
}
if cancelled == nil {
// Raced to another state between the candidate read and the
// write; its new owner is responsible for it now.
candidate := models.ActiveSessionRecoveryCandidate{
TaskID: session.TaskID,
SessionID: session.ID,
ExpectedState: session.State,
ExpectedUpdatedAt: session.UpdatedAt,
ExpectedLastEventAt: lastSessionEventAt(session, lastMessageBySession),
}
if turn != nil {
candidate.ExpectedTurnID = turn.ID
}
if err := s.captureRecoveryExecutorSnapshot(ctx, &candidate); err != nil {
s.logger.Warn("orphaned-session reconciliation: failed to capture executor reservation; skipping session",
zap.String("session_id", session.ID),
zap.Error(err))
continue
}
s.logger.Info("orphaned-session reconciliation: terminalized unbacked session",
// The turn read above can take long enough for a replacement
// execution to register. Re-check immediately before the guarded write
// so a live successor keeps ownership of the session.
if s.executionLivenessChecker.HasLiveExecution(session.ID) {
continue
}
recovered, err := repo.RecoverTaskSessionByCandidate(ctx, candidate, staleBefore)
if err != nil {
s.logger.Warn("orphaned-session reconciliation: failed to recover session",
zap.String("session_id", session.ID),
zap.Error(err))
continue
}
if recovered == nil {
// Raced to another state between the candidate read and the
// write; its new owner is responsible for it now. The next
// sweep will classify any still-interrupted state again.
continue
}
completionCtx, cancelCompletion := context.WithTimeout(
context.WithoutCancel(ctx), taskPublicationTimeout,
)
s.settleRecoveredSession(completionCtx, candidate, recovered)
cancelCompletion()
s.logger.Info("orphaned-session reconciliation: preserved unbacked session for lazy recovery",
zap.String("task_id", session.TaskID),
zap.String("session_id", session.ID),
zap.String("previous_state", string(session.State)))
completionDeadline := time.Now().Add(taskPublicationTimeout)
completionCtx, cancelCompletion := context.WithDeadline(
context.WithoutCancel(ctx), completionDeadline,
)
s.runCancelledSessionEffects(completionCtx, session.TaskID,
[]*models.TaskSession{session}, []*models.TaskSession{cancelled},
completionDeadline, models.SessionOrphanedCancelReason)
cancelCompletion()
}
}
// captureRecoveryExecutorSnapshot records the executor identity observed with
// the session candidate. Post-commit repair must use this immutable snapshot;
// reading the row again after the session write can otherwise stop a successor
// that registered during the effects window.
func (s *Service) captureRecoveryExecutorSnapshot(ctx context.Context, candidate *models.ActiveSessionRecoveryCandidate) error {
if s == nil || s.executors == nil || candidate == nil || candidate.SessionID == "" {
return nil
}
running, err := s.executors.GetExecutorRunningBySessionID(ctx, candidate.SessionID)
if errors.Is(err, models.ErrExecutorRunningNotFound) {
return nil
}
if err != nil {
return fmt.Errorf("failed to capture executor reservation for session recovery: %w", err)
}
if running == nil {
return nil
}
candidate.ExpectedExecutorID = running.ID
candidate.ExpectedExecutorAgentExecutionID = running.AgentExecutionID
candidate.ExpectedExecutorUpdatedAt = running.UpdatedAt
return nil
}
// settleRecoveredSession applies the side effects shared by both execution
// loss sweeps. An interrupted turn is closed with zero duration and pending
// tool calls are settled, but no turn.completed event or workflow transition
// is emitted. The recovery state event is published only after the conditional
// session write has committed.
//
//nolint:cyclop // The ordered effects boundary is intentionally explicit.
func (s *Service) settleRecoveredSession(
ctx context.Context,
candidate models.ActiveSessionRecoveryCandidate,
recovered *models.TaskSession,
) {
if recovered == nil {
return
}
// A retry may observe the stale executor row already repaired by an earlier
// attempt. That stopped row is still the same recovery generation and may
// continue through settlement; a newly running or rotated execution never
// satisfies the repaired-row allowance.
if !s.recoveredSessionStillOwned(ctx, candidate, recovered, true) {
return
}
settlementComplete := true
if err := s.repairRecoveredExecutor(ctx, candidate); err != nil {
settlementComplete = false
}
if candidate.ExpectedTurnID != "" {
turnSettled, successor := s.settleRecoveredTurn(ctx, candidate)
if successor {
return
}
settlementComplete = settlementComplete && turnSettled
}
if !s.recoveredSessionStillOwned(ctx, candidate, recovered, true) {
return
}
if candidate.ExpectedTurnID != "" ||
candidate.ExpectedState == models.TaskSessionStateCreated ||
candidate.ExpectedState == models.TaskSessionStateStarting ||
candidate.ExpectedState == models.TaskSessionStateRunning {
if _, err := s.markTaskInterrupted(ctx, recovered); err != nil {
settlementComplete = false
}
}
if !s.recoveredSessionStillOwned(ctx, candidate, recovered, true) {
return
}
// Publish before clearing the durable settlement. If the clear fails, the
// next sweep republishes this idempotent state event so the recovery cannot
// be lost between the database write and publication.
if err := s.publishSessionRecovered(ctx, candidate.TaskID, candidate.ExpectedState, recovered); err != nil {
settlementComplete = false
}
if settlementComplete {
if err := s.clearRecoverySettlement(ctx, recovered); err != nil {
s.logger.Warn("failed to clear recovered-session settlement marker",
zap.String("session_id", candidate.SessionID), zap.Error(err))
}
}
}
func (s *Service) settleRecoveredTurn(
ctx context.Context,
candidate models.ActiveSessionRecoveryCandidate,
) (settled, successor bool) {
activeTurn, err := s.GetActiveTurn(ctx, candidate.SessionID)
if err != nil {
s.logger.Warn("failed to inspect interrupted turn after recovery",
zap.String("session_id", candidate.SessionID),
zap.String("turn_id", candidate.ExpectedTurnID),
zap.Error(err))
return false, false
}
if activeTurn == nil {
return s.completeRecoveredTurnToolCalls(ctx, candidate), false
}
if activeTurn.ID != candidate.ExpectedTurnID {
// A different active turn proves a successor crossed the effects
// boundary. Leave its turn, marker, and event state alone.
return false, true
}
if err := s.turns.AbandonTurn(ctx, candidate.ExpectedTurnID); err != nil {
s.logger.Warn("failed to abandon interrupted turn after recovery",
zap.String("session_id", candidate.SessionID),
zap.String("turn_id", candidate.ExpectedTurnID),
zap.Error(err))
return false, false
}
return s.completeRecoveredTurnToolCalls(ctx, candidate), false
}
func (s *Service) completeRecoveredTurnToolCalls(
ctx context.Context,
candidate models.ActiveSessionRecoveryCandidate,
) bool {
affected, err := s.turns.CompletePendingToolCallsForTurn(ctx, candidate.ExpectedTurnID)
if err != nil {
s.logger.Warn("failed to complete pending tool calls for recovered turn",
zap.String("session_id", candidate.SessionID),
zap.String("turn_id", candidate.ExpectedTurnID),
zap.Error(err))
return false
}
if affected > 0 {
s.logger.Info("completed pending tool calls for recovered turn",
zap.String("session_id", candidate.SessionID),
zap.String("turn_id", candidate.ExpectedTurnID),
zap.Int64("affected", affected))
}
return true
}
// repairRecoveredExecutor releases the stale runtime reservation while
// preserving resume_token, worktree, and other recovery identity. The
// compare-and-set variant is preferred when the executor repository exposes
// it, so a successor launch cannot be marked stopped by a delayed sweep.
func (s *Service) repairRecoveredExecutor(ctx context.Context, candidate models.ActiveSessionRecoveryCandidate) error {
if s.executors == nil || candidate.SessionID == "" ||
(candidate.ExpectedExecutorID == "" && candidate.ExpectedExecutorAgentExecutionID == "" && candidate.ExpectedExecutorUpdatedAt.IsZero()) {
return nil
}
if cas, ok := s.executors.(interface {
RepairExecutorRunningDeadIfCurrent(context.Context, string, string, time.Time) error
}); ok {
err := cas.RepairExecutorRunningDeadIfCurrent(
ctx, candidate.SessionID, candidate.ExpectedExecutorAgentExecutionID, candidate.ExpectedExecutorUpdatedAt,
)
if errors.Is(err, models.ErrExecutorRunningNotFound) || errors.Is(err, models.ErrExecutionRotated) {
return nil
}
return err
} else {
// Legacy adapters have no compare-and-set primitive. Refuse to claim
// settlement completed: a post-commit read/repair pair could stop a
// successor, so the marker and event must remain retryable instead.
return errors.New("executor repository does not support guarded recovery repair")
}
}
// recoveredSessionStillOwned verifies that the recovery write still owns its
// effects boundary. It compares the session generation and the executor
// identity captured before the write; a newly registered successor therefore
// prevents stale marker, turn, and event effects from being applied.
func (s *Service) recoveredSessionStillOwned(
ctx context.Context,
candidate models.ActiveSessionRecoveryCandidate,
recovered *models.TaskSession,
allowRepairedExecutor bool,
) bool {
if recovered == nil || s.sessions == nil {
return false
}
expectedUpdatedAt := recovered.UpdatedAt
if !candidate.RecoveredUpdatedAt.IsZero() {
expectedUpdatedAt = candidate.RecoveredUpdatedAt
}
current, err := s.sessions.GetTaskSession(ctx, candidate.SessionID)
if err != nil || current == nil || current.TaskID != candidate.TaskID ||
current.State != models.TaskSessionStateWaitingForInput ||
!current.UpdatedAt.Equal(expectedUpdatedAt) {
return false
}
return s.recoveryExecutorStillOwned(ctx, candidate, allowRepairedExecutor)
}
//nolint:cyclop // Identity, timestamp, and repaired-row guards are one CAS contract.
func (s *Service) recoveryExecutorStillOwned(
ctx context.Context,
candidate models.ActiveSessionRecoveryCandidate,
allowRepairedExecutor bool,
) bool {
if s.executors == nil {
return true
}
running, err := s.executors.GetExecutorRunningBySessionID(ctx, candidate.SessionID)
if errors.Is(err, models.ErrExecutorRunningNotFound) {
return candidate.ExpectedExecutorID == "" && candidate.ExpectedExecutorAgentExecutionID == "" && candidate.ExpectedExecutorUpdatedAt.IsZero()
}
if err != nil || running == nil {
return false
}
if candidate.ExpectedExecutorID == "" && candidate.ExpectedExecutorAgentExecutionID == "" && candidate.ExpectedExecutorUpdatedAt.IsZero() {
return false
}
if candidate.ExpectedExecutorID != "" && running.ID != candidate.ExpectedExecutorID {
return false
}
if candidate.ExpectedExecutorAgentExecutionID != "" && running.AgentExecutionID != candidate.ExpectedExecutorAgentExecutionID {
return false
}
if !candidate.ExpectedExecutorUpdatedAt.IsZero() && !running.UpdatedAt.Equal(candidate.ExpectedExecutorUpdatedAt) {
return allowRepairedExecutor && running.Status == models.ExecutorRunningStatusStopped &&
running.AgentExecutionID == candidate.ExpectedExecutorAgentExecutionID
}
return true
}
func (s *Service) clearRecoverySettlement(ctx context.Context, session *models.TaskSession) error {
settlement, ok := models.LoadInterruptedRecoverySettlement(session.Metadata)
if !ok {
return nil
}
remover, ok := s.sessions.(interface {
RemoveSessionMetadataKeyIfJSONValue(context.Context, string, string, interface{}) (bool, error)
})
if !ok {
return errors.New("session repository does not support guarded recovery settlement removal")
}
_, err := remover.RemoveSessionMetadataKeyIfJSONValue(
ctx, session.ID, models.SessionMetaKeyRecoverySettlementPending, settlement,
)
return err
}
type interruptedTaskMetadataSetter interface {
SetTaskMetadataKeyIfNotArchived(ctx context.Context, taskID, key string, value interface{}) (bool, error)
}
type interruptedTaskMetadataAbsentSetter interface {
SetTaskMetadataKeyIfAbsentNotArchived(ctx context.Context, taskID, key string, value interface{}) (bool, error)
}
// interruptedTaskMetadataRecoverySetter adds the session recovery generation
// to the marker write. The task row is updated only while the recovered
// session still owns its WAITING_FOR_INPUT generation and its settlement token
// is present. This closes the gap between the in-memory ownership check and
// the task metadata write when a successor starts concurrently.
type interruptedTaskMetadataRecoverySetter interface {
SetTaskMetadataKeyIfRecoveryCurrent(
ctx context.Context,
taskID, sessionID string,
expectedSessionUpdatedAt time.Time,
expectedRecoveryToken, key string,
value interface{},
) (bool, error)
}
func (s *Service) markTaskInterrupted(ctx context.Context, recovered *models.TaskSession) (bool, error) {
if recovered == nil || recovered.TaskID == "" || recovered.ID == "" {
return false, nil
}
settlement, hasSettlement := models.LoadInterruptedRecoverySettlement(recovered.Metadata)
// The concrete SQLite task repository owns the archive-atomic metadata CAS.
// Keep this optional so focused task-service users and test repositories
// without the newer marker primitive remain compatible.
_, ok := s.tasks.(interruptedTaskMetadataSetter)
if !ok {
return false, errors.New("task repository does not support recovery interruption markers")
}
value := time.Now().UTC().Format(time.RFC3339Nano)
expectedUpdatedAt := recovered.UpdatedAt
if hasSettlement && !settlement.RecoveredUpdatedAt.IsZero() {
expectedUpdatedAt = settlement.RecoveredUpdatedAt
}
var changed bool
var err error
if recoverySetter, supported := s.tasks.(interruptedTaskMetadataRecoverySetter); supported && hasSettlement {
changed, err = recoverySetter.SetTaskMetadataKeyIfRecoveryCurrent(
ctx,
recovered.TaskID,
recovered.ID,
expectedUpdatedAt,
settlement.Token,
models.MetaKeyInterruptedAt,
value,
)
} else if absentSetter, supported := s.tasks.(interruptedTaskMetadataAbsentSetter); supported {
changed, err = absentSetter.SetTaskMetadataKeyIfAbsentNotArchived(ctx, recovered.TaskID, models.MetaKeyInterruptedAt, value)
} else {
// A plain archive guard does not protect a recovery marker from a
// newer interruption arriving between the ownership check and this write.
// Fail closed when the adapter lacks either generation-safe primitive.
s.logger.Warn("skipping recovery interruption marker without generation-safe setter",
zap.String("task_id", recovered.TaskID),
zap.String("session_id", recovered.ID))
return false, errors.New("task repository does not support generation-safe recovery interruption markers")
}
if err != nil {
return false, err
}
task, err := s.tasks.GetTask(ctx, recovered.TaskID)
if err != nil || task == nil {
if err == nil {
err = errors.New("task not found after interruption marker write")
}
return changed, err
}
// Publish after the metadata commit. The task service's per-task FIFO keeps
// this update ahead of later clears, so connected clients see the warning
// without a reload.
s.PublishTaskUpdated(ctx, task)
return changed, nil
}
@@ -54,12 +54,12 @@ func seedOrphanSweepTask(t *testing.T, repo *sqliterepo.Repository, taskID strin
}
}
func TestService_OrphanedSessionReconciliationTerminalizesUnbackedSessions(t *testing.T) {
func TestService_OrphanedSessionReconciliationPreservesUnbackedSessions(t *testing.T) {
svc, eventBus, repo := createTestService(t)
ctx := context.Background()
seedOrphanSweepFixtures(t, repo)
stale := time.Now().UTC().Add(-30 * time.Minute)
stale := time.Now().UTC().Add(-5 * time.Hour)
fresh := time.Now().UTC().Add(-1 * time.Minute)
seedOrphanSweepTask(t, repo, "task-orphaned", false)
@@ -94,7 +94,7 @@ func TestService_OrphanedSessionReconciliationTerminalizesUnbackedSessions(t *te
// Archived task with a stale RUNNING session: the archived pass owns it,
// the orphan pass must not touch it.
seedSession("session-archived-task", "task-archived", models.TaskSessionStateRunning, stale)
// Primary session on the orphaned task: the CANCELLED event must carry
// Primary session on the orphaned task: the recovery event must carry
// is_primary=true so the status-summary projection keeps the durable
// primary assignment.
seedSession("session-primary", "task-primary", models.TaskSessionStateRunning, stale)
@@ -114,45 +114,99 @@ func TestService_OrphanedSessionReconciliationTerminalizesUnbackedSessions(t *te
t.Errorf("session %s state = %q, want %q", id, session.State, want)
}
}
assertState("session-orphaned-running", models.TaskSessionStateCancelled)
assertState("session-orphaned-starting", models.TaskSessionStateCancelled)
assertState("session-orphaned-running", models.TaskSessionStateWaitingForInput)
assertState("session-orphaned-starting", models.TaskSessionStateWaitingForInput)
assertState("session-healthy-sibling", models.TaskSessionStateWaitingForInput)
assertState("session-live-execution", models.TaskSessionStateRunning)
assertState("session-inflight-launch", models.TaskSessionStateRunning)
assertState("session-archived-task", models.TaskSessionStateRunning)
assertState("session-primary", models.TaskSessionStateCancelled)
assertState("session-primary", models.TaskSessionStateWaitingForInput)
orphaned, err := repo.GetTaskSession(ctx, "session-orphaned-running")
if err != nil {
t.Fatalf("GetTaskSession(orphaned): %v", err)
}
if orphaned.ErrorMessage != models.SessionOrphanedCancelReason {
t.Errorf("orphaned session error_message = %q, want %q",
orphaned.ErrorMessage, models.SessionOrphanedCancelReason)
if orphaned.ErrorMessage != "" {
t.Errorf("orphaned session error_message = %q, want empty", orphaned.ErrorMessage)
}
// The event must carry the durable is_primary flag: the primary session's
// cancellation reports is_primary=true rather than the zero value.
if data := sessionCancelledEvent(eventBus, "session-primary"); data != nil {
// recovery reports is_primary=true rather than the zero value.
if data := sessionRecoveredEvent(eventBus, "session-primary"); data != nil {
if isPrimary, ok := data["is_primary"].(bool); !ok || !isPrimary {
t.Errorf("primary session cancellation event is_primary = %v, want true", data["is_primary"])
t.Errorf("primary session recovery event is_primary = %v, want true", data["is_primary"])
}
} else {
t.Error("expected a cancellation event for session-primary")
t.Error("expected a recovery event for session-primary")
}
for _, id := range []string{"session-orphaned-running", "session-orphaned-starting", "session-primary"} {
if !sessionCancelledEventPublished(eventBus, id) {
t.Errorf("expected a session.state_changed event for %s from the orphan sweep, got none", id)
if !sessionRecoveredEventPublished(eventBus, id) {
t.Errorf("expected a recovery session.state_changed event for %s from the orphan sweep, got none", id)
}
}
for _, id := range []string{"session-healthy-sibling", "session-live-execution", "session-inflight-launch", "session-archived-task"} {
if sessionCancelledEventPublished(eventBus, id) {
t.Errorf("unexpected session.state_changed event for %s", id)
if sessionRecoveredEventPublished(eventBus, id) {
t.Errorf("unexpected recovery session.state_changed event for %s", id)
}
}
}
// TestService_OrphanedSessionReconciliationPreservesInterruptedConversation
// covers the restart reconciliation pass. A stale STARTING/RUNNING session
// whose execution disappeared remains the same recoverable conversation,
// including its unfinished turn, instead of becoming CANCELLED.
func TestService_OrphanedSessionReconciliationPreservesInterruptedConversation(t *testing.T) {
svc, eventBus, repo := createTestService(t)
ctx := context.Background()
seedOrphanSweepFixtures(t, repo)
seedOrphanSweepTask(t, repo, "task-interrupted", false)
stale := time.Now().UTC().Add(-30 * time.Minute)
if err := repo.CreateTaskSession(ctx, &models.TaskSession{
ID: "session-interrupted", TaskID: "task-interrupted", State: models.TaskSessionStateRunning,
AgentProfileID: "agent-1", IsPrimary: true, UpdatedAt: stale, StartedAt: stale,
}); err != nil {
t.Fatalf("CreateTaskSession: %v", err)
}
turn, err := svc.StartTurn(ctx, "session-interrupted")
if err != nil {
t.Fatalf("StartTurn: %v", err)
}
if _, err := repo.DB().ExecContext(ctx,
`UPDATE task_session_turns SET started_at = ?, created_at = ?, updated_at = ? WHERE id = ?`,
stale, stale, stale, turn.ID); err != nil {
t.Fatalf("age turn: %v", err)
}
if _, err := repo.DB().ExecContext(ctx,
`UPDATE task_sessions SET updated_at = ? WHERE id = ?`, stale, "session-interrupted"); err != nil {
t.Fatalf("age session: %v", err)
}
svc.SetExecutionLivenessChecker(&fakeExecutionLiveness{})
svc.runOrphanedSessionReconciliation(ctx)
session, err := repo.GetTaskSession(ctx, "session-interrupted")
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
if session.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("session state = %q, want WAITING_FOR_INPUT", session.State)
}
if session.ErrorMessage != "" {
t.Fatalf("session error_message = %q, want empty", session.ErrorMessage)
}
storedTurn, err := svc.turns.GetTurn(ctx, turn.ID)
if err != nil {
t.Fatalf("GetTurn: %v", err)
}
if storedTurn.CompletedAt == nil || !storedTurn.CompletedAt.Equal(storedTurn.StartedAt) {
t.Fatalf("turn completion = %v, want zero-duration completion at %v", storedTurn.CompletedAt, storedTurn.StartedAt)
}
if sessionCancelledEventPublished(eventBus, "session-interrupted") {
t.Fatal("unexpected cancellation event for recovered session")
}
}
func sessionCancelledEventPublished(eventBus *MockEventBus, sessionID string) bool {
for _, evt := range eventBus.GetPublishedEvents() {
if evt.Type != events.TaskSessionStateChanged {
@@ -166,18 +220,27 @@ func sessionCancelledEventPublished(eventBus *MockEventBus, sessionID string) bo
return false
}
// sessionCancelledEvent returns the last session.state_changed payload for
// sessionID, or nil when none published. The orphan sweep's event must carry
// the durable is_primary flag — the status-summary projector demotes the
// session when a cancellation event reports is_primary: false.
func sessionCancelledEvent(eventBus *MockEventBus, sessionID string) map[string]interface{} {
func sessionRecoveredEventPublished(eventBus *MockEventBus, sessionID string) bool {
for _, evt := range eventBus.GetPublishedEvents() {
if evt.Type != events.TaskSessionStateChanged {
continue
}
data, ok := evt.Data.(map[string]interface{})
if ok && data["session_id"] == sessionID && data["new_state"] == string(models.TaskSessionStateWaitingForInput) {
return true
}
}
return false
}
func sessionRecoveredEvent(eventBus *MockEventBus, sessionID string) map[string]interface{} {
var found map[string]interface{}
for _, evt := range eventBus.GetPublishedEvents() {
if evt.Type != events.TaskSessionStateChanged {
continue
}
data, ok := evt.Data.(map[string]interface{})
if ok && data["session_id"] == sessionID && data["new_state"] == string(models.TaskSessionStateCancelled) {
if ok && data["session_id"] == sessionID && data["new_state"] == string(models.TaskSessionStateWaitingForInput) {
found = data
}
}
@@ -254,14 +317,14 @@ type delayedOrphanSessionRepository struct {
once sync.Once
}
func (r *delayedOrphanSessionRepository) CancelRunningTaskSessionByID(
func (r *delayedOrphanSessionRepository) RecoverTaskSessionByCandidate(
ctx context.Context,
sessionID, reason string,
candidate models.ActiveSessionRecoveryCandidate,
staleBefore time.Time,
) (*models.TaskSession, error) {
r.once.Do(func() { close(r.entered) })
<-r.release
return r.Repository.CancelRunningTaskSessionByID(ctx, sessionID, reason, staleBefore)
return r.Repository.RecoverTaskSessionByCandidate(ctx, candidate, staleBefore)
}
func TestService_OrphanedSessionReconciliationUsesFreshEffectsContextAfterDeadline(t *testing.T) {
@@ -287,8 +350,6 @@ func TestService_OrphanedSessionReconciliationUsesFreshEffectsContextAfterDeadli
if err != nil {
t.Fatalf("GetTaskSession: %v", err)
}
clarifications := &recordingTaskClarificationCanceller{}
svc.SetClarificationCanceller(clarifications)
svc.SetExecutionLivenessChecker(&fakeExecutionLiveness{})
passCtx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
@@ -301,7 +362,7 @@ func TestService_OrphanedSessionReconciliationUsesFreshEffectsContextAfterDeadli
select {
case <-delayed.entered:
case <-time.After(time.Second):
t.Fatal("orphan cancellation did not start")
t.Fatal("orphan recovery did not start")
}
select {
case <-passCtx.Done():
@@ -315,11 +376,101 @@ func TestService_OrphanedSessionReconciliationUsesFreshEffectsContextAfterDeadli
t.Fatal("reconciliation did not finish after the delayed write was released")
}
if len(clarifications.contextErrs) != 1 || clarifications.contextErrs[0] != nil {
t.Fatalf("clarification cleanup context errors = %v, want one nil error after a late commit", clarifications.contextErrs)
if session, err := repo.GetTaskSession(setupCtx, "session-late-effects"); err != nil {
t.Fatalf("GetTaskSession after delayed recovery: %v", err)
} else if session.State != models.TaskSessionStateWaitingForInput {
t.Fatalf("session state after delayed recovery = %q, want WAITING_FOR_INPUT", session.State)
}
if !sessionCancelledEventPublished(eventBus, "session-late-effects") {
t.Fatal("expected cancellation event after the delayed write committed")
if !sessionRecoveredEventPublished(eventBus, "session-late-effects") {
t.Fatal("expected recovery event after the delayed write committed")
}
}
type successorAfterRecoveryRepository struct {
*sqliterepo.Repository
afterRecover func()
}
func (r *successorAfterRecoveryRepository) RecoverTaskSessionByCandidate(
ctx context.Context,
candidate models.ActiveSessionRecoveryCandidate,
staleBefore time.Time,
) (*models.TaskSession, error) {
recovered, err := r.Repository.RecoverTaskSessionByCandidate(ctx, candidate, staleBefore)
if recovered != nil && err == nil && r.afterRecover != nil {
r.afterRecover()
}
return recovered, err
}
// A successor can register after the guarded session write commits but before
// post-commit effects run. The recovery must leave that successor's executor
// row and warning marker untouched.
func TestService_OrphanedSessionReconciliationDoesNotRepairSuccessorExecutor(t *testing.T) {
var rawRepo *sqliterepo.Repository
var successorInstalled bool
wrapper := &successorAfterRecoveryRepository{}
svc, _, repo := createTestServiceWithSessionsRepo(t, func(base *sqliterepo.Repository) repository.SessionRepository {
rawRepo = base
wrapper.Repository = base
wrapper.afterRecover = func() {
if successorInstalled {
return
}
successorInstalled = true
now := time.Now().UTC()
if err := base.UpsertExecutorRunning(context.Background(), &models.ExecutorRunning{
ID: "successor-row", SessionID: "session-successor", TaskID: "task-successor",
Status: models.ExecutorRunningStatusRunning, AgentExecutionID: "successor-execution",
LocalPID: 1234, CreatedAt: now, UpdatedAt: now,
}); err != nil {
t.Fatalf("install successor executor: %v", err)
}
if err := base.SetTaskMetadataKey(context.Background(), "task-successor", models.MetaKeyInterruptedAt, "successor-marker"); err != nil {
t.Fatalf("install successor warning marker: %v", err)
}
}
return wrapper
})
ctx := context.Background()
seedOrphanSweepFixtures(t, repo)
seedOrphanSweepTask(t, repo, "task-successor", false)
stale := time.Now().UTC().Add(-5 * time.Hour)
if err := repo.CreateTaskSession(ctx, &models.TaskSession{
ID: "session-successor", TaskID: "task-successor", State: models.TaskSessionStateRunning,
AgentProfileID: "agent-1", IsPrimary: true, UpdatedAt: stale, StartedAt: stale,
}); err != nil {
t.Fatalf("CreateTaskSession: %v", err)
}
oldExecTime := stale
if err := repo.UpsertExecutorRunning(ctx, &models.ExecutorRunning{
ID: "old-row", SessionID: "session-successor", TaskID: "task-successor",
Status: models.ExecutorRunningStatusRunning, AgentExecutionID: "old-execution",
LocalPID: 99, CreatedAt: oldExecTime, UpdatedAt: oldExecTime,
}); err != nil {
t.Fatalf("seed old executor: %v", err)
}
svc.SetExecutionLivenessChecker(&fakeExecutionLiveness{})
// The active sweep uses the same task/session recovery path as restart
// reconciliation, so an expired liveness snapshot cannot stop the successor.
svc.SetSessionExecutionRegistry(&stubExecutionRegistry{})
svc.SetStallDetectionThreshold(2 * time.Hour)
svc.runActiveSessionSweep(ctx, time.Now().UTC())
running, err := rawRepo.GetExecutorRunningBySessionID(ctx, "session-successor")
if err != nil {
t.Fatalf("GetExecutorRunningBySessionID: %v", err)
}
if running.AgentExecutionID != "successor-execution" || running.Status != models.ExecutorRunningStatusRunning || running.LocalPID != 1234 {
t.Fatalf("successor executor = %+v, want running successor unchanged", running)
}
task, err := rawRepo.GetTask(ctx, "task-successor")
if err != nil {
t.Fatalf("GetTask: %v", err)
}
if got := task.Metadata[models.MetaKeyInterruptedAt]; got != "successor-marker" {
t.Fatalf("successor warning marker = %#v, want successor-marker", got)
}
}
@@ -327,7 +478,7 @@ func TestService_OrphanedSessionReconciliationUsesFreshEffectsContextAfterDeadli
// pins the read-then-write guard: a launch CAS-writes its session to STARTING
// (bumping updated_at) before it registers an execution in the in-memory
// store, so the row can be refreshed between the sweep's candidate read and
// its cancellation write. The cancel's staleBefore predicate must fail to
// its recovery write. The recovery's staleBefore predicate must fail to
// match that refreshed row instead of cancelling a launch in progress.
func TestService_OrphanedSessionReconciliationSparesRowRefreshedSinceCandidateRead(t *testing.T) {
svc, eventBus, repo := createTestService(t)
@@ -244,6 +244,54 @@ func (s *Service) publishSessionsCancelled(
}
}
// publishSessionRecovered publishes the state transition produced by
// execution-loss recovery. It mirrors the session.state_changed payload used
// by cancellation while carrying an empty error message and the recoverable
// WAITING_FOR_INPUT state.
func (s *Service) publishSessionRecovered(
ctx context.Context,
taskID string,
oldState models.TaskSessionState,
session *models.TaskSession,
) error {
if s.eventBus == nil || session == nil {
return nil
}
sessCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
data := map[string]interface{}{
sessionEventFieldTaskID: taskID,
sessionEventFieldSessionID: session.ID,
"old_state": string(oldState),
"new_state": string(session.State),
"error_message": "",
"agent_profile_id": session.AgentProfileID,
"agent_profile_snapshot": session.AgentProfileSnapshot,
"is_passthrough": session.IsPassthrough,
"is_primary": session.IsPrimary,
sessionEventFieldUpdatedAt: session.UpdatedAt.Format(time.RFC3339Nano),
sessionEventFieldName: session.Name,
}
if session.ReviewStatus != models.ReviewStatusNone {
data["review_status"] = string(session.ReviewStatus)
}
if len(session.Metadata) > 0 {
data["session_metadata"] = session.Metadata
}
if session.TaskEnvironmentID != "" {
data["task_environment_id"] = session.TaskEnvironmentID
}
event := bus.NewEvent(events.TaskSessionStateChanged, "task-service", data)
if err := s.eventBus.Publish(sessCtx, events.TaskSessionStateChanged, event); err != nil {
s.logger.Error("failed to publish session recovery event",
zap.String(sessionEventFieldTaskID, taskID),
zap.String(sessionEventFieldSessionID, session.ID),
zap.Error(err))
return err
}
return nil
}
// publishTaskEvent publishes task events to the event bus
func (s *Service) publishTaskEvent(ctx context.Context, eventType string, task *models.Task, oldState *v1.TaskState, oldWorkflowIDs ...string) {
s.publishTaskEventWithExtra(ctx, eventType, task, oldState, nil, oldWorkflowIDs...)
@@ -439,6 +487,10 @@ func (s *Service) publishTaskEventNow(ctx context.Context, eventType string, tas
// omitted key here would make clearTaskAutoStartFailedMarker's publish
// as invisible as the set it is meant to undo.
"auto_start_failed": task.Metadata[models.MetaKeyAutoStartFailed] != nil,
// Keep the interruption projection explicit on task.updated so a live
// client receives the warning immediately after reconciliation writes
// the marker; task-merge preserves omitted fields for partial updates.
"interrupted": task.Metadata[models.MetaKeyInterruptedAt] != nil,
// The human assignee, always sent, never omitted when empty, for the
// same reason as auto_start_failed above: the frontend pins the
// previous value when the key is absent, so omitting it would make
+1 -1
View File
@@ -265,7 +265,7 @@ describe("TaskItem queue status", () => {
});
describe("TaskItem interrupted icon", () => {
it("shows the red interrupted icon for a task marked interrupted after a restart", () => {
it("shows the warning triangle for a task marked interrupted after a restart", () => {
renderTaskItem({
state: "REVIEW",
sessionState: "WAITING_FOR_INPUT",
@@ -0,0 +1,41 @@
import { expect, test } from "../../fixtures/test-base";
import { MobileKanbanPage } from "../../pages/mobile-kanban-page";
test.describe("Mobile kanban - interrupted-task icon", () => {
test("shows the warning triangle for an interrupted task", async ({
testPage,
apiClient,
seedData,
}) => {
const interrupted = await apiClient.createTask(
seedData.workspaceId,
"Mobile interrupted fixture",
{
workflow_id: seedData.workflowId,
workflow_step_id: seedData.startStepId,
metadata: { interrupted_at: "2026-08-02T10:00:00Z" },
},
);
await apiClient.updateTaskState(interrupted.id, "REVIEW");
const plain = await apiClient.createTask(seedData.workspaceId, "Mobile plain fixture", {
workflow_id: seedData.workflowId,
workflow_step_id: seedData.startStepId,
});
await apiClient.updateTaskState(plain.id, "REVIEW");
const mobile = new MobileKanbanPage(testPage);
await mobile.goto();
const interruptedCard = mobile.taskCard(interrupted.id);
await expect(interruptedCard).toBeVisible();
const interruptedIcon = interruptedCard.getByTestId("task-state-interrupted");
await expect(interruptedIcon).toBeVisible();
await expect(interruptedIcon).toHaveClass(/tabler-icon-alert-triangle/);
await expect(interruptedIcon).toHaveClass(/text-yellow-500/);
const plainCard = mobile.taskCard(plain.id);
await expect(plainCard).toBeVisible();
await expect(plainCard.getByTestId("task-state-interrupted")).toHaveCount(0);
});
});
@@ -4,13 +4,13 @@ import { SessionPage } from "../../pages/session-page";
// Interrupted-task indicator: when Kandev dies while a task's session is
// mid-turn, startup reconciliation marks the task with the `interrupted_at`
// metadata key and the task list shows a red alert icon until the task is
// metadata key and the task list shows a warning triangle until the task is
// resumed. Seeding `interrupted_at` through the public task-metadata surface
// is the deterministic stand-in for a real crash; the icon itself must render
// from that marker alone.
test.describe("Task list — interrupted-task icon", () => {
test("shows the red interrupted icon only for marked tasks", async ({
test("shows the warning triangle only for marked tasks", async ({
testPage,
apiClient,
seedData,
@@ -58,38 +58,48 @@ test.describe("Task list — interrupted-task icon", () => {
await apiClient.updateTaskState(terminal.id, "COMPLETED");
const kanban = new KanbanPage(testPage);
const workflowSnapshot = testPage.waitForResponse(
(response) =>
response.request().method() === "GET" &&
response.url().includes(`/api/v1/workflows/${seedData.workflowId}/snapshot`) &&
response.ok(),
);
await kanban.goto();
await workflowSnapshot;
const interruptedCard = kanban.taskCard(interrupted.id);
await expect(interruptedCard).toBeVisible();
const cardIcon = interruptedCard.getByTestId("task-state-interrupted");
await expect(cardIcon).toBeVisible();
await expect(cardIcon).toHaveClass(/tabler-icon-alert-triangle/);
await expect(cardIcon).toHaveClass(/text-yellow-500/);
const anchorCard = kanban.taskCardByTitle("Anchor Session");
await expect(anchorCard).toBeVisible({ timeout: 20_000 });
await expect(anchorCard).toBeVisible();
await anchorCard.click();
await expect(testPage).toHaveURL(/\/t\//, { timeout: 20_000 });
await expect(testPage).toHaveURL(/\/t\//);
const session = new SessionPage(testPage);
await session.waitForLoad();
const interruptedRow = session.sidebarTaskItem("Interrupted Fixture");
await expect(interruptedRow).toBeVisible({ timeout: 20_000 });
await expect(interruptedRow.getByTestId("task-state-interrupted")).toBeVisible({
timeout: 20_000,
});
await expect(interruptedRow).toBeVisible();
await expect(interruptedRow.getByTestId("task-state-interrupted")).toBeVisible();
const plainRow = session.sidebarTaskItem("Plain Fixture");
await expect(plainRow).toBeVisible({ timeout: 20_000 });
await expect(plainRow).toBeVisible();
await expect(plainRow.getByTestId("task-state-interrupted")).toHaveCount(0);
const terminalRow = session.sidebarTaskItem("Terminal Fixture");
await expect(terminalRow).toBeVisible({ timeout: 20_000 });
await expect(terminalRow).toBeVisible();
await expect(terminalRow.getByTestId("task-state-interrupted")).toHaveCount(0);
// Reload: the marker must survive SSR hydration from the boot payload.
await testPage.reload();
await session.waitForLoad();
await expect(session.sidebarTaskItem("Interrupted Fixture")).toBeVisible({
timeout: 20_000,
});
await expect(session.sidebarTaskItem("Interrupted Fixture")).toBeVisible();
await expect(
session.sidebarTaskItem("Interrupted Fixture").getByTestId("task-state-interrupted"),
).toBeVisible({ timeout: 20_000 });
).toBeVisible();
await expect(
session.sidebarTaskItem("Plain Fixture").getByTestId("task-state-interrupted"),
).toHaveCount(0);
@@ -300,6 +300,8 @@ const WORKTREE_EXECUTOR_FIELDS = {
primaryExecutorName: "Worktree",
isRemoteExecutor: false,
};
const INTERRUPTED_TASK_ID = "task-1";
const INTERRUPTED_STALE_TASK_TITLE = "Interrupted stale task";
function seedCachedTask(taskOverrides: Record<string, unknown>) {
mockState.kanbanMulti.snapshots = {
@@ -370,6 +372,78 @@ describe("useAllWorkflowSnapshots — snapshot mapping", () => {
await waitFor(() => expect(mockSetWorkflowSnapshot).toHaveBeenCalled());
expect(mockSetWorkflowSnapshot.mock.calls[0][1].tasks[0].autopilot).toBe(false);
});
it("preserves a newer live interruption marker over a stale snapshot value", async () => {
seedCachedTask({ id: INTERRUPTED_TASK_ID, interrupted: false });
let resolveSnapshot: ((value: { steps: unknown[]; tasks: unknown[] }) => void) | undefined;
mockFetchWorkflowSnapshot.mockImplementationOnce(
() =>
new Promise<{ steps: unknown[]; tasks: unknown[] }>((resolve) => {
resolveSnapshot = resolve;
}),
);
renderHook(() => useAllWorkflowSnapshots("ws-A"));
await waitFor(() => expect(mockFetchWorkflowSnapshot).toHaveBeenCalled());
// A live task.updated event marks the task while the older snapshot is in flight.
seedCachedTask({ id: INTERRUPTED_TASK_ID, interrupted: true });
resolveSnapshot?.({
steps: [{ id: "step-1", name: "Review", position: 1 }],
tasks: [
{
id: INTERRUPTED_TASK_ID,
workflow_step_id: "step-1",
title: INTERRUPTED_STALE_TASK_TITLE,
interrupted: false,
},
],
});
await waitFor(() => expect(mockSetWorkflowSnapshot).toHaveBeenCalled());
expect(mockSetWorkflowSnapshot.mock.calls[0][1].tasks[0].interrupted).toBe(true);
});
});
describe("useAllWorkflowSnapshots — interruption marker generations", () => {
beforeEach(() => {
resetMocks([{ id: "wf-A", workspaceId: "ws-A", name: "A" }]);
});
it("preserves a newer false marker after a complete in-flight interruption episode", async () => {
seedCachedTask({ id: INTERRUPTED_TASK_ID, interrupted: false, interruptedGeneration: 1 });
let resolveSnapshot: ((value: { steps: unknown[]; tasks: unknown[] }) => void) | undefined;
mockFetchWorkflowSnapshot.mockImplementationOnce(
() =>
new Promise<{ steps: unknown[]; tasks: unknown[] }>((resolve) => {
resolveSnapshot = resolve;
}),
);
renderHook(() => useAllWorkflowSnapshots("ws-A"));
await waitFor(() => expect(mockFetchWorkflowSnapshot).toHaveBeenCalled());
// The marker changes twice while the snapshot is in flight. The final
// false value is newer even though it equals the value at fetch start.
seedCachedTask({ id: INTERRUPTED_TASK_ID, interrupted: false, interruptedGeneration: 3 });
resolveSnapshot?.({
steps: [{ id: "step-1", name: "Review", position: 1 }],
tasks: [
{
id: INTERRUPTED_TASK_ID,
workflow_step_id: "step-1",
title: INTERRUPTED_STALE_TASK_TITLE,
interrupted: false,
},
],
});
await waitFor(() => expect(mockSetWorkflowSnapshot).toHaveBeenCalled());
expect(mockSetWorkflowSnapshot.mock.calls[0][1].tasks[0]).toMatchObject({
interrupted: false,
interruptedGeneration: 3,
});
});
});
describe("useAllWorkflowSnapshots — executor binding merge", () => {
@@ -60,6 +60,20 @@ function hasNewerLiveAutoStartFailed(
return existing.autoStartFailed !== fetchStart.autoStartFailed;
}
function hasNewerLiveInterrupted(
existing: KanbanTask,
fetchStart: KanbanTask | undefined,
): boolean {
if (!fetchStart) return true;
if (
existing.interruptedGeneration !== undefined ||
fetchStart.interruptedGeneration !== undefined
) {
return existing.interruptedGeneration !== fetchStart.interruptedGeneration;
}
return existing.interrupted !== fetchStart.interrupted;
}
function hasNewerLiveWorkspaceOrphaned(
existing: KanbanTask,
fetchStart: KanbanTask | undefined,
@@ -85,6 +99,12 @@ function preserveLiveMarkerFields(
): void {
// A task.updated event can set or clear either marker while the snapshot
// is in flight. Preserve the newer live value instead of rolling it back.
if (merged.interrupted === undefined || hasNewerLiveInterrupted(existing, fetchStart)) {
merged.interrupted = existing.interrupted;
}
if (existing.interruptedGeneration !== undefined) {
merged.interruptedGeneration = existing.interruptedGeneration;
}
if (merged.autoStartFailed === undefined || hasNewerLiveAutoStartFailed(existing, fetchStart)) {
merged.autoStartFailed = existing.autoStartFailed;
}
@@ -142,6 +142,8 @@ export type KanbanState = {
foregroundActivity?: ForegroundActivity | null;
/** True when the task's session was mid-turn when the backend died. */
interrupted?: boolean;
/** Monotonic client generation for explicit interruption-marker updates. */
interruptedGeneration?: number;
/** True when a workflow step's auto_start_agent on_enter action failed to
* launch a run for this task. */
autoStartFailed?: boolean;
+4 -2
View File
@@ -269,7 +269,7 @@ describe("getTaskStateIcon — task-level activity tri-state", () => {
});
describe("getTaskStateIcon — status markers", () => {
it("renders the accessible interrupted icon with the tooltip label", () => {
it("renders the accessible warning triangle with the tooltip label", () => {
const { container } = render(
<TooltipProvider>
{getTaskStateIcon("REVIEW", undefined, { interrupted: true })}
@@ -277,7 +277,9 @@ describe("getTaskStateIcon — status markers", () => {
);
const icon = container.querySelector('[data-testid="task-state-interrupted"]');
expect(icon).not.toBeNull();
expect(icon?.className).toContain("text-red-500");
expect(icon?.className).toContain("tabler-icon-alert-triangle");
expect(icon?.className).not.toContain("tabler-icon-alert-circle");
expect(icon?.className).toContain("text-yellow-500");
expect(container.querySelector('[aria-label="Interrupted by restart"]')).not.toBeNull();
// The icon itself is decorative; the label lives on the trigger.
expect(icon?.getAttribute("aria-hidden")).toBe("true");
+8 -9
View File
@@ -117,12 +117,11 @@ const PENDING_PERMISSION_ICON: IconConfig = {
};
// The task-level interrupted affordance: the session was mid-turn when the
// backend died and the task has not been resumed. A red alert circle — red is
// otherwise the error/cancelled hue, but the alert shape plus the REVIEW/idle
// coarse state it replaces keeps it distinct from the terminal X affordances.
// backend died and the task has not been resumed. Use the shared warning
// triangle styling so this durable marker reads as attention-needed state.
const TASK_INTERRUPTED_ICON: IconConfig = {
Icon: IconAlertCircle,
className: STYLE_ERROR,
Icon: IconAlertTriangle,
className: STYLE_WARNING,
};
// The task-level auto-start-failed affordance: a workflow step's
@@ -214,7 +213,7 @@ export function isTerminalInterruptedState(
}
/**
* Shared red alert affordance for a task whose session was mid-turn when the
* Shared warning affordance for a task whose session was mid-turn when the
* backend died. Carries the accessible "Interrupted by restart" label and
* tooltip, so every surface that renders the interrupted state (sidebar rows,
* board cards, graph nodes, open-task header) presents it consistently.
@@ -227,12 +226,12 @@ export function InterruptedTaskIcon({ className }: { className?: string }) {
<span
aria-label={t("common:interruptedByRestart")}
tabIndex={0}
className="flex shrink-0 rounded-sm focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-red-500 focus-visible:ring-offset-1"
className="flex shrink-0 rounded-sm focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-yellow-500 focus-visible:ring-offset-1"
>
<IconAlertCircle
<IconAlertTriangle
aria-hidden="true"
data-testid="task-state-interrupted"
className={cn("text-red-500", className)}
className={cn(STYLE_WARNING, className)}
/>
</span>
</TooltipTrigger>
+6
View File
@@ -112,6 +112,7 @@ function preserveMultiSnapshotFields(
| "taskPendingAction"
| "foregroundActivity"
| "interrupted"
| "interruptedGeneration"
| "autoStartFailed"
| "workspaceOrphaned"
| "priority"
@@ -126,6 +127,10 @@ function preserveMultiSnapshotFields(
taskPendingAction: preserveIfUndefined(t.taskPendingAction, fallback?.taskPendingAction),
foregroundActivity: preserveIfUndefined(t.foregroundActivity, fallback?.foregroundActivity),
interrupted: preserveIfUndefined(t.interrupted, fallback?.interrupted),
interruptedGeneration: preserveIfUndefined(
t.interruptedGeneration,
fallback?.interruptedGeneration,
),
autoStartFailed: preserveIfUndefined(t.autoStartFailed, fallback?.autoStartFailed),
workspaceOrphaned: preserveIfUndefined(t.workspaceOrphaned, fallback?.workspaceOrphaned),
priority: preserveIfUndefined(t.priority, fallback?.priority),
@@ -354,6 +359,7 @@ export function registerKanbanHandlers(store: StoreApi<AppState>): WsHandlers {
primarySessionPendingAction: existing?.primarySessionPendingAction,
taskPendingAction: existing?.taskPendingAction,
interrupted: existing?.interrupted,
interruptedGeneration: existing?.interruptedGeneration,
autoStartFailed: existing?.autoStartFailed,
workspaceOrphaned: existing?.workspaceOrphaned,
foregroundActivity: existing?.foregroundActivity,
+11 -1
View File
@@ -101,12 +101,22 @@ export function mergeTaskUpdate(
nextTask: KanbanTask,
payload: TaskEventPayload,
): KanbanTask {
if (!existing) return nextTask;
if (!existing) {
return hasPayloadField(payload, "interrupted")
? { ...nextTask, interruptedGeneration: 1 }
: nextTask;
}
if (isStaleTaskUpdate(existing, nextTask)) return existing;
const merged = {
...nextTask,
...mergeTaskRepositoryFields(existing, nextTask),
};
// A snapshot can observe the same boolean value before and after a live
// false -> true -> false marker episode. Count every explicit marker update
// so an in-flight snapshot cannot erase that newer generation.
merged.interruptedGeneration = hasPayloadField(payload, "interrupted")
? (existing.interruptedGeneration ?? 0) + 1
: existing.interruptedGeneration;
preserveOmittedField(existing, merged, payload, nextTask, {
payloadKey: "parent_id",
taskField: "parentTaskId",
@@ -325,6 +325,22 @@ describe("task.updated interrupted marker (live propagation + safe fallback)", (
expect(interruptedFor(store, "t1")).toEqual({ kanban: false, multi: false });
});
it("increments the marker generation for every explicit update", () => {
const store = storeWithTask({ id: "t1", interrupted: false, interruptedGeneration: 1 });
const handlers = registerTasksHandlers(store);
handlers["task.updated"]!(makeMessage({ ...makeTask("t1"), interrupted: true }));
handlers["task.updated"]!(makeMessage({ ...makeTask("t1"), interrupted: false }));
expect(store.getState().kanban.tasks.find((task) => task.id === "t1")).toMatchObject({
interrupted: false,
interruptedGeneration: 3,
});
expect(
store.getState().kanbanMulti.snapshots.wf1.tasks.find((task) => task.id === "t1"),
).toMatchObject({ interrupted: false, interruptedGeneration: 3 });
});
it("preserves the marker when a partial update omits interrupted", () => {
// Clobber guard: a lightweight task.updated (e.g. a rename) must not wipe
// the interruption reading between real marker events.
@@ -0,0 +1,252 @@
---
created: 2026-09-21
status: complete
requirements:
- REQ-TASKS-SESSION-STALL-VISIBILITY-001
- REQ-TASKS-RESTART-ORPHAN-SESSIONS-002
- REQ-TASKS-INTERRUPTED-TASK-INDICATOR-001
system_design:
- ../../specs/tasks/system-design/session-stall-visibility.md
- ../../specs/tasks/system-design/restart-orphaned-session-terminalization.md
- ../../specs/tasks/system-design/interrupted-task-indicator.md
legacy_specs: []
---
# Implementation plan: Orphaned session recovery on task open
## Overview
Preserve idle and interrupted running conversations during reconciliation.
Both sweeps must leave interrupted sessions recoverable, rather than cancelled.
Focusing a task restores its selected agent. A new user message continues the
conversation. Opening Kandev must not eagerly resume every interrupted agent.
Implement recoverable settlement, then focus recovery, then the persistent warning indicator.
The task system owns this change because it owns session cancellation and
recovery eligibility. The UI uses the existing recovery flow.
## Confirmed evidence
Read-only diagnostics for task `52f801c5-b0ff-4b4f-8f00-0d37ba804d03`
identify session `a136900f-3546-4843-8a99-37194378222e`.
Times below include the Lisbon UTC offset.
- September 20, 11:37:44 +01:00: idle reclaim released the provider runtime.
The log recorded a preserved resume token and worktree.
- September 21, 09:44:24 +01:00: startup reconciled `WAITING_FOR_INPUT`
for lazy recovery, again with a token and worktree.
- September 21, 09:45:25 +01:00: `active-session sweep: healing orphaned sessions`
named this session. The state event recorded `WAITING_FOR_INPUT -> CANCELLED`.
- The session-list MCP response independently recorded the same cancellation time.
- The diagnostic manifest identified deployed commit `1614720ceb`.
The bundle was partial because its oldest logs exceeded the archive limit.
The relevant September 21 cancellation records were present.
PR [#3832](https://github.com/kdlbs/kandev/pull/3832), commit `00c7469d81`,
introduced the active-session sweep. It classifies every execution-less active
session as orphaned, including idle `WAITING_FOR_INPUT` sessions.
PR [#3833](https://github.com/kdlbs/kandev/pull/3833), commit `1614720ceb`,
added a separate restart pass limited to `STARTING` and `RUNNING`.
The logs identify #3832's pass as the cancellation source in this incident.
The merged-PR banner for #3819 is not evidence that its merge cancelled the session.
`GetTaskSessionStatus` automatically recovers archive cancellations, but other
`CANCELLED` sessions fall through to workspace restoration. The orphan reason
therefore requires a manual Resume despite preserved recovery data.
## Scope
### In scope
- Exclude waiting sessions without an unfinished turn from stall detection and healing.
- Replace execution-loss cancellation in both sweeps with recoverable waiting state.
- Recover existing exact orphan cancellations through normal task opening.
- Continue interrupted work only when a new message arrives, including during startup.
- Preserve same-session history, workspace identity, user preferences, and launch ownership.
- Test desktop and phone opening, failure fallback, and explicit-stop exclusions.
- Keep interruption warnings until successful resume. Replace the shared red circle
with a warning-colored triangle in sidebar rows and task cards.
### Out of scope
- Prompt replay, automatic continuation of completed work, or workflow advancement.
- Removing reconciliation, changing thresholds, or adding a runtime flag.
- Live-instance repair, new sessions, database migrations, or provider redesign.
## Technical approach
Task 01 changes both reconciliation passes and their conditional repository
transitions. Reuse startup recovery semantics in `reconcileActiveSessionOnStartup`:
settle the abandoned turn without a success event, preserve recovery identity,
and expose `WAITING_FOR_INPUT`. Include missing-runtime-row sessions. Retain
live-execution, launch-grace, archive/stop, and newer-turn guards. Idle sessions
need no transition or stall event.
Task 02 changes exact cancellation eligibility in `task_operations.go` and
corrects the orphan-reason comment in `models/resume_safety.go`. Reuse existing
resume-token and same-session initialization paths. Keep unsupported recovery
and missing-profile errors actionable. Use the existing frontend recovery hooks
and automatic admission contract. Change frontend logic only if regression
coverage exposes a state-handling gap.
Task 03 updates the shared interruption renderer and moves marker clearing
from STARTING entry to confirmed recovery success. The existing durable marker
survives unsuccessful attempts. Follow its paired design for stale callbacks,
partial updates, warning styling, and desktop/phone coverage.
No separate ADR is needed. The existing
[session-open decision](../../decisions/2026-09-18-session-open-resumes-conversation.md)
already owns prompt-free provider recovery. This package also replaces automatic
orphan cancellation with recoverable interruption settlement, as the user requested.
## ASCII UI preview
UI-01: Focus an interrupted or legacy orphan-cancelled conversation. Labels are illustrative existing
localized states, not proposed new copy.
```text
Current desktop
[orphaned session] [Resume] [Start fresh session]
[existing transcript ]
Proposed desktop
[normal agent startup progress ]
[existing transcript ]
|
v
[existing transcript ]
[message composer Send ]
Proposed phone: task route or task picker
[Task header / session picker]
[normal startup progress ]
[existing transcript ] <- existing chat scroll
[composer Send ]
[existing bottom navigation ]
```
After success, the warning disappears. A new user message continues the work. On actual recovery failure, retain the
existing error, Resume, and Start fresh session controls. With auto-start
prevention enabled, retain manual recovery. Opening never sends a message.
No new overlay, scroll owner, or touch control is added. Keep phone safe areas
and existing touch targets. Structural requirement: recovery occurs in the
selected conversation. Spacing is illustrative.
UI-02: Interruption status in the existing sidebar/card status slot.
```text
Before: (red !) Task title
Desktop after: /!\ Task title [warning color]
Phone after: /!\ Task title [drawer row / board card]
Startup: [normal progress; interruption marker retained]
Resume succeeds: [normal task status] Task title
Resume fails: [marker retained; existing recovery error available]
```
Keep the localized accessible label "Interrupted by restart". The shared
triangle and warning color are required. Existing layouts and scroll owners
remain. No new hover-only information or touch control is introduced.
## Tests
- Task 01 covers stall AC .1-.7 and restart AC 002.1, .4-.7:
idle exemption, resumable interrupted work in both sweeps, missing executor rows,
active-turn read errors, mixed siblings, and completion races.
- Task 02 covers AC .8-.10 with `task_operations_resume_test.go` and frontend
session-resumption tests: exact reason, supported/missing token, profile
validation, duplicate opening, blocked recovery, and failure fallback.
- Task 03 covers interruption AC .2-.7 with marker lifecycle tests, shared icon
tests, and desktop/phone sidebar and board assertions. Verify retained warnings
across failed recovery, reload, and repeated restart, followed by live clearing.
- Retain restart-pass, archive-recovery, explicit-stop, deferred-owner, and
session-ceiling regressions.
## E2E tests
The existing `session/session-resume.spec.ts`,
`session/session-recovery.spec.ts`, and their mobile counterparts already cover
backend interruption, lazy focus recovery, queued messages during startup,
same-session follow-up, auto-start prevention, and failed recovery retry using
the real status-to-launch path. The new backend tests establish the exact
legacy orphan eligibility and reconciliation classification without adding a
production endpoint. The task indicator checks below provide the rendered
desktop and phone assertions for the retained warning state.
Add `task/mobile-task-interrupted-icon.spec.ts` and extend
`task/task-interrupted-icon.spec.ts` for UI-02. Use the shared state-icon unit
tests for every consumer's shape/color contract and the recovery suites for
successful/failed resume behavior. Browser screenshots prove the rendered states.
## Work orders
- [x] [Task 01: Preserve resumable conversations](task-01-preserve-idle-conversations.md)
- [x] [Task 02: Recover orphan conversations on open](task-02-recover-orphan-conversations.md)
- [x] [Task 03: Keep interruption warnings](task-03-interruption-warning.md)
## Related delivery records
The implemented [stall package](../session-stall-visibility/plan.md) and
[restart package](../restart-orphaned-session-terminalization/plan.md) remain
historical records. Their tests remain regression inputs. This package supersedes their execution-loss cancellation policy and retains
their liveness, grace, and concurrency protections. Their historical results do
not prove the revised behavior.
## Verification results
- `python3 scripts/list-docs.py validate`: passed, catalog valid after the warning-indicator revision.
- `python3 scripts/lint-spec-files.py --all`: passed.
- `git diff --check`: passed.
- `go test ./internal/orchestrator ./internal/task/service ./internal/task/repository/sqlite`: passed.
- Full affected Go packages pass after the review remediation, including
missing-row focus recovery, successor barriers, fresh-context settlement,
marker CAS, and committed-generation retry regressions.
- Targeted backend recovery, reconciliation, metadata-CAS, and marker-lifecycle tests: passed.
- Targeted frontend Vitest, typecheck, and ESLint checks: passed.
- Review-fixup regressions for fail-closed executor snapshots, CREATED-session
recovery tokens, abandoned-turn tool-call retry, fresh delayed-commit effects
context, and false-to-true-to-false marker generations: passed.
- Chromium and mobile-chrome interrupted-indicator E2E tests: passed (one test each).
- `node --test scripts/validate-public-docs.test.mjs` and
`node scripts/validate-public-docs.mjs`: passed.
The implementation preserves idle conversations, settles interrupted sessions
to recoverable waiting state with race-safe repository predicates, and lazily
recovers the selected conversation without replaying its old prompt. Exact
legacy orphan cancellations remain compatible with the same recovery path while
archive, explicit-stop, ownership, capacity, and live-execution guards retain
their authority. The shared interruption marker now uses a warning triangle,
survives failed or stale recovery attempts, and clears only after confirmed
provider recovery.
## Integrated review remediation
The three work orders are complete together with the follow-up review fixes:
- Recovered CREATED, STARTING, and RUNNING sessions persist an immutable settlement token
and executor snapshot. A missing `executors_running` row is still resumable,
and task-open status plus focus recovery use the same session and profile.
- Active and restart reconciliation use a bounded post-commit context. Turn,
tool, executor, marker, and event effects remain retryable until the durable
settlement is cleared. Executor repair, session settlement, and marker writes
compare the captured executor and committed session generations, so a
successor cannot be stopped, settled, or marked by a delayed sweep.
- Recovery attempts retain the interruption-marker snapshot through active and
tombstoned attempts. Boot callbacks without a valid generation fail closed,
and a committed marker publishes `task.updated` with the interruption state
so connected desktop and phone clients update without a reload. Live task
merges carry a per-task marker generation, so an in-flight snapshot cannot
erase a newer interruption episode when its final boolean matches the
fetch-start value.
The integrated implementation and regressions are validated by the backend
package tests, the targeted marker and UI checks, the workspace build and lint,
and the specification and public-document validation commands listed above.
## Risks
- A waiting session can still hold an unfinished turn. State alone is insufficient.
- Re-reading a completed turn as absent can defeat the original cancellation snapshot.
- Broadening all cancellations would restart intentionally stopped work.
- A tokenless provider cannot promise native conversation context restoration.
- Partial recovery must not expire questions, duplicate prompts, or advance the workflow.
- Partial diagnostic retention limits reconstruction of earlier runtime history.
@@ -0,0 +1,144 @@
---
id: "01-preserve-idle-conversations"
title: "Preserve resumable conversations during reconciliation"
status: done
wave: 1
depends_on: []
plan: "plan.md"
requirements:
- REQ-TASKS-SESSION-STALL-VISIBILITY-001
- REQ-TASKS-RESTART-ORPHAN-SESSIONS-002
acceptance_criteria:
- AC-TASKS-SESSION-STALL-VISIBILITY-001.1
- AC-TASKS-SESSION-STALL-VISIBILITY-001.2
- AC-TASKS-SESSION-STALL-VISIBILITY-001.3
- AC-TASKS-SESSION-STALL-VISIBILITY-001.4
- AC-TASKS-SESSION-STALL-VISIBILITY-001.5
- AC-TASKS-SESSION-STALL-VISIBILITY-001.6
- AC-TASKS-SESSION-STALL-VISIBILITY-001.7
- AC-TASKS-RESTART-ORPHAN-SESSIONS-002.1
- AC-TASKS-RESTART-ORPHAN-SESSIONS-002.4
- AC-TASKS-RESTART-ORPHAN-SESSIONS-002.5
- AC-TASKS-RESTART-ORPHAN-SESSIONS-002.6
- AC-TASKS-RESTART-ORPHAN-SESSIONS-002.7
system_design:
- ../../specs/tasks/system-design/session-stall-visibility.md
- ../../specs/tasks/system-design/restart-orphaned-session-terminalization.md
---
# Task 01: Preserve resumable conversations during reconciliation
## Summary
Keep idle conversations untouched. Reconcile interrupted running conversations
to recoverable waiting state in both sweeps. Preserve abandoned-turn settlement
without cancellation, prompt replay, or workflow advancement.
## In scope
- Add `TestService_ActiveSessionSweepPreservesIdleWaitingSession` before implementation.
Reuse `newSweepFixture` with five hours of silence. Set `WAITING_FOR_INPUT`
and no active turn. Expect unchanged state and no `task.stalled` event.
Current code cancels this fixture, matching the recorded incident.
- Add `TestService_OrphanedSessionReconciliationPreservesInterruptedConversation`
and `TestService_ActiveSessionSweepRecoversInterruptedConversation` before changes.
Seed old STARTING/RUNNING sessions without live executions. Expect recoverable
waiting state and preserved identity. Current code cancels these sessions.
- Cover waiting sessions with an unfinished turn, missing/retained executor rows,
active-turn read failure, and ordinary `RUNNING`/`STARTING` candidates.
- Cover an idle waiting sibling plus a genuine stale running sibling. Detection
names only the running sibling; task-wide healing remains blocked.
- Cover a turn completing or being replaced between classification and cancellation.
Carry observed identity to the write and retain existing SQL predicates.
- Replace orphan cancellation calls in both sweeps with conditional recoverable
settlement. Reuse `reconcileActiveSessionOnStartup` semantics and existing
abandoned-turn cleanup without importing orchestrator into task service.
- Preserve questions, deferred work, task ownership, and runtime resume identity.
Release only stale execution reservations through their current owner.
- Update root AGENTS.md stall guidance and historical orphan-reason comments.
## Out of scope
Task-open launch behavior, thresholds, explicit-stop/archive policy, and new database columns.
## Acceptance
1. Idle waiting sessions produce neither stall events nor orphan cancellations.
2. Unknown turn state skips evaluation. Mixed-session and racing-turn cases preserve healthy work.
3. Interrupted work becomes resumable waiting state in both passes. It never
becomes CANCELLED because of execution loss, and settlement never advances workflow.
## Verification
From the repository root:
```bash
(cd apps/backend && go test ./internal/task/service -run 'TestService_ActiveSessionSweep|TestService_OrphanedSessionReconciliation' -count=1)
(cd apps/backend && go test ./internal/task/repository/sqlite -run 'Test.*(CancelActiveTaskSessionsByCandidates|RecoverInterrupted)' -count=1)
(cd apps/backend && go test ./internal/orchestrator -run 'Test.*(Startup|Reconcile)' -count=1)
python3 scripts/list-docs.py validate
python3 scripts/lint-spec-files.py --all
git diff --check
```
Record the new idle test's expected failure before the production correction.
If a repository predicate changes, also run `go run ./cmd/sqlguard ./internal`
from `apps/backend` and record the result.
## Files likely touched
- `apps/backend/internal/task/service/active_session_stall.go`
- `apps/backend/internal/task/service/active_session_stall_test.go`
- `apps/backend/internal/task/models/models.go` only if candidate identity needs extension.
- `apps/backend/internal/task/repository/sqlite/session.go` and `session_test.go` only for a proven predicate gap.
- `apps/backend/internal/task/service/archived_session_reconciliation.go`
- `apps/backend/internal/task/service/orphaned_session_reconciliation_test.go`
- `apps/backend/internal/task/service/service_tasks.go` (shared settlement effects).
- `apps/backend/internal/orchestrator/service.go` (startup recovery parity, if needed).
- `apps/backend/internal/orchestrator/service_test.go` (startup recovery regressions).
- `AGENTS.md`
## Dependencies
None. Read the paired requirement and design before implementation.
## Risks
Filtering the all-active-session denominator would weaken the existing task-wide guard.
A second turn read must not replace the classified turn identity with an idle snapshot.
## Parallelism
`sequential`
## Inputs
- [Plan evidence](plan.md#confirmed-evidence)
- [Requirements](../../specs/tasks/requirements/session-stall-visibility.md)
- `active_session_stall_test.go` fixture and mixed-sibling tests.
- `idle_session_reaper.go` documents legitimate execution-less waiting sessions.
## Results
Implemented both reconciliation paths with guarded recoverable settlement.
Idle `WAITING_FOR_INPUT` sessions are excluded from stall classification, while
stale `STARTING` and `RUNNING` sessions, including sessions without a runtime
row, return to `WAITING_FOR_INPUT` after their observed unfinished turn is
abandoned. State, activity, message, turn, archive, and liveness predicates
protect the transition from races; stale executor reservations are repaired
without changing the resume identity. Settlement publishes a recoverable state
event and never emits turn completion or advances workflow state.
The service, SQLite repository, and reconciliation regression suites pass,
including idle siblings, turn read failures, missing executors, archive and
explicit-stop exclusions, and completion races.
The review remediation is included in this work order. Both sweeps now persist
the executorless recovery generation, use a fresh bounded effects context after
detached writes, and retry incomplete settlement from the session metadata.
The retry path accepts the same committed session generation after its stale
executor row was repaired, so an abandoned turn cannot lose its remaining
effects after the session becomes idle or after a successor changes the row.
Executor inventory read failures fail closed before the recovery write, and
CREATED sessions receive the same durable recovery token and interruption
marker as STARTING and RUNNING sessions.
@@ -0,0 +1,151 @@
---
id: "02-recover-orphan-conversations"
title: "Recover orphan conversations on task open"
status: done
wave: 2
depends_on:
- "01-preserve-idle-conversations"
plan: "plan.md"
requirements:
- REQ-TASKS-SESSION-STALL-VISIBILITY-001
- REQ-TASKS-RESTART-ORPHAN-SESSIONS-002
acceptance_criteria:
- AC-TASKS-SESSION-STALL-VISIBILITY-001.8
- AC-TASKS-SESSION-STALL-VISIBILITY-001.9
- AC-TASKS-SESSION-STALL-VISIBILITY-001.10
- AC-TASKS-RESTART-ORPHAN-SESSIONS-002.2
- AC-TASKS-RESTART-ORPHAN-SESSIONS-002.3
- AC-TASKS-RESTART-ORPHAN-SESSIONS-002.8
system_design:
- ../../specs/tasks/system-design/session-stall-visibility.md
- ../../specs/tasks/system-design/restart-orphaned-session-terminalization.md
---
# Task 02: Recover orphan conversations on task open
## Summary
Treat exact system orphan cancellation as eligible for normal automatic recovery.
Keep the selected conversation, existing launch guards, and failure fallback.
## In scope
- Add `TestGetTaskSessionStatus_OrphanCancelledSessionAutoResumes` before implementation.
Seed `CANCELLED`, the exact orphan reason, and a resumable executor row.
Expect `is_resumable` and `needs_resume` true. Current code returns false.
- Cover no token, missing runtime row, non-resumable runtime, missing profile,
existing live execution, archived task, deferred owner, and explicit user stop.
Use existing archive recovery tests as the pattern for same-session initialization.
- Correct the `SessionOrphanedCancelReason` comment that currently equates it with an explicit stop.
- Verify recovery launches with automatic admission, sends no prompt, and preserves
workflow ownership. Exercise repeated status/open calls and capacity refusal.
- Extend frontend hook tests for successful recovery and failure fallback.
- Add desktop and phone scenarios described in the plan using real backend eligibility.
Cover restart during an active turn, focus recovery, and exactly-once delivery
of a new message submitted during startup. Keep another interrupted task
unopened and verify it does not launch. Cover legacy cancelled rows separately.
- Update the relevant recovery section in `docs/public/tasks-and-workflows.md` at implementation,
or the current owning page found by `/docs-maintainer` if that path changes.
Describe recovery, auto-start prevention, and manual fallback as a how-to section.
## Out of scope
New recovery UI, unconditional recovery of terminal states, prompt replay,
replacement sessions, or live database mutations.
## Acceptance
1. Eligible orphan cancellations recover automatically in the same conversation, including pre-existing rows.
2. Launch guards, user stops, and real failures retain their existing behavior without automatic retry loops.
3. Desktop and phone tests prove warning removal, retained history/workspace, no prompt replay, and successful follow-up messaging.
## ASCII UI preview
UI-01 uses the [full plan preview](plan.md#ascii-ui-preview), covering AC .8-.10.
```text
Desktop: [normal startup progress] -> [transcript + composer]
Phone: [task header / session picker]
[startup -> transcript ]
[composer + bottom navigation]
Failure: existing recovery error and explicit retry controls
```
Reuse the existing mobile layout and chat scroll owner. Do not add a dialog.
Retain safe-area clearance and touch-accessible retry and send controls.
## Verification
From the repository root, install dependencies once in this worktree:
```bash
(cd apps && pnpm install --frozen-lockfile)
(cd apps/backend && go test ./internal/orchestrator -run 'TestGetTaskSessionStatus_|TestAutoResumeEligibility|Test.*Resume.*|Test.*Archive.*Resume' -count=1)
(cd apps/web && pnpm exec vitest run hooks/domains/session/use-session-resumption.test.ts hooks/domains/session/use-session-resumption.archive.test.ts hooks/domains/session/use-session-resumption.navigation.test.ts)
(cd apps/web && pnpm e2e:run --project chromium tests/session/session-resume-recovery.spec.ts -- --retries=0)
(cd apps/web && pnpm e2e:run --project mobile-chrome tests/session/mobile-session-resume-recovery.spec.ts -- --retries=0)
python3 scripts/list-docs.py validate
python3 scripts/lint-spec-files.py --all
node --test scripts/validate-public-docs.test.mjs
node scripts/validate-public-docs.mjs
git diff --check
```
Run desktop and mobile sequentially. The managed runner builds current source.
Record RED and GREEN results, actual discovered test counts, and screenshots.
If frontend source changes, run its typecheck and targeted ESLint. If new copy
is necessary, add five-language translations and run `pnpm run i18n:check`.
## Files likely touched
- `apps/backend/internal/orchestrator/task_operations.go`
- `apps/backend/internal/orchestrator/task_operations_resume_test.go`
- `apps/backend/internal/task/models/resume_safety.go`
- `apps/web/hooks/domains/session/use-session-resumption*.ts`
- `apps/web/e2e/tests/session/session-resume-recovery.spec.ts`
- `apps/web/e2e/tests/session/mobile-session-resume-recovery.spec.ts`
- `apps/web/e2e/helpers/session-resume-recovery.ts`
- The existing public task recovery documentation.
## Dependencies
Task 01 establishes recoverable interruption state and prevents idle cancellation.
## Risks
Tokenless initialization cannot restore provider-native history. Retain stored
Kandev history and use the existing fallback contract. Broad reason matching
can silently restart deliberately stopped sessions.
## Parallelism
`sequential`
## Inputs
- [Requirements](../../specs/tasks/requirements/session-stall-visibility.md)
- [Design](../../specs/tasks/system-design/session-stall-visibility.md)
- `task_operations_resume_test.go` archive-cancellation and explicit-stop cases.
- `e2e/tests/task/archived-session-recovery.spec.ts` recovery fixture pattern.
## Results
Implemented exact legacy orphan-cancellation eligibility in task-open status and
resume handling. Rows with the recorded orphan reason use the existing
same-session recovery path when their runtime is resumable; missing, tokenless,
or non-resumable runtime data uses the prompt-free fresh-start fallback. Archive,
explicit-stop, deferred-owner, live-execution, missing-profile, and capacity
guards remain authoritative. Repeated status/open evaluation is safe and does
not replay the previous prompt.
Added backend coverage for the eligibility matrix and resume guards. Existing
desktop and mobile recovery suites cover delayed startup, queued messages,
failure fallback, and same-session follow-up behavior. Public task and
configuration documentation now describes lazy focus recovery and the updated
stall behavior.
The integrated review coverage pairs the active-sweep missing-row regression
with an orchestrator regression that queries the resulting recovery state
through the normal status contract and focuses the selected conversation
without replaying its previous prompt. The existing archive, explicit-stop,
ownership, and capacity guards remain in the same path.
@@ -0,0 +1,144 @@
---
id: "03-interruption-warning"
title: "Keep interruption warnings until recovery succeeds"
status: done
wave: 3
depends_on:
- "02-recover-orphan-conversations"
plan: "plan.md"
requirements:
- REQ-TASKS-INTERRUPTED-TASK-INDICATOR-001
acceptance_criteria:
- AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.2
- AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.3
- AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.4
- AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.5
- AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.6
- AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.7
system_design:
- ../../specs/tasks/system-design/interrupted-task-indicator.md
---
# Task 03: Keep interruption warnings until recovery succeeds
## Summary
Show a warning triangle for interrupted tasks until successful agent recovery.
Preserve the marker through delayed, failed, or blocked resume attempts.
## In scope
- Change both shared interruption icon paths from red circle to warning triangle.
- Replace STARTING-based marker clearing with confirmed recovery success.
Include prompt-free recovery to WAITING_FOR_INPUT and stale callback protection.
- Extend tests for restart marking, missing executor rows, repeated restart,
failed attempts, live removal, and existing icon precedence.
- Update public recovery documentation and affected scoped guidance with implementation.
## Out of scope
Other error icons, new task states, manual warning dismissal, and new mobile controls.
## Acceptance
1. Sidebar and task cards show the warning triangle while interrupted work remains unresumed.
2. Attempts retain the marker; successful recovery clears it across clients. Stale callbacks cannot clear newer interruption state.
3. Desktop and phone coverage proves shape, warning style, accessible meaning, persistence, and recovery lifecycle.
## ASCII UI preview
UI-02, [full plan](plan.md#ascii-ui-preview), covers all criteria in this work order.
```text
Before, sidebar/card: (red !) Task title
After, sidebar/card: /!\ Task title
warning color
Phone drawer/card: /!\ Task title
During startup: [normal progress; marker retained]
After success: [normal task status] Task title
Failed attempt: [interruption retained + existing recovery error]
```
The warning occupies the existing status slot. Desktop and phone share the
semantic change, with existing layouts and scroll owners. Triangle shape and
accessible label are required; ASCII spacing is illustrative. No new touch
control is added. Use the phone task drawer as the shipped mobile exemplar.
## Verification
Write failing marker-lifecycle and shared-icon tests before implementation.
From the repository root, after the package's dependency installation:
```bash
(cd apps/backend && go test ./internal/orchestrator -run 'Test.*(Interrupted|Startup|Retracked|Reconcile)' -count=1)
(cd apps/web && pnpm exec vitest run lib/ui/state-icons.test.tsx)
(cd apps/web && pnpm run typecheck)
(cd apps/web && pnpm exec eslint lib/ui/state-icons.tsx)
(cd apps/web && pnpm e2e:run --project chromium tests/task/task-interrupted-icon.spec.ts tests/session/session-resume-recovery.spec.ts -- --retries=0)
(cd apps/web && pnpm e2e:run --project mobile-chrome tests/task/mobile-task-interrupted-icon.spec.ts tests/session/mobile-session-resume-recovery.spec.ts -- --retries=0)
python3 scripts/list-docs.py validate
python3 scripts/lint-spec-files.py --all
node --test scripts/validate-public-docs.test.mjs
node scripts/validate-public-docs.mjs
git diff --check
```
Add the new mobile file in this work order. Use the configured mobile project
and touch task selection. Extend existing desktop coverage to include actual
board cards, not only the sidebar. Capture both views and assert shape/style,
accessible label, and removal after successful recovery. Keep delayed-resume
and failure cases in the existing recovery suites. Run projects sequentially.
## Files likely touched
- `apps/backend/internal/orchestrator/event_handlers_streaming.go`
- `apps/backend/internal/orchestrator/event_handlers_streaming_test.go`
- `apps/backend/internal/orchestrator/service.go` and restart tests.
- `apps/backend/internal/task/repository/sqlite/` if conditional metadata removal is needed.
- `apps/web/lib/ui/state-icons.tsx` and `state-icons.test.tsx`
- `apps/web/e2e/tests/task/task-interrupted-icon.spec.ts`
- `apps/web/e2e/tests/task/mobile-task-interrupted-icon.spec.ts` (new)
- `apps/web/e2e/tests/session/session-resume-recovery.spec.ts`
- `apps/web/e2e/tests/session/mobile-session-resume-recovery.spec.ts`
- `docs/public/tasks-and-workflows.md`
## Dependencies
Task 02 provides the recovery path whose success clears the marker.
## Risks
Clearing on STARTING hides failed recovery. Clearing on every WAITING_FOR_INPUT
transition would also erase markers during restart reconciliation. Use confirmed
execution readiness, not state names alone. Preserve unrelated metadata keys.
## Parallelism
`sequential`
## Inputs
- [Requirements](../../specs/tasks/requirements/interrupted-task-indicator.md)
- [Design](../../specs/tasks/system-design/interrupted-task-indicator.md)
- `task-interrupted-icon.spec.ts` and `mobile-sidebar-workflow-completion-icon.spec.ts`.
## Results
Implemented warning-triangle rendering for the shared sidebar and task-card
state icon, including the accessible “Interrupted by restart” label and warning
focus styling on desktop and phone. The durable marker is retained through
STARTING entry, request acceptance, failed recovery, and stale snapshots. A
compare-and-set metadata removal clears it only after confirmed provider
recovery, so a late callback cannot erase a newer interruption marker.
Added backend marker-lifecycle and metadata-CAS tests, frontend state-icon and
workflow-snapshot tests, and desktop/mobile E2E assertions for the rendered
triangle and warning color. Typecheck and targeted lint pass.
The review remediation keeps the marker generation through failed, cancelled,
and tombstoned attempts. Empty or unreadable attempt snapshots cannot clear a
newer marker. Reconciliation publishes the committed marker through
`task.updated`, including its explicit interruption projection, so connected
clients see the warning immediately without a reload. The client task merge
also records every explicit marker update, preserving a newer false marker
after a complete interruption episode races an in-flight workflow snapshot.
+7 -5
View File
@@ -181,11 +181,13 @@ startup setting, not a database or Settings value.
sweep. When an unarchived task holds an active session with no live execution
behind it and no session events or messages for longer than this threshold,
Kandev emits a `task.stalled` event and logs a warning. After twice the
threshold of silence, the sweep cancels the orphaned sessions so the task
stops showing a phantom active session. The cancel is all-or-nothing per
task: it runs only when every active session of the task is orphaned and
past the grace window, so a session that still has a live execution blocks
the cancellation of its siblings rather than being swept along with them.
threshold of silence, the sweep returns each classified interrupted session to
`WAITING_FOR_INPUT` and preserves its conversation for recovery when you open
the task. It abandons only the observed unfinished turn, without reporting a
successful completion or advancing the workflow. Idle waiting sessions are
excluded, and recovery runs only when every active session of the task is
execution-less and past the grace window, so a live execution or idle sibling
blocks recovery for the task rather than being swept along with it.
The value is source-specific: an invalid YAML duration fails configuration
parsing, while a zero or negative YAML duration is rejected at startup with
`tasks.stallDetectionThreshold must be positive`. An invalid or non-positive
+10 -3
View File
@@ -453,14 +453,21 @@ preference applies in two situations:
prepared session and the agent stays stopped until you select **Start agent**.
Opening the same task with the preference off keeps the workflow step's
normal auto-start behavior.
- **Opening a task whose agent was stopped by a Kandev restart.** The session
is recovered and shown stopped instead of being resumed automatically. Select
**Start agent** to resume it.
- **Opening a task whose agent was interrupted by a Kandev restart.** Kandev
restores the selected session when the task opens, unless this preference is
enabled. It restores the existing conversation and workspace without
replaying the previous prompt. A new message continues that conversation,
including while the agent is starting. Opening the application does not
resume every interrupted task; each task starts recovery when you focus it.
The preference only gates opening a task. Choosing **Start agent** (or a
workflow step transition) always starts the agent as usual, and a failed or
interrupted session still shows its recovery actions.
An interrupted task keeps a warning indicator until the agent confirms
recovery. Opening the task or starting a recovery attempt does not clear the
indicator; a failed attempt keeps it visible with the existing retry actions.
## Answer clarification questions
When an agent asks a clarification question, answer it from the question panel
@@ -1,192 +1,70 @@
---
status: active
status: draft
system: tasks
created: 2026-08-02
updated: 2026-08-09
updated: 2026-09-21
owners:
- kandev
---
# Interrupted Task Indicator Requirements
# Interrupted task warning
## Overview
This document is the migrated task-system source for the capability. The source detail below remains authoritative while the system is migrated into separate requirement and design records.
A backend restart can interrupt work while preserving its conversation for recovery.
Users must be able to identify affected tasks until their agents resume.
The task system owns the durable interruption marker and its visible lifecycle.
This revision changes the existing red error-style circle to a warning triangle.
It also prevents a failed resume attempt from erasing the interruption marker.
Implementation belongs to the [recovery package](../../../plans/orphaned-session-open-recovery/plan.md).
## Requirements
### REQ-TASKS-INTERRUPTED-TASK-INDICATOR-001: Interrupted Task Indicator
### REQ-TASKS-INTERRUPTED-TASK-INDICATOR-001: Interrupted task warning
**Intent:** Preserve the observable task or workflow behavior recorded by the legacy specification.
**Intent:** Show which tasks lost running work to a restart until recovery succeeds.
#### Acceptance criteria
- **AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.1:** Superseded by criteria
001.2 through 001.7. The previous migrated criterion required a red alert
circle and cleared the marker when startup began.
- **AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.2:** When restart recovery finds
interrupted starting or running work, the system shall retain a durable
interruption marker on its unarchived task. Idle conversations and executions
that survived the restart shall not gain that marker.
- **AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.3:** Unresumed interrupted tasks
shall show a warning triangle in the sidebar and task card. The shared
interruption indicator shall use the existing warning color instead of error red.
Its accessible label shall remain "Interrupted by restart".
- **AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.4:** Opening Kandev, focusing a
task, or accepting a resume request shall not clear the marker. Failed,
cancelled, or blocked recovery attempts shall preserve it. Successful agent
recovery shall clear it and update visible task surfaces without a reload.
- **AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.5:** The marker shall survive
reload, reconnect, repeated restart, and updates that omit interruption state.
A stale callback shall not clear a newer interruption marker.
- **AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.6:** Phone task navigation and
board cards shall show the same warning and recovery lifecycle. The meaning
shall remain accessible without hover or reliance on color alone.
- **AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.7:** Existing permission,
clarification, current activity, and explicit terminal-state precedence shall
remain. Temporary startup progress shall not erase the durable warning.
Failed recovery shall leave the interruption state available alongside the
actual recovery error. Explicit failure and cancellation icons remain unchanged.
- **AC-TASKS-INTERRUPTED-TASK-INDICATOR-001.1:** When a consumer uses this capability, the system shall provide the observable behavior and exclusions documented below.
## Boundaries
## Migrated source detail
Recovery follows [the interruption contract](restart-orphaned-session-terminalization.md).
The warning does not cancel work, start an agent, replay a prompt, or advance
its workflow. Successful recovery can clear the warning before a new message
arrives; restoring the agent and continuing work are separate actions.
## Why
When Kandev crashes, is killed, or restarts while tasks are mid-turn, those
sessions are interrupted: the agent process is gone and the turn was cut short.
Startup reconciliation already flips such sessions to `WAITING_FOR_INPUT` and
the task to `REVIEW`, so after a restart an interrupted task is visually
indistinguishable from a task that finished its turn and is waiting normally.
Operators cannot tell which tasks lost work to the crash. This spec adds a
durable "interrupted" marker set at startup for tasks whose sessions were
mid-turn, surfaced as a red icon in the task list.
## What
- At startup, when reconciliation finds a session that was `STARTING` or
`RUNNING` (mid-turn) when the backend died, the owning task is marked
interrupted with a durable task metadata key `interrupted_at` holding the
reconciliation timestamp (RFC 3339 UTC).
- Sessions that were `WAITING_FOR_INPUT`, `IDLE`, `CREATED`, or terminal at
startup are not marked: they were not interrupted mid-work.
- Archived or deleted tasks are not marked (they do not appear in the active
task list).
- The task API exposes `interrupted: true` on the task DTO while the marker is
present; it is derived from the metadata key, so no schema migration is
required.
- Task list surfaces render a red alert icon beside the task while
`interrupted` is true: sidebar task rows, the mobile task-switcher drawer
(shared row rendering), kanban board cards, graph/swimlane nodes, the rich
task-list row, and the open-task state icon. The icon carries an accessible
label and tooltip ("Interrupted by restart").
- The marker clears when a session of the task next enters `STARTING` or
`RUNNING` — work has resumed, so the task is no longer interrupted — and the
cleared state is published over the existing `task.updated` event so open
clients drop the icon live.
- The red icon renders instead of the idle/done affordances only. Pending
permission, pending clarification, generating/background activity, and
running spinners keep their existing precedence; terminal `COMPLETED`,
`FAILED`, and `CANCELLED` states keep their existing icons.
## Data model
No schema change. The marker lives in the existing `tasks.metadata` JSON
column under a new reserved key:
```
MetaKeyInterruptedAt = "interrupted_at" // value: RFC 3339 UTC timestamp string
```
Task metadata is already serialized into the API task DTO and already
survives restarts. The repository's existing `SetTaskMetadataKey` /
`RemoveTaskMetadataKey` helpers (concurrent-key-safe JSON patch, both SQLite
and Postgres) are the only writers; nothing else may mutate the key.
## API surface
- `v1.Task` gains `interrupted bool json:"interrupted,omitempty"`.
- The field is computed at DTO conversion time in both task serializers
(`internal/task/dto` and `internal/task/models` ToAPI) as
`metadata["interrupted_at"] != nil`. It is not a stored column.
- The field flows through every existing task payload automatically: boot
payload, `kanban.update`, `task.created`, `task.updated`, `task.state_changed`,
and `GET /api/v1/tasks/:id`.
- No new HTTP route, WebSocket action, or event type.
## State machine
The marker follows this lifecycle:
- `unmarked`: no `interrupted_at` key.
- `marked`: `interrupted_at` present; task DTO reports `interrupted: true`.
- `cleared`: key removed; DTO reports `interrupted: false` (omitted).
Transitions:
- `unmarked -> marked` at startup reconciliation when a preserved
`executors_running` row's session was `STARTING` or `RUNNING` and the task
exists and is not archived. Overwriting an existing key is safe: a marked
task cannot have a mid-turn session at the next startup unless it was
resumed and crashed again.
- `marked -> cleared` when any session of the task transitions into
`STARTING` or `RUNNING` (launch, resume, prompt dispatch, agent-ready
wake). The transition chokepoint in the orchestrator
(`updateTaskSessionStateWithHook` / `setSessionStarting`; the implementer
verifies the funnel covers every start path) removes the key and publishes
`task.updated` only when the key was actually removed.
- Terminal task/session transitions do not clear the marker; terminal state
icons take precedence over the red icon, and archiving/deleting the task
removes the row with its metadata.
## Permissions
None. The marker is display metadata on a task the viewer can already see. It
is user-writable only through the generic task metadata update surface and has
no effect beyond the icon.
## Failure modes
- A metadata write fails during reconciliation: the marker is skipped with a
warning and startup continues; the task simply shows no icon until the next
restart.
- A metadata removal fails during a session start: the marker survives and the
next successful start transition clears it; the icon may briefly persist
behind the running spinner, which takes precedence.
- Remote executors: the backend connection is severed even if the remote agent
is still alive. The task is marked. If the remote status poll later reports
the session running again and the session transitions, the marker clears; if
not, the marker stays until the next resume. This is acceptable — from
Kandev's perspective the turn was cut off.
- An already-open client during startup reconnects to the boot payload, which
carries the marker; no startup-time event is required.
- A `task.updated` event that omits `interrupted` must not wipe a set marker in
client state (frontend merge preserves the field when the payload omits it,
mirroring `foreground_activity`).
## Persistence guarantees
- The marker is durable in `tasks.metadata` and survives any number of
restarts until a session start clears it.
- Clearing is a normal key removal; a crash between removal and the
`task.updated` publish leaves the client icon stale until the next payload,
never a corrupt row.
## Scenarios
- **GIVEN** a task whose primary session was `RUNNING` when the backend died,
**WHEN** the backend starts and reconciliation runs, **THEN** the task's
metadata contains `interrupted_at`, the task DTO reports `interrupted: true`,
and the sidebar row, board card, graph node, rich list row, and mobile
switcher row show the red alert icon.
- **GIVEN** that task also has a stored `generating` task summary, **WHEN** reconciliation changes
the primary session to `WAITING_FOR_INPUT`, **THEN** the summary clears `generating` and the red
alert icon is not hidden by a stale running spinner.
- **GIVEN** a task whose session was `WAITING_FOR_INPUT` when the backend
died, **WHEN** reconciliation runs, **THEN** the task is not marked and shows
no red icon.
- **GIVEN** an archived task with a mid-turn session row, **WHEN**
reconciliation runs, **THEN** the task is not marked.
- **GIVEN** a marked task, **WHEN** the user resumes the session and it enters
`STARTING`/`RUNNING`, **THEN** the marker is removed, `task.updated` is
published, and every surface drops the red icon (running spinners take over).
- **GIVEN** a marked task that is never resumed, **WHEN** the backend restarts
again, **THEN** the marker survives and the icon still renders.
- **GIVEN** a marked task whose state is `COMPLETED`, **WHEN** any surface
renders it, **THEN** the green done icon wins and no red icon is shown.
- **GIVEN** a `task.updated` event without an `interrupted` field for a marked
task, **WHEN** a client merges it, **THEN** the client keeps the marker.
## Out of scope
- Changing the task or session state machine (no new task state; the task
remains `REVIEW` with a `WAITING_FOR_INPUT` session after reconciliation).
- Marking sessions or tasks `FAILED` on interruption; resumability behavior is
unchanged.
- Persisting the marker as a typed column; metadata is the single source.
- Office agent cards and the Office dashboard surface, which use their own
status presentation.
- Reconstructing detached remote-work liveness after restart.
- A user-facing action to clear or dismiss the icon manually.
No new task state, manual dismiss action, or Office dashboard behavior is added.
Reuse the existing marker and translated label. This is a task-owned indicator,
not a separate UI-owned requirement.
## Related
- [Task Runtime Cleanup](../system-design/runtime-cleanup.md) — startup reconciliation semantics
the marker hooks into.
- [Sidebar Task Completion Icons](../../ui/requirements/sidebar-task-completion-icons.md) —
the shared task-row icon precedence this feature extends.
- [Backend Runtime-State Ownership](../../../plans/backend-runtime-state-ownership/plan.md) —
startup publication that clears stale generating activity.
- [System design](../system-design/interrupted-task-indicator.md)
- [Task recovery design](../system-design/restart-orphaned-session-terminalization.md)
- [Sidebar task completion icons](../../ui/requirements/sidebar-task-completion-icons.md)
@@ -6,42 +6,61 @@ owners:
- kandev
---
# Restart-Orphaned Session Terminalization Requirements
# Session recovery after backend interruption
## Overview
A task session records the lifecycle of one agent conversation turn. When the
backend process dies while a session's turn is open — a graceful restart, a
crash, or a kill — the component that would have transitioned the session to a
terminal state dies with it. No other actor retries that transition, so the
session row stays in an active state forever and the task appears to wait for
an agent that already finished.
The tasks system owns the task-session lifecycle, so it owns this
reconciliation: the system itself must detect and terminalize sessions that
lost their backing actor, without depending on the dead process, the agent, or
a human SQL edit.
## Terminology
- **Active session state:** A `task_sessions.state` value that implies work is
in flight: `CREATED`, `STARTING`, `RUNNING`, or `WAITING_FOR_INPUT`.
- **Terminal session state:** A state a session never leaves: `COMPLETED`,
`FAILED`, or `CANCELLED`.
- **Orphaned session:** An active-state session whose transition actor is gone:
the backend that would complete or fail its turn terminated, and no live
agent execution backs the session anymore.
- **Live execution:** An agent execution the current backend process tracks in
its in-memory execution store. A session backed by a live execution is not
orphaned even when its row is stale.
- **Launch grace window:** The interval between a session row entering
`STARTING` and its execution necessarily being visible in the in-memory
store. A session inside this window is not eligible for orphan detection.
- **Reconciliation sweep:** The periodic pass that lists candidate sessions
and terminalizes the orphaned ones.
A backend restart can interrupt an agent turn. The conversation remains available
for recovery. Opening Kandev must not cancel interrupted tasks or resume every
agent at once. Focusing a task restores its selected conversation. A new user
message continues the work without replaying the interrupted prompt.
## Requirements
### REQ-TASKS-RESTART-ORPHAN-SESSIONS-002: Interrupted conversations remain resumable
This requirement supersedes requirement 001's automatic cancellation policy.
- **AC-TASKS-RESTART-ORPHAN-SESSIONS-002.1:** After a backend interruption,
an unarchived session that lost its execution shall remain recoverable.
Reconciliation shall represent it as waiting for recovery, never cancelled
solely because its execution is absent or its activity is old.
- **AC-TASKS-RESTART-ORPHAN-SESSIONS-002.2:** Opening Kandev shall not start
all interrupted agents. Focusing an eligible task shall restore its selected
agent and conversation under the existing auto-start preference.
- **AC-TASKS-RESTART-ORPHAN-SESSIONS-002.3:** Focusing a task shall not send
or replay a prompt. A subsequent user message shall continue that conversation
through normal prompt delivery, including during recovery startup.
- **AC-TASKS-RESTART-ORPHAN-SESSIONS-002.4:** Recovery shall preserve session,
workspace, transcript, and available provider resume identity. Missing runtime
records shall not cause cancellation or silent conversation replacement.
- **AC-TASKS-RESTART-ORPHAN-SESSIONS-002.5:** Surviving executions, recent
launches, newer turns, and healthy siblings shall remain untouched. Unknown
liveness or failed reads shall defer reconciliation.
- **AC-TASKS-RESTART-ORPHAN-SESSIONS-002.6:** Settling an interrupted turn
shall not complete the task, advance its workflow, or report successful work.
The session shall remain available for the next message.
- **AC-TASKS-RESTART-ORPHAN-SESSIONS-002.7:** Explicit stops, archive rules,
authorization, capacity admission, and deferred-launch ownership shall retain
their existing authority. Neither sweep shall override these rules.
- **AC-TASKS-RESTART-ORPHAN-SESSIONS-002.8:** Desktop and phone shall support
focus, recovery, and follow-up messaging without an orphan-cancellation banner.
Actual recovery failures shall retain explicit retry actions and stored history.
## Interruption visibility
Interrupted tasks retain the [interruption warning](interrupted-task-indicator.md)
until successful agent recovery. The warning uses the shared triangle and
warning color on sidebar rows and task cards, including phone surfaces.
Focus or an unsuccessful resume attempt alone does not clear it.
## Historical requirement
Requirement 001 and its criteria below record the superseded cancellation policy.
They must not drive new implementation. The pending
[recovery package](../../../plans/orphaned-session-open-recovery/plan.md)
replaces that policy with requirement 002.
### REQ-TASKS-RESTART-ORPHAN-SESSIONS-001: Sessions orphaned by a backend restart reach a terminal state
**Intent:** A backend restart must not leave task sessions stuck in an active
@@ -94,10 +113,10 @@ that the task board stays truthful and the task can move on.
`is_primary` flag, so the status-summary projection keeps the durable
primary-session assignment.
## Out of scope
## Historical exclusions
- Resuming or replaying a turn that was interrupted mid-flight; the orphaned
session is terminalized, not resumed.
- The sweep does not resume or replay interrupted turns. Later task opening
follows the [orphan recovery contract](session-stall-visibility.md).
- Detecting orphans of already-archived tasks; the archived-task
reconciliation pass owns those.
- Agent-side heartbeats or adapter-side turn-state persistence.
@@ -1,5 +1,5 @@
---
status: active
status: draft
system: tasks
created: 2026-09-20
owners:
@@ -10,14 +10,15 @@ owners:
## Overview
An unarchived task whose session is in an active DB state (CREATED, STARTING,
RUNNING, WAITING_FOR_INPUT) but whose backing execution is gone — after a
backend restart, a lost executor, or a launch that never registered — is
invisible to every event-driven surface: no request path owns the session,
no actor will ever advance it, and silence produces no signal. This document
defines the observable contract for detecting that state and healing it,
closing the incident where four finished review tasks sat silently in
`WAITING_FOR_INPUT` for hours with no operator-visible signal.
Sessions with unfinished work can lose their execution after a restart or
executor failure. The system detects these stalls and settles the abandoned work.
A conversation waiting for user input can also have no execution after normal
idle cleanup. That condition is not a stall.
Opening a recoverable conversation restores its agent without requiring a
manual Resume action. The task system owns both classification and recovery.
The September 21 revision is implemented in the
[recovery plan](../../../plans/orphaned-session-open-recovery/plan.md).
## Terms
@@ -25,10 +26,13 @@ closing the incident where four finished review tasks sat silently in
or WAITING_FOR_INPUT.
- **Live execution**: an execution registered in the backend's in-memory
execution store for the session.
- **Idle conversation**: a session waiting for user input with no unfinished
turn. Its execution can be absent after idle cleanup or restart.
- **Stall candidate**: an active session that is not an idle conversation.
- **Event silence**: no persisted activity for the session — the newer of the
session row's last update and its newest persisted message.
- **Stall episode**: the continuous period during which one session is
active, execution-less, and event-silent past the stall threshold. An
a stall candidate, execution-less, and event-silent past the stall threshold. An
episode ends when the session leaves that state (terminal transition,
healing, or a live execution reappearing).
@@ -43,7 +47,7 @@ silently forever.
#### Acceptance criteria
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.1:** When an unarchived task holds
an active session with no live execution and no persisted session activity
a stall candidate with no live execution and no persisted session activity
for longer than the configurable stall threshold (default 2h), the system
shall emit a `task.stalled` event and a warning log naming the task and
its stalled sessions, at most once per session per stall episode.
@@ -51,15 +55,13 @@ silently forever.
execution, transitions to a terminal state, or otherwise shows persisted
activity again, its stall episode shall end, and a later stall of the same
session shall be reported again as a new episode.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.3:** When an active session with
no live execution has been silent for at least twice the stall threshold,
and every active session of the same task is equally orphaned and silent,
the system shall cancel those sessions with the reason `orphaned session`
and publish the session-state change event clients observe.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.4:** The healing cancellation
shall never cancel a session with a live execution, including one that
registered while the sweep was evaluating the task, and shall never cancel
a session outside the set the sweep classified as orphaned.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.3:** Superseded by
`REQ-TASKS-RESTART-ORPHAN-SESSIONS-002`. Execution loss and prolonged silence
shall lead to recoverable interruption settlement, not automatic cancellation.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.4:** The healing transition shall
never alter a session with a live execution, including one that registered
while the sweep was evaluating the task, and shall never alter a session
outside the set the sweep classified as orphaned.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.5:** When a session's activity
clock cannot be read (for example a transient database read failure), the
system shall neither report a stall nor heal for that evaluation, rather
@@ -67,6 +69,30 @@ silently forever.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.6:** When the stall detection
threshold cannot be verified against the live-execution registry, the
sweep shall do nothing rather than guess that executions are gone.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.7:** The system shall preserve idle
conversations regardless of silence duration or execution absence. It shall
not report them as stalled or cancel them as orphaned. If unfinished-turn
evidence is unavailable, classification shall wait for a later evaluation.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.8:** When a user opens a recoverable
orphan-cancelled conversation, the system shall automatically restore the agent
in that session. This applies to existing cancellations and both reconciliation
passes. Recovery shall preserve conversation history and workspace identity.
It shall not replay a prompt, create a replacement session, or advance the workflow.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.9:** Automatic recovery shall honor
auto-start prevention, archive status, authorization, capacity, and deferred
launch ownership. Explicit user stops and unrelated terminal states shall
retain their existing recovery rules. Repeated opening shall not duplicate execution.
- **AC-TASKS-SESSION-STALL-VISIBILITY-001.10:** Desktop and phone task views
shall use normal recovery progress and conversation controls. Successful recovery
shall remove the orphan warning. If recovery fails, existing recovery actions
shall remain available without an automatic retry loop.
## Related contracts
- [Session-open recovery](queued-session-ownership.md)
- [Auto-start preference](prevent-agent-autostart-on-open.md)
- [Restart reconciliation](restart-orphaned-session-terminalization.md)
- [System design](../system-design/session-stall-visibility.md)
## Exclusions
@@ -0,0 +1,75 @@
---
status: draft
system: tasks
requirements:
- REQ-TASKS-INTERRUPTED-TASK-INDICATOR-001
---
# Interrupted task warning design
## Ownership and data
Keep `tasks.metadata.interrupted_at` and the derived task DTO `interrupted` flag.
No schema, route, or event type is added. Startup reconciliation and periodic
interruption recovery mark lost mid-turn work, including missing executor rows.
Normal idle conversations and surviving executions remain unmarked.
Use concurrent-key-safe repository operations and existing archive guards.
Preserve the marker through waiting-state recovery and subsequent restarts.
Client merges retain existing interruption state when a partial payload omits it.
Successful clearing uses the existing authoritative update contract.
## Marker lifecycle
Current `event_handlers_streaming.go` clears the marker at STARTING entry.
Move that clearing to confirmed successful recovery through the existing
provider-ready or accepted-running paths. Audit every `clearTaskInterruptedMarker`
call, including `updateTaskSessionStateWithHook`, `setSessionStarting`, and
execution-correlated activity handlers. STARTING alone proves only an attempt.
A successful prompt-free resume can end in WAITING_FOR_INPUT. Clear the marker
on that success without requiring a new user prompt. Queued, refused, failed,
and cancelled attempts retain it. Existing activity spinners can temporarily
replace the warning while startup is in progress; the durable marker survives.
Use existing execution correlation and compare the observed interruption value
when clearing it. An old readiness callback must not erase a newer interruption.
Keep the update and publication retryable through the existing event path.
Publish `task.updated` only after a successful marker change. Never clear the
marker from a browser mount, navigation handler, or optimistic STARTING projection.
## Shared rendering
In `apps/web/lib/ui/state-icons.tsx`, update both `TASK_INTERRUPTED_ICON` and
`InterruptedTaskIcon`: use `IconAlertTriangle` and the existing `STYLE_WARNING`
color (`text-yellow-500`). Replace the error-colored focus ring consistently.
Keep the test ID `task-state-interrupted` and translated
`common:interruptedByRestart` label. Do not change unrelated error icons.
Consumers include sidebar rows, board cards, mobile task-switcher rows, graph
nodes, rich task lists, and the open-task header. Using the shared renderer
keeps their appearance consistent. Retain current permission, clarification,
activity, and explicit terminal-state precedence.
## Desktop and phone
Desktop keeps the icon in the current task-row and card status position.
Phone reuses the task drawer and board card status slot. No new overlay or
control is needed. Keep the accessible label and visible triangle shape.
Required meaning must not depend on tooltip hover; the task name remains primary.
Existing layout, scroll ownership, safe areas, and touch behavior remain intact.
## Verification and mapping
| Requirement | Design sections |
| --- | --- |
| REQ-TASKS-INTERRUPTED-TASK-INDICATOR-001 | Ownership and data; Marker lifecycle; Shared rendering; Desktop and phone |
Extend the marker lifecycle tests in `event_handlers_streaming_test.go` and
startup reconciliation tests. Extend `state-icons.test.tsx` to assert triangle,
warning styling, accessible label, and precedence. Browser coverage must verify
sidebar and board rendering, reload persistence, failed recovery retention,
and successful recovery removal on desktop and phone.
The [work order](../../../plans/orphaned-session-open-recovery/task-03-interruption-warning.md)
contains exact commands and the UI preview.
@@ -1,130 +1,91 @@
---
status: current
status: draft
system: tasks
requirements:
- REQ-TASKS-RESTART-ORPHAN-SESSIONS-001
- REQ-TASKS-RESTART-ORPHAN-SESSIONS-002
---
# Restart-Orphaned Session Terminalization System Design
# Session recovery after backend interruption
## Purpose and boundaries
## Ownership and control flow
The tasks system owns the task-session lifecycle, including the transition of
a session into a terminal state. This design covers the periodic sweep that
terminalizes `STARTING`/`RUNNING` sessions of unarchived tasks that no live
in-memory agent execution backs — the residue of a backend process that died
mid-turn.
The task system owns recovery state. Both reconciliation passes must preserve
conversation availability after execution loss. This replaces the cancellation
policy shipped in PR #3833 and the equivalent healing action in PR #3832.
It extends the existing reconciliation loop
(`internal/task/service/archived_session_reconciliation.go`) with a second
pass; it does not introduce a new loop, new persistence, or adapter changes.
Adjacent contracts this design uses but does not own: the agent runtime's
in-memory execution store (executors system), session-ceiling reservations
(tasks system, reservation contract), and the archived-task reconciliation
pass that owns archived tasks' stranded sessions.
Use `reconcileActiveSessionOnStartup` in `internal/orchestrator/service.go` as
the behavior precedent. It preserves runtime identity, settles abandoned turns,
and projects non-surviving active sessions as `WAITING_FOR_INPUT` for lazy recovery.
It does not replay the prompt. Surviving executions remain running.
## Requirement mapping
The periodic recovery path must also handle sessions without an executor row.
Keep the existing ticker, launch grace, runtime-aware liveness protections,
retry behavior, and per-session candidate identity. Replace orphan cancellation
with a conditional transition to recoverable `WAITING_FOR_INPUT`.
| Requirement | Design section |
## State and persistence
No new session-state enum is required. Preserve transcript, workspace, resume
token, execution identity, and primary-session assignment. Clear the current
orphan cancellation error on successful recovery. Preserve historical diagnostics.
Settle only the abandoned turn identified by the recovery snapshot. Use the
existing interruption cleanup behavior, not archive cancellation effects.
Do not publish successful `turn.completed`, advance the workflow, or cancel
healthy siblings. Retire stale execution capacity claims through existing owners.
Do not expire questions or clear queued work solely because the backend restarted.
The final write must compare observed state, activity, and turn identity.
Recheck live execution and archive/stop ownership at the transition boundary.
A completed or replaced turn, renewed activity, or concurrent explicit stop
rejects the stale recovery attempt. No database transaction spans runtime I/O.
Recovery must be retryable after partial progress without duplicating effects.
`WAITING_FOR_INPUT` without an unfinished turn is normal idle state. The active
stall sweep must neither alert nor alter it, regardless of elapsed silence.
For interrupted work, retain useful stall diagnostics and recoverable settlement.
The active pass retains its all-active-session eligibility boundary. An idle
sibling blocks that task-wide action; the restart pass remains per-session.
## Focus and message delivery
On task focus, use existing `GetTaskSessionStatus` and `session.launch` resume
flow with automatic admission. Opening the application does not eagerly start
all interrupted sessions. Focusing restores provider context but sends no prompt.
Use normal queued prompt delivery for a message sent during startup. Deliver
that new message once after readiness. Never resend the old prompt to simulate
continuation. Recover legacy `CANCELLED` plus exact orphan-reason rows through
the compatibility exception in the
[stall recovery design](session-stall-visibility.md#recovery-when-opening-a-task).
Other terminal states retain their existing rules.
When provider context cannot be restored, retain the same Kandev transcript and
workspace. Expose the existing recovery fallback without claiming native context
restoration. Honor auto-start prevention and all existing launch ownership guards.
## Interruption marker
Set or preserve `interrupted_at` for lost mid-turn work. Idle reconciliation
must not clear it. Successful correlated provider recovery clears the marker;
STARTING entry alone does not. Follow the
[warning design](interrupted-task-indicator.md) for publication, stale callbacks,
and the shared sidebar/card renderer.
## UI and verification
Reuse the current desktop chat and dedicated phone task layout. Show normal
recovery progress and the retained transcript. Use existing retry controls for
actual failures. See the [plan](../../../plans/orphaned-session-open-recovery/plan.md)
for previews and the desktop/phone test matrix.
| Requirement | Design sections |
| --- | --- |
| `REQ-TASKS-RESTART-ORPHAN-SESSIONS-001` | [Control flow](#control-flow), [Failure and recovery](#failure-and-recovery) |
| REQ-TASKS-RESTART-ORPHAN-SESSIONS-002 | Ownership and control flow; State and persistence; Focus and message delivery; UI and verification |
## Components and responsibilities
## Related decision
- **Reconciliation loop** (`Service.StartArchivedSessionReconciliationLoop`):
ticks every minute and runs both passes — the archived pass first, then the
orphan pass.
- **Orphan pass** (`Service.runOrphanedSessionReconciliation`): lists stale
candidates, filters by live-execution absence, cancels each, runs
post-cancellation effects.
- **Session repository** (`sqlite.Repository`): candidate query
`ListStaleRunningSessionsOnUnarchivedTasks` and per-session cancel
`CancelRunningTaskSessionByID`. Not part of the `SessionRepository`
interface; reached through the narrow `orphanedSessionRepository`
capability so test doubles stay unchanged.
- **Execution-liveness seam** (`TaskExecutionLivenessChecker`): reports
whether a session has a live in-memory execution. Satisfied by the
lifecycle adapter (`HasLiveExecution`), which answers from the in-memory
store only and never lazily creates an execution.
- **Shared post-cancellation effects** (`Service.notifyCancelledSessions`):
extracted from the archive cancellation path and reused verbatim:
clarification expiry, parked-projection clearing, ceiling release, and the
`session.state_changed` publish.
## Data and contracts
- Candidate query joins `task_sessions` to `tasks` and selects rows where
`ts.state IN ('STARTING','RUNNING')`, `ts.updated_at < staleBefore`, and
`t.archived_at IS NULL`, ordered by `updated_at` ascending.
- `staleBefore` is `now - (constants.AgentLaunchTimeout + 5m grace
allowance)`: one full launch budget plus the start-deadline allowance, read
(not copied) from `constants.AgentLaunchTimeout` so an operator-raised
preparation budget is honored. The same cutoff is passed into the cancel
statement and re-asserted as `updated_at < staleBefore` there, so a row
refreshed between the candidate read and the write no longer matches.
- `CancelRunningTaskSessionByID(sessionID, reason, staleBefore)` is an
`UPDATE ... RETURNING` over rows still in `STARTING`/`RUNNING` whose
`updated_at` still predates the cutoff; it returns `nil` when the session
raced to another state or was refreshed, and the write detaches from the
caller context with a 10-second timeout. The RETURNING clause selects
`is_primary` alongside the other event-payload fields, so the published
cancellation event carries the durable primary flag.
- Cancellation reason constant: `models.SessionOrphanedCancelReason`
(`"orphaned by backend restart"`), distinct from the archive reasons;
`IsArchiveCancelReason` returns false for it.
## Control flow
1. Every tick, pass 2 runs after the archived pass.
2. If no execution-liveness checker is wired, the pass is inert: absence
from the store is the only dead signal, and a nil checker can never prove
a session unbacked.
3. List candidates with the stale-before cutoff computed from the launch
budget.
4. For each candidate, re-check `HasLiveExecution(session.ID)` at the moment
of the write — a launch that raced the grace window since the read must
not be reaped.
5. Cancel via `CancelRunningTaskSessionByID`; the statement re-asserts both
the state set (`STARTING`/`RUNNING`) and the staleness cutoff
(`updated_at < staleBefore`), so a row refreshed by an in-flight launch
between the liveness check and this write matches nothing; on `nil`
(raced or refreshed) skip; on error warn and let the next tick retry.
6. Run the shared post-cancellation effects with the orphan reason.
## Failure and recovery
- A failed candidate read logs an error and returns; the next tick retries.
- A failed cancel logs a warn and continues to the next candidate; the next
tick retries that session. The sweep never gives up for as long as the
process runs.
- The cancel write is context-detached (`context.WithoutCancel` + 10s bound),
so the terminal transition survives a client disconnect and cannot block a
sweep pass indefinitely on a locked writer.
- Pass 2 is a no-op on clean trees: fresh rows are filtered by the grace
window, live-backed rows by the liveness check, archived tasks' rows by the
archived-pass filter, and terminal rows by the candidate query.
## Persistence
No schema change. The sweep writes only `task_sessions.state`,
`error_message`, `completed_at`, and `updated_at`. Survivors re-tracked by
startup recovery are in the in-memory store and stay untouched, so the sweep
composes with agent-survival re-tracking rather than racing it.
## Security
No new trust boundary. The sweep runs in-process with the task service's own
credentials and touches only `task_sessions` rows of unarchived tasks.
## Observability
- `task-session reconciliation loop started (every 1 minute)` at loop start.
- `orphaned-session reconciliation: found candidates` / `terminalized
unbacked session` (Info, with task and session IDs, previous state).
- `failed to list candidates` / `failed to cancel session` (Error/Warn with
the underlying error).
## Related decisions
- Issue #3711 and its approved fix plan: extend the existing sweep rather
than drain-on-shutdown, turn-state persistence, heartbeats, or manual SQL.
The [session-open decision](../../../decisions/2026-09-18-session-open-resumes-conversation.md)
already separates provider recovery from prompt dispatch. The user's September 21
clarification extends that behavior to interrupted running tasks. The requirement
and this design preserve the rationale without a separate ADR.
@@ -1,5 +1,5 @@
---
status: current
status: draft
system: tasks
created: 2026-09-20
owners:
@@ -32,7 +32,12 @@ Each tick:
1. `ListUnarchivedTasksWithActiveSessions` returns candidate tasks.
2. Per task, `ListActiveTaskSessionsByTaskID` loads the active sessions and
the registry reports live ones (`LiveSessionIDsForTask`); sessions
outside that snapshot are orphaned.
outside that snapshot need lifecycle classification. A missing execution
alone does not prove a stall.
For `WAITING_FOR_INPUT`, read the active turn through `GetActiveTurn`.
A nil turn identifies an idle conversation and excludes it from detection
and healing. A read error skips the task for this tick.
`IDLE` remains outside the active-session query.
3. Each orphaned session's activity clock is the newer of its row's
`updated_at` and its newest `task_session_messages` row
(`GetLastMessageTimeBySessionIDs`). A read failure skips the task this
@@ -43,13 +48,15 @@ Each tick:
5. Stalled sessions not yet reported in this episode publish one
`task.stalled` event; delivery is recorded only after the publish
succeeds, so a failed publish retries next tick (AC .1).
6. A healable set covering every active session of the task re-checks
liveness at the cancellation boundary and then cancels exactly the
classified session IDs through `finalizeCancelledSessionIDs`, reusing the
archived pass's transition: `CANCELLED` with `SessionOrphanedCancelReason`
(`"orphaned session"`), clarification expiry, parked-projection cleanup,
session-ceiling release, and one `session.state_changed` per session
(AC .3, .4).
6. A recoverable set covering every active session rechecks liveness and
settles abandoned work to waiting state under the
[interruption recovery design](restart-orphaned-session-terminalization.md).
Neither this pass nor the restart pass writes `CANCELLED` for execution loss.
Keep every active session in the task-wide eligibility denominator. An idle
sibling blocks this task-wide action. The restart pass remains per-session.
Carry the observed activity and turn identity through the conditional recovery
write. A completed or replaced turn rejects a stale recovery snapshot.
## Session activity clock
@@ -70,10 +77,10 @@ on the sweep's single goroutine and resets when no candidates remain.
- No registry wired: the pass skips entirely (AC .6).
- Activity read failure: the task is skipped this tick (AC .5).
- Cancellation write failure: retried with the archived pass's bounded
- Recovery write failure: retried with the archived pass's bounded
in-line retry, and the next tick retries again.
- A live execution appearing between classification and the cancellation
boundary aborts the heal; the ID-scoped predicate structurally excludes
- A live execution appearing between classification and the recovery
boundary aborts the transition; the ID-scoped predicate structurally excludes
sessions that became active after classification (AC .4).
## Configuration
@@ -84,12 +91,67 @@ on the sweep's single goroutine and resets when no candidates remain.
negative YAML durations are rejected at startup; invalid or non-positive
environment values fall back to `2h`.
## Recovery when opening a task
Extend `GetTaskSessionStatus` in `internal/orchestrator/task_operations.go` with
an exact orphan-reason exception for `CANCELLED`. Use
`models.SessionOrphanedCancelReason`; do not classify arbitrary cancellation
messages as recoverable. Keep archive cancellation reasons distinct.
Use the existing archive-recovery branch as the implementation pattern:
- With a supported resume token, validate the profile and return
`is_resumable=true`, `needs_resume=true`, and a distinct orphan recovery reason.
- Without a usable token, use existing same-session initialization eligibility.
Preserve stored messages and workspace identity. Do not claim provider context
was restored when the provider cannot resume it.
- Missing profiles and unavailable prerequisites retain actionable recovery
errors. Do not mark an impossible launch as automatic recovery.
- Existing live executions return the running status without a launch request.
The frontend continues through `use-session-resumption-operations.ts` and
`session.launch` with `intent=resume`. Opening sends no prompt and never invokes
the new-session action. Keep the automatic admission flag, existing request
deduplication, archive fences, preference gate, and deferred-owner checks.
On successful startup, clear the current cancellation error through the normal
session transition. Preserve historical diagnostic events.
Use existing silent workspace restoration and explicit retry after a failed
resume. A failed attempt must not remount into an endless automatic retry.
No schema migration or live-data rewrite is required: old orphan reasons are
recognized on read.
## Desktop and phone behavior
Use the existing task conversation recovery surface. Desktop retains its chat
panel. Phone retains `session-mobile-layout.tsx`, its session picker, one chat
scroll owner, fixed navigation, and safe-area handling. No new dialog or banner
is introduced. Both surfaces show ordinary startup progress followed by the
same transcript and composer. Existing recovery controls remain on actual failure.
The [implementation plan](../../../plans/orphaned-session-open-recovery/plan.md)
contains the state preview and desktop/phone regression matrix. Reuse translated
copy. Any required new copy must enter all five locale catalogs.
## Compatibility and decision
This draft corrects the classification shipped in PR #3832. The new recovery
exception also covers cancellations from PR #3833. Keep both reconciliation
passes, but replace their orphan cancellation with recoverable interruption
settlement. Archive and explicit-stop cancellation remain unchanged.
The [session-open decision](../../../decisions/2026-09-18-session-open-resumes-conversation.md)
already separates provider recovery from prompt dispatch and workflow ownership.
This local extension needs no additional architecture decision.
## Requirement mapping
| Requirement | Satisfied by |
| --- | --- |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .1, .2) | Control flow steps 4-5; Episode dedupe |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .3) | Control flow step 6 |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .4) | Cancellation boundary re-check; ID-scoped cancel |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .4) | Recovery boundary re-check; conditional session transition |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .5) | Control flow step 3 |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .6) | Failure and recovery: no registry |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .7) | Lifecycle classification and recovery boundary |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .8, .9) | Recovery when opening a task |
| REQ-TASKS-SESSION-STALL-VISIBILITY-001 (AC .10) | Desktop and phone behavior |