Files
kandev/apps/backend/internal/orchestrator/service.go
T
João Salavisa 97c1f689e4 feat(workflow): explicit step-completion signal for auto-advance (#1276)
* feat(workflow): explicit step-completion signal for auto-advance

Adds `step_complete_kandev` MCP tool + `WorkflowStep.auto_advance_requires_signal`
column so workflow steps can gate their `on_turn_complete` transitions on an
agent-emitted completion signal instead of inferring "done" from any turn-end.
See docs/decisions/0015-explicit-completion-signal-for-auto-advance.md for the
motivation, alternatives considered, and trade-offs.

* fix(workflow): address PR review feedback for explicit completion signal

- Dedup `LoadPendingStepSignal` into `internal/task/models` so the MCP
  handler (write site) and the orchestrator (read site) share one decoder.
- Gate the out-of-band subscriber on `WorkflowStep.AutoAdvanceRequiresSignal`
  so `step_complete_kandev` is not an unintended manual-advance path for
  signalless steps.
- Fail closed on step-load errors in `processOnTurnCompleteViaEngine`: a
  loader hiccup must block the transition rather than silently bypass the
  gate.
- Reject `step_complete_kandev` for terminal sessions
  (Completed / Failed / Cancelled) so the agent gets a clear error instead
  of `accepted: true` followed by a no-op.
- Surface bus-publish failures back to the agent instead of swallowing
  them — the idempotency guard makes the retry safe and prevents the
  session from getting stuck.
- New `TestOnStepCompletionSignaled` covers the subscriber paths
  (RUNNING / WAITING+match / stale step / not signal-gated).
- ADR doc fixes: payload field list now matches the wire shape, writer
  ownership is documented, the unsubstantiated 250ms-debounce claim is
  removed, and the late-signal re-close race is recorded as a deferred
  open question.
- Drop the new toggle's HelpTip in favour of an inline description — the
  shadcn tooltip portal was intercepting clicks on the agent-profile
  dropdown options in the workflow settings E2E.

* fix(workflow): round-2 review fixes for explicit completion signal

- Split `handleStepComplete` into `resolveStepCompleteTarget` +
  `publishStepCompletionEvent` helpers so the entrypoint clears funlen
  (88 → under 80 lines) and the two halves are independently testable.
- Classify session/task load errors instead of returning `NotFound` for
  every failure: use the exported `taskrepo.ErrTaskNotFound` sentinel for
  task lookups, substring-match for the session repo (no sentinel today),
  and surface anything else as `InternalError` with the raw error logged
  but not leaked to the agent — so a transient DB hiccup triggers a retry
  rather than session abandonment.
- Pin `step_complete_kandev` explicitly in the mode-specific tool-count
  tests: `Contains` in `ModeTask`, `NotContains` in `ModeConfig` and
  `ModeOffice`, so accidental dereg or cross-mode registration is caught
  by the diff instead of by a still-equal count.

* fix(workflow): round-3 review fixes + defer pipeline-editor toggle

- Defer the "Wait for agent completion signal" sub-toggle to a follow-up
  PR. Adding it to the step config card shifts the layout enough that the
  agent profile select's Radix popper opens into a sibling tooltip's hover
  path, breaking workflow-agent-profile.spec.ts on shard 7. The backend
  column ships in this PR — workflows can opt in via the YAML import or
  the MCP `update_workflow_step_kandev` tool — and the UI affordance gets
  its own iteration where the test interference can be investigated
  without blocking the rest of the change.
- Re-publish the bus event on the `already_signaled` short-circuit when
  the session is WAITING_FOR_INPUT. Without this, an agent's retry after
  the first call's publish failed would short-circuit on the persisted
  bag, never fire the subscriber, and leave the session stuck.
- handleStepComplete docstring: drop the non-existent feature-flag claim
  and the "audit entry" wording — gating is purely
  `WorkflowStep.AutoAdvanceRequiresSignal` and the bag is cleared on the
  next turn start with no separate audit trail.
- `step_complete_kandev` tool description: same fix — clarify that the
  call succeeds for non-gated steps but the signal is discarded on the
  next turn start (no audit history to query later).

* test(workflow): cover handleStepComplete + reword defer-comment

- Add `TestHandleStepComplete_*` (6 subtests, all passing) covering the
  uncovered handler paths flagged by Claude/cubic on round 3:
  missing-fields validation, session-does-not-belong-to-task ownership
  guard, terminal-session rejection (Completed/Failed/Cancelled),
  first-call happy path (bag persisted + bus payload shape pinned), dedup
  while RUNNING (no re-publish), and dedup while WAITING_FOR_INPUT (the
  retry-after-publish-failure path — must re-publish so the subscriber
  fires).
- Reword the deferred-toggle comment to drop the YAML-export claim:
  the workflow export model doesn't yet serialize
  `auto_advance_requires_signal`, so MCP `update_workflow_step_kandev`
  is the only configuration path that round-trips today. Note the
  follow-up needs to extend the export schema alongside the UI toggle.

* test(workflow): pin signaled_at + drop redundant GetTaskSession on bag-clear

- Assert `signaled_at` is present and parses as RFC3339Nano in
  `TestHandleStepComplete_FirstCallAccepted`. The field is part of the
  handler's documented response contract; pinning it catches accidental
  drop/rename regressions.
- Add `clearPendingStepSignalByID(ctx, sessionID)` as a thin DB-only
  variant of `clearPendingStepSignal` and use it from
  `executeStepTransition`, where the session struct isn't read after the
  clear. Avoids an extra `GetTaskSession` round-trip per transition.
  `clearPendingStepSignal(session)` still exists for the
  in-memory-mutation case (subscriber's stale-step path).

* fix(workflow): subscriber stale-clear guard + ADR/comment drift fixes

- onStepCompletionSignaled's stale-step path now reads the bag and only
  clears it when its current StepID matches the (stale) event's stepID.
  Previously the unconditional clear could destroy a valid signal that
  the new step's agent had just written when sessions are reused across
  steps without auto_start_agent. (Greptile P1.)
- Drop the "feature flag" phrase from WorkflowStep.AutoAdvanceRequiresSignal
  docstring — leftover from an earlier draft. The gating is purely the
  boolean field; the global flag was removed two rounds ago.
- Merge the two adjacent `if triggerOnEnter` blocks in
  executeStepTransition so the full triggerOnEnter=true path reads top-
  to-bottom in one place. (CodeRabbit + cubic.)
- ADR 0015 Mechanism section: re-label the SessionStepHistory.Metadata
  audit annotation, the `step_complete_signaled` system message, and the
  expvar telemetry counters as deferred. They're planned shapes, not
  current behaviour — calling them "Audit" / "Chat visibility" /
  "Telemetry" in present tense was misleading for future readers.

* fix(workflow): drop subscriber complexity + reload session on stale-clear

- Extract `parseStepCompletionEvent` and `stepIsSignalGated` helpers from
  `onStepCompletionSignaled`. Cyclomatic complexity goes 17 → under 15
  (cyclop on new code, CI lint regression after the round-6
  stale-clear guard).
- The stale-step branch now re-loads the session from the DB before
  deciding whether to clear the bag. The previous round-6 fix checked
  the local `session.Metadata` snapshot, which can be older than a
  concurrent write from the new step's agent — that check could see a
  stale signal that had since been overwritten with a valid one and
  erase the new bag entry. The narrower load-on-clear is cheap and
  closes the cubic P1 hole. (The broader race in
  `processOnTurnCompleteViaEngine` reading session.Metadata is the
  monotonic-marker open question in ADR 0015; out of scope here.)

* test(workflow): pin stale-event guard's preserve-current-bag side + dedup StringFromAny

- Add `TestOnStepCompletionSignaled` subtest "stale event → valid bag
  for CURRENT step is preserved" — fires a stale step-A event while
  the bag holds a valid step-B signal, asserts the bag still contains
  StepID=step_b after the subscriber runs. Pins the negative side of
  the round-6 `existing.StepID == stepID` guard, so a regression that
  drops or inverts the guard (and would silently leave signal-gated
  steps stuck) fails the test instead of slipping through.
- Promote `pendingSignalString` to exported `models.StringFromAny`
  and reuse it from the orchestrator subscriber's
  `parseStepCompletionEvent`. Eliminates the byte-for-byte duplicate
  helper across the two packages; the bag-shape decoder and the bus-
  event decoder now share one source of truth for "missing string".

---------

Co-authored-by: Kandev Agent <agent@kandev.dev>
2026-06-05 18:32:27 +01:00

1241 lines
50 KiB
Go

// Package orchestrator provides the main orchestrator service that coordinates
// task execution across agents. It manages:
//
// - Task queuing and scheduling via the Scheduler
// - Agent lifecycle through the AgentManager
// - Event handling and propagation
// - Session management and resume
//
// The orchestrator acts as the central coordinator between the task service,
// agent lifecycle manager, and the event bus.
package orchestrator
import (
"context"
"errors"
"regexp"
"strings"
"sync"
"time"
"go.uber.org/zap"
"github.com/kandev/kandev/internal/agentctl/types/streams"
"github.com/kandev/kandev/internal/common/logger"
"github.com/kandev/kandev/internal/events/bus"
"github.com/kandev/kandev/internal/gitlab"
"github.com/kandev/kandev/internal/orchestrator/executor"
"github.com/kandev/kandev/internal/orchestrator/messagequeue"
"github.com/kandev/kandev/internal/orchestrator/queue"
"github.com/kandev/kandev/internal/orchestrator/scheduler"
"github.com/kandev/kandev/internal/orchestrator/watcher"
"github.com/kandev/kandev/internal/secrets"
"github.com/kandev/kandev/internal/task/models"
"github.com/kandev/kandev/internal/workflow/engine"
wfmodels "github.com/kandev/kandev/internal/workflow/models"
"github.com/kandev/kandev/internal/worktree"
v1 "github.com/kandev/kandev/pkg/api/v1"
)
// Common errors
var (
ErrServiceAlreadyRunning = errors.New("service is already running")
ErrServiceNotRunning = errors.New("service is not running")
)
// ServiceConfig holds orchestrator service configuration
type ServiceConfig struct {
Scheduler scheduler.SchedulerConfig
QueueSize int
QueueGroup string
}
// DefaultServiceConfig returns default configuration
func DefaultServiceConfig() ServiceConfig {
return ServiceConfig{
Scheduler: scheduler.DefaultSchedulerConfig(),
QueueSize: 1000,
QueueGroup: "orchestrator",
}
}
// MessageCreator is an interface for creating messages on tasks
type MessageCreator interface {
CreateAgentMessage(ctx context.Context, taskID, content, agentSessionID, turnID string) error
CreateUserMessage(ctx context.Context, taskID, content, agentSessionID, turnID string, metadata map[string]interface{}) error
// CreateToolCallMessage creates a message for a tool call.
// normalized contains the typed tool payload data.
// parentToolCallID is the parent Task tool call ID for subagent nesting (empty for top-level).
CreateToolCallMessage(ctx context.Context, taskID, toolCallID, parentToolCallID, title, status, agentSessionID, turnID string, normalized *streams.NormalizedPayload) error
// UpdateToolCallMessage updates a tool call message's status and optionally its normalized data.
// If the message doesn't exist, it creates it using taskID, turnID, and msgType.
// normalized contains the typed tool payload data.
// parentToolCallID is the parent Task tool call ID for subagent nesting (empty for top-level).
UpdateToolCallMessage(ctx context.Context, taskID, toolCallID, parentToolCallID, status, result, agentSessionID, title, turnID, msgType string, normalized *streams.NormalizedPayload) error
CreateSessionMessage(ctx context.Context, taskID, content, agentSessionID, messageType, turnID string, metadata map[string]interface{}, requestsInput bool) error
CreatePermissionRequestMessage(ctx context.Context, taskID, sessionID, pendingID, toolCallID, title, turnID string, options []map[string]interface{}, actionType string, actionDetails map[string]interface{}) (string, error)
UpdatePermissionMessage(ctx context.Context, sessionID, pendingID string, status models.PermissionStatus) error
// CreateAgentMessageStreaming creates a new agent message with a pre-generated ID for streaming updates
CreateAgentMessageStreaming(ctx context.Context, messageID, taskID, content, agentSessionID, turnID string) error
// AppendAgentMessage appends additional content to an existing streaming message
AppendAgentMessage(ctx context.Context, messageID, additionalContent string) error
// CreateThinkingMessageStreaming creates a new thinking message with a pre-generated ID for streaming updates
CreateThinkingMessageStreaming(ctx context.Context, messageID, taskID, content, agentSessionID, turnID string) error
// AppendThinkingMessage appends additional content to an existing streaming thinking message
AppendThinkingMessage(ctx context.Context, messageID, additionalContent string) error
// InvalidateModelCache clears any cached model for a session, forcing the next
// message to re-read the model from the DB. Called after model switches.
InvalidateModelCache(sessionID string)
}
// TurnService is an interface for managing session turns
type TurnService interface {
StartTurn(ctx context.Context, sessionID string) (*models.Turn, error)
CompleteTurn(ctx context.Context, turnID string) error
GetActiveTurn(ctx context.Context, sessionID string) (*models.Turn, error)
// AbandonOpenTurns buries any open turns for a session by setting
// completed_at = started_at (zero duration), so a subsequent prompt starts
// a fresh turn instead of adopting one that was orphaned by a crash or
// restart. Used on session resume.
AbandonOpenTurns(ctx context.Context, sessionID string) error
}
// TaskEventPublisher is the orchestrator's collaborator for publishing
// task.updated events. Implemented by the task service (which owns the rich
// payload build — session counts, primary session info, repositories,
// metadata, parent_id). The orchestrator does not construct task.updated
// payloads itself.
type TaskEventPublisher interface {
PublishTaskUpdated(ctx context.Context, task *models.Task)
}
// WorkflowStepGetter retrieves workflow step information for prompt building.
type WorkflowStepGetter interface {
GetStep(ctx context.Context, stepID string) (*wfmodels.WorkflowStep, error)
GetNextStepByPosition(ctx context.Context, workflowID string, currentPosition int) (*wfmodels.WorkflowStep, error)
GetPreviousStepByPosition(ctx context.Context, workflowID string, currentPosition int) (*wfmodels.WorkflowStep, error)
GetWorkflowAgentProfileID(ctx context.Context, workflowID string) (string, error)
}
// repoStore is the repository interface accepted by NewService.
// It covers both the orchestrator's own needs (sessionExecutorStore) and
// the executor package's needs (executor.executorStore).
type repoStore interface {
sessionExecutorStore
// Additional methods needed by executor
UpdateTaskState(ctx context.Context, id string, state v1.TaskState) error
GetPrimaryTaskRepository(ctx context.Context, taskID string) (*models.TaskRepository, error)
ListTaskRepositories(ctx context.Context, taskID string) ([]*models.TaskRepository, error)
CreateTaskSession(ctx context.Context, session *models.TaskSession) error
UpdateTaskSession(ctx context.Context, session *models.TaskSession) error
ListActiveTaskSessions(ctx context.Context) ([]*models.TaskSession, error)
ListActiveTaskSessionsByTaskID(ctx context.Context, taskID string) ([]*models.TaskSession, error)
CreateTaskSessionWorktree(ctx context.Context, sessionWorktree *models.TaskSessionWorktree) error
ListSessionsWithBranches(ctx context.Context) ([]models.SessionBranchInfo, error)
GetRepository(ctx context.Context, id string) (*models.Repository, error)
UpdateRepository(ctx context.Context, repository *models.Repository) error
ListRepositories(ctx context.Context, workspaceID string) ([]*models.Repository, error)
GetExecutorProfile(ctx context.Context, id string) (*models.ExecutorProfile, error)
// Multi-repo task environment children
CreateTaskEnvironmentRepo(ctx context.Context, repo *models.TaskEnvironmentRepo) error
ListTaskEnvironmentRepos(ctx context.Context, envID string) ([]*models.TaskEnvironmentRepo, error)
// Session history + plan (for context handover)
GetTaskPlan(ctx context.Context, taskID string) (*models.TaskPlan, error)
}
// sessionExecutorStore is the minimal repository interface needed by the orchestrator service.
type sessionExecutorStore interface {
// Session
GetTaskSession(ctx context.Context, id string) (*models.TaskSession, error)
GetActiveTaskSessionByTaskID(ctx context.Context, taskID string) (*models.TaskSession, error)
SetSessionPrimary(ctx context.Context, sessionID string) error
UpdateTaskSession(ctx context.Context, session *models.TaskSession) error
UpdateTaskSessionState(ctx context.Context, id string, state models.TaskSessionState, errorMessage string) error
UpdateTaskSessionBaseCommit(ctx context.Context, id string, baseCommitSHA string) error
GetTaskSessionByTaskAndAgent(ctx context.Context, taskID, agentInstanceID string) (*models.TaskSession, error)
UpdateTaskSessionWorktreeBranch(ctx context.Context, sessionID, branch string) error
UpdateSessionReviewStatus(ctx context.Context, sessionID string, status string) error
UpdateSessionMetadata(ctx context.Context, sessionID string, metadata map[string]interface{}) error
SetSessionMetadataKey(ctx context.Context, sessionID, key string, value interface{}) error
// Executor running state
ListExecutorsRunning(ctx context.Context) ([]*models.ExecutorRunning, error)
UpsertExecutorRunning(ctx context.Context, running *models.ExecutorRunning) error
GetExecutorRunningBySessionID(ctx context.Context, sessionID string) (*models.ExecutorRunning, error)
DeleteExecutorRunningBySessionID(ctx context.Context, sessionID string) error
HasExecutorRunningRow(ctx context.Context, sessionID string) (bool, error)
UpdateResumeToken(ctx context.Context, sessionID, expectedExecID, resumeToken, lastMessageUUID string) error
UpdateExecutorRunningStatus(ctx context.Context, sessionID, status string) error
// Executor
GetExecutor(ctx context.Context, id string) (*models.Executor, error)
// Task
GetTask(ctx context.Context, id string) (*models.Task, error)
UpdateTask(ctx context.Context, task *models.Task) error
// Git snapshots and commits
GetLatestGitSnapshot(ctx context.Context, sessionID string) (*models.GitSnapshot, error)
CreateGitSnapshot(ctx context.Context, snapshot *models.GitSnapshot) error
DeleteLiveMonitorSnapshots(ctx context.Context, sessionID string) error
UpsertLatestLiveGitSnapshot(ctx context.Context, snapshot *models.GitSnapshot) error
CreateSessionCommit(ctx context.Context, commit *models.SessionCommit) error
GetSessionCommits(ctx context.Context, sessionID string) ([]*models.SessionCommit, error)
DeleteSessionCommit(ctx context.Context, id string) error
// Session listing + delete
ListTaskSessions(ctx context.Context, taskID string) ([]*models.TaskSession, error)
ListNonTerminalSessionsByAgentInstance(ctx context.Context, agentInstanceID string) ([]*models.TaskSession, error)
DeleteTaskSession(ctx context.Context, id string) error
// Messages — used by resume to backfill the initial user prompt when a
// prior launch failed before recordInitialMessage ran.
ListMessages(ctx context.Context, sessionID string) ([]*models.Message, error)
// Pending clarification rows — durable guard for on_turn_complete while the user is answering.
FindPendingClarificationMessagesBySessionID(ctx context.Context, sessionID string) ([]*models.Message, error)
// Workspace
GetWorkspace(ctx context.Context, id string) (*models.Workspace, error)
// Task environment
GetTaskEnvironment(ctx context.Context, id string) (*models.TaskEnvironment, error)
GetTaskEnvironmentByTaskID(ctx context.Context, taskID string) (*models.TaskEnvironment, error)
CreateTaskEnvironment(ctx context.Context, env *models.TaskEnvironment) error
UpdateTaskEnvironment(ctx context.Context, env *models.TaskEnvironment) error
}
// Service is the main orchestrator service
type Service struct {
config ServiceConfig
logger *logger.Logger
eventBus bus.EventBus
taskRepo scheduler.TaskRepository
repo sessionExecutorStore
agentManager executor.AgentManagerClient
// Components
queue *queue.TaskQueue
executor *executor.Executor
scheduler *scheduler.Scheduler
watcher *watcher.Watcher
// Message queue service for queueing messages while agent is running
messageQueue *messagequeue.Service
// Message creator for saving agent responses
messageCreator MessageCreator
// Turn service for managing session turns
turnService TurnService
// Task event publisher for emitting task.updated events.
// Task service owns the rich payload; orchestrator delegates.
taskEvents TaskEventPublisher
// Workflow step getter for prompt building
workflowStepGetter WorkflowStepGetter
// Workflow engine for typed state-machine evaluation of step transitions
workflowEngine *engine.Engine
workflowStore *workflowStore
// engineOptions are applied each time initWorkflowEngine runs. Wired
// from cmd/kandev (Phase 3.2) to plug Phase 2 ADR-0004 dependencies
// — RunQueueAdapter, ParticipantStore, DecisionStore, and the CEO /
// Primary agent resolvers — without coupling the orchestrator to
// the office or runs packages.
engineOptions []engine.Option
// Phase 2 (ADR-0004) callback dependencies, set via dedicated
// setters from cmd/kandev. When set, buildWorkflowCallbacks
// registers the queue_run / clear_decisions /
// queue_run_for_each_participant callbacks with these
// dependencies. Nil-safe: missing deps simply skip registration.
engineRunQueue engine.RunQueueAdapter
engineParticipants engine.ParticipantStore
engineDecisions engine.DecisionStore
enginePrimary engine.PrimaryAgentResolver
engineCEOResolver engine.CEOAgentResolver
// Phase 8 dependencies — also nil-safe.
engineTaskCreator engine.TaskCreator
engineWorkflowSwitcher engine.WorkflowSwitcher
// GitHub service for PR auto-detection on push
githubService GitHubService
// Office task-handoffs materializer (phase 6 wiring) — invoked from
// PrepareTaskSession to flip workspace groups to materialized once
// their owner task has a session with worktree details. Optional;
// nil-safe.
workspaceMaterializer WorkspaceMaterializer
// Review task creator for auto-creating tasks from review watch PRs
reviewTaskCreator ReviewTaskCreator
// Issue task creator for auto-creating tasks from issue watch events
issueTaskCreator IssueTaskCreator
// Watcher dispatch coordinator: single seam for the watcher→task pipeline.
// Constructed lazily once issueTaskCreator is wired (see SetIssueTaskCreator).
watcherCoordinator *WatcherDispatchCoordinator
// Watcher throttle state. watcherTaskCount counts committed open watcher-
// created tasks (DB-backed source of truth across restarts). pendingByWatch
// tracks in-process events whose dedup row has not yet been written —
// without it, a burst of events read the same stale COUNT(*) before any
// goroutine commits, and the cap is silently overshot. Both reads and the
// pending increment happen under watcherMu to prevent the burst race.
watcherTaskCount WatcherTaskCounter
watcherMu sync.Mutex
pendingByWatch map[string]int
// watcherSaturated tracks per-watch whether the last gate result was
// "deferred", so we log the state-transition Warn ("cap reached" /
// "cap cleared") only once per transition instead of every event.
watcherSaturated map[string]bool
// profileLookup answers "is this agent profile still live?" for the
// dispatch pre-flight. Set via SetProfileLookup from main; nil-safe so
// the legacy code path (and tests without profile wiring) keep working.
profileLookup ProfileLookup
// Jira service for issue watch dedup operations
jiraService JiraService
// jiraSource adapts jiraService onto WatcherSource. Built once in
// SetJiraService so handlers don't allocate per bus event.
jiraSource *JiraWatcherSource
// Linear service for issue watch dedup operations
linearService LinearService
// linearSource adapts linearService onto WatcherSource. Built once in
// SetLinearService so handlers don't allocate per bus event.
linearSource *LinearWatcherSource
// Sentry service for issue watch dedup operations
sentryService SentryService
// sentrySource adapts sentryService onto WatcherSource. Built once in
// SetSentryService so handlers don't allocate per bus event.
sentrySource *SentryWatcherSource
// GitLab service + task creators for auto-creating tasks from review /
// issue watch events. When the task creators are nil the events are
// logged but no tasks are created — matches the GitHub flow when a
// workspace has no task creator wired.
gitlabService *gitlab.Service
gitlabReviewTaskCreator GitLabReviewTaskCreator
gitlabIssueTaskCreator GitLabIssueTaskCreator
// Repository resolver for cloning + finding/creating repos for review tasks
repositoryResolver RepositoryResolver
// Automation service for handling automation triggers
automationService AutomationService
// Worktree manager — used to clean up ephemeral worktrees for run-mode
// automation tasks immediately on completion rather than waiting for
// the 24h Office GC. Nil-safe.
worktreeMgr *worktree.Manager
// Clarification canceller — cancels pending clarifications when agent's turn completes
clarificationCanceller ClarificationCanceller
// Push tracker: "<sessionID>|<repository_name>" -> last known ahead count.
// The repository_name segment is required for multi-repo: each repo emits
// its own git status events, and keying by sessionID alone made events
// from different repos overwrite each other's ahead counts (so only one
// push got detected per session). Single-repo / repo-less sessions key
// against an empty repository_name.
pushTracker sync.Map
// gitSnapshotCache throttles per-session writes of the live git status
// snapshot (used by handleGitStatusUpdate -> persistGitStatusSnapshot).
gitSnapshotCache *gitSnapshotCache
// Active turns map: sessionID -> turnID
activeTurns sync.Map
// Session reset flags: sessionID -> true while resetAgentContext is restarting process.
// Used to suppress stale ready events and avoid draining queued prompts mid-reset.
resetInProgressSessions sync.Map
// suppressToast: sessionID -> true. Set by failure handlers that create
// guidance messages with actions. Cleared by updateTaskSessionState which
// adds suppress_toast to the WS event so the frontend skips the error toast.
suppressToast sync.Map
// clarificationWatchdogs tracks active clarification primary-path resume watchdogs.
// key: "<session_id>::<pending_id>", value: *clarificationWatchdogEntry
clarificationWatchdogs sync.Map
// clarificationWatchdogTimeout controls how long to wait for post-clarification
// activity before triggering fallback resume.
clarificationWatchdogTimeout time.Duration
// cancelInFlight tracks sessionIDs whose CancelAgent call is currently in
// progress. It deduplicates impatient retries from the UI and lets late
// agent.ready/boot_ready events from the cancelled turn return before they
// evaluate workflow transitions or drain queued messages.
cancelInFlight sync.Map
// transientRetries tracks in-progress transient-provider-error (529
// Overloaded) retry loops. key: sessionID, value: *transientRetryEntry.
// A backoff timer per session re-drives the failed prompt; cancelled on
// success, user-cancel, or service shutdown.
transientRetries sync.Map
// lastTurnPrompt caches the most recent outbound prompt per session so a
// transient-failure retry can re-drive the same turn without the caller's
// context. key: sessionID, value: capturedPrompt. Replaced every turn.
lastTurnPrompt sync.Map
// Service state
mu sync.RWMutex
running bool
startedAt time.Time
}
// Status contains orchestrator status information
type Status struct {
Running bool `json:"running"`
ActiveAgents int `json:"active_agents"`
QueuedTasks int `json:"queued_tasks"`
TotalProcessed int64 `json:"total_processed"`
TotalFailed int64 `json:"total_failed"`
UptimeSeconds int64 `json:"uptime_seconds"`
LastHeartbeat time.Time `json:"last_heartbeat"`
}
// NewService creates a new orchestrator service. msgQueue is the persistent
// message queue service (constructed externally so cmd/kandev can wire it to
// the shared SQLite pool); pass nil to default to an in-memory queue, which
// suits tests but loses entries on restart.
func NewService(
cfg ServiceConfig,
eventBus bus.EventBus,
agentManager executor.AgentManagerClient,
taskRepo scheduler.TaskRepository,
repo repoStore,
shellPrefs executor.ShellPreferenceProvider,
secretStore secrets.SecretStore,
msgQueue *messagequeue.Service,
log *logger.Logger,
) *Service {
svcLogger := log.WithFields(zap.String("component", "orchestrator"))
// Create the task queue with configured size
taskQueue := queue.NewTaskQueue(cfg.QueueSize)
// Create the executor with the agent manager client and repository for persistent sessions
execCfg := executor.ExecutorConfig{
ShellPrefs: shellPrefs,
SecretStore: secretStore,
}
exec := executor.NewExecutor(agentManager, repo, log, execCfg)
// Create the scheduler with queue, executor, and task repository
sched := scheduler.NewScheduler(taskQueue, exec, taskRepo, log, cfg.Scheduler)
if msgQueue == nil {
msgQueue = messagequeue.NewServiceMemory(log)
}
// Create the service (watcher will be created after we have handlers)
s := &Service{
config: cfg,
logger: svcLogger,
eventBus: eventBus,
taskRepo: taskRepo,
repo: repo,
agentManager: agentManager,
queue: taskQueue,
executor: exec,
scheduler: sched,
messageQueue: msgQueue,
clarificationWatchdogTimeout: 15 * time.Second,
gitSnapshotCache: newGitSnapshotCache(),
}
// Wire executor state changes through the orchestrator so events are published
// (e.g. WebSocket notifications to the frontend). Must be set after service
// construction so the session callback can reference s.updateTaskSessionState.
exec.SetOnTaskStateChange(func(ctx context.Context, taskID string, state v1.TaskState) error {
return taskRepo.UpdateTaskState(ctx, taskID, state)
})
exec.SetOnSessionStateChange(func(ctx context.Context, taskID, sessionID string, state models.TaskSessionState, errorMessage string) error {
s.updateTaskSessionState(ctx, taskID, sessionID, state, errorMessage, true)
return nil
})
exec.SetOnLaunchFailed(s.handleSessionLaunchFailed)
exec.SetOnAgentStartFailed(s.handleAgentStartFailed)
if caps, ok := agentManager.(executor.ExecutorTypeCapabilities); ok {
exec.SetCapabilities(caps)
}
// Create the watcher with event handlers that wire everything together
handlers := watcher.EventHandlers{
OnTaskDeleted: s.handleTaskDeleted,
OnAgentRunning: s.handleAgentRunning,
OnAgentBootReady: s.handleAgentBootReady,
OnAgentReady: s.handleAgentReady,
OnAgentCompleted: s.handleAgentCompleted,
OnAgentFailed: s.handleAgentFailed,
OnAgentStopped: s.handleAgentStopped,
OnAgentStreamEvent: s.handleAgentStreamEvent,
OnACPSessionCreated: s.handleACPSessionCreated,
OnPermissionRequest: s.handlePermissionRequest,
OnGitEvent: s.handleGitEvent,
OnContextWindowUpdated: s.handleContextWindowUpdated,
OnTaskMoved: s.handleTaskMoved,
}
s.watcher = watcher.NewWatcher(eventBus, handlers, cfg.QueueGroup, log)
return s
}
// SetMessageCreator sets the message creator for saving agent responses to the database.
//
// This must be called before starting the orchestrator if you want agent messages, tool calls,
// and streaming content to be persisted to the database. The MessageCreator interface provides
// methods for creating and updating messages associated with task sessions.
//
// The MessageCreator is typically the task service, which owns the message persistence logic.
// Event handlers in the orchestrator call these methods when agent events occur:
// - AgentStreamEvent → CreateAgentMessage, AppendAgentMessage
// - Tool calls → CreateToolCallMessage, UpdateToolCallMessage
// - Permission requests → CreatePermissionRequestMessage
//
// When to call: During orchestrator initialization, after creating the task service.
//
// If not set: Agent messages won't be saved to the database (events will still be published).
func (s *Service) SetMessageCreator(mc MessageCreator) {
s.messageCreator = mc
}
// SetOnPrimarySessionSet sets a callback on the executor for when the first session
// of a task is marked primary. Used to publish a task.updated event so the frontend
// receives the primary_session_id.
func (s *Service) SetOnPrimarySessionSet(fn executor.PrimarySessionSetFunc) {
s.executor.SetOnPrimarySessionSet(fn)
}
// SetRepoCloner sets the repository cloner and updater on the executor, enabling automatic
// cloning of provider-backed repositories (e.g. from a GitHub URL) when they are launched
// for local/worktree execution and have no local path yet.
func (s *Service) SetRepoCloner(cloner executor.RepoCloner, updater executor.RepoUpdater) {
s.executor.SetRepoCloner(cloner, updater)
}
// SetTurnService sets the turn service for tracking conversation turns.
//
// A "turn" represents a single conversation round-trip: user prompt → agent response.
// The TurnService tracks turn timing and duration for analytics and UI display (e.g., showing
// how long each agent response took).
//
// The TurnService is typically the task service, which owns turn persistence logic.
// The orchestrator calls these methods:
// - StartTurn: When agent begins processing a prompt
// - CompleteTurn: When agent finishes and returns to ready state
// - GetActiveTurn: To associate messages with current turn
//
// When to call: During orchestrator initialization, after creating the task service.
//
// If not set: Turns won't be tracked (orchestrator continues functioning normally, but
// no timing data is recorded and turn IDs in messages will be empty).
func (s *Service) SetTurnService(turnService TurnService) {
s.turnService = turnService
}
// SetTaskEventPublisher wires the publisher used for task.updated events.
//
// The task service is the canonical publisher: it loads session counts,
// primary session info, repositories, metadata, and parent_id, and emits a
// single rich payload. Orchestrator code paths that mutate a task (workflow
// transitions, primary session assignment, workflow-step moves) delegate to
// this publisher instead of constructing their own partial payloads.
//
// If not set: orchestrator task.updated publishers no-op (nothing is emitted
// on those paths). Task service's own publishTaskEvent calls are unaffected.
func (s *Service) SetTaskEventPublisher(publisher TaskEventPublisher) {
s.taskEvents = publisher
}
// publishTaskUpdated forwards to the configured TaskEventPublisher.
// No-op when the publisher isn't wired (tests, or before SetTaskEventPublisher
// has been called during startup).
func (s *Service) publishTaskUpdated(ctx context.Context, task *models.Task) {
if s.taskEvents == nil || task == nil {
return
}
s.taskEvents.PublishTaskUpdated(ctx, task)
}
// SetWorkflowStepGetter sets the workflow step getter for prompt building.
//
// When workflow_step_id is provided to StartTask, the orchestrator uses this getter
// to retrieve the step's prompt_prefix, prompt_suffix, and plan_mode settings to
// build the effective prompt.
//
// If not set: workflow_step_id in StartTask is ignored and the prompt is used as-is.
func (s *Service) SetWorkflowStepGetter(getter WorkflowStepGetter) {
s.workflowStepGetter = getter
s.initWorkflowEngine()
}
// ClarificationCanceller detaches in-memory clarification waiters when an agent's
// turn completes while questions are still pending in the DB.
type ClarificationCanceller interface {
DetachSessionAndNotify(ctx context.Context, sessionID string) int
}
// SetClarificationCanceller sets the canceller for cleaning up pending clarifications on turn complete.
func (s *Service) SetClarificationCanceller(c ClarificationCanceller) {
s.clarificationCanceller = c
}
// initWorkflowEngine creates the workflow engine with store and callbacks.
// Called after the workflow step getter is set. Safe to call multiple times —
// previously-applied engine options are re-applied on reinit.
func (s *Service) initWorkflowEngine() {
if s.workflowStepGetter == nil {
return
}
store := newWorkflowStore(s.repo, s.workflowStepGetter, s.agentManager, s.publishTaskUpdated, s.logger)
callbacks := buildWorkflowCallbacks(s)
s.workflowStore = store
s.workflowEngine = engine.New(store, callbacks, s.engineOptions...)
}
// SetEngineRunQueue wires the engine's RunQueueAdapter dependency. Used
// both as an engine option (so the engine can resolve queue_run targets)
// and as a callback dependency (so QueueRunCallback can enqueue runs).
func (s *Service) SetEngineRunQueue(adapter engine.RunQueueAdapter) {
s.engineRunQueue = adapter
s.engineOptions = append(s.engineOptions, engine.WithRunQueue(adapter))
s.reinitWorkflowEngine()
}
// SetEngineParticipantStore wires the engine's ParticipantStore.
func (s *Service) SetEngineParticipantStore(store engine.ParticipantStore) {
s.engineParticipants = store
s.engineOptions = append(s.engineOptions, engine.WithParticipantStore(store))
s.reinitWorkflowEngine()
}
// SetEngineDecisionStore wires the engine's DecisionStore.
func (s *Service) SetEngineDecisionStore(store engine.DecisionStore) {
s.engineDecisions = store
s.engineOptions = append(s.engineOptions, engine.WithDecisionStore(store))
s.reinitWorkflowEngine()
}
// SetEngineCEOResolver wires the engine's CEOAgentResolver.
func (s *Service) SetEngineCEOResolver(resolver engine.CEOAgentResolver) {
s.engineCEOResolver = resolver
s.engineOptions = append(s.engineOptions, engine.WithCEOAgentResolver(resolver))
s.reinitWorkflowEngine()
}
// SetPrimaryAgentResolver wires the engine's PrimaryAgentResolver. The
// engine has no constructor option for this — the resolver is consumed
// by QueueRunCallback at registration time, so the orchestrator
// re-runs initWorkflowEngine to pick it up.
func (s *Service) SetPrimaryAgentResolver(resolver engine.PrimaryAgentResolver) {
s.enginePrimary = resolver
s.reinitWorkflowEngine()
}
// SetEngineTaskCreator wires the engine's TaskCreator dependency for
// the create_child_task action. The orchestrator captures it both as an
// engine option and as a callback dependency.
func (s *Service) SetEngineTaskCreator(creator engine.TaskCreator) {
s.engineTaskCreator = creator
s.engineOptions = append(s.engineOptions, engine.WithTaskCreator(creator))
s.reinitWorkflowEngine()
}
// SetEngineWorkflowSwitcher wires the engine's WorkflowSwitcher for the
// switch_workflow action.
func (s *Service) SetEngineWorkflowSwitcher(switcher engine.WorkflowSwitcher) {
s.engineWorkflowSwitcher = switcher
s.engineOptions = append(s.engineOptions, engine.WithWorkflowSwitcher(switcher))
s.reinitWorkflowEngine()
}
func (s *Service) reinitWorkflowEngine() {
if s.workflowStepGetter != nil {
s.initWorkflowEngine()
}
}
// WorkflowEngine returns the workflow engine, or nil if it has not been
// initialised yet (the workflow step getter must be set first).
//
// Exposed for cross-package wiring (e.g. office service plumbing the
// engine into its dispatcher). The engine itself is safe for concurrent
// HandleTrigger calls.
func (s *Service) WorkflowEngine() *engine.Engine {
return s.workflowEngine
}
// startTurnForSession ensures the session has an active turn and returns its ID.
//
// Idempotent: if an open turn already exists (either tracked in activeTurns or
// only present in the DB — the latter happens when service.CreateMessage lazily
// started a turn for an inbound user message before the orchestrator's prompt
// cycle began, or when the backend restarted with open turns in the DB), it is
// adopted rather than duplicated. A new turn is created only when none exists.
//
// This avoids the classic dual-creation leak: user message → service.CreateMessage
// starts turn X (DB only) → PromptTask → startTurnForSession → would create turn Y
// (DB + activeTurns), leaving X open forever because nothing tracks it.
func (s *Service) startTurnForSession(ctx context.Context, sessionID string) string {
if s.turnService == nil {
return ""
}
if turnIDVal, ok := s.activeTurns.Load(sessionID); ok {
if turnID, ok := turnIDVal.(string); ok && turnID != "" {
return turnID
}
}
if turn, err := s.turnService.GetActiveTurn(ctx, sessionID); turn != nil {
s.activeTurns.Store(sessionID, turn.ID)
return turn.ID
} else if err != nil {
// A real DB read failure here would otherwise be silently dropped, and
// we'd fall through to StartTurn — potentially writing a duplicate next
// to an existing open turn we couldn't see. Log it; the next sweep via
// completeTurnForSession will mop up any duplicate.
s.logger.Warn("failed to look up active turn before starting a new one",
zap.String("session_id", sessionID),
zap.Error(err))
}
turn, err := s.turnService.StartTurn(ctx, sessionID)
if err != nil {
s.logger.Warn("failed to start turn",
zap.String("session_id", sessionID),
zap.Error(err))
return ""
}
s.activeTurns.Store(sessionID, turn.ID)
return turn.ID
}
// completeTurnForSession closes any open turn for the session.
//
// The DB is the source of truth — activeTurns is just an in-memory cache that
// can drift (see startTurnForSession) or be wiped by a backend restart. We
// query the DB for any open turn and close it. Loops to mop up multiple
// zombies (e.g. left over from before this fix) with a small sanity bound.
func (s *Service) completeTurnForSession(ctx context.Context, sessionID string) {
if s.turnService == nil {
return
}
s.activeTurns.Delete(sessionID)
const maxIterations = 16
closed := 0
for closed < maxIterations {
turn, err := s.turnService.GetActiveTurn(ctx, sessionID)
if err != nil {
s.logger.Warn("failed to look up active turn",
zap.String("session_id", sessionID),
zap.Error(err))
return
}
if turn == nil {
return
}
if err := s.turnService.CompleteTurn(ctx, turn.ID); err != nil {
// GetActiveTurn returns the latest open turn — retrying here
// would just hit the same row and loop. Bail; the next
// completeTurnForSession call will pick it up.
s.logger.Warn("failed to complete turn; will retry on next sweep",
zap.String("session_id", sessionID),
zap.String("turn_id", turn.ID),
zap.Error(err))
return
}
closed++
}
// Only warn if turns are *still* accumulating after the cap. Closing
// exactly maxIterations turns and then finding the session clean is not a
// runaway.
if turn, err := s.turnService.GetActiveTurn(ctx, sessionID); err == nil && turn != nil {
s.logger.Warn("completeTurnForSession iteration cap hit; possible turn close loop",
zap.String("session_id", sessionID),
zap.Int("max_iterations", maxIterations))
}
}
// getActiveTurnID returns the active turn ID for a session.
// If no active turn exists and the session ID is provided, it will start a new turn.
// This ensures messages always have a valid turn ID even in edge cases like resumed sessions.
func (s *Service) getActiveTurnID(sessionID string) string {
if sessionID == "" {
return ""
}
turnIDVal, ok := s.activeTurns.Load(sessionID)
if ok {
turnID, _ := turnIDVal.(string)
if turnID != "" {
return turnID
}
}
// No active turn exists - start one lazily
// This handles edge cases like resumed sessions or race conditions
return s.startTurnForSession(context.Background(), sessionID)
}
func (s *Service) setSessionResetInProgress(sessionID string, inProgress bool) {
if sessionID == "" {
return
}
if inProgress {
s.resetInProgressSessions.Store(sessionID, true)
return
}
s.resetInProgressSessions.Delete(sessionID)
}
func (s *Service) isSessionResetInProgress(sessionID string) bool {
if sessionID == "" {
return false
}
v, ok := s.resetInProgressSessions.Load(sessionID)
if !ok {
return false
}
inProgress, _ := v.(bool)
return inProgress
}
// Start starts all orchestrator components
func (s *Service) Start(ctx context.Context) error {
s.mu.Lock()
if s.running {
s.mu.Unlock()
return ErrServiceAlreadyRunning
}
s.running = true
s.startedAt = time.Now()
s.mu.Unlock()
s.logger.Info("starting orchestrator service")
// Reconcile session state from persisted runtime state on startup.
// This does NOT launch any agent processes — sessions are recovered lazily
// when the user opens them (via task.session.status → task.session.resume).
s.reconcileSessionsOnStartup(ctx)
// Start the watcher first to begin receiving events
if err := s.watcher.Start(ctx); err != nil {
s.mu.Lock()
s.running = false
s.mu.Unlock()
return err
}
// Start the scheduler processing loop
if err := s.scheduler.Start(ctx); err != nil {
if stopErr := s.watcher.Stop(); stopErr != nil {
s.logger.Warn("failed to stop watcher after scheduler start failure", zap.Error(stopErr))
}
s.mu.Lock()
s.running = false
s.mu.Unlock()
return err
}
// Subscribe to GitHub integration events
s.subscribeGitHubEvents()
// Subscribe to GitLab integration events
s.subscribeGitLabEvents()
// Subscribe to JIRA integration events
s.subscribeJiraEvents()
// Subscribe to Linear integration events
s.subscribeLinearEvents()
// Subscribe to Sentry integration events
s.subscribeSentryEvents()
// Subscribe to automation events
s.subscribeAutomationEvents()
// Subscribe to clarification events (cancel-and-resume flow)
s.subscribeClarificationEvents()
// Subscribe to prepare events (persist result in session metadata)
s.subscribePrepareEvents()
// Subscribe to ADR-0015 step-completion signals (out-of-band path:
// signal arrives after turn-end).
s.subscribeStepCompletionEvents()
s.logger.Info("orchestrator service started successfully")
return nil
}
// Stop stops all orchestrator components
func (s *Service) Stop() error {
s.mu.Lock()
if !s.running {
s.mu.Unlock()
return ErrServiceNotRunning
}
s.running = false
s.mu.Unlock()
s.logger.Info("stopping orchestrator service")
// Stop components in reverse order
var errs []error
if err := s.scheduler.Stop(); err != nil {
s.logger.Error("failed to stop scheduler", zap.Error(err))
errs = append(errs, err)
}
if err := s.watcher.Stop(); err != nil {
s.logger.Error("failed to stop watcher", zap.Error(err))
errs = append(errs, err)
}
s.cancelAllClarificationWatchdogs()
s.cancelAllTransientRetries()
if len(errs) > 0 {
return errs[0]
}
s.logger.Info("orchestrator service stopped successfully")
return nil
}
// reconcileSessionsOnStartup adjusts database state for sessions that were active before restart.
// It does NOT launch any agent processes — sessions are recovered lazily when the user opens them
// (via task.session.status → task.session.resume or by sending a prompt).
//
// Strategy:
//
// 1. Terminal states (Completed/Cancelled/Failed) → clean up executor record
// 2. Never-started (Created) → clean up executor record
// 3. Active states (Starting/Running/WaitingForInput) → set session to WAITING_FOR_INPUT,
// clear stale execution IDs, fix task state, preserve ExecutorRunning record
// 4. Pre-poll remote executor status for remote runtimes (sprites, remote_docker)
//
// Called by: Start() method during orchestrator initialization.
func (s *Service) reconcileSessionsOnStartup(ctx context.Context) {
runningExecutors, err := s.repo.ListExecutorsRunning(ctx)
if err != nil {
s.logger.Warn("failed to list executors running on startup", zap.Error(err))
return
}
if len(runningExecutors) == 0 {
s.logger.Info("no executors to reconcile on startup")
return
}
s.logger.Info("reconciling sessions on startup (lazy recovery)", zap.Int("count", len(runningExecutors)))
var remoteRecords []executor.RemoteStatusPollRequest
for _, running := range runningExecutors {
if models.IsRemoteExecutorType(models.ExecutorType(running.Runtime)) {
remoteRecords = append(remoteRecords, executor.RemoteStatusPollRequest{
SessionID: running.SessionID,
Runtime: running.Runtime,
AgentExecutionID: running.AgentExecutionID,
ContainerID: running.ContainerID,
Metadata: running.Metadata,
})
}
s.reconcileOneSessionOnStartup(ctx, running)
}
// Pre-poll remote executor status so task lists show accurate state
if len(remoteRecords) > 0 && s.agentManager != nil {
s.agentManager.PollRemoteStatusForRecords(ctx, remoteRecords)
}
}
// reconcileOneSessionOnStartup adjusts DB state for a single session without launching agents.
func (s *Service) reconcileOneSessionOnStartup(ctx context.Context, running *models.ExecutorRunning) {
sessionID := running.SessionID
if sessionID == "" {
return
}
session, err := s.repo.GetTaskSession(ctx, sessionID)
if err != nil {
s.logger.Warn("failed to load session for reconciliation",
zap.String("session_id", sessionID),
zap.Error(err))
return
}
previousState := session.State
// Handle terminal and never-started states (reuse existing cleanup logic)
if skip := s.handleTerminalSessionOnStartup(ctx, session, running, previousState); skip {
return
}
if previousState == models.TaskSessionStateCreated {
s.logger.Info("session was never started; cleaning up",
zap.String("session_id", sessionID))
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
s.logger.Warn("failed to remove executor record for unstarted session",
zap.String("session_id", sessionID),
zap.Error(err))
}
return
}
// IDLE is the office "between turns" resting state — agent process
// and executor are already torn down, conversation is preserved
// in the existing executors_running row for the next run. There
// is nothing to recover; flipping to WAITING_FOR_INPUT here would
// (a) lie about the session having pending user input and (b) make
// the chat UI render as "working" because the office session shape
// uses IDLE specifically to avoid that.
if previousState == models.TaskSessionStateIdle {
s.logger.Info("session reconciled for lazy recovery (idle, no state change)",
zap.String("session_id", sessionID),
zap.String("task_id", running.TaskID),
zap.Bool("has_resume_token", running.ResumeToken != ""),
zap.Bool("has_worktree", running.WorktreePath != ""))
return
}
// Active states: STARTING, RUNNING, WAITING_FOR_INPUT
// Set session to WAITING_FOR_INPUT (idle, ready for lazy resume when user opens it)
if previousState != models.TaskSessionStateWaitingForInput {
if err := s.repo.UpdateTaskSessionState(ctx, sessionID, models.TaskSessionStateWaitingForInput, ""); err != nil {
s.logger.Warn("failed to set session to WAITING_FOR_INPUT on startup",
zap.String("session_id", sessionID),
zap.Error(err))
}
}
// PRESERVE executors_running.agent_execution_id post-restart. The in-memory
// process is gone, but the stored ID still serves as a "this session was
// previously launched" marker that drives resume detection in
// applyRunningRecordToResumeRequest → isResumedSession → ResumePassthroughSession
// (which adds the --resume flag). Clearing it here makes the next launch take
// the fresh-start path and skip the resume CLI flag, so passthrough TUIs lose
// their conversation context after every backend restart. The stale ID is
// harmless: persistExecutorRunning UPSERTs it atomically alongside the
// in-memory Add on the next launch, closing the divergence window we set out
// to fix in this refactor.
// Ensure task is in REVIEW state (not stuck IN_PROGRESS)
if running.TaskID != "" {
task, taskErr := s.taskRepo.GetTask(ctx, running.TaskID)
if taskErr == nil && task != nil && task.State == v1.TaskStateInProgress {
if updateErr := s.taskRepo.UpdateTaskState(ctx, running.TaskID, v1.TaskStateReview); updateErr != nil {
s.logger.Warn("failed to update task to REVIEW on startup",
zap.String("task_id", running.TaskID),
zap.Error(updateErr))
}
}
}
// PRESERVE the ExecutorRunning record — it holds the resume token and worktree info
// needed for lazy recovery when the user opens the session.
s.logger.Info("session reconciled for lazy recovery",
zap.String("session_id", sessionID),
zap.String("task_id", running.TaskID),
zap.String("previous_state", string(previousState)),
zap.Bool("has_resume_token", running.ResumeToken != ""),
zap.Bool("has_worktree", running.WorktreePath != ""))
}
// handleTerminalSessionOnStartup processes sessions in terminal states during startup.
// Returns true if the session should be skipped (no further processing needed).
func (s *Service) handleTerminalSessionOnStartup(ctx context.Context, session *models.TaskSession, running *models.ExecutorRunning, previousState models.TaskSessionState) bool {
sessionID := session.ID
switch previousState {
case models.TaskSessionStateCompleted, models.TaskSessionStateCancelled:
s.logger.Info("session in terminal state; cleaning up executor record",
zap.String("session_id", sessionID),
zap.String("task_id", session.TaskID),
zap.String("state", string(previousState)))
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
s.logger.Warn("failed to remove executor record for terminal session",
zap.String("session_id", sessionID),
zap.Error(err))
}
return true
case models.TaskSessionStateFailed:
s.handleFailedSessionOnStartup(ctx, session, running)
return true
}
return false
}
// handleFailedSessionOnStartup handles a failed session during startup recovery.
func (s *Service) handleFailedSessionOnStartup(ctx context.Context, session *models.TaskSession, running *models.ExecutorRunning) {
sessionID := session.ID
// If session failed, ensure task is in REVIEW state (not stuck IN_PROGRESS)
if session.TaskID != "" {
task, taskErr := s.taskRepo.GetTask(ctx, session.TaskID)
if taskErr == nil && task.State == v1.TaskStateInProgress {
s.logger.Info("fixing task state: session failed but task still IN_PROGRESS",
zap.String("task_id", session.TaskID),
zap.String("session_id", sessionID))
if updateErr := s.taskRepo.UpdateTaskState(ctx, session.TaskID, v1.TaskStateReview); updateErr != nil {
s.logger.Warn("failed to update task state to REVIEW",
zap.String("task_id", session.TaskID),
zap.Error(updateErr))
}
}
}
if canResumeRunning(running) {
s.logger.Info("preserving executor record for resumable failed session",
zap.String("session_id", sessionID),
zap.String("task_id", session.TaskID))
} else {
s.logger.Info("cleaning up executor record for non-resumable failed session",
zap.String("session_id", sessionID),
zap.String("task_id", session.TaskID))
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
s.logger.Warn("failed to remove executor record for failed session",
zap.String("session_id", sessionID),
zap.Error(err))
}
}
}
func canResumeRunning(running *models.ExecutorRunning) bool {
if running == nil || running.ResumeToken == "" {
return false
}
if running.Resumable {
return true
}
return models.IsAlwaysResumableRuntime(running.Runtime)
}
func isMissingMergedPRBranchError(err error) bool {
if err == nil {
return false
}
msg := strings.ToLower(err.Error())
return strings.Contains(msg, "couldn't find remote ref") ||
(strings.Contains(msg, "not found locally or on remote") && strings.Contains(msg, "branch")) ||
(strings.Contains(msg, "pathspec") && strings.Contains(msg, "did not match"))
}
var (
quotedBranchPattern = regexp.MustCompile(`branch "([^"]+)"`)
remoteRefPattern = regexp.MustCompile(`remote ref ([^\s]+)`)
pathspecBranchPattern = regexp.MustCompile(`pathspec '([^']+)'`)
)
func extractMissingBranchName(err error) string {
if err == nil {
return ""
}
msg := err.Error()
if match := quotedBranchPattern.FindStringSubmatch(msg); len(match) == 2 {
return strings.TrimSpace(match[1])
}
if match := remoteRefPattern.FindStringSubmatch(msg); len(match) == 2 {
return strings.TrimSpace(match[1])
}
if match := pathspecBranchPattern.FindStringSubmatch(msg); len(match) == 2 {
return strings.TrimSpace(match[1])
}
return ""
}
func (s *Service) handleSessionLaunchFailed(ctx context.Context, taskID, sessionID string, launchErr error) {
if s.messageCreator == nil || !isMissingMergedPRBranchError(launchErr) {
return
}
branch := extractMissingBranchName(launchErr)
content := "This task references a PR branch that no longer exists on remote (likely merged and deleted)."
if branch != "" {
content = "The remote PR branch \"" + branch + "\" no longer exists (likely merged and deleted)."
}
metadata := map[string]interface{}{
"variant": "warning",
"failure_kind": "missing_pr_branch",
"missing_branch": branch,
"actions": []map[string]interface{}{
{
actionMetaKeyType: "archive_task",
actionMetaKeyLabel: "Archive task",
actionMetaKeyTooltip: "Keep task history and hide it from active work",
actionMetaKeyIcon: "archive",
actionMetaKeyTestID: "missing-branch-archive-button",
},
{
actionMetaKeyType: "delete_task",
actionMetaKeyLabel: "Delete task",
actionMetaKeyTooltip: "Permanently remove this task",
"variant": "destructive",
actionMetaKeyIcon: "trash",
actionMetaKeyTestID: "missing-branch-delete-button",
},
},
}
s.suppressToast.Store(sessionID, true)
msgCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if err := s.messageCreator.CreateSessionMessage(
msgCtx,
taskID,
content,
sessionID,
string(v1.MessageTypeStatus),
"", // No turn — agent never started, avoid lazily creating a synthetic turn.
metadata,
false,
); err != nil {
s.logger.Warn("failed to create missing PR branch launch failure message",
zap.String("task_id", taskID),
zap.String("session_id", sessionID),
zap.Error(err))
}
}
// IsRunning returns true if the service is running
func (s *Service) IsRunning() bool {
s.mu.RLock()
defer s.mu.RUnlock()
return s.running
}
// GetStatus returns the orchestrator status
func (s *Service) GetStatus() *Status {
s.mu.RLock()
running := s.running
startedAt := s.startedAt
s.mu.RUnlock()
queueStatus := s.scheduler.GetQueueStatus()
var uptimeSeconds int64
if running {
uptimeSeconds = int64(time.Since(startedAt).Seconds())
}
return &Status{
Running: running,
ActiveAgents: queueStatus.ActiveExecutions,
QueuedTasks: queueStatus.QueuedTasks,
TotalProcessed: queueStatus.TotalProcessed,
TotalFailed: queueStatus.TotalFailed,
UptimeSeconds: uptimeSeconds,
LastHeartbeat: time.Now(),
}
}
// GetMessageQueue returns the message queue service
func (s *Service) GetMessageQueue() *messagequeue.Service {
return s.messageQueue
}
// GetEventBus returns the event bus
func (s *Service) GetEventBus() bus.EventBus {
return s.eventBus
}