97c1f689e4
* feat(workflow): explicit step-completion signal for auto-advance Adds `step_complete_kandev` MCP tool + `WorkflowStep.auto_advance_requires_signal` column so workflow steps can gate their `on_turn_complete` transitions on an agent-emitted completion signal instead of inferring "done" from any turn-end. See docs/decisions/0015-explicit-completion-signal-for-auto-advance.md for the motivation, alternatives considered, and trade-offs. * fix(workflow): address PR review feedback for explicit completion signal - Dedup `LoadPendingStepSignal` into `internal/task/models` so the MCP handler (write site) and the orchestrator (read site) share one decoder. - Gate the out-of-band subscriber on `WorkflowStep.AutoAdvanceRequiresSignal` so `step_complete_kandev` is not an unintended manual-advance path for signalless steps. - Fail closed on step-load errors in `processOnTurnCompleteViaEngine`: a loader hiccup must block the transition rather than silently bypass the gate. - Reject `step_complete_kandev` for terminal sessions (Completed / Failed / Cancelled) so the agent gets a clear error instead of `accepted: true` followed by a no-op. - Surface bus-publish failures back to the agent instead of swallowing them — the idempotency guard makes the retry safe and prevents the session from getting stuck. - New `TestOnStepCompletionSignaled` covers the subscriber paths (RUNNING / WAITING+match / stale step / not signal-gated). - ADR doc fixes: payload field list now matches the wire shape, writer ownership is documented, the unsubstantiated 250ms-debounce claim is removed, and the late-signal re-close race is recorded as a deferred open question. - Drop the new toggle's HelpTip in favour of an inline description — the shadcn tooltip portal was intercepting clicks on the agent-profile dropdown options in the workflow settings E2E. * fix(workflow): round-2 review fixes for explicit completion signal - Split `handleStepComplete` into `resolveStepCompleteTarget` + `publishStepCompletionEvent` helpers so the entrypoint clears funlen (88 → under 80 lines) and the two halves are independently testable. - Classify session/task load errors instead of returning `NotFound` for every failure: use the exported `taskrepo.ErrTaskNotFound` sentinel for task lookups, substring-match for the session repo (no sentinel today), and surface anything else as `InternalError` with the raw error logged but not leaked to the agent — so a transient DB hiccup triggers a retry rather than session abandonment. - Pin `step_complete_kandev` explicitly in the mode-specific tool-count tests: `Contains` in `ModeTask`, `NotContains` in `ModeConfig` and `ModeOffice`, so accidental dereg or cross-mode registration is caught by the diff instead of by a still-equal count. * fix(workflow): round-3 review fixes + defer pipeline-editor toggle - Defer the "Wait for agent completion signal" sub-toggle to a follow-up PR. Adding it to the step config card shifts the layout enough that the agent profile select's Radix popper opens into a sibling tooltip's hover path, breaking workflow-agent-profile.spec.ts on shard 7. The backend column ships in this PR — workflows can opt in via the YAML import or the MCP `update_workflow_step_kandev` tool — and the UI affordance gets its own iteration where the test interference can be investigated without blocking the rest of the change. - Re-publish the bus event on the `already_signaled` short-circuit when the session is WAITING_FOR_INPUT. Without this, an agent's retry after the first call's publish failed would short-circuit on the persisted bag, never fire the subscriber, and leave the session stuck. - handleStepComplete docstring: drop the non-existent feature-flag claim and the "audit entry" wording — gating is purely `WorkflowStep.AutoAdvanceRequiresSignal` and the bag is cleared on the next turn start with no separate audit trail. - `step_complete_kandev` tool description: same fix — clarify that the call succeeds for non-gated steps but the signal is discarded on the next turn start (no audit history to query later). * test(workflow): cover handleStepComplete + reword defer-comment - Add `TestHandleStepComplete_*` (6 subtests, all passing) covering the uncovered handler paths flagged by Claude/cubic on round 3: missing-fields validation, session-does-not-belong-to-task ownership guard, terminal-session rejection (Completed/Failed/Cancelled), first-call happy path (bag persisted + bus payload shape pinned), dedup while RUNNING (no re-publish), and dedup while WAITING_FOR_INPUT (the retry-after-publish-failure path — must re-publish so the subscriber fires). - Reword the deferred-toggle comment to drop the YAML-export claim: the workflow export model doesn't yet serialize `auto_advance_requires_signal`, so MCP `update_workflow_step_kandev` is the only configuration path that round-trips today. Note the follow-up needs to extend the export schema alongside the UI toggle. * test(workflow): pin signaled_at + drop redundant GetTaskSession on bag-clear - Assert `signaled_at` is present and parses as RFC3339Nano in `TestHandleStepComplete_FirstCallAccepted`. The field is part of the handler's documented response contract; pinning it catches accidental drop/rename regressions. - Add `clearPendingStepSignalByID(ctx, sessionID)` as a thin DB-only variant of `clearPendingStepSignal` and use it from `executeStepTransition`, where the session struct isn't read after the clear. Avoids an extra `GetTaskSession` round-trip per transition. `clearPendingStepSignal(session)` still exists for the in-memory-mutation case (subscriber's stale-step path). * fix(workflow): subscriber stale-clear guard + ADR/comment drift fixes - onStepCompletionSignaled's stale-step path now reads the bag and only clears it when its current StepID matches the (stale) event's stepID. Previously the unconditional clear could destroy a valid signal that the new step's agent had just written when sessions are reused across steps without auto_start_agent. (Greptile P1.) - Drop the "feature flag" phrase from WorkflowStep.AutoAdvanceRequiresSignal docstring — leftover from an earlier draft. The gating is purely the boolean field; the global flag was removed two rounds ago. - Merge the two adjacent `if triggerOnEnter` blocks in executeStepTransition so the full triggerOnEnter=true path reads top- to-bottom in one place. (CodeRabbit + cubic.) - ADR 0015 Mechanism section: re-label the SessionStepHistory.Metadata audit annotation, the `step_complete_signaled` system message, and the expvar telemetry counters as deferred. They're planned shapes, not current behaviour — calling them "Audit" / "Chat visibility" / "Telemetry" in present tense was misleading for future readers. * fix(workflow): drop subscriber complexity + reload session on stale-clear - Extract `parseStepCompletionEvent` and `stepIsSignalGated` helpers from `onStepCompletionSignaled`. Cyclomatic complexity goes 17 → under 15 (cyclop on new code, CI lint regression after the round-6 stale-clear guard). - The stale-step branch now re-loads the session from the DB before deciding whether to clear the bag. The previous round-6 fix checked the local `session.Metadata` snapshot, which can be older than a concurrent write from the new step's agent — that check could see a stale signal that had since been overwritten with a valid one and erase the new bag entry. The narrower load-on-clear is cheap and closes the cubic P1 hole. (The broader race in `processOnTurnCompleteViaEngine` reading session.Metadata is the monotonic-marker open question in ADR 0015; out of scope here.) * test(workflow): pin stale-event guard's preserve-current-bag side + dedup StringFromAny - Add `TestOnStepCompletionSignaled` subtest "stale event → valid bag for CURRENT step is preserved" — fires a stale step-A event while the bag holds a valid step-B signal, asserts the bag still contains StepID=step_b after the subscriber runs. Pins the negative side of the round-6 `existing.StepID == stepID` guard, so a regression that drops or inverts the guard (and would silently leave signal-gated steps stuck) fails the test instead of slipping through. - Promote `pendingSignalString` to exported `models.StringFromAny` and reuse it from the orchestrator subscriber's `parseStepCompletionEvent`. Eliminates the byte-for-byte duplicate helper across the two packages; the bag-shape decoder and the bus- event decoder now share one source of truth for "missing string". --------- Co-authored-by: Kandev Agent <agent@kandev.dev>
1241 lines
50 KiB
Go
1241 lines
50 KiB
Go
// Package orchestrator provides the main orchestrator service that coordinates
|
|
// task execution across agents. It manages:
|
|
//
|
|
// - Task queuing and scheduling via the Scheduler
|
|
// - Agent lifecycle through the AgentManager
|
|
// - Event handling and propagation
|
|
// - Session management and resume
|
|
//
|
|
// The orchestrator acts as the central coordinator between the task service,
|
|
// agent lifecycle manager, and the event bus.
|
|
package orchestrator
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"regexp"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
|
|
"go.uber.org/zap"
|
|
|
|
"github.com/kandev/kandev/internal/agentctl/types/streams"
|
|
"github.com/kandev/kandev/internal/common/logger"
|
|
"github.com/kandev/kandev/internal/events/bus"
|
|
"github.com/kandev/kandev/internal/gitlab"
|
|
"github.com/kandev/kandev/internal/orchestrator/executor"
|
|
"github.com/kandev/kandev/internal/orchestrator/messagequeue"
|
|
"github.com/kandev/kandev/internal/orchestrator/queue"
|
|
"github.com/kandev/kandev/internal/orchestrator/scheduler"
|
|
"github.com/kandev/kandev/internal/orchestrator/watcher"
|
|
"github.com/kandev/kandev/internal/secrets"
|
|
"github.com/kandev/kandev/internal/task/models"
|
|
"github.com/kandev/kandev/internal/workflow/engine"
|
|
wfmodels "github.com/kandev/kandev/internal/workflow/models"
|
|
"github.com/kandev/kandev/internal/worktree"
|
|
v1 "github.com/kandev/kandev/pkg/api/v1"
|
|
)
|
|
|
|
// Common errors
|
|
var (
|
|
ErrServiceAlreadyRunning = errors.New("service is already running")
|
|
ErrServiceNotRunning = errors.New("service is not running")
|
|
)
|
|
|
|
// ServiceConfig holds orchestrator service configuration
|
|
type ServiceConfig struct {
|
|
Scheduler scheduler.SchedulerConfig
|
|
QueueSize int
|
|
QueueGroup string
|
|
}
|
|
|
|
// DefaultServiceConfig returns default configuration
|
|
func DefaultServiceConfig() ServiceConfig {
|
|
return ServiceConfig{
|
|
Scheduler: scheduler.DefaultSchedulerConfig(),
|
|
QueueSize: 1000,
|
|
QueueGroup: "orchestrator",
|
|
}
|
|
}
|
|
|
|
// MessageCreator is an interface for creating messages on tasks
|
|
type MessageCreator interface {
|
|
CreateAgentMessage(ctx context.Context, taskID, content, agentSessionID, turnID string) error
|
|
CreateUserMessage(ctx context.Context, taskID, content, agentSessionID, turnID string, metadata map[string]interface{}) error
|
|
// CreateToolCallMessage creates a message for a tool call.
|
|
// normalized contains the typed tool payload data.
|
|
// parentToolCallID is the parent Task tool call ID for subagent nesting (empty for top-level).
|
|
CreateToolCallMessage(ctx context.Context, taskID, toolCallID, parentToolCallID, title, status, agentSessionID, turnID string, normalized *streams.NormalizedPayload) error
|
|
// UpdateToolCallMessage updates a tool call message's status and optionally its normalized data.
|
|
// If the message doesn't exist, it creates it using taskID, turnID, and msgType.
|
|
// normalized contains the typed tool payload data.
|
|
// parentToolCallID is the parent Task tool call ID for subagent nesting (empty for top-level).
|
|
UpdateToolCallMessage(ctx context.Context, taskID, toolCallID, parentToolCallID, status, result, agentSessionID, title, turnID, msgType string, normalized *streams.NormalizedPayload) error
|
|
CreateSessionMessage(ctx context.Context, taskID, content, agentSessionID, messageType, turnID string, metadata map[string]interface{}, requestsInput bool) error
|
|
CreatePermissionRequestMessage(ctx context.Context, taskID, sessionID, pendingID, toolCallID, title, turnID string, options []map[string]interface{}, actionType string, actionDetails map[string]interface{}) (string, error)
|
|
UpdatePermissionMessage(ctx context.Context, sessionID, pendingID string, status models.PermissionStatus) error
|
|
// CreateAgentMessageStreaming creates a new agent message with a pre-generated ID for streaming updates
|
|
CreateAgentMessageStreaming(ctx context.Context, messageID, taskID, content, agentSessionID, turnID string) error
|
|
// AppendAgentMessage appends additional content to an existing streaming message
|
|
AppendAgentMessage(ctx context.Context, messageID, additionalContent string) error
|
|
// CreateThinkingMessageStreaming creates a new thinking message with a pre-generated ID for streaming updates
|
|
CreateThinkingMessageStreaming(ctx context.Context, messageID, taskID, content, agentSessionID, turnID string) error
|
|
// AppendThinkingMessage appends additional content to an existing streaming thinking message
|
|
AppendThinkingMessage(ctx context.Context, messageID, additionalContent string) error
|
|
// InvalidateModelCache clears any cached model for a session, forcing the next
|
|
// message to re-read the model from the DB. Called after model switches.
|
|
InvalidateModelCache(sessionID string)
|
|
}
|
|
|
|
// TurnService is an interface for managing session turns
|
|
type TurnService interface {
|
|
StartTurn(ctx context.Context, sessionID string) (*models.Turn, error)
|
|
CompleteTurn(ctx context.Context, turnID string) error
|
|
GetActiveTurn(ctx context.Context, sessionID string) (*models.Turn, error)
|
|
// AbandonOpenTurns buries any open turns for a session by setting
|
|
// completed_at = started_at (zero duration), so a subsequent prompt starts
|
|
// a fresh turn instead of adopting one that was orphaned by a crash or
|
|
// restart. Used on session resume.
|
|
AbandonOpenTurns(ctx context.Context, sessionID string) error
|
|
}
|
|
|
|
// TaskEventPublisher is the orchestrator's collaborator for publishing
|
|
// task.updated events. Implemented by the task service (which owns the rich
|
|
// payload build — session counts, primary session info, repositories,
|
|
// metadata, parent_id). The orchestrator does not construct task.updated
|
|
// payloads itself.
|
|
type TaskEventPublisher interface {
|
|
PublishTaskUpdated(ctx context.Context, task *models.Task)
|
|
}
|
|
|
|
// WorkflowStepGetter retrieves workflow step information for prompt building.
|
|
type WorkflowStepGetter interface {
|
|
GetStep(ctx context.Context, stepID string) (*wfmodels.WorkflowStep, error)
|
|
GetNextStepByPosition(ctx context.Context, workflowID string, currentPosition int) (*wfmodels.WorkflowStep, error)
|
|
GetPreviousStepByPosition(ctx context.Context, workflowID string, currentPosition int) (*wfmodels.WorkflowStep, error)
|
|
GetWorkflowAgentProfileID(ctx context.Context, workflowID string) (string, error)
|
|
}
|
|
|
|
// repoStore is the repository interface accepted by NewService.
|
|
// It covers both the orchestrator's own needs (sessionExecutorStore) and
|
|
// the executor package's needs (executor.executorStore).
|
|
type repoStore interface {
|
|
sessionExecutorStore
|
|
// Additional methods needed by executor
|
|
UpdateTaskState(ctx context.Context, id string, state v1.TaskState) error
|
|
GetPrimaryTaskRepository(ctx context.Context, taskID string) (*models.TaskRepository, error)
|
|
ListTaskRepositories(ctx context.Context, taskID string) ([]*models.TaskRepository, error)
|
|
CreateTaskSession(ctx context.Context, session *models.TaskSession) error
|
|
UpdateTaskSession(ctx context.Context, session *models.TaskSession) error
|
|
ListActiveTaskSessions(ctx context.Context) ([]*models.TaskSession, error)
|
|
ListActiveTaskSessionsByTaskID(ctx context.Context, taskID string) ([]*models.TaskSession, error)
|
|
CreateTaskSessionWorktree(ctx context.Context, sessionWorktree *models.TaskSessionWorktree) error
|
|
ListSessionsWithBranches(ctx context.Context) ([]models.SessionBranchInfo, error)
|
|
GetRepository(ctx context.Context, id string) (*models.Repository, error)
|
|
UpdateRepository(ctx context.Context, repository *models.Repository) error
|
|
ListRepositories(ctx context.Context, workspaceID string) ([]*models.Repository, error)
|
|
GetExecutorProfile(ctx context.Context, id string) (*models.ExecutorProfile, error)
|
|
// Multi-repo task environment children
|
|
CreateTaskEnvironmentRepo(ctx context.Context, repo *models.TaskEnvironmentRepo) error
|
|
ListTaskEnvironmentRepos(ctx context.Context, envID string) ([]*models.TaskEnvironmentRepo, error)
|
|
// Session history + plan (for context handover)
|
|
GetTaskPlan(ctx context.Context, taskID string) (*models.TaskPlan, error)
|
|
}
|
|
|
|
// sessionExecutorStore is the minimal repository interface needed by the orchestrator service.
|
|
type sessionExecutorStore interface {
|
|
// Session
|
|
GetTaskSession(ctx context.Context, id string) (*models.TaskSession, error)
|
|
GetActiveTaskSessionByTaskID(ctx context.Context, taskID string) (*models.TaskSession, error)
|
|
SetSessionPrimary(ctx context.Context, sessionID string) error
|
|
UpdateTaskSession(ctx context.Context, session *models.TaskSession) error
|
|
UpdateTaskSessionState(ctx context.Context, id string, state models.TaskSessionState, errorMessage string) error
|
|
UpdateTaskSessionBaseCommit(ctx context.Context, id string, baseCommitSHA string) error
|
|
GetTaskSessionByTaskAndAgent(ctx context.Context, taskID, agentInstanceID string) (*models.TaskSession, error)
|
|
UpdateTaskSessionWorktreeBranch(ctx context.Context, sessionID, branch string) error
|
|
UpdateSessionReviewStatus(ctx context.Context, sessionID string, status string) error
|
|
UpdateSessionMetadata(ctx context.Context, sessionID string, metadata map[string]interface{}) error
|
|
SetSessionMetadataKey(ctx context.Context, sessionID, key string, value interface{}) error
|
|
// Executor running state
|
|
ListExecutorsRunning(ctx context.Context) ([]*models.ExecutorRunning, error)
|
|
UpsertExecutorRunning(ctx context.Context, running *models.ExecutorRunning) error
|
|
GetExecutorRunningBySessionID(ctx context.Context, sessionID string) (*models.ExecutorRunning, error)
|
|
DeleteExecutorRunningBySessionID(ctx context.Context, sessionID string) error
|
|
HasExecutorRunningRow(ctx context.Context, sessionID string) (bool, error)
|
|
UpdateResumeToken(ctx context.Context, sessionID, expectedExecID, resumeToken, lastMessageUUID string) error
|
|
UpdateExecutorRunningStatus(ctx context.Context, sessionID, status string) error
|
|
// Executor
|
|
GetExecutor(ctx context.Context, id string) (*models.Executor, error)
|
|
// Task
|
|
GetTask(ctx context.Context, id string) (*models.Task, error)
|
|
UpdateTask(ctx context.Context, task *models.Task) error
|
|
// Git snapshots and commits
|
|
GetLatestGitSnapshot(ctx context.Context, sessionID string) (*models.GitSnapshot, error)
|
|
CreateGitSnapshot(ctx context.Context, snapshot *models.GitSnapshot) error
|
|
DeleteLiveMonitorSnapshots(ctx context.Context, sessionID string) error
|
|
UpsertLatestLiveGitSnapshot(ctx context.Context, snapshot *models.GitSnapshot) error
|
|
CreateSessionCommit(ctx context.Context, commit *models.SessionCommit) error
|
|
GetSessionCommits(ctx context.Context, sessionID string) ([]*models.SessionCommit, error)
|
|
DeleteSessionCommit(ctx context.Context, id string) error
|
|
// Session listing + delete
|
|
ListTaskSessions(ctx context.Context, taskID string) ([]*models.TaskSession, error)
|
|
ListNonTerminalSessionsByAgentInstance(ctx context.Context, agentInstanceID string) ([]*models.TaskSession, error)
|
|
DeleteTaskSession(ctx context.Context, id string) error
|
|
// Messages — used by resume to backfill the initial user prompt when a
|
|
// prior launch failed before recordInitialMessage ran.
|
|
ListMessages(ctx context.Context, sessionID string) ([]*models.Message, error)
|
|
// Pending clarification rows — durable guard for on_turn_complete while the user is answering.
|
|
FindPendingClarificationMessagesBySessionID(ctx context.Context, sessionID string) ([]*models.Message, error)
|
|
// Workspace
|
|
GetWorkspace(ctx context.Context, id string) (*models.Workspace, error)
|
|
// Task environment
|
|
GetTaskEnvironment(ctx context.Context, id string) (*models.TaskEnvironment, error)
|
|
GetTaskEnvironmentByTaskID(ctx context.Context, taskID string) (*models.TaskEnvironment, error)
|
|
CreateTaskEnvironment(ctx context.Context, env *models.TaskEnvironment) error
|
|
UpdateTaskEnvironment(ctx context.Context, env *models.TaskEnvironment) error
|
|
}
|
|
|
|
// Service is the main orchestrator service
|
|
type Service struct {
|
|
config ServiceConfig
|
|
logger *logger.Logger
|
|
eventBus bus.EventBus
|
|
taskRepo scheduler.TaskRepository
|
|
repo sessionExecutorStore
|
|
agentManager executor.AgentManagerClient
|
|
|
|
// Components
|
|
queue *queue.TaskQueue
|
|
executor *executor.Executor
|
|
scheduler *scheduler.Scheduler
|
|
watcher *watcher.Watcher
|
|
|
|
// Message queue service for queueing messages while agent is running
|
|
messageQueue *messagequeue.Service
|
|
|
|
// Message creator for saving agent responses
|
|
messageCreator MessageCreator
|
|
|
|
// Turn service for managing session turns
|
|
turnService TurnService
|
|
|
|
// Task event publisher for emitting task.updated events.
|
|
// Task service owns the rich payload; orchestrator delegates.
|
|
taskEvents TaskEventPublisher
|
|
|
|
// Workflow step getter for prompt building
|
|
workflowStepGetter WorkflowStepGetter
|
|
|
|
// Workflow engine for typed state-machine evaluation of step transitions
|
|
workflowEngine *engine.Engine
|
|
workflowStore *workflowStore
|
|
// engineOptions are applied each time initWorkflowEngine runs. Wired
|
|
// from cmd/kandev (Phase 3.2) to plug Phase 2 ADR-0004 dependencies
|
|
// — RunQueueAdapter, ParticipantStore, DecisionStore, and the CEO /
|
|
// Primary agent resolvers — without coupling the orchestrator to
|
|
// the office or runs packages.
|
|
engineOptions []engine.Option
|
|
|
|
// Phase 2 (ADR-0004) callback dependencies, set via dedicated
|
|
// setters from cmd/kandev. When set, buildWorkflowCallbacks
|
|
// registers the queue_run / clear_decisions /
|
|
// queue_run_for_each_participant callbacks with these
|
|
// dependencies. Nil-safe: missing deps simply skip registration.
|
|
engineRunQueue engine.RunQueueAdapter
|
|
engineParticipants engine.ParticipantStore
|
|
engineDecisions engine.DecisionStore
|
|
enginePrimary engine.PrimaryAgentResolver
|
|
engineCEOResolver engine.CEOAgentResolver
|
|
// Phase 8 dependencies — also nil-safe.
|
|
engineTaskCreator engine.TaskCreator
|
|
engineWorkflowSwitcher engine.WorkflowSwitcher
|
|
|
|
// GitHub service for PR auto-detection on push
|
|
githubService GitHubService
|
|
|
|
// Office task-handoffs materializer (phase 6 wiring) — invoked from
|
|
// PrepareTaskSession to flip workspace groups to materialized once
|
|
// their owner task has a session with worktree details. Optional;
|
|
// nil-safe.
|
|
workspaceMaterializer WorkspaceMaterializer
|
|
|
|
// Review task creator for auto-creating tasks from review watch PRs
|
|
reviewTaskCreator ReviewTaskCreator
|
|
|
|
// Issue task creator for auto-creating tasks from issue watch events
|
|
issueTaskCreator IssueTaskCreator
|
|
|
|
// Watcher dispatch coordinator: single seam for the watcher→task pipeline.
|
|
// Constructed lazily once issueTaskCreator is wired (see SetIssueTaskCreator).
|
|
watcherCoordinator *WatcherDispatchCoordinator
|
|
|
|
// Watcher throttle state. watcherTaskCount counts committed open watcher-
|
|
// created tasks (DB-backed source of truth across restarts). pendingByWatch
|
|
// tracks in-process events whose dedup row has not yet been written —
|
|
// without it, a burst of events read the same stale COUNT(*) before any
|
|
// goroutine commits, and the cap is silently overshot. Both reads and the
|
|
// pending increment happen under watcherMu to prevent the burst race.
|
|
watcherTaskCount WatcherTaskCounter
|
|
watcherMu sync.Mutex
|
|
pendingByWatch map[string]int
|
|
// watcherSaturated tracks per-watch whether the last gate result was
|
|
// "deferred", so we log the state-transition Warn ("cap reached" /
|
|
// "cap cleared") only once per transition instead of every event.
|
|
watcherSaturated map[string]bool
|
|
|
|
// profileLookup answers "is this agent profile still live?" for the
|
|
// dispatch pre-flight. Set via SetProfileLookup from main; nil-safe so
|
|
// the legacy code path (and tests without profile wiring) keep working.
|
|
profileLookup ProfileLookup
|
|
|
|
// Jira service for issue watch dedup operations
|
|
jiraService JiraService
|
|
// jiraSource adapts jiraService onto WatcherSource. Built once in
|
|
// SetJiraService so handlers don't allocate per bus event.
|
|
jiraSource *JiraWatcherSource
|
|
|
|
// Linear service for issue watch dedup operations
|
|
linearService LinearService
|
|
// linearSource adapts linearService onto WatcherSource. Built once in
|
|
// SetLinearService so handlers don't allocate per bus event.
|
|
linearSource *LinearWatcherSource
|
|
|
|
// Sentry service for issue watch dedup operations
|
|
sentryService SentryService
|
|
// sentrySource adapts sentryService onto WatcherSource. Built once in
|
|
// SetSentryService so handlers don't allocate per bus event.
|
|
sentrySource *SentryWatcherSource
|
|
// GitLab service + task creators for auto-creating tasks from review /
|
|
// issue watch events. When the task creators are nil the events are
|
|
// logged but no tasks are created — matches the GitHub flow when a
|
|
// workspace has no task creator wired.
|
|
gitlabService *gitlab.Service
|
|
gitlabReviewTaskCreator GitLabReviewTaskCreator
|
|
gitlabIssueTaskCreator GitLabIssueTaskCreator
|
|
|
|
// Repository resolver for cloning + finding/creating repos for review tasks
|
|
repositoryResolver RepositoryResolver
|
|
|
|
// Automation service for handling automation triggers
|
|
automationService AutomationService
|
|
|
|
// Worktree manager — used to clean up ephemeral worktrees for run-mode
|
|
// automation tasks immediately on completion rather than waiting for
|
|
// the 24h Office GC. Nil-safe.
|
|
worktreeMgr *worktree.Manager
|
|
|
|
// Clarification canceller — cancels pending clarifications when agent's turn completes
|
|
clarificationCanceller ClarificationCanceller
|
|
|
|
// Push tracker: "<sessionID>|<repository_name>" -> last known ahead count.
|
|
// The repository_name segment is required for multi-repo: each repo emits
|
|
// its own git status events, and keying by sessionID alone made events
|
|
// from different repos overwrite each other's ahead counts (so only one
|
|
// push got detected per session). Single-repo / repo-less sessions key
|
|
// against an empty repository_name.
|
|
pushTracker sync.Map
|
|
|
|
// gitSnapshotCache throttles per-session writes of the live git status
|
|
// snapshot (used by handleGitStatusUpdate -> persistGitStatusSnapshot).
|
|
gitSnapshotCache *gitSnapshotCache
|
|
|
|
// Active turns map: sessionID -> turnID
|
|
activeTurns sync.Map
|
|
|
|
// Session reset flags: sessionID -> true while resetAgentContext is restarting process.
|
|
// Used to suppress stale ready events and avoid draining queued prompts mid-reset.
|
|
resetInProgressSessions sync.Map
|
|
|
|
// suppressToast: sessionID -> true. Set by failure handlers that create
|
|
// guidance messages with actions. Cleared by updateTaskSessionState which
|
|
// adds suppress_toast to the WS event so the frontend skips the error toast.
|
|
suppressToast sync.Map
|
|
|
|
// clarificationWatchdogs tracks active clarification primary-path resume watchdogs.
|
|
// key: "<session_id>::<pending_id>", value: *clarificationWatchdogEntry
|
|
clarificationWatchdogs sync.Map
|
|
|
|
// clarificationWatchdogTimeout controls how long to wait for post-clarification
|
|
// activity before triggering fallback resume.
|
|
clarificationWatchdogTimeout time.Duration
|
|
|
|
// cancelInFlight tracks sessionIDs whose CancelAgent call is currently in
|
|
// progress. It deduplicates impatient retries from the UI and lets late
|
|
// agent.ready/boot_ready events from the cancelled turn return before they
|
|
// evaluate workflow transitions or drain queued messages.
|
|
cancelInFlight sync.Map
|
|
|
|
// transientRetries tracks in-progress transient-provider-error (529
|
|
// Overloaded) retry loops. key: sessionID, value: *transientRetryEntry.
|
|
// A backoff timer per session re-drives the failed prompt; cancelled on
|
|
// success, user-cancel, or service shutdown.
|
|
transientRetries sync.Map
|
|
|
|
// lastTurnPrompt caches the most recent outbound prompt per session so a
|
|
// transient-failure retry can re-drive the same turn without the caller's
|
|
// context. key: sessionID, value: capturedPrompt. Replaced every turn.
|
|
lastTurnPrompt sync.Map
|
|
|
|
// Service state
|
|
mu sync.RWMutex
|
|
running bool
|
|
startedAt time.Time
|
|
}
|
|
|
|
// Status contains orchestrator status information
|
|
type Status struct {
|
|
Running bool `json:"running"`
|
|
ActiveAgents int `json:"active_agents"`
|
|
QueuedTasks int `json:"queued_tasks"`
|
|
TotalProcessed int64 `json:"total_processed"`
|
|
TotalFailed int64 `json:"total_failed"`
|
|
UptimeSeconds int64 `json:"uptime_seconds"`
|
|
LastHeartbeat time.Time `json:"last_heartbeat"`
|
|
}
|
|
|
|
// NewService creates a new orchestrator service. msgQueue is the persistent
|
|
// message queue service (constructed externally so cmd/kandev can wire it to
|
|
// the shared SQLite pool); pass nil to default to an in-memory queue, which
|
|
// suits tests but loses entries on restart.
|
|
func NewService(
|
|
cfg ServiceConfig,
|
|
eventBus bus.EventBus,
|
|
agentManager executor.AgentManagerClient,
|
|
taskRepo scheduler.TaskRepository,
|
|
repo repoStore,
|
|
shellPrefs executor.ShellPreferenceProvider,
|
|
secretStore secrets.SecretStore,
|
|
msgQueue *messagequeue.Service,
|
|
log *logger.Logger,
|
|
) *Service {
|
|
svcLogger := log.WithFields(zap.String("component", "orchestrator"))
|
|
|
|
// Create the task queue with configured size
|
|
taskQueue := queue.NewTaskQueue(cfg.QueueSize)
|
|
|
|
// Create the executor with the agent manager client and repository for persistent sessions
|
|
execCfg := executor.ExecutorConfig{
|
|
ShellPrefs: shellPrefs,
|
|
SecretStore: secretStore,
|
|
}
|
|
exec := executor.NewExecutor(agentManager, repo, log, execCfg)
|
|
|
|
// Create the scheduler with queue, executor, and task repository
|
|
sched := scheduler.NewScheduler(taskQueue, exec, taskRepo, log, cfg.Scheduler)
|
|
|
|
if msgQueue == nil {
|
|
msgQueue = messagequeue.NewServiceMemory(log)
|
|
}
|
|
|
|
// Create the service (watcher will be created after we have handlers)
|
|
s := &Service{
|
|
config: cfg,
|
|
logger: svcLogger,
|
|
eventBus: eventBus,
|
|
taskRepo: taskRepo,
|
|
repo: repo,
|
|
agentManager: agentManager,
|
|
queue: taskQueue,
|
|
executor: exec,
|
|
scheduler: sched,
|
|
messageQueue: msgQueue,
|
|
clarificationWatchdogTimeout: 15 * time.Second,
|
|
gitSnapshotCache: newGitSnapshotCache(),
|
|
}
|
|
|
|
// Wire executor state changes through the orchestrator so events are published
|
|
// (e.g. WebSocket notifications to the frontend). Must be set after service
|
|
// construction so the session callback can reference s.updateTaskSessionState.
|
|
exec.SetOnTaskStateChange(func(ctx context.Context, taskID string, state v1.TaskState) error {
|
|
return taskRepo.UpdateTaskState(ctx, taskID, state)
|
|
})
|
|
exec.SetOnSessionStateChange(func(ctx context.Context, taskID, sessionID string, state models.TaskSessionState, errorMessage string) error {
|
|
s.updateTaskSessionState(ctx, taskID, sessionID, state, errorMessage, true)
|
|
return nil
|
|
})
|
|
exec.SetOnLaunchFailed(s.handleSessionLaunchFailed)
|
|
exec.SetOnAgentStartFailed(s.handleAgentStartFailed)
|
|
if caps, ok := agentManager.(executor.ExecutorTypeCapabilities); ok {
|
|
exec.SetCapabilities(caps)
|
|
}
|
|
|
|
// Create the watcher with event handlers that wire everything together
|
|
handlers := watcher.EventHandlers{
|
|
OnTaskDeleted: s.handleTaskDeleted,
|
|
OnAgentRunning: s.handleAgentRunning,
|
|
OnAgentBootReady: s.handleAgentBootReady,
|
|
OnAgentReady: s.handleAgentReady,
|
|
OnAgentCompleted: s.handleAgentCompleted,
|
|
OnAgentFailed: s.handleAgentFailed,
|
|
OnAgentStopped: s.handleAgentStopped,
|
|
OnAgentStreamEvent: s.handleAgentStreamEvent,
|
|
OnACPSessionCreated: s.handleACPSessionCreated,
|
|
OnPermissionRequest: s.handlePermissionRequest,
|
|
OnGitEvent: s.handleGitEvent,
|
|
OnContextWindowUpdated: s.handleContextWindowUpdated,
|
|
OnTaskMoved: s.handleTaskMoved,
|
|
}
|
|
s.watcher = watcher.NewWatcher(eventBus, handlers, cfg.QueueGroup, log)
|
|
|
|
return s
|
|
}
|
|
|
|
// SetMessageCreator sets the message creator for saving agent responses to the database.
|
|
//
|
|
// This must be called before starting the orchestrator if you want agent messages, tool calls,
|
|
// and streaming content to be persisted to the database. The MessageCreator interface provides
|
|
// methods for creating and updating messages associated with task sessions.
|
|
//
|
|
// The MessageCreator is typically the task service, which owns the message persistence logic.
|
|
// Event handlers in the orchestrator call these methods when agent events occur:
|
|
// - AgentStreamEvent → CreateAgentMessage, AppendAgentMessage
|
|
// - Tool calls → CreateToolCallMessage, UpdateToolCallMessage
|
|
// - Permission requests → CreatePermissionRequestMessage
|
|
//
|
|
// When to call: During orchestrator initialization, after creating the task service.
|
|
//
|
|
// If not set: Agent messages won't be saved to the database (events will still be published).
|
|
func (s *Service) SetMessageCreator(mc MessageCreator) {
|
|
s.messageCreator = mc
|
|
}
|
|
|
|
// SetOnPrimarySessionSet sets a callback on the executor for when the first session
|
|
// of a task is marked primary. Used to publish a task.updated event so the frontend
|
|
// receives the primary_session_id.
|
|
func (s *Service) SetOnPrimarySessionSet(fn executor.PrimarySessionSetFunc) {
|
|
s.executor.SetOnPrimarySessionSet(fn)
|
|
}
|
|
|
|
// SetRepoCloner sets the repository cloner and updater on the executor, enabling automatic
|
|
// cloning of provider-backed repositories (e.g. from a GitHub URL) when they are launched
|
|
// for local/worktree execution and have no local path yet.
|
|
func (s *Service) SetRepoCloner(cloner executor.RepoCloner, updater executor.RepoUpdater) {
|
|
s.executor.SetRepoCloner(cloner, updater)
|
|
}
|
|
|
|
// SetTurnService sets the turn service for tracking conversation turns.
|
|
//
|
|
// A "turn" represents a single conversation round-trip: user prompt → agent response.
|
|
// The TurnService tracks turn timing and duration for analytics and UI display (e.g., showing
|
|
// how long each agent response took).
|
|
//
|
|
// The TurnService is typically the task service, which owns turn persistence logic.
|
|
// The orchestrator calls these methods:
|
|
// - StartTurn: When agent begins processing a prompt
|
|
// - CompleteTurn: When agent finishes and returns to ready state
|
|
// - GetActiveTurn: To associate messages with current turn
|
|
//
|
|
// When to call: During orchestrator initialization, after creating the task service.
|
|
//
|
|
// If not set: Turns won't be tracked (orchestrator continues functioning normally, but
|
|
// no timing data is recorded and turn IDs in messages will be empty).
|
|
func (s *Service) SetTurnService(turnService TurnService) {
|
|
s.turnService = turnService
|
|
}
|
|
|
|
// SetTaskEventPublisher wires the publisher used for task.updated events.
|
|
//
|
|
// The task service is the canonical publisher: it loads session counts,
|
|
// primary session info, repositories, metadata, and parent_id, and emits a
|
|
// single rich payload. Orchestrator code paths that mutate a task (workflow
|
|
// transitions, primary session assignment, workflow-step moves) delegate to
|
|
// this publisher instead of constructing their own partial payloads.
|
|
//
|
|
// If not set: orchestrator task.updated publishers no-op (nothing is emitted
|
|
// on those paths). Task service's own publishTaskEvent calls are unaffected.
|
|
func (s *Service) SetTaskEventPublisher(publisher TaskEventPublisher) {
|
|
s.taskEvents = publisher
|
|
}
|
|
|
|
// publishTaskUpdated forwards to the configured TaskEventPublisher.
|
|
// No-op when the publisher isn't wired (tests, or before SetTaskEventPublisher
|
|
// has been called during startup).
|
|
func (s *Service) publishTaskUpdated(ctx context.Context, task *models.Task) {
|
|
if s.taskEvents == nil || task == nil {
|
|
return
|
|
}
|
|
s.taskEvents.PublishTaskUpdated(ctx, task)
|
|
}
|
|
|
|
// SetWorkflowStepGetter sets the workflow step getter for prompt building.
|
|
//
|
|
// When workflow_step_id is provided to StartTask, the orchestrator uses this getter
|
|
// to retrieve the step's prompt_prefix, prompt_suffix, and plan_mode settings to
|
|
// build the effective prompt.
|
|
//
|
|
// If not set: workflow_step_id in StartTask is ignored and the prompt is used as-is.
|
|
func (s *Service) SetWorkflowStepGetter(getter WorkflowStepGetter) {
|
|
s.workflowStepGetter = getter
|
|
s.initWorkflowEngine()
|
|
}
|
|
|
|
// ClarificationCanceller detaches in-memory clarification waiters when an agent's
|
|
// turn completes while questions are still pending in the DB.
|
|
type ClarificationCanceller interface {
|
|
DetachSessionAndNotify(ctx context.Context, sessionID string) int
|
|
}
|
|
|
|
// SetClarificationCanceller sets the canceller for cleaning up pending clarifications on turn complete.
|
|
func (s *Service) SetClarificationCanceller(c ClarificationCanceller) {
|
|
s.clarificationCanceller = c
|
|
}
|
|
|
|
// initWorkflowEngine creates the workflow engine with store and callbacks.
|
|
// Called after the workflow step getter is set. Safe to call multiple times —
|
|
// previously-applied engine options are re-applied on reinit.
|
|
func (s *Service) initWorkflowEngine() {
|
|
if s.workflowStepGetter == nil {
|
|
return
|
|
}
|
|
store := newWorkflowStore(s.repo, s.workflowStepGetter, s.agentManager, s.publishTaskUpdated, s.logger)
|
|
callbacks := buildWorkflowCallbacks(s)
|
|
s.workflowStore = store
|
|
s.workflowEngine = engine.New(store, callbacks, s.engineOptions...)
|
|
}
|
|
|
|
// SetEngineRunQueue wires the engine's RunQueueAdapter dependency. Used
|
|
// both as an engine option (so the engine can resolve queue_run targets)
|
|
// and as a callback dependency (so QueueRunCallback can enqueue runs).
|
|
func (s *Service) SetEngineRunQueue(adapter engine.RunQueueAdapter) {
|
|
s.engineRunQueue = adapter
|
|
s.engineOptions = append(s.engineOptions, engine.WithRunQueue(adapter))
|
|
s.reinitWorkflowEngine()
|
|
}
|
|
|
|
// SetEngineParticipantStore wires the engine's ParticipantStore.
|
|
func (s *Service) SetEngineParticipantStore(store engine.ParticipantStore) {
|
|
s.engineParticipants = store
|
|
s.engineOptions = append(s.engineOptions, engine.WithParticipantStore(store))
|
|
s.reinitWorkflowEngine()
|
|
}
|
|
|
|
// SetEngineDecisionStore wires the engine's DecisionStore.
|
|
func (s *Service) SetEngineDecisionStore(store engine.DecisionStore) {
|
|
s.engineDecisions = store
|
|
s.engineOptions = append(s.engineOptions, engine.WithDecisionStore(store))
|
|
s.reinitWorkflowEngine()
|
|
}
|
|
|
|
// SetEngineCEOResolver wires the engine's CEOAgentResolver.
|
|
func (s *Service) SetEngineCEOResolver(resolver engine.CEOAgentResolver) {
|
|
s.engineCEOResolver = resolver
|
|
s.engineOptions = append(s.engineOptions, engine.WithCEOAgentResolver(resolver))
|
|
s.reinitWorkflowEngine()
|
|
}
|
|
|
|
// SetPrimaryAgentResolver wires the engine's PrimaryAgentResolver. The
|
|
// engine has no constructor option for this — the resolver is consumed
|
|
// by QueueRunCallback at registration time, so the orchestrator
|
|
// re-runs initWorkflowEngine to pick it up.
|
|
func (s *Service) SetPrimaryAgentResolver(resolver engine.PrimaryAgentResolver) {
|
|
s.enginePrimary = resolver
|
|
s.reinitWorkflowEngine()
|
|
}
|
|
|
|
// SetEngineTaskCreator wires the engine's TaskCreator dependency for
|
|
// the create_child_task action. The orchestrator captures it both as an
|
|
// engine option and as a callback dependency.
|
|
func (s *Service) SetEngineTaskCreator(creator engine.TaskCreator) {
|
|
s.engineTaskCreator = creator
|
|
s.engineOptions = append(s.engineOptions, engine.WithTaskCreator(creator))
|
|
s.reinitWorkflowEngine()
|
|
}
|
|
|
|
// SetEngineWorkflowSwitcher wires the engine's WorkflowSwitcher for the
|
|
// switch_workflow action.
|
|
func (s *Service) SetEngineWorkflowSwitcher(switcher engine.WorkflowSwitcher) {
|
|
s.engineWorkflowSwitcher = switcher
|
|
s.engineOptions = append(s.engineOptions, engine.WithWorkflowSwitcher(switcher))
|
|
s.reinitWorkflowEngine()
|
|
}
|
|
|
|
func (s *Service) reinitWorkflowEngine() {
|
|
if s.workflowStepGetter != nil {
|
|
s.initWorkflowEngine()
|
|
}
|
|
}
|
|
|
|
// WorkflowEngine returns the workflow engine, or nil if it has not been
|
|
// initialised yet (the workflow step getter must be set first).
|
|
//
|
|
// Exposed for cross-package wiring (e.g. office service plumbing the
|
|
// engine into its dispatcher). The engine itself is safe for concurrent
|
|
// HandleTrigger calls.
|
|
func (s *Service) WorkflowEngine() *engine.Engine {
|
|
return s.workflowEngine
|
|
}
|
|
|
|
// startTurnForSession ensures the session has an active turn and returns its ID.
|
|
//
|
|
// Idempotent: if an open turn already exists (either tracked in activeTurns or
|
|
// only present in the DB — the latter happens when service.CreateMessage lazily
|
|
// started a turn for an inbound user message before the orchestrator's prompt
|
|
// cycle began, or when the backend restarted with open turns in the DB), it is
|
|
// adopted rather than duplicated. A new turn is created only when none exists.
|
|
//
|
|
// This avoids the classic dual-creation leak: user message → service.CreateMessage
|
|
// starts turn X (DB only) → PromptTask → startTurnForSession → would create turn Y
|
|
// (DB + activeTurns), leaving X open forever because nothing tracks it.
|
|
func (s *Service) startTurnForSession(ctx context.Context, sessionID string) string {
|
|
if s.turnService == nil {
|
|
return ""
|
|
}
|
|
|
|
if turnIDVal, ok := s.activeTurns.Load(sessionID); ok {
|
|
if turnID, ok := turnIDVal.(string); ok && turnID != "" {
|
|
return turnID
|
|
}
|
|
}
|
|
|
|
if turn, err := s.turnService.GetActiveTurn(ctx, sessionID); turn != nil {
|
|
s.activeTurns.Store(sessionID, turn.ID)
|
|
return turn.ID
|
|
} else if err != nil {
|
|
// A real DB read failure here would otherwise be silently dropped, and
|
|
// we'd fall through to StartTurn — potentially writing a duplicate next
|
|
// to an existing open turn we couldn't see. Log it; the next sweep via
|
|
// completeTurnForSession will mop up any duplicate.
|
|
s.logger.Warn("failed to look up active turn before starting a new one",
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
}
|
|
|
|
turn, err := s.turnService.StartTurn(ctx, sessionID)
|
|
if err != nil {
|
|
s.logger.Warn("failed to start turn",
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
return ""
|
|
}
|
|
|
|
s.activeTurns.Store(sessionID, turn.ID)
|
|
return turn.ID
|
|
}
|
|
|
|
// completeTurnForSession closes any open turn for the session.
|
|
//
|
|
// The DB is the source of truth — activeTurns is just an in-memory cache that
|
|
// can drift (see startTurnForSession) or be wiped by a backend restart. We
|
|
// query the DB for any open turn and close it. Loops to mop up multiple
|
|
// zombies (e.g. left over from before this fix) with a small sanity bound.
|
|
func (s *Service) completeTurnForSession(ctx context.Context, sessionID string) {
|
|
if s.turnService == nil {
|
|
return
|
|
}
|
|
|
|
s.activeTurns.Delete(sessionID)
|
|
|
|
const maxIterations = 16
|
|
closed := 0
|
|
for closed < maxIterations {
|
|
turn, err := s.turnService.GetActiveTurn(ctx, sessionID)
|
|
if err != nil {
|
|
s.logger.Warn("failed to look up active turn",
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
return
|
|
}
|
|
if turn == nil {
|
|
return
|
|
}
|
|
if err := s.turnService.CompleteTurn(ctx, turn.ID); err != nil {
|
|
// GetActiveTurn returns the latest open turn — retrying here
|
|
// would just hit the same row and loop. Bail; the next
|
|
// completeTurnForSession call will pick it up.
|
|
s.logger.Warn("failed to complete turn; will retry on next sweep",
|
|
zap.String("session_id", sessionID),
|
|
zap.String("turn_id", turn.ID),
|
|
zap.Error(err))
|
|
return
|
|
}
|
|
closed++
|
|
}
|
|
// Only warn if turns are *still* accumulating after the cap. Closing
|
|
// exactly maxIterations turns and then finding the session clean is not a
|
|
// runaway.
|
|
if turn, err := s.turnService.GetActiveTurn(ctx, sessionID); err == nil && turn != nil {
|
|
s.logger.Warn("completeTurnForSession iteration cap hit; possible turn close loop",
|
|
zap.String("session_id", sessionID),
|
|
zap.Int("max_iterations", maxIterations))
|
|
}
|
|
}
|
|
|
|
// getActiveTurnID returns the active turn ID for a session.
|
|
// If no active turn exists and the session ID is provided, it will start a new turn.
|
|
// This ensures messages always have a valid turn ID even in edge cases like resumed sessions.
|
|
func (s *Service) getActiveTurnID(sessionID string) string {
|
|
if sessionID == "" {
|
|
return ""
|
|
}
|
|
turnIDVal, ok := s.activeTurns.Load(sessionID)
|
|
if ok {
|
|
turnID, _ := turnIDVal.(string)
|
|
if turnID != "" {
|
|
return turnID
|
|
}
|
|
}
|
|
// No active turn exists - start one lazily
|
|
// This handles edge cases like resumed sessions or race conditions
|
|
return s.startTurnForSession(context.Background(), sessionID)
|
|
}
|
|
|
|
func (s *Service) setSessionResetInProgress(sessionID string, inProgress bool) {
|
|
if sessionID == "" {
|
|
return
|
|
}
|
|
if inProgress {
|
|
s.resetInProgressSessions.Store(sessionID, true)
|
|
return
|
|
}
|
|
s.resetInProgressSessions.Delete(sessionID)
|
|
}
|
|
|
|
func (s *Service) isSessionResetInProgress(sessionID string) bool {
|
|
if sessionID == "" {
|
|
return false
|
|
}
|
|
v, ok := s.resetInProgressSessions.Load(sessionID)
|
|
if !ok {
|
|
return false
|
|
}
|
|
inProgress, _ := v.(bool)
|
|
return inProgress
|
|
}
|
|
|
|
// Start starts all orchestrator components
|
|
func (s *Service) Start(ctx context.Context) error {
|
|
s.mu.Lock()
|
|
if s.running {
|
|
s.mu.Unlock()
|
|
return ErrServiceAlreadyRunning
|
|
}
|
|
s.running = true
|
|
s.startedAt = time.Now()
|
|
s.mu.Unlock()
|
|
|
|
s.logger.Info("starting orchestrator service")
|
|
|
|
// Reconcile session state from persisted runtime state on startup.
|
|
// This does NOT launch any agent processes — sessions are recovered lazily
|
|
// when the user opens them (via task.session.status → task.session.resume).
|
|
s.reconcileSessionsOnStartup(ctx)
|
|
|
|
// Start the watcher first to begin receiving events
|
|
if err := s.watcher.Start(ctx); err != nil {
|
|
s.mu.Lock()
|
|
s.running = false
|
|
s.mu.Unlock()
|
|
return err
|
|
}
|
|
|
|
// Start the scheduler processing loop
|
|
if err := s.scheduler.Start(ctx); err != nil {
|
|
if stopErr := s.watcher.Stop(); stopErr != nil {
|
|
s.logger.Warn("failed to stop watcher after scheduler start failure", zap.Error(stopErr))
|
|
}
|
|
s.mu.Lock()
|
|
s.running = false
|
|
s.mu.Unlock()
|
|
return err
|
|
}
|
|
|
|
// Subscribe to GitHub integration events
|
|
s.subscribeGitHubEvents()
|
|
|
|
// Subscribe to GitLab integration events
|
|
s.subscribeGitLabEvents()
|
|
|
|
// Subscribe to JIRA integration events
|
|
s.subscribeJiraEvents()
|
|
|
|
// Subscribe to Linear integration events
|
|
s.subscribeLinearEvents()
|
|
|
|
// Subscribe to Sentry integration events
|
|
s.subscribeSentryEvents()
|
|
|
|
// Subscribe to automation events
|
|
s.subscribeAutomationEvents()
|
|
|
|
// Subscribe to clarification events (cancel-and-resume flow)
|
|
s.subscribeClarificationEvents()
|
|
|
|
// Subscribe to prepare events (persist result in session metadata)
|
|
s.subscribePrepareEvents()
|
|
|
|
// Subscribe to ADR-0015 step-completion signals (out-of-band path:
|
|
// signal arrives after turn-end).
|
|
s.subscribeStepCompletionEvents()
|
|
|
|
s.logger.Info("orchestrator service started successfully")
|
|
return nil
|
|
}
|
|
|
|
// Stop stops all orchestrator components
|
|
func (s *Service) Stop() error {
|
|
s.mu.Lock()
|
|
if !s.running {
|
|
s.mu.Unlock()
|
|
return ErrServiceNotRunning
|
|
}
|
|
s.running = false
|
|
s.mu.Unlock()
|
|
|
|
s.logger.Info("stopping orchestrator service")
|
|
|
|
// Stop components in reverse order
|
|
var errs []error
|
|
|
|
if err := s.scheduler.Stop(); err != nil {
|
|
s.logger.Error("failed to stop scheduler", zap.Error(err))
|
|
errs = append(errs, err)
|
|
}
|
|
|
|
if err := s.watcher.Stop(); err != nil {
|
|
s.logger.Error("failed to stop watcher", zap.Error(err))
|
|
errs = append(errs, err)
|
|
}
|
|
|
|
s.cancelAllClarificationWatchdogs()
|
|
s.cancelAllTransientRetries()
|
|
|
|
if len(errs) > 0 {
|
|
return errs[0]
|
|
}
|
|
|
|
s.logger.Info("orchestrator service stopped successfully")
|
|
return nil
|
|
}
|
|
|
|
// reconcileSessionsOnStartup adjusts database state for sessions that were active before restart.
|
|
// It does NOT launch any agent processes — sessions are recovered lazily when the user opens them
|
|
// (via task.session.status → task.session.resume or by sending a prompt).
|
|
//
|
|
// Strategy:
|
|
//
|
|
// 1. Terminal states (Completed/Cancelled/Failed) → clean up executor record
|
|
// 2. Never-started (Created) → clean up executor record
|
|
// 3. Active states (Starting/Running/WaitingForInput) → set session to WAITING_FOR_INPUT,
|
|
// clear stale execution IDs, fix task state, preserve ExecutorRunning record
|
|
// 4. Pre-poll remote executor status for remote runtimes (sprites, remote_docker)
|
|
//
|
|
// Called by: Start() method during orchestrator initialization.
|
|
func (s *Service) reconcileSessionsOnStartup(ctx context.Context) {
|
|
runningExecutors, err := s.repo.ListExecutorsRunning(ctx)
|
|
if err != nil {
|
|
s.logger.Warn("failed to list executors running on startup", zap.Error(err))
|
|
return
|
|
}
|
|
if len(runningExecutors) == 0 {
|
|
s.logger.Info("no executors to reconcile on startup")
|
|
return
|
|
}
|
|
|
|
s.logger.Info("reconciling sessions on startup (lazy recovery)", zap.Int("count", len(runningExecutors)))
|
|
|
|
var remoteRecords []executor.RemoteStatusPollRequest
|
|
for _, running := range runningExecutors {
|
|
if models.IsRemoteExecutorType(models.ExecutorType(running.Runtime)) {
|
|
remoteRecords = append(remoteRecords, executor.RemoteStatusPollRequest{
|
|
SessionID: running.SessionID,
|
|
Runtime: running.Runtime,
|
|
AgentExecutionID: running.AgentExecutionID,
|
|
ContainerID: running.ContainerID,
|
|
Metadata: running.Metadata,
|
|
})
|
|
}
|
|
s.reconcileOneSessionOnStartup(ctx, running)
|
|
}
|
|
|
|
// Pre-poll remote executor status so task lists show accurate state
|
|
if len(remoteRecords) > 0 && s.agentManager != nil {
|
|
s.agentManager.PollRemoteStatusForRecords(ctx, remoteRecords)
|
|
}
|
|
}
|
|
|
|
// reconcileOneSessionOnStartup adjusts DB state for a single session without launching agents.
|
|
func (s *Service) reconcileOneSessionOnStartup(ctx context.Context, running *models.ExecutorRunning) {
|
|
sessionID := running.SessionID
|
|
if sessionID == "" {
|
|
return
|
|
}
|
|
|
|
session, err := s.repo.GetTaskSession(ctx, sessionID)
|
|
if err != nil {
|
|
s.logger.Warn("failed to load session for reconciliation",
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
return
|
|
}
|
|
|
|
previousState := session.State
|
|
|
|
// Handle terminal and never-started states (reuse existing cleanup logic)
|
|
if skip := s.handleTerminalSessionOnStartup(ctx, session, running, previousState); skip {
|
|
return
|
|
}
|
|
if previousState == models.TaskSessionStateCreated {
|
|
s.logger.Info("session was never started; cleaning up",
|
|
zap.String("session_id", sessionID))
|
|
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
|
|
s.logger.Warn("failed to remove executor record for unstarted session",
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
}
|
|
return
|
|
}
|
|
|
|
// IDLE is the office "between turns" resting state — agent process
|
|
// and executor are already torn down, conversation is preserved
|
|
// in the existing executors_running row for the next run. There
|
|
// is nothing to recover; flipping to WAITING_FOR_INPUT here would
|
|
// (a) lie about the session having pending user input and (b) make
|
|
// the chat UI render as "working" because the office session shape
|
|
// uses IDLE specifically to avoid that.
|
|
if previousState == models.TaskSessionStateIdle {
|
|
s.logger.Info("session reconciled for lazy recovery (idle, no state change)",
|
|
zap.String("session_id", sessionID),
|
|
zap.String("task_id", running.TaskID),
|
|
zap.Bool("has_resume_token", running.ResumeToken != ""),
|
|
zap.Bool("has_worktree", running.WorktreePath != ""))
|
|
return
|
|
}
|
|
|
|
// Active states: STARTING, RUNNING, WAITING_FOR_INPUT
|
|
// Set session to WAITING_FOR_INPUT (idle, ready for lazy resume when user opens it)
|
|
if previousState != models.TaskSessionStateWaitingForInput {
|
|
if err := s.repo.UpdateTaskSessionState(ctx, sessionID, models.TaskSessionStateWaitingForInput, ""); err != nil {
|
|
s.logger.Warn("failed to set session to WAITING_FOR_INPUT on startup",
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
}
|
|
}
|
|
|
|
// PRESERVE executors_running.agent_execution_id post-restart. The in-memory
|
|
// process is gone, but the stored ID still serves as a "this session was
|
|
// previously launched" marker that drives resume detection in
|
|
// applyRunningRecordToResumeRequest → isResumedSession → ResumePassthroughSession
|
|
// (which adds the --resume flag). Clearing it here makes the next launch take
|
|
// the fresh-start path and skip the resume CLI flag, so passthrough TUIs lose
|
|
// their conversation context after every backend restart. The stale ID is
|
|
// harmless: persistExecutorRunning UPSERTs it atomically alongside the
|
|
// in-memory Add on the next launch, closing the divergence window we set out
|
|
// to fix in this refactor.
|
|
|
|
// Ensure task is in REVIEW state (not stuck IN_PROGRESS)
|
|
if running.TaskID != "" {
|
|
task, taskErr := s.taskRepo.GetTask(ctx, running.TaskID)
|
|
if taskErr == nil && task != nil && task.State == v1.TaskStateInProgress {
|
|
if updateErr := s.taskRepo.UpdateTaskState(ctx, running.TaskID, v1.TaskStateReview); updateErr != nil {
|
|
s.logger.Warn("failed to update task to REVIEW on startup",
|
|
zap.String("task_id", running.TaskID),
|
|
zap.Error(updateErr))
|
|
}
|
|
}
|
|
}
|
|
|
|
// PRESERVE the ExecutorRunning record — it holds the resume token and worktree info
|
|
// needed for lazy recovery when the user opens the session.
|
|
|
|
s.logger.Info("session reconciled for lazy recovery",
|
|
zap.String("session_id", sessionID),
|
|
zap.String("task_id", running.TaskID),
|
|
zap.String("previous_state", string(previousState)),
|
|
zap.Bool("has_resume_token", running.ResumeToken != ""),
|
|
zap.Bool("has_worktree", running.WorktreePath != ""))
|
|
}
|
|
|
|
// handleTerminalSessionOnStartup processes sessions in terminal states during startup.
|
|
// Returns true if the session should be skipped (no further processing needed).
|
|
func (s *Service) handleTerminalSessionOnStartup(ctx context.Context, session *models.TaskSession, running *models.ExecutorRunning, previousState models.TaskSessionState) bool {
|
|
sessionID := session.ID
|
|
switch previousState {
|
|
case models.TaskSessionStateCompleted, models.TaskSessionStateCancelled:
|
|
s.logger.Info("session in terminal state; cleaning up executor record",
|
|
zap.String("session_id", sessionID),
|
|
zap.String("task_id", session.TaskID),
|
|
zap.String("state", string(previousState)))
|
|
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
|
|
s.logger.Warn("failed to remove executor record for terminal session",
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
}
|
|
return true
|
|
case models.TaskSessionStateFailed:
|
|
s.handleFailedSessionOnStartup(ctx, session, running)
|
|
return true
|
|
}
|
|
return false
|
|
}
|
|
|
|
// handleFailedSessionOnStartup handles a failed session during startup recovery.
|
|
func (s *Service) handleFailedSessionOnStartup(ctx context.Context, session *models.TaskSession, running *models.ExecutorRunning) {
|
|
sessionID := session.ID
|
|
// If session failed, ensure task is in REVIEW state (not stuck IN_PROGRESS)
|
|
if session.TaskID != "" {
|
|
task, taskErr := s.taskRepo.GetTask(ctx, session.TaskID)
|
|
if taskErr == nil && task.State == v1.TaskStateInProgress {
|
|
s.logger.Info("fixing task state: session failed but task still IN_PROGRESS",
|
|
zap.String("task_id", session.TaskID),
|
|
zap.String("session_id", sessionID))
|
|
if updateErr := s.taskRepo.UpdateTaskState(ctx, session.TaskID, v1.TaskStateReview); updateErr != nil {
|
|
s.logger.Warn("failed to update task state to REVIEW",
|
|
zap.String("task_id", session.TaskID),
|
|
zap.Error(updateErr))
|
|
}
|
|
}
|
|
}
|
|
if canResumeRunning(running) {
|
|
s.logger.Info("preserving executor record for resumable failed session",
|
|
zap.String("session_id", sessionID),
|
|
zap.String("task_id", session.TaskID))
|
|
} else {
|
|
s.logger.Info("cleaning up executor record for non-resumable failed session",
|
|
zap.String("session_id", sessionID),
|
|
zap.String("task_id", session.TaskID))
|
|
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
|
|
s.logger.Warn("failed to remove executor record for failed session",
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
}
|
|
}
|
|
}
|
|
|
|
func canResumeRunning(running *models.ExecutorRunning) bool {
|
|
if running == nil || running.ResumeToken == "" {
|
|
return false
|
|
}
|
|
if running.Resumable {
|
|
return true
|
|
}
|
|
return models.IsAlwaysResumableRuntime(running.Runtime)
|
|
}
|
|
|
|
func isMissingMergedPRBranchError(err error) bool {
|
|
if err == nil {
|
|
return false
|
|
}
|
|
msg := strings.ToLower(err.Error())
|
|
return strings.Contains(msg, "couldn't find remote ref") ||
|
|
(strings.Contains(msg, "not found locally or on remote") && strings.Contains(msg, "branch")) ||
|
|
(strings.Contains(msg, "pathspec") && strings.Contains(msg, "did not match"))
|
|
}
|
|
|
|
var (
|
|
quotedBranchPattern = regexp.MustCompile(`branch "([^"]+)"`)
|
|
remoteRefPattern = regexp.MustCompile(`remote ref ([^\s]+)`)
|
|
pathspecBranchPattern = regexp.MustCompile(`pathspec '([^']+)'`)
|
|
)
|
|
|
|
func extractMissingBranchName(err error) string {
|
|
if err == nil {
|
|
return ""
|
|
}
|
|
msg := err.Error()
|
|
if match := quotedBranchPattern.FindStringSubmatch(msg); len(match) == 2 {
|
|
return strings.TrimSpace(match[1])
|
|
}
|
|
if match := remoteRefPattern.FindStringSubmatch(msg); len(match) == 2 {
|
|
return strings.TrimSpace(match[1])
|
|
}
|
|
if match := pathspecBranchPattern.FindStringSubmatch(msg); len(match) == 2 {
|
|
return strings.TrimSpace(match[1])
|
|
}
|
|
return ""
|
|
}
|
|
|
|
func (s *Service) handleSessionLaunchFailed(ctx context.Context, taskID, sessionID string, launchErr error) {
|
|
if s.messageCreator == nil || !isMissingMergedPRBranchError(launchErr) {
|
|
return
|
|
}
|
|
|
|
branch := extractMissingBranchName(launchErr)
|
|
content := "This task references a PR branch that no longer exists on remote (likely merged and deleted)."
|
|
if branch != "" {
|
|
content = "The remote PR branch \"" + branch + "\" no longer exists (likely merged and deleted)."
|
|
}
|
|
metadata := map[string]interface{}{
|
|
"variant": "warning",
|
|
"failure_kind": "missing_pr_branch",
|
|
"missing_branch": branch,
|
|
"actions": []map[string]interface{}{
|
|
{
|
|
actionMetaKeyType: "archive_task",
|
|
actionMetaKeyLabel: "Archive task",
|
|
actionMetaKeyTooltip: "Keep task history and hide it from active work",
|
|
actionMetaKeyIcon: "archive",
|
|
actionMetaKeyTestID: "missing-branch-archive-button",
|
|
},
|
|
{
|
|
actionMetaKeyType: "delete_task",
|
|
actionMetaKeyLabel: "Delete task",
|
|
actionMetaKeyTooltip: "Permanently remove this task",
|
|
"variant": "destructive",
|
|
actionMetaKeyIcon: "trash",
|
|
actionMetaKeyTestID: "missing-branch-delete-button",
|
|
},
|
|
},
|
|
}
|
|
s.suppressToast.Store(sessionID, true)
|
|
msgCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
|
defer cancel()
|
|
if err := s.messageCreator.CreateSessionMessage(
|
|
msgCtx,
|
|
taskID,
|
|
content,
|
|
sessionID,
|
|
string(v1.MessageTypeStatus),
|
|
"", // No turn — agent never started, avoid lazily creating a synthetic turn.
|
|
metadata,
|
|
false,
|
|
); err != nil {
|
|
s.logger.Warn("failed to create missing PR branch launch failure message",
|
|
zap.String("task_id", taskID),
|
|
zap.String("session_id", sessionID),
|
|
zap.Error(err))
|
|
}
|
|
}
|
|
|
|
// IsRunning returns true if the service is running
|
|
func (s *Service) IsRunning() bool {
|
|
s.mu.RLock()
|
|
defer s.mu.RUnlock()
|
|
return s.running
|
|
}
|
|
|
|
// GetStatus returns the orchestrator status
|
|
func (s *Service) GetStatus() *Status {
|
|
s.mu.RLock()
|
|
running := s.running
|
|
startedAt := s.startedAt
|
|
s.mu.RUnlock()
|
|
|
|
queueStatus := s.scheduler.GetQueueStatus()
|
|
|
|
var uptimeSeconds int64
|
|
if running {
|
|
uptimeSeconds = int64(time.Since(startedAt).Seconds())
|
|
}
|
|
|
|
return &Status{
|
|
Running: running,
|
|
ActiveAgents: queueStatus.ActiveExecutions,
|
|
QueuedTasks: queueStatus.QueuedTasks,
|
|
TotalProcessed: queueStatus.TotalProcessed,
|
|
TotalFailed: queueStatus.TotalFailed,
|
|
UptimeSeconds: uptimeSeconds,
|
|
LastHeartbeat: time.Now(),
|
|
}
|
|
}
|
|
|
|
// GetMessageQueue returns the message queue service
|
|
func (s *Service) GetMessageQueue() *messagequeue.Service {
|
|
return s.messageQueue
|
|
}
|
|
|
|
// GetEventBus returns the event bus
|
|
func (s *Service) GetEventBus() bus.EventBus {
|
|
return s.eventBus
|
|
}
|