Files
kandev/apps/backend/internal/orchestrator/service.go
T
João Salavisa c11a6bbfd2 feat(github): add issue watcher with task creation and auto-cleanup (#672)
* feat(github): add issue watcher with task creation and auto-cleanup

Add a GitHub Issue Watcher feature that polls for new issues matching
configurable filters and automatically creates tasks for them. This
complements the existing PR Review Watcher with the same patterns.

Backend:
- Issue watch CRUD (models, store, service, controller, WS handlers)
- Poller integration with configurable intervals
- Event-driven task creation via orchestrator
- Repository resolution and worktree support for issue tasks
- Auto-cleanup of tasks for closed issues (CleanupClosedIssueTasks)
- GetIssueState client method across all implementations
- Default branch auto-detection from cloned repo HEAD

Frontend:
- Issue watch settings UI (table, dialog, placeholders)
- Issue link icon on task cards (green circle-dot)
- API client, store slice, and hook for issue watches
- SSR and WS hydration of issue metadata

* fix: resolve golangci-lint errors (goconst, nestif, dupl) and prettier formatting

* fix(github): fix issue watch update shape, trigger key, and label quoting

- httpUpdateIssueWatch now returns the full IssueWatch object instead of
  {updated:true} so the frontend store correctly refreshes after edits
- httpTriggerIssueWatch returns new_issues_found to match TriggerIssueResponse
  type; was returning new_issues so the toast count was always 0
- buildIssueFilter uses conditional quoting instead of fmt.Sprintf %q to
  avoid Go escape sequences in GitHub search qualifiers

Co-authored-by: João Salavisa <undefined@users.noreply.github.com>

* fix: resolve eslint warnings (max-lines-per-function, no-nested-ternary)

* test(github): add issue watcher tests and fix review feedback

- Add TestInterpolateIssuePrompt (4 cases: empty/all/none/partial)
- Add TestCreateIssueTask (3 cases: skip/reserve+assign/release on fail)
- Add TestShouldDeleteIssueTask (4 cases: open/closed+sessions/closed/error)
- Fix WS response key new_issues -> new_issues_found
- Fix fmt.Printf -> fmt.Fprintf(os.Stderr) in store.go
- Add CleanupClosedIssueTasks call in WS trigger handler

* fix: address PR review feedback and CI failures

- Split test files to fix revive file-length-limit (service_test.go,
  event_handlers_github_test.go)
- Add session checker guard in CleanupClosedIssueTasks
- Normalize poll intervals on UpdateIssueWatch
- Add cleanup call in TriggerAllIssueChecks loop
- Log cleanup errors in controller and WS handler
- Prefer refs/remotes/origin/HEAD in detectGitDefaultBranch
- Add aria-label to issue task icon
- Fix TypeScript type errors: add linkedIssues to GitHubSlice,
  fix hook exports, remove invalid imports

* fix: split test files exceeding revive limit and clean up minor issues

- Split service_test.go (883→779 lines): move TestTriggerPRSync_* to
  service_pr_sync_test.go
- Split event_handlers_github_test.go (806→677 lines): move
  TestCreateReviewTask_* and helpers to event_handlers_github_review_test.go
- Remove redundant s.taskSessionChecker nil-check inside shouldDeleteIssueTask
  (early guard at CleanupClosedIssueTasks already ensures it is non-nil)
- Remove unreachable dead guard in detectAndPersistDefaultBranch
  (repo.DefaultBranch is always "" at that call site)
- Fix bogus #0 issue badge in task-session-sidebar.tsx when issueNumber is absent

Co-authored-by: João Salavisa <undefined@users.noreply.github.com>

* fix: address Claude review — poll clamp and prompt injection

- Handle v <= 0 in UpdateIssueWatch poll interval (fall back to default)
- Remove {{issue.body}} from default template to prevent prompt injection
- Update test expected output for new default template

* fix: remove {{issue.body}} from frontend DEFAULT_ISSUE_WATCH_PROMPT

Mirrors the backend fix — the UI pre-fills the prompt so the backend
fallback was never reached. The placeholder stays in
ISSUE_WATCH_PLACEHOLDERS for explicit opt-in.

* feat(github): add isIssueWatch sidebar filter dimension

Add isIssueWatch boolean to KanbanState, TaskSwitcherItem, and sidebar
filter registry — mirroring the isPRReview pattern. Issue-watch tasks
(created from issue_watch_id in metadata) can now be explicitly filtered
in the sidebar, just like PR review tasks.

Files changed:
- sidebar-view-types: add isIssueWatch dimension
- filter-dimension-registry: add Issue watch filter meta
- apply-view: add isIssueWatch extractor
- kanban types: add isIssueWatch field
- TaskSwitcherItem: add isIssueWatch field
- use-all-workflow-snapshots: derive isIssueWatch from metadata
- ssr/mapper: derive isIssueWatch from metadata
- task-session-sidebar: pass isIssueWatch, extract helpers to fix complexity

* fix: correct deleteIssueWatch response type to match backend

Backend httpDeleteIssueWatch returns {"deleted": true}; the frontend
type was { success: boolean } — a mismatch with no functional impact
(return value is ignored) but wrong per TypeScript contract.

Co-authored-by: João Salavisa <undefined@users.noreply.github.com>

* refactor: extract metadata helpers to shared lib/metadata-utils.ts

- Move isPRReviewFromMetadata, isIssueWatchFromMetadata, and
  issueFieldsFromMetadata to a shared module to eliminate duplicate
  definitions (sonarjs/no-identical-functions).
- Fix deleteIssueWatch response type: { success: boolean } -> { deleted: boolean }
  to match the actual backend response.

* style: fix prettier formatting

* test(web): add tests for isIssueWatch filter dimension and metadata-utils

Co-authored-by: João Salavisa <undefined@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: fix prettier formatting in test files

* fix(github): cascade-delete issue watch tasks on watch deletion and add isPRReview to SSR mapper

- DeleteIssueWatch now wraps in a transaction and deletes related
  github_issue_watch_tasks rows before removing the watch; prevents
  orphaned dedup rows accumulating after a watch is deleted
- snapshotToState in mapper.ts now also sets isPRReview from metadata,
  matching the mapping already done in use-all-workflow-snapshots.ts

Co-authored-by: João Salavisa <undefined@users.noreply.github.com>

* style: extract duplicate string constant in metadata-utils test

* style: extract repeated WS error strings into constants in handlers.go

goconst min-occurrences: 3 — 'invalid payload' (6x) and 'id required' (4x)
both exceed the threshold. Extract to package-level constants.

Co-authored-by: João Salavisa <undefined@users.noreply.github.com>

* style: extract repeated issue key strings into constants in event_handlers_github.go

goconst min-occurrences: 3 — 'issue_number' (9x) and 'issue_watch_id' (5x)
both newly introduced in this PR exceed the threshold.

Co-authored-by: João Salavisa <undefined@users.noreply.github.com>

* style: fix prettier formatting in metadata-utils test

---------

Co-authored-by: Kandev Agent <agent@kandev.dev>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: João Salavisa <undefined@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 17:32:46 +01:00

899 lines
34 KiB
Go

// Package orchestrator provides the main orchestrator service that coordinates
// task execution across agents. It manages:
//
// - Task queuing and scheduling via the Scheduler
// - Agent lifecycle through the AgentManager
// - Event handling and propagation
// - Session management and resume
//
// The orchestrator acts as the central coordinator between the task service,
// agent lifecycle manager, and the event bus.
package orchestrator
import (
"context"
"errors"
"regexp"
"strings"
"sync"
"time"
"go.uber.org/zap"
"github.com/kandev/kandev/internal/agentctl/types/streams"
"github.com/kandev/kandev/internal/common/logger"
"github.com/kandev/kandev/internal/events/bus"
"github.com/kandev/kandev/internal/orchestrator/executor"
"github.com/kandev/kandev/internal/orchestrator/messagequeue"
"github.com/kandev/kandev/internal/orchestrator/queue"
"github.com/kandev/kandev/internal/orchestrator/scheduler"
"github.com/kandev/kandev/internal/orchestrator/watcher"
"github.com/kandev/kandev/internal/secrets"
"github.com/kandev/kandev/internal/task/models"
"github.com/kandev/kandev/internal/workflow/engine"
wfmodels "github.com/kandev/kandev/internal/workflow/models"
v1 "github.com/kandev/kandev/pkg/api/v1"
)
// Common errors
var (
ErrServiceAlreadyRunning = errors.New("service is already running")
ErrServiceNotRunning = errors.New("service is not running")
)
// ServiceConfig holds orchestrator service configuration
type ServiceConfig struct {
Scheduler scheduler.SchedulerConfig
QueueSize int
QueueGroup string
}
// DefaultServiceConfig returns default configuration
func DefaultServiceConfig() ServiceConfig {
return ServiceConfig{
Scheduler: scheduler.DefaultSchedulerConfig(),
QueueSize: 1000,
QueueGroup: "orchestrator",
}
}
// MessageCreator is an interface for creating messages on tasks
type MessageCreator interface {
CreateAgentMessage(ctx context.Context, taskID, content, agentSessionID, turnID string) error
CreateUserMessage(ctx context.Context, taskID, content, agentSessionID, turnID string, metadata map[string]interface{}) error
// CreateToolCallMessage creates a message for a tool call.
// normalized contains the typed tool payload data.
// parentToolCallID is the parent Task tool call ID for subagent nesting (empty for top-level).
CreateToolCallMessage(ctx context.Context, taskID, toolCallID, parentToolCallID, title, status, agentSessionID, turnID string, normalized *streams.NormalizedPayload) error
// UpdateToolCallMessage updates a tool call message's status and optionally its normalized data.
// If the message doesn't exist, it creates it using taskID, turnID, and msgType.
// normalized contains the typed tool payload data.
// parentToolCallID is the parent Task tool call ID for subagent nesting (empty for top-level).
UpdateToolCallMessage(ctx context.Context, taskID, toolCallID, parentToolCallID, status, result, agentSessionID, title, turnID, msgType string, normalized *streams.NormalizedPayload) error
CreateSessionMessage(ctx context.Context, taskID, content, agentSessionID, messageType, turnID string, metadata map[string]interface{}, requestsInput bool) error
CreatePermissionRequestMessage(ctx context.Context, taskID, sessionID, pendingID, toolCallID, title, turnID string, options []map[string]interface{}, actionType string, actionDetails map[string]interface{}) (string, error)
UpdatePermissionMessage(ctx context.Context, sessionID, pendingID, status string) error
// CreateAgentMessageStreaming creates a new agent message with a pre-generated ID for streaming updates
CreateAgentMessageStreaming(ctx context.Context, messageID, taskID, content, agentSessionID, turnID string) error
// AppendAgentMessage appends additional content to an existing streaming message
AppendAgentMessage(ctx context.Context, messageID, additionalContent string) error
// CreateThinkingMessageStreaming creates a new thinking message with a pre-generated ID for streaming updates
CreateThinkingMessageStreaming(ctx context.Context, messageID, taskID, content, agentSessionID, turnID string) error
// AppendThinkingMessage appends additional content to an existing streaming thinking message
AppendThinkingMessage(ctx context.Context, messageID, additionalContent string) error
// InvalidateModelCache clears any cached model for a session, forcing the next
// message to re-read the model from the DB. Called after model switches.
InvalidateModelCache(sessionID string)
}
// TurnService is an interface for managing session turns
type TurnService interface {
StartTurn(ctx context.Context, sessionID string) (*models.Turn, error)
CompleteTurn(ctx context.Context, turnID string) error
GetActiveTurn(ctx context.Context, sessionID string) (*models.Turn, error)
}
// WorkflowStepGetter retrieves workflow step information for prompt building.
type WorkflowStepGetter interface {
GetStep(ctx context.Context, stepID string) (*wfmodels.WorkflowStep, error)
GetNextStepByPosition(ctx context.Context, workflowID string, currentPosition int) (*wfmodels.WorkflowStep, error)
GetPreviousStepByPosition(ctx context.Context, workflowID string, currentPosition int) (*wfmodels.WorkflowStep, error)
GetWorkflowAgentProfileID(ctx context.Context, workflowID string) (string, error)
}
// repoStore is the repository interface accepted by NewService.
// It covers both the orchestrator's own needs (sessionExecutorStore) and
// the executor package's needs (executor.executorStore).
type repoStore interface {
sessionExecutorStore
// Additional methods needed by executor
UpdateTaskState(ctx context.Context, id string, state v1.TaskState) error
GetPrimaryTaskRepository(ctx context.Context, taskID string) (*models.TaskRepository, error)
CreateTaskSession(ctx context.Context, session *models.TaskSession) error
UpdateTaskSession(ctx context.Context, session *models.TaskSession) error
ListActiveTaskSessions(ctx context.Context) ([]*models.TaskSession, error)
ListActiveTaskSessionsByTaskID(ctx context.Context, taskID string) ([]*models.TaskSession, error)
CreateTaskSessionWorktree(ctx context.Context, sessionWorktree *models.TaskSessionWorktree) error
ListSessionsWithBranches(ctx context.Context) ([]models.SessionBranchInfo, error)
GetRepository(ctx context.Context, id string) (*models.Repository, error)
UpdateRepository(ctx context.Context, repository *models.Repository) error
GetExecutorProfile(ctx context.Context, id string) (*models.ExecutorProfile, error)
GetWorkspace(ctx context.Context, id string) (*models.Workspace, error)
// Session history + plan (for context handover)
GetTaskPlan(ctx context.Context, taskID string) (*models.TaskPlan, error)
}
// sessionExecutorStore is the minimal repository interface needed by the orchestrator service.
type sessionExecutorStore interface {
// Session
GetTaskSession(ctx context.Context, id string) (*models.TaskSession, error)
GetActiveTaskSessionByTaskID(ctx context.Context, taskID string) (*models.TaskSession, error)
SetSessionPrimary(ctx context.Context, sessionID string) error
UpdateTaskSession(ctx context.Context, session *models.TaskSession) error
UpdateTaskSessionState(ctx context.Context, id string, state models.TaskSessionState, errorMessage string) error
UpdateTaskSessionBaseCommit(ctx context.Context, id string, baseCommitSHA string) error
UpdateTaskSessionWorktreeBranch(ctx context.Context, sessionID, branch string) error
ClearSessionExecutionID(ctx context.Context, id string) error
UpdateSessionReviewStatus(ctx context.Context, sessionID string, status string) error
UpdateSessionMetadata(ctx context.Context, sessionID string, metadata map[string]interface{}) error
SetSessionMetadataKey(ctx context.Context, sessionID, key string, value interface{}) error
// Executor running state
ListExecutorsRunning(ctx context.Context) ([]*models.ExecutorRunning, error)
UpsertExecutorRunning(ctx context.Context, running *models.ExecutorRunning) error
GetExecutorRunningBySessionID(ctx context.Context, sessionID string) (*models.ExecutorRunning, error)
DeleteExecutorRunningBySessionID(ctx context.Context, sessionID string) error
// Executor
GetExecutor(ctx context.Context, id string) (*models.Executor, error)
// Task
GetTask(ctx context.Context, id string) (*models.Task, error)
UpdateTask(ctx context.Context, task *models.Task) error
// Git snapshots and commits
GetLatestGitSnapshot(ctx context.Context, sessionID string) (*models.GitSnapshot, error)
CreateGitSnapshot(ctx context.Context, snapshot *models.GitSnapshot) error
UpsertLatestLiveGitSnapshot(ctx context.Context, snapshot *models.GitSnapshot) error
CreateSessionCommit(ctx context.Context, commit *models.SessionCommit) error
GetSessionCommits(ctx context.Context, sessionID string) ([]*models.SessionCommit, error)
DeleteSessionCommit(ctx context.Context, id string) error
// Session listing + delete
ListTaskSessions(ctx context.Context, taskID string) ([]*models.TaskSession, error)
DeleteTaskSession(ctx context.Context, id string) error
// Task environment
GetTaskEnvironmentByTaskID(ctx context.Context, taskID string) (*models.TaskEnvironment, error)
CreateTaskEnvironment(ctx context.Context, env *models.TaskEnvironment) error
UpdateTaskEnvironment(ctx context.Context, env *models.TaskEnvironment) error
}
// Service is the main orchestrator service
type Service struct {
config ServiceConfig
logger *logger.Logger
eventBus bus.EventBus
taskRepo scheduler.TaskRepository
repo sessionExecutorStore
agentManager executor.AgentManagerClient
// Components
queue *queue.TaskQueue
executor *executor.Executor
scheduler *scheduler.Scheduler
watcher *watcher.Watcher
// Message queue service for queueing messages while agent is running
messageQueue *messagequeue.Service
// Message creator for saving agent responses
messageCreator MessageCreator
// Turn service for managing session turns
turnService TurnService
// Workflow step getter for prompt building
workflowStepGetter WorkflowStepGetter
// Workflow engine for typed state-machine evaluation of step transitions
workflowEngine *engine.Engine
workflowStore *workflowStore
// GitHub service for PR auto-detection on push
githubService GitHubService
// Review task creator for auto-creating tasks from review watch PRs
reviewTaskCreator ReviewTaskCreator
// Issue task creator for auto-creating tasks from issue watch events
issueTaskCreator IssueTaskCreator
// Repository resolver for cloning + finding/creating repos for review tasks
repositoryResolver RepositoryResolver
// Clarification canceller — cancels pending clarifications when agent's turn completes
clarificationCanceller ClarificationCanceller
// Push tracker: sessionID -> last known ahead count
pushTracker sync.Map
// gitSnapshotCache throttles per-session writes of the live git status
// snapshot (used by handleGitStatusUpdate -> persistGitStatusSnapshot).
gitSnapshotCache *gitSnapshotCache
// Active turns map: sessionID -> turnID
activeTurns sync.Map
// Session reset flags: sessionID -> true while resetAgentContext is restarting process.
// Used to suppress stale ready events and avoid draining queued prompts mid-reset.
resetInProgressSessions sync.Map
// suppressToast: sessionID -> true. Set by failure handlers that create
// guidance messages with actions. Cleared by updateTaskSessionState which
// adds suppress_toast to the WS event so the frontend skips the error toast.
suppressToast sync.Map
// clarificationWatchdogs tracks active clarification primary-path resume watchdogs.
// key: "<session_id>::<pending_id>", value: *clarificationWatchdogEntry
clarificationWatchdogs sync.Map
// clarificationWatchdogTimeout controls how long to wait for post-clarification
// activity before triggering fallback resume.
clarificationWatchdogTimeout time.Duration
// Service state
mu sync.RWMutex
running bool
startedAt time.Time
}
// Status contains orchestrator status information
type Status struct {
Running bool `json:"running"`
ActiveAgents int `json:"active_agents"`
QueuedTasks int `json:"queued_tasks"`
TotalProcessed int64 `json:"total_processed"`
TotalFailed int64 `json:"total_failed"`
UptimeSeconds int64 `json:"uptime_seconds"`
LastHeartbeat time.Time `json:"last_heartbeat"`
}
// NewService creates a new orchestrator service
func NewService(
cfg ServiceConfig,
eventBus bus.EventBus,
agentManager executor.AgentManagerClient,
taskRepo scheduler.TaskRepository,
repo repoStore,
shellPrefs executor.ShellPreferenceProvider,
secretStore secrets.SecretStore,
log *logger.Logger,
) *Service {
svcLogger := log.WithFields(zap.String("component", "orchestrator"))
// Create the task queue with configured size
taskQueue := queue.NewTaskQueue(cfg.QueueSize)
// Create the executor with the agent manager client and repository for persistent sessions
execCfg := executor.ExecutorConfig{
ShellPrefs: shellPrefs,
SecretStore: secretStore,
}
exec := executor.NewExecutor(agentManager, repo, log, execCfg)
// Create the scheduler with queue, executor, and task repository
sched := scheduler.NewScheduler(taskQueue, exec, taskRepo, log, cfg.Scheduler)
// Create the message queue service
msgQueue := messagequeue.NewService(log)
// Create the service (watcher will be created after we have handlers)
s := &Service{
config: cfg,
logger: svcLogger,
eventBus: eventBus,
taskRepo: taskRepo,
repo: repo,
agentManager: agentManager,
queue: taskQueue,
executor: exec,
scheduler: sched,
messageQueue: msgQueue,
clarificationWatchdogTimeout: 15 * time.Second,
gitSnapshotCache: newGitSnapshotCache(),
}
// Wire executor state changes through the orchestrator so events are published
// (e.g. WebSocket notifications to the frontend). Must be set after service
// construction so the session callback can reference s.updateTaskSessionState.
exec.SetOnTaskStateChange(func(ctx context.Context, taskID string, state v1.TaskState) error {
return taskRepo.UpdateTaskState(ctx, taskID, state)
})
exec.SetOnSessionStateChange(func(ctx context.Context, taskID, sessionID string, state models.TaskSessionState, errorMessage string) error {
s.updateTaskSessionState(ctx, taskID, sessionID, state, errorMessage, true)
return nil
})
exec.SetOnLaunchFailed(s.handleSessionLaunchFailed)
exec.SetOnAgentStartFailed(s.handleAgentStartFailed)
if caps, ok := agentManager.(executor.ExecutorTypeCapabilities); ok {
exec.SetCapabilities(caps)
}
// Create the watcher with event handlers that wire everything together
handlers := watcher.EventHandlers{
OnTaskDeleted: s.handleTaskDeleted,
OnAgentRunning: s.handleAgentRunning,
OnAgentReady: s.handleAgentReady,
OnAgentCompleted: s.handleAgentCompleted,
OnAgentFailed: s.handleAgentFailed,
OnAgentStopped: s.handleAgentStopped,
OnAgentStreamEvent: s.handleAgentStreamEvent,
OnACPSessionCreated: s.handleACPSessionCreated,
OnPermissionRequest: s.handlePermissionRequest,
OnGitEvent: s.handleGitEvent,
OnContextWindowUpdated: s.handleContextWindowUpdated,
OnTaskMoved: s.handleTaskMoved,
}
s.watcher = watcher.NewWatcher(eventBus, handlers, cfg.QueueGroup, log)
return s
}
// SetMessageCreator sets the message creator for saving agent responses to the database.
//
// This must be called before starting the orchestrator if you want agent messages, tool calls,
// and streaming content to be persisted to the database. The MessageCreator interface provides
// methods for creating and updating messages associated with task sessions.
//
// The MessageCreator is typically the task service, which owns the message persistence logic.
// Event handlers in the orchestrator call these methods when agent events occur:
// - AgentStreamEvent → CreateAgentMessage, AppendAgentMessage
// - Tool calls → CreateToolCallMessage, UpdateToolCallMessage
// - Permission requests → CreatePermissionRequestMessage
//
// When to call: During orchestrator initialization, after creating the task service.
//
// If not set: Agent messages won't be saved to the database (events will still be published).
func (s *Service) SetMessageCreator(mc MessageCreator) {
s.messageCreator = mc
}
// SetOnPrimarySessionSet sets a callback on the executor for when the first session
// of a task is marked primary. Used to publish a task.updated event so the frontend
// receives the primary_session_id.
func (s *Service) SetOnPrimarySessionSet(fn executor.PrimarySessionSetFunc) {
s.executor.SetOnPrimarySessionSet(fn)
}
// SetRepoCloner sets the repository cloner and updater on the executor, enabling automatic
// cloning of provider-backed repositories (e.g. from a GitHub URL) when they are launched
// for local/worktree execution and have no local path yet.
func (s *Service) SetRepoCloner(cloner executor.RepoCloner, updater executor.RepoUpdater) {
s.executor.SetRepoCloner(cloner, updater)
}
// SetTurnService sets the turn service for tracking conversation turns.
//
// A "turn" represents a single conversation round-trip: user prompt → agent response.
// The TurnService tracks turn timing and duration for analytics and UI display (e.g., showing
// how long each agent response took).
//
// The TurnService is typically the task service, which owns turn persistence logic.
// The orchestrator calls these methods:
// - StartTurn: When agent begins processing a prompt
// - CompleteTurn: When agent finishes and returns to ready state
// - GetActiveTurn: To associate messages with current turn
//
// When to call: During orchestrator initialization, after creating the task service.
//
// If not set: Turns won't be tracked (orchestrator continues functioning normally, but
// no timing data is recorded and turn IDs in messages will be empty).
func (s *Service) SetTurnService(turnService TurnService) {
s.turnService = turnService
}
// SetWorkflowStepGetter sets the workflow step getter for prompt building.
//
// When workflow_step_id is provided to StartTask, the orchestrator uses this getter
// to retrieve the step's prompt_prefix, prompt_suffix, and plan_mode settings to
// build the effective prompt.
//
// If not set: workflow_step_id in StartTask is ignored and the prompt is used as-is.
func (s *Service) SetWorkflowStepGetter(getter WorkflowStepGetter) {
s.workflowStepGetter = getter
s.initWorkflowEngine()
}
// ClarificationCanceller cancels pending clarifications when an agent's turn completes.
type ClarificationCanceller interface {
CancelSessionAndNotify(ctx context.Context, sessionID string) int
}
// SetClarificationCanceller sets the canceller for cleaning up pending clarifications on turn complete.
func (s *Service) SetClarificationCanceller(c ClarificationCanceller) {
s.clarificationCanceller = c
}
// initWorkflowEngine creates the workflow engine with store and callbacks.
// Called after the workflow step getter is set. Safe to call multiple times.
func (s *Service) initWorkflowEngine() {
if s.workflowStepGetter == nil {
return
}
store := newWorkflowStore(s.repo, s.workflowStepGetter, s.agentManager, s.eventBus, s.logger)
callbacks := buildWorkflowCallbacks(s)
s.workflowStore = store
s.workflowEngine = engine.New(store, callbacks)
}
// startTurnForSession starts a new turn for the session and stores it.
func (s *Service) startTurnForSession(ctx context.Context, sessionID string) string {
if s.turnService == nil {
return ""
}
turn, err := s.turnService.StartTurn(ctx, sessionID)
if err != nil {
s.logger.Warn("failed to start turn",
zap.String("session_id", sessionID),
zap.Error(err))
return ""
}
s.activeTurns.Store(sessionID, turn.ID)
return turn.ID
}
// completeTurnForSession completes the active turn for the session.
func (s *Service) completeTurnForSession(ctx context.Context, sessionID string) {
if s.turnService == nil {
return
}
turnIDVal, ok := s.activeTurns.LoadAndDelete(sessionID)
if !ok {
return
}
turnID, ok := turnIDVal.(string)
if !ok || turnID == "" {
return
}
if err := s.turnService.CompleteTurn(ctx, turnID); err != nil {
s.logger.Warn("failed to complete turn",
zap.String("session_id", sessionID),
zap.String("turn_id", turnID),
zap.Error(err))
}
}
// getActiveTurnID returns the active turn ID for a session.
// If no active turn exists and the session ID is provided, it will start a new turn.
// This ensures messages always have a valid turn ID even in edge cases like resumed sessions.
func (s *Service) getActiveTurnID(sessionID string) string {
if sessionID == "" {
return ""
}
turnIDVal, ok := s.activeTurns.Load(sessionID)
if ok {
turnID, _ := turnIDVal.(string)
if turnID != "" {
return turnID
}
}
// No active turn exists - start one lazily
// This handles edge cases like resumed sessions or race conditions
return s.startTurnForSession(context.Background(), sessionID)
}
func (s *Service) setSessionResetInProgress(sessionID string, inProgress bool) {
if sessionID == "" {
return
}
if inProgress {
s.resetInProgressSessions.Store(sessionID, true)
return
}
s.resetInProgressSessions.Delete(sessionID)
}
func (s *Service) isSessionResetInProgress(sessionID string) bool {
if sessionID == "" {
return false
}
v, ok := s.resetInProgressSessions.Load(sessionID)
if !ok {
return false
}
inProgress, _ := v.(bool)
return inProgress
}
// Start starts all orchestrator components
func (s *Service) Start(ctx context.Context) error {
s.mu.Lock()
if s.running {
s.mu.Unlock()
return ErrServiceAlreadyRunning
}
s.running = true
s.startedAt = time.Now()
s.mu.Unlock()
s.logger.Info("starting orchestrator service")
// Reconcile session state from persisted runtime state on startup.
// This does NOT launch any agent processes — sessions are recovered lazily
// when the user opens them (via task.session.status → task.session.resume).
s.reconcileSessionsOnStartup(ctx)
// Start the watcher first to begin receiving events
if err := s.watcher.Start(ctx); err != nil {
s.mu.Lock()
s.running = false
s.mu.Unlock()
return err
}
// Start the scheduler processing loop
if err := s.scheduler.Start(ctx); err != nil {
if stopErr := s.watcher.Stop(); stopErr != nil {
s.logger.Warn("failed to stop watcher after scheduler start failure", zap.Error(stopErr))
}
s.mu.Lock()
s.running = false
s.mu.Unlock()
return err
}
// Subscribe to GitHub integration events
s.subscribeGitHubEvents()
// Subscribe to clarification events (cancel-and-resume flow)
s.subscribeClarificationEvents()
// Subscribe to prepare events (persist result in session metadata)
s.subscribePrepareEvents()
s.logger.Info("orchestrator service started successfully")
return nil
}
// Stop stops all orchestrator components
func (s *Service) Stop() error {
s.mu.Lock()
if !s.running {
s.mu.Unlock()
return ErrServiceNotRunning
}
s.running = false
s.mu.Unlock()
s.logger.Info("stopping orchestrator service")
// Stop components in reverse order
var errs []error
if err := s.scheduler.Stop(); err != nil {
s.logger.Error("failed to stop scheduler", zap.Error(err))
errs = append(errs, err)
}
if err := s.watcher.Stop(); err != nil {
s.logger.Error("failed to stop watcher", zap.Error(err))
errs = append(errs, err)
}
s.cancelAllClarificationWatchdogs()
if len(errs) > 0 {
return errs[0]
}
s.logger.Info("orchestrator service stopped successfully")
return nil
}
// reconcileSessionsOnStartup adjusts database state for sessions that were active before restart.
// It does NOT launch any agent processes — sessions are recovered lazily when the user opens them
// (via task.session.status → task.session.resume or by sending a prompt).
//
// Strategy:
//
// 1. Terminal states (Completed/Cancelled/Failed) → clean up executor record
// 2. Never-started (Created) → clean up executor record
// 3. Active states (Starting/Running/WaitingForInput) → set session to WAITING_FOR_INPUT,
// clear stale execution IDs, fix task state, preserve ExecutorRunning record
// 4. Pre-poll remote executor status for remote runtimes (sprites, remote_docker)
//
// Called by: Start() method during orchestrator initialization.
func (s *Service) reconcileSessionsOnStartup(ctx context.Context) {
runningExecutors, err := s.repo.ListExecutorsRunning(ctx)
if err != nil {
s.logger.Warn("failed to list executors running on startup", zap.Error(err))
return
}
if len(runningExecutors) == 0 {
s.logger.Info("no executors to reconcile on startup")
return
}
s.logger.Info("reconciling sessions on startup (lazy recovery)", zap.Int("count", len(runningExecutors)))
var remoteRecords []executor.RemoteStatusPollRequest
for _, running := range runningExecutors {
if models.IsRemoteExecutorType(models.ExecutorType(running.Runtime)) {
remoteRecords = append(remoteRecords, executor.RemoteStatusPollRequest{
SessionID: running.SessionID,
Runtime: running.Runtime,
AgentExecutionID: running.AgentExecutionID,
ContainerID: running.ContainerID,
Metadata: running.Metadata,
})
}
s.reconcileOneSessionOnStartup(ctx, running)
}
// Pre-poll remote executor status so task lists show accurate state
if len(remoteRecords) > 0 && s.agentManager != nil {
s.agentManager.PollRemoteStatusForRecords(ctx, remoteRecords)
}
}
// reconcileOneSessionOnStartup adjusts DB state for a single session without launching agents.
func (s *Service) reconcileOneSessionOnStartup(ctx context.Context, running *models.ExecutorRunning) {
sessionID := running.SessionID
if sessionID == "" {
return
}
session, err := s.repo.GetTaskSession(ctx, sessionID)
if err != nil {
s.logger.Warn("failed to load session for reconciliation",
zap.String("session_id", sessionID),
zap.Error(err))
return
}
previousState := session.State
// Handle terminal and never-started states (reuse existing cleanup logic)
if skip := s.handleTerminalSessionOnStartup(ctx, session, running, previousState); skip {
return
}
if previousState == models.TaskSessionStateCreated {
s.logger.Info("session was never started; cleaning up",
zap.String("session_id", sessionID))
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
s.logger.Warn("failed to remove executor record for unstarted session",
zap.String("session_id", sessionID),
zap.Error(err))
}
return
}
// Active states: STARTING, RUNNING, WAITING_FOR_INPUT
// Set session to WAITING_FOR_INPUT (idle, ready for lazy resume when user opens it)
if previousState != models.TaskSessionStateWaitingForInput {
if err := s.repo.UpdateTaskSessionState(ctx, sessionID, models.TaskSessionStateWaitingForInput, ""); err != nil {
s.logger.Warn("failed to set session to WAITING_FOR_INPUT on startup",
zap.String("session_id", sessionID),
zap.Error(err))
}
}
// Clear stale execution references (agent process is gone after restart)
_ = s.repo.ClearSessionExecutionID(ctx, sessionID)
// Ensure task is in REVIEW state (not stuck IN_PROGRESS)
if running.TaskID != "" {
task, taskErr := s.taskRepo.GetTask(ctx, running.TaskID)
if taskErr == nil && task != nil && task.State == v1.TaskStateInProgress {
if updateErr := s.taskRepo.UpdateTaskState(ctx, running.TaskID, v1.TaskStateReview); updateErr != nil {
s.logger.Warn("failed to update task to REVIEW on startup",
zap.String("task_id", running.TaskID),
zap.Error(updateErr))
}
}
}
// PRESERVE the ExecutorRunning record — it holds the resume token and worktree info
// needed for lazy recovery when the user opens the session.
s.logger.Info("session reconciled for lazy recovery",
zap.String("session_id", sessionID),
zap.String("task_id", running.TaskID),
zap.String("previous_state", string(previousState)),
zap.Bool("has_resume_token", running.ResumeToken != ""),
zap.Bool("has_worktree", running.WorktreePath != ""))
}
// handleTerminalSessionOnStartup processes sessions in terminal states during startup.
// Returns true if the session should be skipped (no further processing needed).
func (s *Service) handleTerminalSessionOnStartup(ctx context.Context, session *models.TaskSession, running *models.ExecutorRunning, previousState models.TaskSessionState) bool {
sessionID := session.ID
switch previousState {
case models.TaskSessionStateCompleted, models.TaskSessionStateCancelled:
s.logger.Info("session in terminal state; cleaning up executor record",
zap.String("session_id", sessionID),
zap.String("task_id", session.TaskID),
zap.String("state", string(previousState)))
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
s.logger.Warn("failed to remove executor record for terminal session",
zap.String("session_id", sessionID),
zap.Error(err))
}
return true
case models.TaskSessionStateFailed:
s.handleFailedSessionOnStartup(ctx, session, running)
return true
}
return false
}
// handleFailedSessionOnStartup handles a failed session during startup recovery.
func (s *Service) handleFailedSessionOnStartup(ctx context.Context, session *models.TaskSession, running *models.ExecutorRunning) {
sessionID := session.ID
// If session failed, ensure task is in REVIEW state (not stuck IN_PROGRESS)
if session.TaskID != "" {
task, taskErr := s.taskRepo.GetTask(ctx, session.TaskID)
if taskErr == nil && task.State == v1.TaskStateInProgress {
s.logger.Info("fixing task state: session failed but task still IN_PROGRESS",
zap.String("task_id", session.TaskID),
zap.String("session_id", sessionID))
if updateErr := s.taskRepo.UpdateTaskState(ctx, session.TaskID, v1.TaskStateReview); updateErr != nil {
s.logger.Warn("failed to update task state to REVIEW",
zap.String("task_id", session.TaskID),
zap.Error(updateErr))
}
}
}
if canResumeRunning(running) {
s.logger.Info("preserving executor record for resumable failed session",
zap.String("session_id", sessionID),
zap.String("task_id", session.TaskID))
} else {
s.logger.Info("cleaning up executor record for non-resumable failed session",
zap.String("session_id", sessionID),
zap.String("task_id", session.TaskID))
if err := s.repo.DeleteExecutorRunningBySessionID(ctx, sessionID); err != nil {
s.logger.Warn("failed to remove executor record for failed session",
zap.String("session_id", sessionID),
zap.Error(err))
}
}
}
func canResumeRunning(running *models.ExecutorRunning) bool {
if running == nil || running.ResumeToken == "" {
return false
}
if running.Resumable {
return true
}
return models.IsAlwaysResumableRuntime(running.Runtime)
}
func isMissingMergedPRBranchError(err error) bool {
if err == nil {
return false
}
msg := strings.ToLower(err.Error())
return strings.Contains(msg, "couldn't find remote ref") ||
(strings.Contains(msg, "not found locally or on remote") && strings.Contains(msg, "branch")) ||
(strings.Contains(msg, "pathspec") && strings.Contains(msg, "did not match"))
}
var (
quotedBranchPattern = regexp.MustCompile(`branch "([^"]+)"`)
remoteRefPattern = regexp.MustCompile(`remote ref ([^\s]+)`)
pathspecBranchPattern = regexp.MustCompile(`pathspec '([^']+)'`)
)
func extractMissingBranchName(err error) string {
if err == nil {
return ""
}
msg := err.Error()
if match := quotedBranchPattern.FindStringSubmatch(msg); len(match) == 2 {
return strings.TrimSpace(match[1])
}
if match := remoteRefPattern.FindStringSubmatch(msg); len(match) == 2 {
return strings.TrimSpace(match[1])
}
if match := pathspecBranchPattern.FindStringSubmatch(msg); len(match) == 2 {
return strings.TrimSpace(match[1])
}
return ""
}
func (s *Service) handleSessionLaunchFailed(ctx context.Context, taskID, sessionID string, launchErr error) {
if s.messageCreator == nil || !isMissingMergedPRBranchError(launchErr) {
return
}
branch := extractMissingBranchName(launchErr)
content := "This task references a PR branch that no longer exists on remote (likely merged and deleted)."
if branch != "" {
content = "The remote PR branch \"" + branch + "\" no longer exists (likely merged and deleted)."
}
metadata := map[string]interface{}{
"variant": "warning",
"failure_kind": "missing_pr_branch",
"missing_branch": branch,
"actions": []map[string]interface{}{
{
"type": "archive_task",
"label": "Archive task",
"tooltip": "Keep task history and hide it from active work",
"icon": "archive",
"test_id": "missing-branch-archive-button",
},
{
"type": "delete_task",
"label": "Delete task",
"tooltip": "Permanently remove this task",
"variant": "destructive",
"icon": "trash",
"test_id": "missing-branch-delete-button",
},
},
}
s.suppressToast.Store(sessionID, true)
msgCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if err := s.messageCreator.CreateSessionMessage(
msgCtx,
taskID,
content,
sessionID,
string(v1.MessageTypeStatus),
"", // No turn — agent never started, avoid lazily creating a synthetic turn.
metadata,
false,
); err != nil {
s.logger.Warn("failed to create missing PR branch launch failure message",
zap.String("task_id", taskID),
zap.String("session_id", sessionID),
zap.Error(err))
}
}
// IsRunning returns true if the service is running
func (s *Service) IsRunning() bool {
s.mu.RLock()
defer s.mu.RUnlock()
return s.running
}
// GetStatus returns the orchestrator status
func (s *Service) GetStatus() *Status {
s.mu.RLock()
running := s.running
startedAt := s.startedAt
s.mu.RUnlock()
queueStatus := s.scheduler.GetQueueStatus()
var uptimeSeconds int64
if running {
uptimeSeconds = int64(time.Since(startedAt).Seconds())
}
return &Status{
Running: running,
ActiveAgents: queueStatus.ActiveExecutions,
QueuedTasks: queueStatus.QueuedTasks,
TotalProcessed: queueStatus.TotalProcessed,
TotalFailed: queueStatus.TotalFailed,
UptimeSeconds: uptimeSeconds,
LastHeartbeat: time.Now(),
}
}
// GetMessageQueue returns the message queue service
func (s *Service) GetMessageQueue() *messagequeue.Service {
return s.messageQueue
}
// GetEventBus returns the event bus
func (s *Service) GetEventBus() bus.EventBus {
return s.eventBus
}