Files
kandev/docs/specs/office/scheduler.md
T
2026-08-21 22:04:17 +01:00

50 KiB

status, created, owner
status created owner
draft 2026-04-25 cfl

Office Scheduler

Why

Kandev's base task scheduler is reactive: tasks enter the queue only when a user explicitly starts them or sends a prompt. Office adds autonomous agent operation, which requires the system to wake agents on its own when events happen (assignments, comments, blocker resolutions, approvals), on a schedule (routines), and on heartbeat ticks (periodic coordinator checks). Without an autonomous wakeup pipeline, every interaction needs a human to initiate it, and the cost / reliability story (idle skips, rate-limit retries, staleness, recovery) has nowhere to live.

Office supplies autonomous run producers and Office-specific maintenance. The persisted runs queue and its single backend-wide consumer are shared workflow infrastructure, not one scheduler per Office workspace. The shared ownership, scoping, and shutdown contract is defined by queued run scheduling and ADR-2026-08-01-global-run-scheduler-ownership.

What

Shared scheduler boundary

  • One shared runs scheduler processes persisted work for every workspace.
  • Office event subscribers and unstarted-task recovery act only on tasks whose project/workflow identity makes Task.IsFromOffice true. A runner on an ordinary Kanban task is not sufficient.
  • Explicit workflow queue_run actions remain available to every workflow style and are not filtered by Office identity.
  • Office recovery is maintenance, not part of every five-second queue drain. When no workspace has adopted Office, it skips the task scan.
  • The shared runs scheduler and cron loop stop and join before database cleanup during graceful shutdown.

Wakeup queue

A SQLite-persisted queue of "wake this agent up" requests. Every periodic, event-driven, and reactive trigger flows through this queue before becoming an agent run. Each request:

  • Has a source discriminator (see table below) plus a typed payload.
  • Carries an idempotency_key for source-level dedup within a 24-hour window.
  • Is coalesced into an in-flight run when one exists for the same agent (claim-time merge).
  • Produces exactly one runs row on successful claim; the run is the execution record, the wakeup-request is the dispatch record.

Wakeup sources

Source Trigger Payload
routine A routine's cron / webhook / manual trigger fires {routine_id, variables, missed_ticks?}
comment Comment posted on a task assigned to this agent (non-self). Also the channel pathway: inbound Telegram/Slack messages become comments on a channel task. {task_id, comment_id}
agent_error A sub-agent's session failed (escalation to coordinator) {agent_profile_id, run_id, error}
self Agent self-wake via tool call {reason, payload?}
user User mention / explicit wake from the UI {user_id, context?}
task_assigned An authoritative Office task's assignee_agent_instance_id is set or changed, including a newly-created task/subtask that already has a runner {task_id}
task_blockers_resolved All blocking tasks of an assigned task reach done {task_id, resolved_blocker_ids}
task_children_completed All child tasks of an assigned task reach terminal state {task_id}
approval_resolved An approval requested by this agent is approved/rejected {approval_id, status, decision_note}
budget_alert Budget threshold crossed for a sub-agent (coordinator only) {agent_instance_id, budget_pct}

The task-event sources are dispatched by office event subscribers when the corresponding task event fires. The legacy heartbeat source is retired: all periodic wakes flow through the routine source - each coordinator agent gets a pre-installed routine at onboarding (see Routines), and that routine's cron tick is what wakes the coordinator.

Manual status changes (kanban drag-drop)

Office tasks live on the system office workflow and appear on the kanban board. Users can drag them between columns (steps), which emits a task.moved event. The office event subscribers handle task.moved events for office tasks (identified by assignee_agent_instance_id != null) and fire the appropriate wakeups:

  • Move to "In Progress": task_assigned wakeup for the assignee agent.
  • Move to "Done" or "Cancelled": resolve blocker dependencies, fire task_blockers_resolved / task_children_completed wakeups for affected agents.
  • Move to "In Review": if execution_policy has reviewers, wake reviewer agents.
  • Move from "In Review" back to "In Progress": comment wakeup with rejection context for the assignee.

Non-office tasks (those without assignee_agent_instance_id) are ignored by office subscribers.

Coalescing - three layers, one job each

Coalescing happens entirely at the wakeup-request layer; the runs table is the execution record.

  1. Source-level dedup via idempotency_key (UNIQUE column). Format:

    • routine:<routine_id>:<trigger_id>:<unix_minute> for cron routines.
    • comment:<comment_id> for comments.
    • task_assigned:<task_id>:<agent_instance_id> for task creation/assignment events. Duplicate inserts in the same window are rejected silently. Handles webhook re-delivery, event-bus replay, and restart recovery.
  2. Claim-time merge. When the dispatcher processes a wakeup-request, it looks for an in-flight run for the same agent (queued -> scheduled-retry -> running, in that order). If one exists: insert the new request with status="coalesced", run_id=<existing>, merge the new request's payload into the existing run's context_snapshot, and increment coalesced_count on the in-flight wakeup-request. The agent sees the merged context when it actually runs. If none exists: insert the wakeup-request with status="queued" and create the corresponding runs row.

  3. runs.idempotency_key is kept as a defensive secondary key. Rarely tripped now (the wakeup-request layer handles the common case), but useful for the rare "two cron processes fired the same tick from different leaders" scenario during a leadership change.

A coalescing window (default 5 seconds) also merges wakeups for the same agent and same source if no run is yet in flight. Example: 5 subtasks complete within 3 seconds generating 5 task_children_completed wakeups, coalesced into 1 with coalesced_count=5.

Routines

A routine is a task template (or taskless wake spec) with one or more triggers. Routines are the only mechanism for periodic wakes; there is no system-level agent heartbeat cron.

Routine fields:

  • id, workspace_id, name, description.
  • assignee_agent_instance_id - who gets the resulting task / run.
  • status: active | paused | archived.
  • concurrency_policy: coalesce_if_active (default) | skip_if_active | always_enqueue.
  • catch_up_policy: enqueue_missed_with_cap (default, cap 25) | skip_missed.
  • catch_up_max: integer, default 25.
  • task_template: JSON. Empty means lightweight routine (taskless run per fire). Non-empty means heavy routine (fresh task created on the routine workflow).
  • variables: declared template variables (type, default, required).

Triggers:

  • Schedule (cron): cron_expression, timezone, computed next_run_at, last_fired_at.
  • Webhook: public_id (URL path component), signing_mode (none | bearer | hmac_sha256), secret. URL: POST /api/routine-triggers/<public_id>/fire. Webhook payload is available as variables.
  • Manual: fired only via UI or API.

Variables use {{name}} syntax in title/description with types text, number, boolean, select. Built-ins: {{date}}, {{datetime}}. Resolution order (later wins): built-ins -> declared defaults -> provided values (manual UI or webhook payload). Adding {{new_var}} to a template auto-creates the variable declaration on save (text, no default, not required).

Routine runs

Each trigger firing creates a routine run record (office_routine_runs) with routine_id, trigger_id, source (cron | webhook | manual), status (received -> task_created | skipped | coalesced | failed), trigger_payload (resolved variable values), linked_task_id (heavy only), coalesced_into_run_id, dispatch_fingerprint (hash of resolved template + assignee), and lifecycle timestamps.

Heavy vs lightweight routines

  • Lightweight (task_template empty): fire produces a taskless agent run. Continuation summary keyed by routine:<routine_id>. Use case: "check upstream PRs" without a trackable artifact.
  • Heavy (task_template set): fire creates a fresh task in the routine workflow (a single auto-completing in_progress -> done step, system-flagged via SystemWorkflowTemplateIDs so heavy routine tasks inherit the hide-by-default UX), then a normal task-bound run. Use case: "daily review" where output should be a trackable item.

Concurrency policy

Evaluated at dispatch by querying for an in-flight run for the same routine fingerprint:

  • skip_if_active: do not create a new task / run. Mark skipped.
  • coalesce_if_active (default): merge into the existing run. Mark coalesced.
  • always_enqueue / always_create: always proceed.

"Active" means the linked task / run is not in a terminal state.

Catch-up policy

If the scheduler was down and missed cron ticks:

  • skip_missed: fire only the current tick.
  • enqueue_missed_with_cap (default, cap 25): fire missed ticks up to the cap; dropped ticks are not recorded individually but summarized into the next prompt's wake context ("you missed N ticks since X").

The pre-installed coordinator routine

At agent-create time, when an agent's role is coordinator/CEO, the system creates one routine:

name:                "Coordinator heartbeat"
description:         "Wakes the coordinator every 5 minutes to check workspace activity, react to errors and budget signals, and decide what to do next."
assignee_agent_id:   <new coordinator agent id>
status:              active
concurrency_policy:  coalesce_if_active
catch_up_policy:     enqueue_missed_with_cap
catch_up_max:        25
task_template:       ""
variables:           []
trigger:
  kind:              schedule
  cron_expression:   "*/5 * * * *"
  timezone:          (workspace TZ, fall back to UTC)
  enabled:           true

This is a regular routine - no system flag, no lock, no badge. The user can edit / pause / delete it. If deleted, the coordinator only wakes via reactive sources. Default cadence is every five minutes (the prior 60s default was too aggressive); users can crank it up.

Idle wakeup skip

Before processing a routine-fired heartbeat-style wakeup (lightweight routine, no task payload), the scheduler checks whether the agent has any actionable tasks. If none, the wakeup is skipped, no session is launched.

Actionable states: TODO and IN_PROGRESS. Terminal (DONE, CANCELLED, ARCHIVED) and review-gated (IN_REVIEW) do not count.

Skip conditions (all must hold):

  • Wakeup is a periodic / heartbeat-style routine fire (lightweight, no task in payload).
  • Agent has skip_idle_wakeups = true.
  • CountActionableTasksForAgent returns 0.

Per-agent skip_idle_wakeups defaults:

Role Default
worker true
specialist true
assistant true
ceo / coordinator false

Coordinator agents default to false because their heartbeat purpose is self-directed coordination (surveying projects, reassigning tasks, checking budgets) which does not require a directly assigned task. Users can override per agent.

Event-triggered wakeups always proceed - the skip applies only to periodic wakes:

Source Skippable?
routine (lightweight, heartbeat-style) Yes
task_assigned, comment, task_blockers_resolved, task_children_completed, approval_resolved, agent_error, budget_alert, self, user No

Skipped wakeups are not silently discarded:

  1. Logged at INFO with wakeup_id, agent_instance_id, agent_name, reason="no_actionable_tasks".
  2. Marked finished (normal terminal state).
  3. Recorded as a wakeup_idle_skipped activity entry.

Skip check uses a single indexed count:

SELECT COUNT(*) FROM tasks
WHERE assignee_agent_instance_id = $agentID
  AND state IN ('TODO', 'IN_PROGRESS')
  AND archived_at IS NULL

Executor resolution at launch

When the scheduler claims a run, it resolves the executor using the agent preference first. If the agent has no executor preference and the run payload carries task_id, the scheduler resolves the task's project and allows that project executor config to satisfy the launch. This prevents assigned task runs from retrying indefinitely solely because the worker's agent row has an empty executor preference.

Staleness check (before claim)

Before processWakeup proceeds past the agent status guard, the scheduler checks whether the wakeup's context is still valid. This runs on every wakeup that carries a task_id in its payload.

Staleness conditions (each produces a distinct cancel reason):

Condition Cancel reason
Task not found task_not_found
Task assignee changed (task.AssigneeAgentInstanceID != wakeup.AgentInstanceID) assignee_changed
Task reached terminal state (DONE, CANCELLED, ARCHIVED) task_terminal
Task's review-stage participant changed review_participant_changed

A stale wakeup is cancelled (status cancelled), not retried. Cancellation is idempotent and logged. Any held checkout lock is released.

Retry cancellation on reassignment

At retry promotion time (scheduleRetry / scheduleRetryAt), before re-queuing a scheduled retry, the service checks whether:

  • scheduledRetry.AgentInstanceID still matches task.AssigneeAgentInstanceID, or
  • The task is now CANCELLED.

If either holds, the retry is cancelled with reason retry_stale_assignee or retry_task_cancelled. Execution locks held by the old agent are cleared.

Additionally, when a task's assignee is updated via the API, any pending scheduled_retry wakeups for the previous assignee are cancelled immediately, without waiting for the retry-promotion path.

Rate-limit retry with parsed reset time

When HandleWakeupFailure is called, if the error is a rate-limit error the service tries to extract a reset timestamp from the message text. If one is found, the wakeup is scheduled for parsed_reset_time + 30s regardless of the RetryCount position in the backoff table. RetryCount is still incremented so MaxRetryCount escalation still applies.

Rate-limit detection - any of (case-insensitive where stated):

  • "rate limit", "rate_limit", "429", "too many requests", "quota exceeded".

Reset-time patterns:

  • "resets at HH:MM AM/PM" - absolute wall-clock time on the current day (next occurrence if past).
  • "Retry-After: N" - N seconds from now.
  • "try again in X minutes" / "try again in X seconds" - relative duration.
  • "rate limit exceeded ... reset_time: <unix timestamp>" - Unix epoch seconds.

A 30-second buffer is added to any parsed time. If no pattern matches or the parsed time is in the past after buffer, the existing exponential backoff applies unchanged.

Log fields gain source: "rate_limit_parsed" vs source: "backoff", plus parsed_reset_at (UTC).

Default backoff for non-rate-limit retries: 4 attempts at [2m, 10m, 30m, 2h] with 25% jitter. After MaxRetryCount (4) failures, escalateFailure is called - the wakeup is marked failed, an agent.error inbox item is created, and the coordinator receives an agent_error wakeup.

Recovery sweep (unstarted Office tasks)

Office maintenance performs a recovery sweep separately from the shared queue-drain tick. It finds authoritative Office TODO tasks created inside the workspace recovery lookback window and dispatches them as task_assigned runs only when no queued, claimed, or finished run exists for the task. The task-creation timestamp is bounded by the lookback; matching run rows are not, so a task that already started is never reclassified as unstarted merely because its prior run is old. Failed and cancelled rows do not block recovery. Assignment on an ordinary Kanban task does not imply autonomy.

Selection:

SELECT t.id FROM tasks t
WHERE t.state = 'TODO'
  AND t.assignee_agent_instance_id IS NOT NULL
  AND (
    COALESCE(t.project_id, '') != ''
    OR t.workflow_id = (
      SELECT w.office_workflow_id FROM workspaces w WHERE w.id = t.workspace_id
    )
  )
  AND t.archived_at IS NULL
  AND t.created_at >= NOW() - INTERVAL '<lookback_hours> hours'
  AND NOT EXISTS (
      SELECT 1 FROM runs r
      WHERE r.payload->>'task_id' = t.id
        AND r.status IN ('queued', 'claimed', 'finished')
  )

Per-candidate guards:

  • Skip if agent is paused or stopped.
  • Skip if a wakeup is already queued for this task (prevents duplicates on concurrent ticks).
  • Skip if the agent's invocation budget is exhausted.

Logged: recovery_dispatch per dispatched task, recovery_sweep_complete summary entry with dispatched_count per sweep.

Lookback is a workspace setting recovery_lookback_hours, default 24, range 1-720, clamped on write.

Scheduler processing pipeline

processWakeup:
  1. Claim                    -- atomic UPDATE: status='queued' -> 'claimed', claimed_at=now()
  2. Guard: agent status      -- paused/stopped -> mark finished, no action
  3. Staleness check          -- task scope still valid (assignee, terminal, review participant)
  4. Idle skip                -- routine/heartbeat-style + skip_idle_wakeups=true + 0 actionable tasks -> finished
  5. Checkout                 -- atomic CAS lock on the task (only when payload has task_id)
  6. Budget pre-check         -- workspace -> project -> agent budgets; pause_agent action skips
  7. Build context            -- assemble prompt from source, payload, context_snapshot
  8. Resolve executor         -- task override -> agent preference -> project -> workspace default
  9. Resolve execution agent  -- Office binding -> concrete profile or shared dynamic router
  10. Create session          -- one logical TaskSession through the orchestrator pipeline
  11. Launch                  -- lifecycle -> compute executor -> dynamic conductor or concrete agentctl
  12. Finish                  -- mark wakeup finished; parse output for follow-up actions

Atomic task checkout

When an agent starts working on a task, it acquires an exclusive lock via CAS:

UPDATE tasks SET checkout_agent_id = $agent, checkout_at = now()
WHERE id = $task AND checkout_agent_id IS NULL
RETURNING *

Zero rows = another agent already holds the lock -> wakeup skipped, no retry. Released on finish or failure. Prevents two agents racing on the same task when concurrency > 1 or multiple agents are assigned.

Agent concurrency

Each agent instance has max_concurrent_sessions (default 1) and cooldown_sec (default 10s). The claim query skips agents at capacity and skips agents whose last_wakeup_finished_at is within the cooldown window. Wakeups for busy or cooling-down agents stay queued and are picked up naturally when eligible. No re-queuing, no retry limits, no expiry: a slow QA agent with 20 tasks queued processes them sequentially. Concurrency > 1 is useful for agents handling independent tasks (e.g. multiple code reviews in parallel).

Claim query

SELECT w.* FROM office_wakeup_queue w
JOIN office_agent_instances a ON a.id = w.agent_instance_id
WHERE w.status = 'queued'
  AND a.status IN ('idle', 'working')
  AND (w.scheduled_retry_at IS NULL OR w.scheduled_retry_at <= now())
  AND (a.last_wakeup_finished_at IS NULL OR a.last_wakeup_finished_at <= now() - a.cooldown_sec)
  AND (
    SELECT COUNT(*) FROM task_sessions ts
    WHERE ts.agent_instance_id = w.agent_instance_id
      AND ts.state IN ('STARTING', 'RUNNING', 'WAITING_FOR_INPUT')
  ) < a.max_concurrent_sessions
ORDER BY w.requested_at
LIMIT 1

One-shot session model + continuation summary

Each wakeup produces a single agent session that runs to completion and exits. The agent receives a structured prompt describing why it was woken.

Taskless runs always start a fresh session. A defensive taskID=="" short-circuit in HasPriorSessionForAgent ensures we never resume across taskless fires.

Task-bound wakeups use session resume by default: each subsequent wakeup for a (task, agent) pair reloads the prior ACP session via session/load, falling back to session/new on error. See office-task-session-lifecycle for the per-pair model.

Continuation summary

To bridge context across taskless fires (heartbeats, lightweight routines) without unbounded conversation growth, the system maintains a per-(agent, scope) markdown summary.

Table agent_continuation_summaries:

agent_profile_id  TEXT  NOT NULL
scope             TEXT  NOT NULL  -- "routine:<routine_id>" (per-routine summary chain)
content           TEXT  NOT NULL  DEFAULT ''  -- markdown body, capped at 8 KB
content_tokens    INT   NOT NULL  DEFAULT 0
updated_at        TIMESTAMP NOT NULL
updated_by_run_id TEXT NOT NULL DEFAULT ''
PRIMARY KEY (agent_profile_id, scope)

Writes are upsert (one current row per scope, no history). The prompt slice is truncated to 1,500 chars; full content cap is 8 KB.

Summary structure (markdown sections):

## Active focus
2-3 lines. What the coordinator is currently watching/driving.

## Open blockers
Bullet list. Each: blocker + what's needed to unblock + when surfaced.

## Recent decisions
Bullet list. Last ~5 things the coordinator committed to. Date-stamped.

## Next action
One sentence. The single next thing to do on the next wake-up.

Generation is server-synthesized, not agent-written. A builder (internal/office/summary/builder.go) composes the markdown deterministically from structured inputs after each successful run:

Input Source Used for
run.result_json Adapter-populated structured output; fallback chain result_json.summary -> .result -> .message -> .error Recent actions / decisions
Workspace activity stats office_activity_log + runs (counts of completed/failed tasks, agent-error escalations, budget signals) Active focus, opening blocker context
Active blockers Tasks in BLOCKED state assigned to managed agents Open blockers section
Previous summary body Prior row in agent_continuation_summaries Continuity / fallback for unchanged sections
Inferred next action Decision table on (workspace state, last run status); falls back to "Continue monitoring." Next action

Idempotent: re-running with the same inputs produces the same output. Called from the AgentCompleted event subscriber on successful taskless runs. On failure, the previous summary is left intact (last-good wins).

Resume delta prompt

When resuming a task-bound session (same agent, same task, session ID preserved), the agent receives only a resume delta - the new information since the last run. Full instructions and context are skipped (the agent CLI retains them from the previous session), saving ~5-10K tokens per fire.

Subtask sequencing via blockers

Office does not have a separate workflow/template engine for subtask ordering. The agent's instructions (via skills) define how to decompose work and which subtasks to create. Sequencing is enforced through the existing blocker system: the agent creates subtask 2 with blocked_by: [subtask 1].

The scheduler respects blockers: a task_assigned wakeup for a blocked task is held until blockers resolve. When a subtask completes:

  1. If requires_approval=true: task moves to in_review, inbox item created. On user approval, task moves to done.
  2. If requires_approval=false: task moves directly to done.
  3. On reaching done: any sibling tasks that had this task as a blocker receive a task_blockers_resolved wakeup for their assigned agent.

This creates a natural pipeline: Spec (requires_approval) -> Build (blocked_by Spec) -> Review (blocked_by Build) -> Ship (blocked_by Review). The user only intervenes at approval gates.

The coordinator is woken via task_children_completed when all subtasks under a parent reach terminal state.

Pre-execution budget check

Before launching an agent session, the scheduler checks all applicable budget policies (workspace -> project -> agent). If any budget is exceeded with action_on_exceed=pause_agent, the agent is paused and the wakeup is skipped. Prevents wasting tokens on a run that would immediately be followed by a budget-exceeded pause.

Integration with existing scheduler

User-initiated task execution continues through the orchestrator directly. Engine-emitted work uses the single shared runs scheduler across all workspaces. Office contributes event-driven producers and maintenance handlers to that shared path; it does not own a per-workspace scheduler. Both paths converge at the agent runtime.

Data model

agent_wakeup_requests

id                       TEXT PRIMARY KEY
agent_profile_id         TEXT NOT NULL
source                   TEXT NOT NULL  -- routine | comment | agent_error | self | user
                                        --   plus task-event sources: task_assigned, task_blockers_resolved,
                                        --   task_children_completed, approval_resolved, budget_alert
reason                   TEXT NOT NULL  -- short label for telemetry
payload                  TEXT NOT NULL DEFAULT '{}'  -- JSON, typed at boundary (see Payload schema)
status                   TEXT NOT NULL  -- queued | claimed | coalesced | skipped | finished | failed | cancelled
cancel_reason            TEXT           -- task_not_found | assignee_changed | task_terminal | review_participant_changed
                                        --   | retry_stale_assignee | retry_task_cancelled
coalesced_count          INT  NOT NULL DEFAULT 1
idempotency_key          TEXT UNIQUE
retry_count              INT  NOT NULL DEFAULT 0
scheduled_retry_at       TIMESTAMP
context_snapshot         TEXT NOT NULL DEFAULT '{}'  -- pre-computed prompt context (task summary, new comments)
run_id                   TEXT  -- the run this request fulfilled (when status terminal)
requested_at             TIMESTAMP NOT NULL
claimed_at               TIMESTAMP
finished_at              TIMESTAMP
INDEX (agent_profile_id, status)
INDEX (idempotency_key)

agent_continuation_summaries

agent_profile_id   TEXT NOT NULL
scope              TEXT NOT NULL  -- routine:<routine_id>
content            TEXT NOT NULL DEFAULT ''  -- markdown, capped 8 KB
content_tokens     INT  NOT NULL DEFAULT 0
updated_at         TIMESTAMP NOT NULL
updated_by_run_id  TEXT NOT NULL DEFAULT ''
PRIMARY KEY (agent_profile_id, scope)

office_routines

id                    TEXT PRIMARY KEY
workspace_id          TEXT NOT NULL
name                  TEXT NOT NULL
description           TEXT
assignee_agent_id     TEXT NOT NULL
status                TEXT NOT NULL  -- active | paused | archived
concurrency_policy    TEXT NOT NULL  -- coalesce_if_active | skip_if_active | always_enqueue
catch_up_policy       TEXT NOT NULL  -- enqueue_missed_with_cap | skip_missed
catch_up_max          INT  NOT NULL DEFAULT 25
task_template         TEXT NOT NULL DEFAULT ''  -- JSON; empty -> lightweight
variables             TEXT NOT NULL DEFAULT '[]'  -- JSON array
priority              INT  NOT NULL DEFAULT 0
created_at, updated_at

office_routine_triggers

id                TEXT PRIMARY KEY
routine_id        TEXT NOT NULL
kind              TEXT NOT NULL  -- schedule | webhook | manual
cron_expression   TEXT           -- when kind=schedule
timezone          TEXT
next_run_at       TIMESTAMP      -- computed; atomically claimed for cron scheduling
last_fired_at     TIMESTAMP
public_id         TEXT UNIQUE    -- when kind=webhook
signing_mode      TEXT           -- none | bearer | hmac_sha256
secret            TEXT
enabled           BOOL NOT NULL DEFAULT TRUE

office_routine_runs

id                       TEXT PRIMARY KEY
routine_id               TEXT NOT NULL
trigger_id               TEXT NOT NULL
source                   TEXT NOT NULL  -- cron | webhook | manual
status                   TEXT NOT NULL  -- received | task_created | skipped | coalesced | failed
trigger_payload          TEXT NOT NULL DEFAULT '{}'
linked_task_id           TEXT
coalesced_into_run_id    TEXT
dispatch_fingerprint     TEXT NOT NULL
started_at, completed_at

runs (additions)

result_json       TEXT  NOT NULL  DEFAULT '{}'  -- structured adapter output for summary builder
assembled_prompt  TEXT  NOT NULL  DEFAULT ''    -- final prompt as the agent saw it (inspection)
summary_injected  TEXT  NOT NULL  DEFAULT ''    -- the summary that was prepended (per-run snapshot)

runs.payload.task_id is empty for taskless runs; all other run lifecycle fields (idempotency_key, claim/coalesce, cost rollup) carry over unchanged.

Workspace settings (additions)

recovery_lookback_hours   INT  NOT NULL DEFAULT 24   -- range 1-720, clamped on write

Agent instance / profile (additions)

skip_idle_wakeups       BOOL  NOT NULL  -- default true for worker/specialist/assistant, false for ceo
max_concurrent_sessions INT   NOT NULL DEFAULT 1
cooldown_sec            INT   NOT NULL DEFAULT 10
last_wakeup_finished_at TIMESTAMP

Payload schema

agent_wakeup_requests.payload is TEXT DEFAULT '{}'. The DB column is free-form so adding a new source needs no migration. Type safety lives in code (internal/office/wakeup/payloads.go): one Go struct per source, unmarshaled at the dispatcher boundary.

type RoutinePayload struct {
    RoutineID   string         `json:"routine_id"`
    Variables   map[string]any `json:"variables,omitempty"`
    MissedTicks int            `json:"missed_ticks,omitempty"` // when catch-up cap collapsed N fires
}
type CommentPayload struct {
    TaskID    string `json:"task_id"`
    CommentID string `json:"comment_id"`
}
type AgentErrorPayload struct {
    AgentProfileID string `json:"agent_profile_id"`
    RunID          string `json:"run_id"`
    Error          string `json:"error"`
}
type TaskAssignedPayload struct { TaskID string `json:"task_id"` }
// ... one per source. Switched on agent_wakeup_requests.source.

API surface

Routines

  • GET /api/office/routines?workspace_id=... - list.
  • POST /api/office/routines - create.
  • GET /api/office/routines/:id - detail (includes triggers, recent runs).
  • PATCH /api/office/routines/:id - update name / description / status / concurrency / catch-up / variables / assignee.
  • DELETE /api/office/routines/:id - delete.
  • POST /api/office/routines/:id/fire - manual fire (with variable payload).
  • POST /api/routine-triggers/:public_id/fire - webhook fire (signed per signing_mode).

Wakeup queue (internal)

  • office.wakeup.Service.Enqueue(req) - source-keyed insert (returns the existing wakeup if idempotency_key collides).
  • office.wakeup.Service.HandleWakeupFailure(wakeupID, err) - rate-limit parsing + scheduleRetry / escalateFailure.
  • office.wakeup.Service.CancelStale(wakeupID, reason) - mark cancelled, release locks, log activity.
  • office.wakeup.Dispatcher.processWakeup(wakeupID) - the pipeline described above.

Activity codes (surfaced in office activity feed)

  • wakeup_idle_skipped - heartbeat-style skip on no actionable tasks.
  • wakeup_budget_blocked - pre-flight budget pause.
  • wakeup_stale_cancelled - stale check cancelled a queued wakeup (reason, task_id, agent).
  • wakeup_retry_cancelled - retry cancelled due to reassignment (task_id, old_agent).
  • recovery_dispatch - unstarted task dispatched by recovery sweep (task_id, agent).
  • recovery_sweep_complete - summary entry per sweep (dispatched_count).

Run inspection (existing /office/agents/[id]/runs/[runId])

GetRunDetail returns the RunDetail aggregate including the new columns (result_json, assembled_prompt, summary_injected) plus existing context_snapshot, output_summary. The detail UI surfaces a "Prompt" tab rendering the assembled prompt + injected summary. WS live updates via useRunLiveSync are unchanged.

Frontend UI

  • /office/routines (list): each row shows name, trigger type, cron expression, next-fire-at, status, last-run, assignee. Enabled toggle flips status between active and paused. "Create Routine" opens a dialog (name, description, task template fields, trigger configuration, concurrency policy, catch-up policy + max, variable declarations).
  • /office/routines/[id] (detail/edit): same field surface plus status radio, last-fired indicator, "Run now" button, and a run history table (status, trigger payload, linked task link).
  • Coordinator-empty-state: banner on agent detail page when role is coordinator and no enabled routines target the agent - "This coordinator has no scheduled wake-ups. It will only fire on comments, errors, or manual triggers." Linkable to routines page.
  • Wakeup list page gains status badges for new terminal states: "Cancelled - assignee changed", "Cancelled - task completed", "Cancelled - task cancelled", "Cancelled - retry stale", "Recovered".
  • Workspace settings advanced section: "Recovery lookback window" numeric input (hours, default 24, range 1-720).

State machine

Wakeup request

queued -> claimed -> finished       (normal completion)
queued -> claimed -> failed         (after MaxRetryCount retries)
queued -> claimed -> failed (retrying) -> scheduled-retry -> claimed -> ... -> finished | failed
queued -> coalesced                 (merged into in-flight at claim time)
queued -> skipped                   (concurrency_policy=skip_if_active or coalesce drop)
queued -> cancelled                 (staleness check, retry-stale at promotion, API-driven reassign)
claimed -> finished (idle skip)     (no actionable tasks; informational only)

Transition triggers:

  • queued -> claimed: atomic UPDATE in claim query (one process wins).
  • claimed -> finished: dispatch pipeline succeeds OR idle skip / paused agent guard.
  • claimed -> failed (retrying): HandleWakeupFailure schedules retry (backoff or parsed rate-limit reset).
  • claimed -> cancelled: staleness check fails.
  • queued -> cancelled: API reassign cancels prior-assignee retries; retry promotion finds task no longer owned.
  • queued -> coalesced / skipped: dispatcher applies concurrency policy.

Routine

active <-> paused
active -> archived (terminal)

Triggered manually via UI / API.

Routine run

received -> task_created    (heavy routine: real task created)
received -> skipped         (concurrency_policy=skip_if_active and active run exists)
received -> coalesced       (concurrency_policy=coalesce_if_active and active run exists)
received -> failed          (dispatch error)

Permissions

The scheduler's reach is workspace-scoped: every routine, wakeup-request, run, and recovery dispatch is keyed by workspace_id (routines) or by an agent_profile_id that resolves to a single workspace. Cross-workspace dispatch is not possible by construction - the dispatcher only loads agents and tasks from the routine / wakeup's own workspace.

Action Who can perform it
Create / update / delete / pause routines in a workspace The workspace's UI user (no per-field permission model in v1). Routines API endpoints under /api/office/routines are not gated by agent capability today.
Fire a routine manually (POST /api/office/routines/:id/fire) UI user. Agent callers cannot manually fire routines from the runtime (no CapabilitySpawnRoutine).
Fire a webhook trigger (POST /api/routine-triggers/:public_id/fire) Any external caller possessing a valid signature per the trigger's signing_mode (none accepts unauthenticated requests). Failed signature returns 401; no routine run is created.
Enqueue a wakeup (office.wakeup.Service.Enqueue) Internal-only Go API. Office event subscribers, the workflow engine, comment handlers, approval handlers, and the recovery sweep are the legitimate callers. Agents cannot synthesize wakeups directly - the self-source wakeup is produced by the runtime in response to a recognized agent tool call, not by an HTTP route.
Override / cancel an in-flight wakeup or scheduled retry Indirect only. Reassigning a task via the task API or cancelling / completing the underlying task triggers the staleness and retry-cancellation paths described above. There is no direct "cancel this wakeup" endpoint.
Read run inspection (assembled_prompt, summary_injected, result_json) UI user for the workspace. No agent-side capability exposes raw prompts cross-agent.
Adjust workspace settings (recovery_lookback_hours) UI user. Workspace-level setting; no per-agent or per-project override.

Agent capabilities (CapabilityPostComment, CapabilityUpdateTaskStatus, etc., per agents.md) govern what an agent can do inside a run. They do not gate which agents the scheduler will wake - that is determined entirely by routine assignment, task assignment, and the reviewers / approvers lists managed in tasks.md.

A formal per-route authorization model (workspace membership, admin role, RBAC over routines / wakeups) is out of scope here; see Out of scope.

Failure modes

Dependency / invariant Behavior
SQLite write failure during enqueue Source caller surfaces error; idempotency-key collision is silent (treated as already-enqueued).
Claim query returns 0 rows Another process won the claim or no eligible wakeup; tick exits cleanly.
Office is enabled with no Office workflow/project Office recovery skips its task scan; ordinary Kanban assignments are not launched.
Agent paused / stopped at claim Wakeup marked finished with no action; not retried.
Task referenced by payload is missing Staleness check cancels with task_not_found; activity logged.
Task assignee changed before claim Cancelled with assignee_changed.
Task reached terminal state before claim Cancelled with task_terminal.
Agent session fails (crash, timeout, unrecoverable) HandleWakeupFailure -> parse rate-limit reset OR exponential backoff (4 attempts at 2m / 10m / 30m / 2h with 25% jitter). After cap: mark failed, create agent.error inbox item, fire agent_error wakeup for coordinator.
Rate-limit error with parseable reset Wakeup scheduled for reset + 30s; backoff skipped; RetryCount still incremented.
Rate-limit error without parseable reset Falls through to exponential backoff.
Retry promoted but task reassigned Cancelled with retry_stale_assignee; execution locks cleared.
Retry promoted but task cancelled Cancelled with retry_task_cancelled.
Budget pre-check fails Agent paused per policy; wakeup skipped (not retried); wakeup_budget_blocked activity entry.
Atomic checkout finds task locked by another agent Wakeup skipped, no retry.
Summary builder fails on successful taskless run Previous summary left intact; failure logged; next successful run rebuilds.
Coordinator's pre-installed routine fails to install at onboarding Logged + warned; coordinator's agent detail UI shows "no scheduled wake-ups" empty state. User can install one manually.
Routine cron tick missed (scheduler down) enqueue_missed_with_cap (default): fire missed ticks up to cap=25 with "missed N ticks" in next prompt context. skip_missed: fire current tick only.
Webhook signature verification fails Trigger rejected with 401; no routine run created.
Graceful shutdown begins Queue and cron loops cancel and join before repositories and SQLite close; no scheduler logs database is closed.

Persistence guarantees

Survives a kandev process restart: all agent_wakeup_requests rows including queued, claimed (re-claimed on restart), and scheduled_retry_at; all office_routines, office_routine_triggers (with next_run_at advanced), office_routine_runs history; agent_continuation_summaries (last-good); runs rows including result_json, assembled_prompt, summary_injected snapshots.

Does NOT survive (reconstructed on next tick): in-memory claim leases - a claimed wakeup whose process died is picked up by the staleness/recovery path; the scheduler's claim query is the source of truth. The unstarted-task recovery sweep suppresses duplicates with its NOT EXISTS check on runs: any queued, claimed, or finished run of any age blocks redispatch, while failed and cancelled runs do not.

Retention: idempotency-key dedup window 24 hours; summary cap 8 KB per row; routine run history retained for inspection (no automatic prune in scope here); catch-up cap (default 25) drops missed routine ticks beyond it (not recorded individually).

The scheduler reads all queued and unexpired-retry wakeup requests on boot and resumes processing them.

Scenarios

  • GIVEN an authoritative Office task is assigned to a worker agent instance, WHEN the assignment is saved, THEN a task_assigned wakeup is queued for that agent. The scheduler claims it, creates a session, and the agent starts working on the task.

  • GIVEN Office mode is enabled but a workspace contains only Kanban workflows, WHEN a Kanban task with a runner remains in TODO, THEN Office subscribers and recovery do not queue a task_assigned run for it.

  • GIVEN Kandev is shutting down while the shared runs scheduler or cron loop is active, WHEN graceful shutdown reaches database cleanup, THEN those loops have already stopped and joined and no sql: database is closed scheduler error is emitted.

  • GIVEN a worker agent is currently running a session (at capacity), WHEN a comment wakeup arrives for the same agent, THEN the wakeup stays in queued status. When the current session completes, the next scheduler tick picks up the wakeup and the agent processes the comment.

  • GIVEN a task with three subtasks assigned to different workers, WHEN all three subtasks reach done, THEN a single task_children_completed wakeup (coalesced) is queued for the parent task's assignee.

  • GIVEN a coordinator agent with the pre-installed "Coordinator heartbeat" routine (cron */5 * * * *), WHEN the wall clock crosses a 5-minute boundary, THEN the routines cron tick inserts a routine-source wakeup request, the dispatcher creates a taskless run, the agent receives the prompt with the continuation summary prepended, and on completion the summary is upserted under scope="routine:<id>".

  • GIVEN a wakeup for a paused agent instance, WHEN the scheduler claims it, THEN the wakeup is marked finished with no session created. The wakeup is not retried.

  • GIVEN a backend restart, WHEN the scheduler starts, THEN it reads all queued wakeup requests from SQLite and resumes processing them. No wakeups are lost.

  • GIVEN a parent task with subtasks [Spec (requires_approval, assigned to planner), Build (blocked_by Spec, assigned to developer)], WHEN the planner completes the Spec subtask, THEN Spec moves to in_review. WHEN the user approves, THEN Spec moves to done, Build's blocker resolves, and a task_blockers_resolved wakeup is queued for the developer agent. The developer starts working on Build automatically.

  • GIVEN a heavy routine "Daily Dep Update" with cron 0 9 * * * in UTC and assignee "Frontend Worker", WHEN the clock reaches 09:00 UTC, THEN a task titled "Daily Dep Update - 2026-04-25" is created on the routine workflow, assigned to Frontend Worker, and a task_assigned wakeup is queued.

  • GIVEN a routine with concurrency_policy=skip_if_active and an active task from a previous run, WHEN the routine fires again, THEN no new task is created and the routine run is recorded with status skipped.

  • GIVEN a routine with a webhook trigger and signing_mode=hmac_sha256, WHEN an external system POSTs to the trigger URL with valid signature and payload {"branch": "release/2.0"}, THEN the routine fires with {{branch}} resolved to "release/2.0" in the task template.

  • GIVEN the scheduler was down for 3 hours, WHEN it restarts, THEN routines with catch_up_policy=skip_missed fire only the current tick; routines with enqueue_missed_with_cap fire missed ticks up to the cap (default 25), with overflow summarized as "missed N ticks" in the next prompt context.

  • GIVEN a user on the routines page, WHEN they click "Run Now" on a routine with a required {{reason}} variable, THEN a modal prompts for the variable value before firing.

  • GIVEN a worker agent with skip_idle_wakeups = true and no tasks in TODO or IN_PROGRESS state, WHEN a lightweight-routine wakeup is claimed, THEN the scheduler logs wakeup_idle_skipped, marks the wakeup finished, records an activity entry, and does not launch a session. A task_assigned wakeup arriving for the same agent skips the check and proceeds normally.

  • GIVEN a coordinator agent with default skip_idle_wakeups = false, WHEN a routine-fired wakeup is claimed and the coordinator has no directly assigned tasks, THEN the skip check is not performed and the wakeup proceeds normally so the coordinator can do proactive coordination work.

  • GIVEN a wakeup fails with "rate_limit_error: resets at 4:00 AM UTC", WHEN HandleWakeupFailure is called at 3:45 AM UTC, THEN the wakeup is scheduled for 04:00:30 AM UTC (parsed time + 30s buffer). The exponential backoff table is not consulted. Similarly "Retry-After: 3600" schedules for now + 3600s + 30s, and "try again in 5 minutes" schedules for now + 5m30s.

  • GIVEN a wakeup fails with a generic network timeout (no rate-limit keywords), WHEN HandleWakeupFailure is called, THEN the existing exponential backoff applies unchanged. If retries exhaust MaxRetryCount, escalateFailure is called as normal - the rate-limit path does not suppress escalation.

  • GIVEN a task_assigned wakeup is queued for agent A on task T, WHEN task T is reassigned to agent B before the wakeup is claimed, THEN the wakeup is cancelled with reason assignee_changed, agent A is not launched, and a wakeup_stale_cancelled activity entry is logged.

  • GIVEN a wakeup for task T fails and a retry is scheduled 10 minutes out, WHEN task T is reassigned (or cancelled) before the retry fires, THEN the retry is cancelled at promotion time with reason retry_stale_assignee (or retry_task_cancelled), execution locks are cleared, and a wakeup_retry_cancelled activity entry is logged. A PATCH reassign on the API cancels pending retries for the previous assignee immediately.

  • GIVEN an authoritative Office task in TODO state assigned to agent A has no queued, claimed, or finished run and was created within the lookback window, WHEN the recovery sweep runs, THEN a task_assigned run is dispatched and a recovery_dispatch activity entry is logged. Ordinary Kanban tasks, tasks with a queued, claimed, or finished run of any age, and tasks created outside recovery_lookback_hours are skipped.

  • GIVEN a wakeup for a task that has reached DONE state, WHEN the staleness check runs at claim time, THEN the wakeup is cancelled with reason task_terminal and the agent is not launched.

  • GIVEN a coordinator's pre-installed "Coordinator heartbeat" routine is deleted by the user, WHEN the next scheduler tick runs, THEN no routine wakeup is queued for that coordinator; the coordinator only wakes via reactive sources (comments, errors, manual, self, user).

  • GIVEN two routine triggers fire for the same coordinator within the coalescing window, WHEN the dispatcher claims the first, THEN the second wakeup-request is inserted with status="coalesced", its payload is merged into the first run's context_snapshot, and coalesced_count is incremented.

Out of scope

  • Distributed scheduling across multiple backend instances (single-process scheduler); deduplication of recovery dispatches across instances (idempotency key is sufficient).
  • Priority ordering beyond FIFO within the queue (task priority handled at assignment time).
  • Wakeup scheduling with future timestamps as a primary API (routines handle scheduled execution).
  • Rate limiting per agent beyond the single-concurrency guard and cooldown.
  • Dynamically adjusting heartbeat cadence based on workload (backpressure scheduling).
  • Complex workflow chains (routine A triggers routine B); routines create independent tasks, inter-task dependencies use the blocker system.
  • Routine templates shared across workspaces; routine-level budget limits (use agent or project budgets); plugin-managed routines (pluginManagedResources pattern); routine revisions / rollback.
  • Webhook trigger UI polish beyond the create dialog and detail page (covered in a separate routine-webhooks spec).
  • A full agent-memory subsystem (vector store, semantic recall) - the continuation summary is deliberately the minimum viable memory layer.
  • Web UI for editing summaries directly (read-only display in the agent's overview tab is enough); backfilling historical heartbeat conversations into summaries.
  • Modifying MaxRetryCount for rate-limit errors (stays at 4); surfacing parsed reset times in the UI or inbox; per-provider configuration of reset-time parsing patterns; handling Retry-After as an HTTP response header (this spec covers only error message text).
  • Per-agent or per-project recovery lookback window overrides (workspace-level is sufficient).
  • Surfacing retry cancellation reasons in the agent detail page runs tab (existing retry count display is adequate); automatic reassignment when a stale wakeup is cancelled (cancellation only - the scheduler does not infer intent).
  • Suppressing non-heartbeat wakeups based on task state; configuring which task states count as "actionable" per agent or workspace.
  • Recovery sweeps for non-TODO states (IN_PROGRESS tasks with no active session are covered separately by the blocked-task-escalation spec).
  • Event-based triggers from external systems beyond webhooks (GitHub event subscriptions use webhooks as transport).