feat(orchestrator): add instance-wide agent session concurrency ceiling (#3687)
Co-authored-by: nova28 <17953305+nova28@users.noreply.github.com> Co-authored-by: Carlos Florencio <carlosmflorencio@gmail.com>
This commit is contained in:
Henry
@@ -0,0 +1,49 @@
|
||||
---
|
||||
status: done
|
||||
created: 2026-09-15
|
||||
requirements:
|
||||
- REQ-AGENTS-SESSION-CEILING-001
|
||||
system_design:
|
||||
- ../../specs/agents/system-design/session-concurrency-ceiling.md
|
||||
---
|
||||
|
||||
# Session Concurrency Ceiling Delivery Plan
|
||||
|
||||
## Overview
|
||||
|
||||
Add a shared instance-wide admission ceiling for agent sessions. Preserve the
|
||||
existing orchestrator launch seams, durable task metadata, and executor
|
||||
callbacks. Close the recovery, replay, stale-callback, and queue ownership
|
||||
gaps found during review.
|
||||
|
||||
## Delivery flow
|
||||
|
||||
```text
|
||||
launch seam -> admission controller -> reservation or deferred_launch CAS
|
||||
-> executor callback -> confirm/release -> sweep replay -> task update
|
||||
```
|
||||
|
||||
The controller owns live admission state. The task repository owns the durable
|
||||
retry record. The executor owns process identity. These ownership boundaries
|
||||
remain separate during repair.
|
||||
|
||||
## Verification strategy
|
||||
|
||||
- Run focused orchestrator tests for cold resume, callback identity, Office
|
||||
replay, dynamic relaunch outcomes, queue retry, and deferred-record conflicts.
|
||||
- Run focused task-service tests for nested ceiling prompt edits.
|
||||
- Run `go test -race` for the orchestrator package because reservations and
|
||||
callback release are concurrent.
|
||||
- Run the documentation coverage evaluator through the PR workflow after push.
|
||||
|
||||
## Work orders
|
||||
|
||||
- [x] [Task 01: Preserve ceiling launch ownership](task-01-session-concurrency-ceiling.md)
|
||||
|
||||
## Risks
|
||||
|
||||
- A repository read failure before a process callback is treated as not owned,
|
||||
so a reservation can wait for the sweep instead of being released by an
|
||||
ambiguous callback.
|
||||
- The startup ceiling is environment-only. Operators must restart the backend
|
||||
after changing it.
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
id: "01-session-concurrency-ceiling"
|
||||
title: "Preserve session ceiling launch ownership"
|
||||
status: done
|
||||
wave: 1
|
||||
depends_on: []
|
||||
plan: "plan.md"
|
||||
requirements:
|
||||
- REQ-AGENTS-SESSION-CEILING-001
|
||||
acceptance_criteria:
|
||||
- AC-AGENTS-SESSION-CEILING-001.1
|
||||
- AC-AGENTS-SESSION-CEILING-001.2
|
||||
- AC-AGENTS-SESSION-CEILING-001.3
|
||||
- AC-AGENTS-SESSION-CEILING-001.4
|
||||
- AC-AGENTS-SESSION-CEILING-001.5
|
||||
- AC-AGENTS-SESSION-CEILING-001.6
|
||||
- AC-AGENTS-SESSION-CEILING-001.7
|
||||
system_design:
|
||||
- ../../specs/agents/system-design/session-concurrency-ceiling.md
|
||||
---
|
||||
|
||||
# Task 01: Preserve Session Ceiling Launch Ownership
|
||||
|
||||
## Outcome
|
||||
|
||||
Keep every accepted or deferred launch owned by one component across refusal,
|
||||
callback, replay, and prompt-edit paths.
|
||||
|
||||
## In scope
|
||||
|
||||
- Return the existing resume attempt when seam-3 admission refuses a cold
|
||||
resume.
|
||||
- Stop a successful early prompt-fallback result before callback preparation.
|
||||
- Fence process callbacks by the current session execution identity.
|
||||
- Preserve manual origin for the compound resume-and-prompt operation.
|
||||
- Distinguish dynamic relaunch success, deferral, and failure.
|
||||
- Keep Office start records out of workflow auto-start drop checks.
|
||||
- Report a different pending payload as an explicit conflict.
|
||||
- Requeue ordinary messages after seam-3 refusal.
|
||||
- Update nested ceiling launch payload prompts.
|
||||
|
||||
## Exclusions
|
||||
|
||||
- New launch seams, new storage tables, or a frontend settings surface.
|
||||
- Changes to provider protocols or the task card presentation.
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. Automatic refusals either retain their exact durable replay record or return
|
||||
an explicit conflict. Manual actions remain admitted and audited.
|
||||
2. A stale execution callback cannot release a successor reservation, and a
|
||||
deferred detached relaunch does not finalize its automation run.
|
||||
3. Cold resume, queue retry, Office replay, dynamic outcome, and nested prompt
|
||||
regressions pass in focused tests.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cd apps/backend && go test ./internal/orchestrator ./internal/task/service -count=1
|
||||
cd apps/backend && go test -race ./internal/orchestrator -count=1
|
||||
git diff --check
|
||||
```
|
||||
|
||||
## Likely files
|
||||
|
||||
- `apps/backend/internal/orchestrator/task_operations.go`
|
||||
- `apps/backend/internal/orchestrator/dynamic_launch.go`
|
||||
- `apps/backend/internal/orchestrator/ceiling_defer.go`
|
||||
- `apps/backend/internal/orchestrator/ceiling_replay.go`
|
||||
- `apps/backend/internal/orchestrator/ceiling_release.go`
|
||||
- `apps/backend/internal/orchestrator/event_handlers_agent.go`
|
||||
- `apps/backend/internal/task/service/service_deferred_launch.go`
|
||||
- matching regression tests
|
||||
|
||||
## Results
|
||||
|
||||
- Focused orchestrator regressions pass for cold-resume ownership, stale
|
||||
callback fencing, Office replay, duplicate conflicts, queue retry, and the
|
||||
dynamic deferred outcome.
|
||||
- Focused task-service regressions pass for top-level and nested prompt edits.
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
status: active
|
||||
system: agents
|
||||
created: 2026-09-15
|
||||
owners:
|
||||
- kandev
|
||||
---
|
||||
|
||||
# Session Concurrency Ceiling Requirements
|
||||
|
||||
## Overview
|
||||
|
||||
One Kandev instance can start more agent sessions than its host can serve. A
|
||||
shared ceiling must limit new automatic starts while preserving explicit user
|
||||
actions and accepted work. This contract belongs to the agent system because it
|
||||
controls admission of agent executions. Tasks own their durable deferral data.
|
||||
|
||||
## Terminology
|
||||
|
||||
- **Session ceiling:** The maximum number of sessions in `STARTING` or `RUNNING`
|
||||
state, including launches admitted in the current process.
|
||||
- **Deferred launch:** A durable task record that contains the complete launch
|
||||
kind and payload for an automatic request that the ceiling refused.
|
||||
- **Manual origin:** A direct user action. It can use a manual override when the
|
||||
ceiling is full.
|
||||
|
||||
## Requirements
|
||||
|
||||
### REQ-AGENTS-SESSION-CEILING-001: Bound instance agent sessions
|
||||
|
||||
**Intent:** Keep automatic agent starts within the instance capacity while
|
||||
keeping accepted work and manual recovery available.
|
||||
|
||||
**User story:** As an operator, I want automatic agent starts to respect a
|
||||
bounded instance capacity, so that one installation does not overload its host.
|
||||
|
||||
#### Acceptance criteria
|
||||
|
||||
- **AC-AGENTS-SESSION-CEILING-001.1:** When an automatic launch would exceed the
|
||||
configured ceiling, the system shall refuse the launch and persist its kind,
|
||||
replay payload, origin, reason, and enqueue time on the task.
|
||||
- **AC-AGENTS-SESSION-CEILING-001.2:** When a manual launch would exceed the
|
||||
ceiling, the system shall admit it and record that it used a manual override.
|
||||
- **AC-AGENTS-SESSION-CEILING-001.3:** The ceiling shall count sessions in
|
||||
`STARTING` and `RUNNING` state together with in-flight reservations, and shall
|
||||
release a reservation only after its launch reaches a counted state or fails.
|
||||
- **AC-AGENTS-SESSION-CEILING-001.4:** A deferred launch shall retain its first
|
||||
payload when a different launch for the same task arrives, and the second
|
||||
caller shall receive an explicit conflict so it retains ownership.
|
||||
- **AC-AGENTS-SESSION-CEILING-001.5:** A retry sweep shall replay deferred
|
||||
launches by their stored kind. It shall preserve a record when capacity is
|
||||
still full or replay fails for a non-ceiling reason.
|
||||
- **AC-AGENTS-SESSION-CEILING-001.6:** A callback from an older execution shall
|
||||
not release or confirm a reservation held by a successor execution for the
|
||||
same session.
|
||||
- **AC-AGENTS-SESSION-CEILING-001.7:** The startup environment variable
|
||||
`KANDEV_MAX_CONCURRENT_SESSIONS` shall accept a non-negative integer, use zero
|
||||
for unlimited, and use a derived default with a minimum of two when unset or
|
||||
invalid.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Per-workspace or per-user ceilings.
|
||||
- A frontend settings control for this startup-only value.
|
||||
- Replacing the orchestrator's existing launch seams or task repository.
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
status: current
|
||||
system: agents
|
||||
requirements:
|
||||
- REQ-AGENTS-SESSION-CEILING-001
|
||||
---
|
||||
|
||||
# Session Concurrency Ceiling System Design
|
||||
|
||||
## Purpose and boundaries
|
||||
|
||||
The orchestrator owns one admission controller shared by manual starts,
|
||||
resumes, workflow starts, queue drains, and dynamic relaunches. The controller
|
||||
does not own task metadata or provider execution. It returns a reservation or a
|
||||
typed refusal. The task repository owns the durable deferred-launch record.
|
||||
|
||||
## Requirement mapping
|
||||
|
||||
| Requirement | Design section |
|
||||
| --- | --- |
|
||||
| `REQ-AGENTS-SESSION-CEILING-001` | [Admission and replay](#admission-and-replay), [Failure and recovery](#failure-and-recovery) |
|
||||
|
||||
## Components and responsibilities
|
||||
|
||||
| Component | Responsibility |
|
||||
| --- | --- |
|
||||
| `sessionCeilingController` | Counts persisted `STARTING`/`RUNNING` sessions and process-local reservations. It admits, refuses, confirms, and releases launches. |
|
||||
| Orchestrator launch seams | Pass the launch origin and complete replay payload to the controller. They consume a reservation only after launch success. |
|
||||
| `deferCeilingRefusal` | Merges the ceiling-owned keys into task `deferred_launch` with compare-and-set semantics. |
|
||||
| Ceiling sweep | Lists tasks with ceiling records, validates eligibility, and dispatches the stored launch kind. |
|
||||
| Task repository and service | Store and update the shared deferred record. Prompt edits update both legacy top-level data and the nested ceiling payload. |
|
||||
| Executor callbacks | Confirm or release only when the callback execution still owns the session row. |
|
||||
|
||||
## Data and contracts
|
||||
|
||||
The shared record uses `ceiling_deferred`, `ceiling_launch_kind`,
|
||||
`ceiling_launch_payload`, `ceiling_launch_origin`, `ceiling_reason_code`, and
|
||||
`ceiling_queued_at`. The payload is nested so replay does not confuse a resume,
|
||||
prompt ensure, or dynamic relaunch with a task start. A different pending
|
||||
payload returns `ErrCeilingLaunchConflict`; the existing record stays unchanged.
|
||||
|
||||
The controller resolves `KANDEV_MAX_CONCURRENT_SESSIONS` once at startup. An
|
||||
unset or invalid value uses half the host CPU count, with a minimum of two. Zero
|
||||
means unlimited. The value is an environment-only startup setting.
|
||||
|
||||
## Admission and replay
|
||||
|
||||
The flow is:
|
||||
|
||||
```text
|
||||
launch request
|
||||
-> orchestrator seam
|
||||
-> sessionCeilingController.admit
|
||||
-> reservation, or typed refusal
|
||||
-> deferCeilingRefusal(task CAS) when automatic refusal
|
||||
-> executor launch and callback
|
||||
-> confirm/release reservation
|
||||
-> ceiling sweep replays the stored kind when capacity is available
|
||||
```
|
||||
|
||||
Manual origins bypass refusal and write a manual-override audit entry. Automatic
|
||||
origins never become manual during replay. The sweep clears a record only after
|
||||
successful dispatch. A repeated refusal leaves the original timestamp and
|
||||
payload in place.
|
||||
|
||||
## Failure and recovery
|
||||
|
||||
Reservation ownership is session-keyed. A stale process-start callback first
|
||||
checks the persisted session execution identity; it cannot release a successor's
|
||||
reservation. A dynamic relaunch reports three outcomes: succeeded, deferred, or
|
||||
failed. A deferred detached relaunch leaves the automation run and its durable
|
||||
record for the sweep. Queue dispatch treats a seam-3 refusal as retryable so the
|
||||
original message remains in the queue.
|
||||
|
||||
Office automatic starts do not use workflow-step auto-start eligibility when the
|
||||
sweep evaluates a `start` record. This keeps Office scheduling ownership in the
|
||||
Office path.
|
||||
|
||||
## Persistence
|
||||
|
||||
`deferred_launch` is updated with a read-compare-write retry loop. The ceiling
|
||||
keys share the task record with other launch intents and are removed as a group
|
||||
after replay or a terminal drop. A prompt edit preserves unrelated keys and
|
||||
updates the nested replay payload.
|
||||
|
||||
## Security
|
||||
|
||||
The controller receives launch data from trusted backend paths. The nested
|
||||
payload is excluded from plugin host data. No prompt or environment value is
|
||||
executed during replay; it is passed to the same existing launch seam after
|
||||
task and session eligibility checks.
|
||||
|
||||
## Observability
|
||||
|
||||
Admission decisions, refusal reasons, reservation expiry, replay drops, and
|
||||
manual overrides use the existing orchestrator logs and task status messages.
|
||||
The stored reason and population snapshot make a refusal diagnosable after a
|
||||
restart.
|
||||
Reference in New Issue
Block a user