feat(office): rate-limit agent-initiated task-assigned wakes per task (#3814)
Co-authored-by: nova28 <17953305+nova28@users.noreply.github.com> Co-authored-by: Carlos Florencio <carlosmflorencio@gmail.com>
This commit is contained in:
@@ -0,0 +1,57 @@
|
||||
---
|
||||
created: 2026-09-17
|
||||
status: done
|
||||
requirements:
|
||||
- REQ-OFFICE-ASSIGN-RATE-001
|
||||
- REQ-OFFICE-ASSIGN-RATE-002
|
||||
- REQ-OFFICE-ASSIGN-RATE-003
|
||||
system_design:
|
||||
- ../../specs/office/system-design/assignment-wake-rate-limit-01.md
|
||||
---
|
||||
|
||||
# Implementation Plan: Office Assignment Wake Rate Limit
|
||||
|
||||
## Overview
|
||||
|
||||
Bound the number of agent-initiated `task_assigned` wakes one task can produce
|
||||
in a rolling window. Removing the same-agent equality gate in
|
||||
`ApplyTaskMutation` let an agent holding `can_assign_tasks` reassign a task to
|
||||
itself repeatedly, spaced past the five-second coalescing window, minting an
|
||||
unbounded stream of genuinely distinct wakes. Neither existing bound catches
|
||||
this: the dedup key is generation-aware by design, and `task_assigned` is on the
|
||||
attended-provenance allowlist, so the daily spend ceiling never applies.
|
||||
|
||||
## Scope
|
||||
|
||||
- One admission gate in `SchedulerService.queueRun`, after the recent-duplicate
|
||||
lookup and coalescing, before `CreateRun`. It binds the task-mutation
|
||||
producer, which is the only `task_assigned` producer routing through that
|
||||
function and the only one carrying an actor.
|
||||
- A task-scoped window count derived from existing `runs` rows; no new table,
|
||||
column, migration, or background job.
|
||||
- A new `QueueOutcome` value, declared in both places the enum lives.
|
||||
- One `expvar` counter labelled by a closed three-value reason set, plus a Warn
|
||||
log at the decision site.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Dedup key identity, the coalescing window, and the 24-hour lookback.
|
||||
- The event-subscriber producer of assignment wakes; a per-task allowance
|
||||
cannot bind it.
|
||||
- The provenance classifier and the budget admission gates.
|
||||
- Rate-limiting the assignment write itself; a refused wake still persists the
|
||||
assignee and bumps the generation.
|
||||
- Any frontend. No screen renders the allowance, the counters, or a refusal.
|
||||
|
||||
## Implementation Wave
|
||||
|
||||
- [ ] [task-01-assignment-wake-rate-limit](task-01-assignment-wake-rate-limit.md)
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cd apps/backend && go test ./internal/office/scheduler ./internal/office/service \
|
||||
./internal/runs/service ./internal/workflow/engine
|
||||
python3 scripts/lint-spec-files.py --all
|
||||
python3 scripts/list-docs.py validate
|
||||
```
|
||||
@@ -0,0 +1,145 @@
|
||||
---
|
||||
id: "01-assignment-wake-rate-limit"
|
||||
title: "Bound agent-initiated assignment wakes per task"
|
||||
status: done
|
||||
wave: 1
|
||||
depends_on: []
|
||||
plan: "plan.md"
|
||||
requirements:
|
||||
- REQ-OFFICE-ASSIGN-RATE-001
|
||||
- REQ-OFFICE-ASSIGN-RATE-002
|
||||
- REQ-OFFICE-ASSIGN-RATE-003
|
||||
acceptance_criteria:
|
||||
- AC-OFFICE-ASSIGN-RATE-001.1
|
||||
- AC-OFFICE-ASSIGN-RATE-001.2
|
||||
- AC-OFFICE-ASSIGN-RATE-001.3
|
||||
- AC-OFFICE-ASSIGN-RATE-001.4
|
||||
- AC-OFFICE-ASSIGN-RATE-001.5
|
||||
- AC-OFFICE-ASSIGN-RATE-001.6
|
||||
- AC-OFFICE-ASSIGN-RATE-001.7
|
||||
- AC-OFFICE-ASSIGN-RATE-001.8
|
||||
- AC-OFFICE-ASSIGN-RATE-001.9
|
||||
- AC-OFFICE-ASSIGN-RATE-001.10
|
||||
- AC-OFFICE-ASSIGN-RATE-001.11
|
||||
- AC-OFFICE-ASSIGN-RATE-001.12
|
||||
- AC-OFFICE-ASSIGN-RATE-001.13
|
||||
- AC-OFFICE-ASSIGN-RATE-002.1
|
||||
- AC-OFFICE-ASSIGN-RATE-002.2
|
||||
- AC-OFFICE-ASSIGN-RATE-002.3
|
||||
- AC-OFFICE-ASSIGN-RATE-002.4
|
||||
- AC-OFFICE-ASSIGN-RATE-002.5
|
||||
- AC-OFFICE-ASSIGN-RATE-002.6
|
||||
- AC-OFFICE-ASSIGN-RATE-003.1
|
||||
- AC-OFFICE-ASSIGN-RATE-003.2
|
||||
- AC-OFFICE-ASSIGN-RATE-003.3
|
||||
- AC-OFFICE-ASSIGN-RATE-003.4
|
||||
- AC-OFFICE-ASSIGN-RATE-003.5
|
||||
- AC-OFFICE-ASSIGN-RATE-003.6
|
||||
system_design:
|
||||
- ../../specs/office/system-design/assignment-wake-rate-limit-01.md
|
||||
---
|
||||
|
||||
# Task 01: Bound agent-initiated assignment wakes per task
|
||||
|
||||
## Summary
|
||||
|
||||
Add one admission gate to the Office run queue that refuses an agent-initiated
|
||||
`task_assigned` wake once a task has already produced `N` of them inside a
|
||||
rolling window `W`. The gate is the last check before insertion, so "admitted"
|
||||
means "a row was written" and the count needs no state of its own.
|
||||
|
||||
## Scope
|
||||
|
||||
- A window-count repository read over the **stored** `runs` rows, filtered by
|
||||
`reason`, the payload's `task_id` and `actor_type`, and `requested_at`, using
|
||||
`dialect.JSONExtract` and the string-type guard `CoalesceRun` already needs.
|
||||
- The **incoming** wake's `actor_type` and `task_id` read in Go, by unmarshalling
|
||||
the payload `queueRun` was handed, the way `taskIDFromPayload` does.
|
||||
`dialect.JSONExtract` is SQL over a column and does not apply to a wake that is
|
||||
not a row yet; the two reads are separate.
|
||||
- The gate in `SchedulerService.queueRun`, positioned after `CheckIdempotencyKey`
|
||||
and `CoalesceRun` and before `CreateRun`. That position binds the
|
||||
task-mutation producer only (`AC-OFFICE-ASSIGN-RATE-001.12`); no other
|
||||
`task_assigned` producer routes through this function.
|
||||
- `RunContext.ActorType` reaching the payload for the mutation producer's
|
||||
agent-actor wakes, pinned by a test
|
||||
(`AC-OFFICE-ASSIGN-RATE-001.13`). A wake carrying no `actor_type` is out of
|
||||
scope: admit, no allowance consumed, no counter moved.
|
||||
- `N = 5` and `W = 10 * time.Minute` declared once beside
|
||||
`CoalesceWindowSeconds` and `IdempotencyWindowHours`.
|
||||
- A new `QueueOutcome` value added to **both** declarations
|
||||
(`internal/runs/service` and `internal/workflow/engine/adapters.go`).
|
||||
- A reporting helper beside `ReportWindowedDedup` / `ReportDurableDedup`, a new
|
||||
`expvar.Map` beside `office_run_dedup_total`, and a Warn log carrying task id,
|
||||
assignee profile id, acting agent id, observed count and allowance.
|
||||
|
||||
## Exclusions
|
||||
|
||||
- No change to dedup key construction, the coalescing predicate, the 24-hour
|
||||
lookback, provenance classification, or budget admission.
|
||||
- No change to `SetTaskAssigneeAsAgent`'s permission check or to the mutation
|
||||
itself: a refused wake still persists the assignee, bumps
|
||||
`assignment_generation`, publishes the task update, and interrupts a displaced
|
||||
assignee's session.
|
||||
- No bound on the other four `task_assigned` producers: the task-lifecycle event
|
||||
subscriber (`service/event_subscribers.go`), onboarding, the recovery sweep,
|
||||
and the orchestrator's workflow auto-start. None carries an actor, and a
|
||||
per-task allowance cannot bind a producer that wakes a task just created,
|
||||
onboarded or swept. Do not "fix" this by giving one of them an actor.
|
||||
- No ordering or tiebreak logic. `AC-OFFICE-ASSIGN-RATE-001.10` makes the
|
||||
decision order-independent; no `ORDER BY` belongs in the window count.
|
||||
- No frontend, no migration, no new configuration surface.
|
||||
|
||||
## Implementation acceptance conditions
|
||||
|
||||
1. `N` agent-actor assignment wakes for one task admit and the next refuses,
|
||||
writing no row and returning the new outcome; a `user`-actor wake and an
|
||||
empty-`callerAgentID` call are never refused, and neither is a wake carrying
|
||||
no `actor_type` at all.
|
||||
2. A wake merged by coalescing or suppressed by the idempotency lookup neither
|
||||
consumes the allowance nor can be refused by it; a refused wake does not
|
||||
consume it either.
|
||||
3. A failing count read and an unattributable task both **admit**, each
|
||||
incrementing the counter under its own `reason` label, with no identifier
|
||||
used as a label. Unattributable covers an absent, null or empty `task_id` and
|
||||
a present-but-non-string one, identically; an unparseable payload is *not*
|
||||
unattributable — it fails the actor step first and leaves scope with no
|
||||
counter moved.
|
||||
4. A test enumerates the `task_assigned` producers and asserts the actor type
|
||||
each emits, pinning `AC-OFFICE-ASSIGN-RATE-001.12`'s invariant that no other
|
||||
producer sets `actor_type` `agent`.
|
||||
5. A mixed-actor coalesce is covered in both directions and asserts the count the
|
||||
gate then observes, which is the accepted off-by-one in the requirement's
|
||||
`## Accepted consequence` rather than an exact count.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cd apps/backend && go test ./internal/office/scheduler ./internal/office/service \
|
||||
./internal/runs/service ./internal/workflow/engine
|
||||
cd apps/backend && make lint
|
||||
```
|
||||
|
||||
## Likely files
|
||||
|
||||
- `apps/backend/internal/office/scheduler/run.go`
|
||||
- `apps/backend/internal/office/scheduler/reactivity.go`
|
||||
- `apps/backend/internal/runs/service/dedup.go`
|
||||
- `apps/backend/internal/runs/service/metrics_vars.go`
|
||||
- `apps/backend/internal/workflow/engine/adapters.go`
|
||||
- `apps/backend/internal/runs/repository/**` (window-count query, SQLite and
|
||||
PostgreSQL twins)
|
||||
|
||||
## Risks
|
||||
|
||||
- The `QueueOutcome` enum is declared twice and must stay in sync; adding the
|
||||
value to one declaration only breaks the engine adapter's reporting.
|
||||
- A hand-written `json_extract` diverges between SQLite and PostgreSQL. Use
|
||||
`dialect.JSONExtract` and guard the JSON string type, or a numeric `task_id`
|
||||
in an older payload silently joins another task's allowance.
|
||||
- Tests must inject time (`synctest`), never `time.Sleep`.
|
||||
- `CoalesceRun` overwrites the surviving row's payload and does not match on
|
||||
`actor_type`, so a mixed-actor merge relabels the actor the count reads. This
|
||||
is an accepted consequence, not a bug to fix here: do not add `actor_type` to
|
||||
the coalescing predicate, and do not write a test that asserts an exact count
|
||||
across such a merge.
|
||||
@@ -0,0 +1,354 @@
|
||||
---
|
||||
status: draft
|
||||
system: office
|
||||
created: 2026-09-17
|
||||
owners:
|
||||
- kandev
|
||||
---
|
||||
|
||||
# Office: Assignment Wake Rate Limit Requirements
|
||||
|
||||
## Overview
|
||||
|
||||
An Office agent holding `can_assign_tasks` — the default `worker` role
|
||||
permission — can call the dashboard's assignee mutator repeatedly with itself as
|
||||
the target. Each call commits a new `tasks.assignment_generation`, so each
|
||||
produces a genuinely new wake occurrence with a distinct dedup key. Spaced past
|
||||
the coalescing window, the agent mints an unbounded stream of never-suppressed
|
||||
`task_assigned` runs against its own task.
|
||||
|
||||
Two mechanisms look like they should bound this and do not.
|
||||
|
||||
**Deduplication does not**, by design. `AC-OFFICE-RUN-DEDUP-001.3` *requires* a
|
||||
second assignment to the same agent to insert a run: a key must suppress a
|
||||
redelivery and must not suppress a repeat. Every wake in this stream is a
|
||||
repeat.
|
||||
|
||||
**The spend ceiling does not**, and this is the load-bearing finding.
|
||||
`ClassifyRunProvenance` (`AC-OFFICE-BUDGET-007.1`) is a total function of
|
||||
`reason` alone, and `task_assigned` sits on its attended allowlist because a
|
||||
*person* assigning a task is plausibly waiting on it. Attended runs are exempt
|
||||
from operator policies and from the default ceiling, and a reason-only
|
||||
classifier cannot tell an agent actor from a human one, so these runs never
|
||||
reach it. The spend is unbounded, not loosely bounded.
|
||||
|
||||
This document owns the bound: a per-task allowance on agent-initiated
|
||||
assignment wakes, a separate guard rather than a change to coalescing.
|
||||
|
||||
## Terminology
|
||||
|
||||
- **Agent-initiated assignment wake:** a `task_assigned` wake for a mutation
|
||||
whose actor type is `agent`; defined normatively by
|
||||
`AC-OFFICE-ASSIGN-RATE-001.1`.
|
||||
- **Allowance:** the admitted agent-initiated assignment wakes permitted for one
|
||||
task in one rolling window.
|
||||
- **Admitted:** a wake that inserted a `runs` row. One suppressed by dedup or
|
||||
merged by coalescing is neither admitted nor refused; an earlier gate decided
|
||||
it.
|
||||
- **Refused:** a wake this capability declined, inserting no row and merging into
|
||||
none.
|
||||
|
||||
## Prior art
|
||||
|
||||
### Our own recorded reasoning (wiki)
|
||||
|
||||
**Searched:** QMD collection `wiki` (564 documents, re-indexed 2026-09-17),
|
||||
semantic query with reranking off. A semantic result, not a degraded keyword
|
||||
scan.
|
||||
|
||||
[[agent-budget-governance]] states the position this capability departs from: an
|
||||
operator-initiated agent is bounded by the operator's attention, a cron- or
|
||||
webhook-triggered one by nothing except a budget. Its companion
|
||||
[[event-triggered-agent-activation]] frames the axis as *who starts it*.
|
||||
|
||||
Our case is a third kind neither page enumerates: **agent-initiated** activation
|
||||
that the classifier deciding whether a budget applies cannot tell from
|
||||
operator-initiated. It has an actor, so Kandev reads it as attended and bounded
|
||||
by an attention nobody is paying. That is why the budget is not the bound here.
|
||||
|
||||
Two of its anti-patterns are adopted: an org cap without a per-principal cap
|
||||
(Kandev's default ceiling is exactly an org cap, this adds a narrower per-task
|
||||
one, and the missing aggregate is named under `## Out of scope`), and treating a
|
||||
cap as cost control when it is also the blast radius of an agent's autonomy
|
||||
(queue rows, scheduler churn and session launches are blast radius no spend
|
||||
ceiling touches). We depart from its window shape, where the corpus converges on
|
||||
`calendar_month_utc`, because a calendar window lets an attacker spend a full
|
||||
allowance either side of the boundary.
|
||||
|
||||
### What other products shipped (saas-kb)
|
||||
|
||||
Recorded in the system design's `## Prior art considered`, beside the choices it
|
||||
informs: a count cap rather than a cooldown, and why the near-universal
|
||||
per-principal concurrency cap is not this axis.
|
||||
|
||||
### Prior art inside this repository
|
||||
|
||||
Three precedents shape the criteria below: Office automations'
|
||||
`max_concurrent_runs` cap, the workspace pause gate's refused admission, and
|
||||
`checkIdleSkip`'s open failure, which `AC-OFFICE-ASSIGN-RATE-002.2` follows and
|
||||
the pause gate deliberately does not. Call sites and the contrast are in the
|
||||
system design.
|
||||
|
||||
## Requirements
|
||||
|
||||
### REQ-OFFICE-ASSIGN-RATE-001: A per-task allowance bounds agent-initiated assignment wakes
|
||||
|
||||
**Intent:** A repeat assignment must still wake the agent, so the bound cannot
|
||||
suppress the class. It bounds how often the class may fire, scoped so a human
|
||||
operator is never throttled.
|
||||
|
||||
**User story:** As an Office operator, I want an agent that can assign tasks to
|
||||
be unable to mint an endless stream of runs against a task, so that an
|
||||
unattended workspace cannot spend without limit while nobody is watching.
|
||||
|
||||
#### Acceptance criteria
|
||||
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.1:** An *agent-initiated assignment wake* is a
|
||||
`task_assigned` wake produced for a task mutation whose actor type is exactly
|
||||
`agent`. A wake carrying any other actor type — `user`, an unrecognised value,
|
||||
or no actor type at all — is not one, and no criterion here applies to it: it
|
||||
shall be admitted without consuming an allowance, never refused, and shall
|
||||
move none of the counters in `REQ-OFFICE-ASSIGN-RATE-003`. The internal and administrative caller that
|
||||
supplies an empty acting agent identifier is classified `user`, so it is
|
||||
never limited. An absent actor type is a normal classification outcome, not a
|
||||
degraded one: the out-of-scope producers `AC-OFFICE-ASSIGN-RATE-001.12` names
|
||||
carry none.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.2:** The system shall admit at most `N`
|
||||
agent-initiated assignment wakes for one task within any rolling window of
|
||||
duration `W` in serial evaluation. Concurrent evaluation may exceed `N`;
|
||||
see AC-OFFICE-ASSIGN-RATE-002.1.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.3:** The allowance shall be scoped to the task
|
||||
alone. Wakes for one task count against one allowance whichever agent is being
|
||||
assigned, so alternating assignment between two agents shall not yield two
|
||||
allowances.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.4:** Only an admitted wake shall consume the
|
||||
allowance. A refused wake shall not consume it: each admitted wake frees its
|
||||
share of the allowance one window after its own request instant, however many
|
||||
refusals fell in between, subject to `## Accepted consequence`'s coalescing
|
||||
relabel.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.5:** The allowance shall be evaluated after the
|
||||
recent-duplicate lookup and after the coalescing attempt, and before the run
|
||||
row is inserted. A wake that an earlier gate already suppressed shall be
|
||||
neither refused by this one nor counted against the allowance, subject to
|
||||
`## Accepted consequence`'s coalescing relabel.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.6:** When the allowance is exhausted, the system
|
||||
shall refuse the wake: insert no run row, merge into no existing run, and
|
||||
report neither a queued nor a coalesced nor a deduplicated outcome for it.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.7:** A refusal shall not abort or roll back the
|
||||
mutation. The assignee change stays persisted, the assignment generation stays
|
||||
bumped, and the mutation's other effects — the previous assignee's session
|
||||
interrupt, that assignee's office session row transition, and the task-updated
|
||||
publication — shall still occur. This mutation produces no
|
||||
comment wake and no mention wake, so none is required here and a regression
|
||||
test for this criterion shall not assert one.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.8:** Within the producer scope fixed by
|
||||
`AC-OFFICE-ASSIGN-RATE-001.12`, the allowance shall apply whether or not the
|
||||
wake carries a dedup key. A wake enqueued keyless because its generation could
|
||||
not be resolved, per `REQ-OFFICE-RUN-DEDUP-003`, is still bounded, so the
|
||||
keyless path is not an unbounded bypass of this allowance.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.9:** An unassignment — a mutation whose new
|
||||
assignee identifier is empty — produces no assignment wake, and shall
|
||||
therefore be neither refused nor counted against the allowance.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.10:** The admission decision shall be a count of
|
||||
the wakes falling in the window and shall not depend on the order in which
|
||||
they fall within it. No ordering rule and no tiebreak between two wakes
|
||||
sharing one persisted request instant is part of this contract; window
|
||||
membership is fixed by `AC-OFFICE-ASSIGN-RATE-001.11` and the limit by
|
||||
`AC-OFFICE-ASSIGN-RATE-001.2`. A test that must identify *which* wake was the
|
||||
`N`th shall arrange distinct persisted request instants rather than rely on a
|
||||
tiebreak.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.11:** The window shall be the half-open interval
|
||||
from the evaluation instant less `W`, exclusive, to the evaluation instant,
|
||||
inclusive. A wake whose request instant is exactly one window old falls
|
||||
outside the window and shall not count.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.12:** The boundary is payload shape, not a count
|
||||
of producers: the gate binds exactly those `task_assigned` wakes that reach it
|
||||
carrying actor type `agent`, and every criterion here is scoped to that shape.
|
||||
One producer does so today, the task-mutation reactivity path, the only one
|
||||
carrying the mutation's actor. No other producer of a
|
||||
`task_assigned` wake shall set actor type `agent`; that invariant shall be
|
||||
pinned by a test enumerating the producers and asserting the actor type each
|
||||
emits. Producers carrying no actor are out of scope per `## Out of scope`.
|
||||
- **AC-OFFICE-ASSIGN-RATE-001.13:** A wake produced by the in-scope producer for
|
||||
a mutation whose actor is an agent shall carry the actor type `agent` on the
|
||||
wake itself, so the gate classifies it without re-reading the mutation.
|
||||
Losing that field would take every such wake out of scope under
|
||||
`AC-OFFICE-ASSIGN-RATE-001.1` and silently stop the allowance binding.
|
||||
|
||||
### REQ-OFFICE-ASSIGN-RATE-002: The allowance degrades safely and predictably
|
||||
|
||||
**Intent:** A cost guard that turns a transient database error into dropped
|
||||
legitimate wakes trades a bounded cost problem for an unbounded liveness one.
|
||||
Failure direction, race behaviour and constants must be stated, not discovered.
|
||||
|
||||
**User story:** As a Kandev maintainer, I want the allowance's failure and race
|
||||
behaviour fixed in the contract, so that an implementation cannot pick the
|
||||
convenient answer.
|
||||
|
||||
#### Acceptance criteria
|
||||
|
||||
- **AC-OFFICE-ASSIGN-RATE-002.1:** When two agent-initiated assignment wakes
|
||||
for one task are evaluated concurrently, each may observe an allowance with
|
||||
room and both may be admitted. The system shall permit over-admission bounded
|
||||
by the number of concurrent evaluations, and shall never refuse a wake before
|
||||
`N` have been admitted in the window.
|
||||
- **AC-OFFICE-ASSIGN-RATE-002.2:** When the read deriving the current count
|
||||
fails, the system shall admit the wake and shall count and log that degraded
|
||||
admission under its own distinct reason. It shall not refuse the wake. The
|
||||
degraded admission shall be logged at warning level carrying the task
|
||||
identifier, the assignee agent profile identifier, the acting agent
|
||||
identifier, the allowance, and the underlying read failure; it shall not carry
|
||||
an observed count, which the failed read did not produce. This is
|
||||
deliberately the opposite of the workspace pause gate, whose closed failure is
|
||||
correct because a kill switch failing open is not a kill switch. The precedent
|
||||
is `checkIdleSkip`.
|
||||
- **AC-OFFICE-ASSIGN-RATE-002.3:** When a wake's task cannot be determined, the
|
||||
system shall admit it and count it under a distinct unattributed reason, so a
|
||||
payload-shape change cannot silently disable the gate by making every wake
|
||||
unattributable. This case arises only for a wake already in scope
|
||||
under `AC-OFFICE-ASSIGN-RATE-001.1`: a wake out of scope by actor type is
|
||||
admitted there and never reaches this criterion, so a wake carrying neither an
|
||||
actor type nor a task is not counted here. A task cannot be determined when
|
||||
the identifier is absent, null, empty, or present but not a string; those
|
||||
shapes are treated identically.
|
||||
That admission shall be logged at warning level carrying the reason, the
|
||||
allowance, and the acting agent identifier when the wake carries one; it shall
|
||||
not carry a task identifier, which is by definition the value that could not
|
||||
be determined.
|
||||
- **AC-OFFICE-ASSIGN-RATE-002.4:** The gate shall hold no state of its own: the
|
||||
count shall be derived from already-persisted run rows. A restart, a crash
|
||||
between evaluation and insert, or a deleted run row therefore changes the
|
||||
observed count without leaving the allowance inconsistent; no reconciliation
|
||||
step is required.
|
||||
- **AC-OFFICE-ASSIGN-RATE-002.5:** Re-delivering one assignment occurrence that
|
||||
carries a dedup key shall not consume the allowance twice: redelivery is
|
||||
decided by the dedup identity, which `AC-OFFICE-ASSIGN-RATE-001.5` places
|
||||
before this gate. A keyless wake has no such identity, so re-delivering one
|
||||
does consume the allowance again — intended, because
|
||||
`AC-OFFICE-ASSIGN-RATE-001.8` bounds that path precisely where nothing
|
||||
upstream suppresses it. A test for this criterion shall use a keyed wake.
|
||||
- **AC-OFFICE-ASSIGN-RATE-002.6:** `N` shall be 5 and `W` shall be 10 minutes,
|
||||
fixed values of this capability rather than operator-configurable, stated in
|
||||
one place the implementation and its tests share. `N` shall be at least 1; 0,
|
||||
which would refuse every agent-initiated assignment wake, is not permitted.
|
||||
|
||||
### REQ-OFFICE-ASSIGN-RATE-003: A refusal is never silent
|
||||
|
||||
**Intent:** A wake that vanishes without trace is indistinguishable from one
|
||||
never produced, and an operator debugging a stalled task cannot tell. The
|
||||
sibling deduplication capability set the floor at operator telemetry.
|
||||
|
||||
**User story:** As an Office operator, I want a refused assignment wake to be
|
||||
visible in telemetry and logs, so that an agent not waking up is diagnosable
|
||||
instead of mysterious.
|
||||
|
||||
#### Acceptance criteria
|
||||
|
||||
- **AC-OFFICE-ASSIGN-RATE-003.1:** Every refusal and every degraded admission
|
||||
shall increment a counter labelled by reason, drawn from a closed set of
|
||||
three values: allowance exhausted, count read failed, and task
|
||||
unattributed.
|
||||
- **AC-OFFICE-ASSIGN-RATE-003.2:** No task, agent, workspace, session or run
|
||||
identifier shall ever be a counter label, so cardinality cannot grow with
|
||||
traffic.
|
||||
- **AC-OFFICE-ASSIGN-RATE-003.3:** A refusal shall be logged at warning level
|
||||
carrying the task identifier, the assignee agent profile identifier, the
|
||||
acting agent identifier, the observed count and the allowance.
|
||||
- **AC-OFFICE-ASSIGN-RATE-003.4:** An admitted wake this capability did not
|
||||
alter shall move none of these counters; ordinary admitted volume is already
|
||||
visible on the runs queue.
|
||||
- **AC-OFFICE-ASSIGN-RATE-003.5:** The outcome reported to the producer for a
|
||||
refused wake shall be distinguishable from queued, coalesced and deduplicated
|
||||
outcomes, and shall not be the outcome meaning no enqueue was attempted. A
|
||||
caller shall not infer a rate-limit refusal from an absence.
|
||||
- **AC-OFFICE-ASSIGN-RATE-003.6:** The counter shall be published on the same
|
||||
operator telemetry surface as the run-deduplication counters.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **The five-second coalescing window, and which reasons coalesce.** Held
|
||||
unchanged by repeated human decision on the run-dedup-generation card. This
|
||||
capability is a separate guard placed after it; a coalesced wake neither
|
||||
consumes the allowance nor can be refused by it.
|
||||
- **The 24-hour recent-duplicate lookback and the durable unique index.**
|
||||
Unchanged. Both decide *identity*, which this capability never touches.
|
||||
- **What makes a wake occurrence distinct.** `REQ-OFFICE-RUN-DEDUP-001` through
|
||||
`-004` own dedup key identity. This capability refuses wakes whose identity is
|
||||
genuinely new; it never redefines identity.
|
||||
- **A per-acting-agent allowance.** [[agent-budget-governance]] and the vendor
|
||||
corpus both argue for a per-principal cap. Not adopted: it would throttle a
|
||||
coordinator legitimately assigning many *different* tasks in a burst, the
|
||||
behaviour Office's coordinator role exists to perform. Sizing one needs a
|
||||
legitimate-coordinator rate measured from real workspace data, which does not
|
||||
exist yet.
|
||||
- **A workspace-aggregate launch budget, and a causation depth cap.** Card
|
||||
`7dbc0b94` owns work-in-progress limits, depth caps and self-trigger
|
||||
suppression. This is the per-task half only: across `T` tasks an agent retains
|
||||
`5T` admitted wakes per window, and only an aggregate budget bounds that. The
|
||||
two compose; neither substitutes for the other.
|
||||
- **Reclassifying an agent-initiated `task_assigned` run as unattended.** The
|
||||
adjacent fix, deliberately not taken here: `AC-OFFICE-BUDGET-007.1` makes
|
||||
provenance a total function of `reason` and `-007.2` pins the attended
|
||||
allowlist to a literal set including `task_assigned`, so an actor-aware
|
||||
classifier changes `budget-run-provenance.md`'s contract and belongs to its
|
||||
owner. It is incomplete alone in any case — a daily ceiling permits a full
|
||||
day's allowance to burn in minutes and bounds no queue rows, session launches
|
||||
or scheduler churn.
|
||||
- **Rate-limiting the assignment mutation itself.** The assignee mutator keeps
|
||||
accepting every call its permission check allows and the assignment generation
|
||||
keeps bumping; only the wake is refused. Refusing the write would make the
|
||||
board un-editable by agents.
|
||||
- **A persisted record of a refused wake.** Office automations record a
|
||||
`skipped` row, the better surface in the abstract. Not adopted: `runs` has no
|
||||
such status and adding one to queue infrastructure shared by every workflow
|
||||
style is larger than this guard warrants. `REQ-OFFICE-RUN-DEDUP-004` set the
|
||||
floor for a high-frequency suppression outcome at operator telemetry and logs,
|
||||
which `REQ-OFFICE-ASSIGN-RATE-003` meets.
|
||||
- **Notifying anybody.** No inbox item, approval, activity entry, task-timeline
|
||||
row or escalation. The counter added here is the evidence needed to size such
|
||||
a surface later.
|
||||
- **Operator tunability of `N` and `W`.** They ship as fixed values, like the
|
||||
coalescing window. A settings surface, its API and its frontend are separate
|
||||
work.
|
||||
- **A dedicated runtime feature toggle.** Office is already off in the prod
|
||||
profile, so this ships behind the flag gating the surface it guards. A second
|
||||
toggle inside a disabled feature buys no rollout safety.
|
||||
- **Any frontend.** No screen renders the allowance, the counters, or a
|
||||
refusal.
|
||||
- **Every other producer of a `task_assigned` wake.** Besides the in-scope
|
||||
mutation path the repository has four: the task-lifecycle event subscriber,
|
||||
onboarding, the unstarted-task recovery sweep, and the orchestrator's workflow
|
||||
auto-start action. Two reasons, not convenience. None carries
|
||||
an actor, so none is agent-initiated under `AC-OFFICE-ASSIGN-RATE-001.1`,
|
||||
which `AC-OFFICE-ASSIGN-RATE-001.12` makes a standing invariant rather than an
|
||||
observation. And a *per-task* allowance structurally cannot bind them: each
|
||||
wakes a task just created, onboarded or swept, where a fresh or long-idle task
|
||||
brings a fresh allowance, and the subscriber's update path is a redelivery
|
||||
already owned by the dedup identity. Bounding them needs the aggregate scope
|
||||
named above.
|
||||
- **Existing run rows.** No migration, backfill or reinterpretation. Already
|
||||
persisted rows count toward a window exactly as new ones do.
|
||||
|
||||
## Accepted consequence
|
||||
|
||||
When the allowance is exhausted and the refused wake targets a *different* agent
|
||||
from the current assignee, that agent is left assigned and not woken until a
|
||||
later occurrence or a user-initiated assignment wakes it. Accepted, not
|
||||
mitigated: it needs five agent-driven reassignments of one task inside ten
|
||||
minutes; user-initiated assignment is never limited, so an operator can recover
|
||||
the task by hand; and the refusal is counted and logged under
|
||||
`REQ-OFFICE-ASSIGN-RATE-003`. Exempting cross-agent
|
||||
reassignment would reopen the exploit: alternating assignment between two agents
|
||||
was already unbounded.
|
||||
|
||||
A coalescing merge can also relabel the actor type the count reads, because
|
||||
`CoalesceRun` does not match on actor type and overwrites the surviving row's
|
||||
payload. Per mixed-actor merge the observed count may run one high or one low.
|
||||
Accepted: both available fixes are excluded, actor-aware coalescing by
|
||||
`## Out of scope` and state of this gate's own by
|
||||
`AC-OFFICE-ASSIGN-RATE-002.4`. The mechanism is in the design's
|
||||
`## Counting without new state`.
|
||||
|
||||
## System design
|
||||
|
||||
[Assignment wake rate limit](../system-design/assignment-wake-rate-limit-01.md).
|
||||
@@ -0,0 +1,472 @@
|
||||
---
|
||||
status: draft
|
||||
system: office
|
||||
requirements:
|
||||
- REQ-OFFICE-ASSIGN-RATE-001
|
||||
- REQ-OFFICE-ASSIGN-RATE-002
|
||||
- REQ-OFFICE-ASSIGN-RATE-003
|
||||
---
|
||||
|
||||
# Office: Assignment Wake Rate Limit System Design
|
||||
|
||||
## Purpose and boundaries
|
||||
|
||||
This design places one additional admission gate in the Office run queue: an
|
||||
allowance on agent-initiated `task_assigned` wakes, scoped per task and to one
|
||||
producer — the task-mutation reactivity path, per
|
||||
`AC-OFFICE-ASSIGN-RATE-001.12`. It adds no table, no column, no migration, and
|
||||
no background job. The count is derived from rows the queue already writes.
|
||||
|
||||
It does not touch dedup key identity
|
||||
(`docs/specs/office/requirements/run-dedup-generation.md`), the five-second
|
||||
coalescing predicate, the 24-hour recent-duplicate lookback, the attended and
|
||||
unattended provenance classifier, or the pre-launch budget gates.
|
||||
|
||||
## Requirement mapping
|
||||
|
||||
| Requirement | Mechanism |
|
||||
|---|---|
|
||||
| `REQ-OFFICE-ASSIGN-RATE-001` | A gate in `SchedulerService.queueRun`, evaluated after the idempotency and coalescing checks and before `CreateRun`, backed by a task-scoped window count over `runs`. Its position fixes the producer scope: see below. |
|
||||
| `REQ-OFFICE-ASSIGN-RATE-002` | The gate is a pure read plus the insert it guards; it fails open, admits under a race, and derives its count from persisted rows only. |
|
||||
| `REQ-OFFICE-ASSIGN-RATE-003` | A new `expvar.Map` beside the run-dedup counters, plus a Warn log at the decision site, plus a new `QueueOutcome` value. |
|
||||
|
||||
## Where the gate sits
|
||||
|
||||
`apps/backend/internal/office/scheduler/run.go`'s `queueRun` is where the
|
||||
mutation producer's wakes converge — `reactToAssigneeChange` reaches it through
|
||||
`QueueRunCtx` — and it already owns the other two admission decisions.
|
||||
|
||||
It is **not** a choke point every Office producer reaches, and nothing here may
|
||||
assume it is. The repository has five `task_assigned` producers, listed by
|
||||
inventory rather than inferred from the two this capability happens to discuss:
|
||||
|
||||
| Producer | Route to an insert | Actor on its payload |
|
||||
|---|---|---|
|
||||
| `scheduler/reactivity.go`'s `reactToAssigneeChange` — **in scope** | `QueueRunCtx` → `SchedulerService.queueRun` | `agent` or `user`, from the mutation |
|
||||
| `service/event_subscribers.go`'s `queueTaskAssignedRun` | `Service.QueueRun` | none |
|
||||
| `office/onboarding/service.go` (its own comment calls it "a third `task_assigned` producer") | `Service.QueueRun` | none |
|
||||
| `service/scheduler_recovery.go`'s `recoverUnstartedTasks` sweep | `Service.QueueRun` | none |
|
||||
| `orchestrator/event_handlers_workflow.go`'s `queueOfficeAutoStartRun`, the `auto_start_agent` step action (`officeAutoStartRunReason`) | `engineRunQueue.QueueRun` → `runsservice.QueueRun` | none |
|
||||
|
||||
`Service.QueueRun` (`service/run.go`) delegates to `runsservice.QueueRun` when a
|
||||
runs service is wired and otherwise re-implements the idempotency, coalescing
|
||||
and insert steps inline in `queueRunInline`; neither route, and neither does the
|
||||
orchestrator's, passes through `SchedulerService.queueRun`. So a gate placed
|
||||
here binds the mutation producer and only the mutation producer.
|
||||
|
||||
Two independent facts agree today — one producer reaches this function, and one
|
||||
producer puts an actor on the payload — and `AC-OFFICE-ASSIGN-RATE-001.12`
|
||||
deliberately makes the **second** the contract. Call position is an
|
||||
implementation fact a refactor can move without anyone noticing; payload shape
|
||||
is checkable from the payload alone, and that criterion's enumeration test is
|
||||
what keeps the two from drifting apart silently. A future producer that reaches
|
||||
`queueRun` without an actor is out of scope by the same rule that covers the
|
||||
four below, and one that sets `actor_type: agent` violates a written invariant
|
||||
instead of quietly joining the gate.
|
||||
|
||||
Excluding the other four is a scope decision, not an accepted hole: a *per-task*
|
||||
allowance could not bind any of them. The subscriber's live path is
|
||||
`handleTaskCreated`, which fires once per task creation, where a fresh task
|
||||
identifier always brings a fresh allowance; its `handleTaskUpdated` is
|
||||
documented in its own doc comment as a redelivery and defensive path, and today
|
||||
reaches no wake at all, because neither `task.updated` publisher puts
|
||||
`assignee_agent_profile_id` in the payload and `queueTaskAssignedRun` returns
|
||||
early without one. Onboarding wakes a task it has just created. The recovery
|
||||
sweep wakes a task that was never started, and is capped at `maxRecoveryPerTick`
|
||||
rows per tick anyway. The orchestrator's is workflow-driven: it fires on a step
|
||||
transition running `auto_start_agent`, and its one reference to the participant
|
||||
slate is how it *resolves* which agent to wake, not a second producer. None of
|
||||
the four is an agent minting fresh occurrences against one task in a loop, which
|
||||
is the shape this capability bounds.
|
||||
|
||||
The gate belongs in `queueRun` rather than in `reactToAssigneeChange` or in
|
||||
`DashboardService.SetTaskAssigneeAsAgent`:
|
||||
|
||||
- In `reactToAssigneeChange` it would bind the same single producer, but
|
||||
*upstream* of the recent-duplicate lookup and the coalescing attempt. A wake
|
||||
later suppressed or merged would already have consumed the allowance,
|
||||
contradicting `AC-OFFICE-ASSIGN-RATE-001.5` and `.4`. Position, not producer
|
||||
coverage, is what rules it out.
|
||||
- In `SetTaskAssigneeAsAgent` it would have to refuse the *mutation*, which
|
||||
`REQ-OFFICE-ASSIGN-RATE-001.7` and the `## Out of scope` entry on
|
||||
rate-limiting the write both forbid.
|
||||
|
||||
The resulting order inside `queueRun`, satisfying
|
||||
`AC-OFFICE-ASSIGN-RATE-001.5`:
|
||||
|
||||
1. `guardAgentStatus`
|
||||
2. `checkPauseGate`
|
||||
3. `CheckIdempotencyKey` — windowed dedup (unchanged)
|
||||
4. `CoalesceRun` — five-second merge (unchanged)
|
||||
5. **the assignment allowance** (new)
|
||||
6. `CreateRun`
|
||||
|
||||
Placing it at 5 is what makes "admitted" mean exactly "a row was inserted".
|
||||
A wake suppressed at 3 or merged at 4 never reaches the gate, so it neither
|
||||
consumes the allowance nor can be refused by it — `AC-OFFICE-ASSIGN-RATE-001.5`
|
||||
and the coalescing exclusion in `## Out of scope` are the same fact.
|
||||
|
||||
## The predicate
|
||||
|
||||
The gate applies only when all three hold, per
|
||||
`AC-OFFICE-ASSIGN-RATE-001.1`, `.2` and `.9`, and in this order:
|
||||
|
||||
1. `reason == RunReasonTaskAssigned`
|
||||
2. the wake's `actor_type` is `agent`
|
||||
3. the wake names a task
|
||||
|
||||
The order is contract, not style. A wake failing 2 is out of scope and admitted
|
||||
silently; a wake that passes 2 but fails 3 is the counted unattributed case of
|
||||
`AC-OFFICE-ASSIGN-RATE-002.3`. Evaluating 3 first would count a wake that
|
||||
carries neither field against the unattributed reason, which is the one the
|
||||
event-subscriber producer's payload would land on if it ever reached here.
|
||||
|
||||
`actor_type` and `task_id` are already carried on `RunContext` and already
|
||||
serialised into the payload (`run.go`'s `RunContext.ActorType`, `.TaskID`).
|
||||
`queueRun` receives the encoded payload rather than the struct, so it reads both
|
||||
back out of it **in Go**, with one `json.Unmarshal` of the incoming payload —
|
||||
exactly what `CoalesceRun` does through `taskIDFromPayload` before it builds any
|
||||
SQL.
|
||||
|
||||
The two reads are not interchangeable and the build must not collapse them.
|
||||
`dialect.JSONExtract(driver, col, path)` returns a SQL fragment over a *column*,
|
||||
so it applies to the **stored** rows the window count scans
|
||||
(`## Counting without new state`) and cannot apply to the **incoming** wake,
|
||||
which is not a row yet. Incoming: Go. Stored: SQL.
|
||||
|
||||
Parsing the incoming payload has three failure shapes, which `CoalesceRun`
|
||||
deliberately keeps apart. The evaluation order above decides where each one
|
||||
lands, and the mapping is deliberately not symmetric:
|
||||
|
||||
| Incoming payload | Where it lands |
|
||||
|---|---|
|
||||
| `task_id` a non-empty JSON string | step 3 passes, the gate applies |
|
||||
| `task_id` absent, JSON null, or `""` | step 3 fails: `AC-OFFICE-ASSIGN-RATE-002.3`, admit and count unattributed |
|
||||
| `task_id` present but not a string | step 3 fails: `AC-OFFICE-ASSIGN-RATE-002.3`, admit and count unattributed |
|
||||
| the payload does not parse at all | no readable `actor_type` either, so it fails **step 2** first: out of scope under `AC-OFFICE-ASSIGN-RATE-001.1`, admitted, no counter moved |
|
||||
|
||||
The last row is the one to be careful about, and it is part of why the order is
|
||||
contract. A payload this gate cannot parse is not an unattributable *in-scope*
|
||||
wake, it is a wake never shown to be in scope at all, and
|
||||
`AC-OFFICE-ASSIGN-RATE-002.3` says in terms that a wake out of scope by actor
|
||||
type never reaches it. Counting it as unattributed would count wakes never shown
|
||||
to be agent-initiated, which `AC-OFFICE-ASSIGN-RATE-001.1` exempts by
|
||||
definition.
|
||||
|
||||
The two task-shaped failures do share one bucket, unlike `CoalesceRun`, which
|
||||
must hold `invalidTaskID` apart from taskless because it *merges* rows and a
|
||||
wrong merge overwrites another launch's payload. This gate only counts and both
|
||||
shapes admit, so splitting them would buy nothing and would add a counter value
|
||||
`AC-OFFICE-ASSIGN-RATE-003.1`'s closed set does not have.
|
||||
|
||||
Step 2 needs no equivalent table: an `actor_type` that is absent, not a string,
|
||||
or any string other than `agent` simply is not `agent`, which
|
||||
`AC-OFFICE-ASSIGN-RATE-001.1` already covers as "any other actor type ... or no
|
||||
actor type at all" — out of scope, admitted, no counter moved.
|
||||
|
||||
`actor_type` can be *absent* rather than merely different: the out-of-scope
|
||||
event-subscriber producer builds its payload as `{"task_id": …}` and carries no
|
||||
actor at all. Absent therefore has to classify the same way `"user"` does — out
|
||||
of scope, admitted, no allowance consumed, no counter moved, per
|
||||
`AC-OFFICE-ASSIGN-RATE-001.1` — and not as a degraded case with its own reason
|
||||
value. A counter that increments on every Office task creation is not a
|
||||
regression signal, it is background noise that hides one. What guards against
|
||||
the in-scope producer actually losing the field is
|
||||
`AC-OFFICE-ASSIGN-RATE-001.13`'s payload contract, pinned by a test, rather than
|
||||
a runtime counter that could not tell that loss apart from ordinary traffic.
|
||||
|
||||
Because `actor_type` is `"user"` whenever the acting agent identifier is empty
|
||||
(`runReactivityForAssigneeChange`'s `userSentinel` default), the
|
||||
internal and administrative caller that passes `callerAgentID == ""` classifies
|
||||
as `user` and is never limited. `AC-OFFICE-ASSIGN-RATE-001.1` states that as a
|
||||
contract rather than leaving it as a consequence of the sentinel's value.
|
||||
|
||||
## Counting without new state
|
||||
|
||||
`AC-OFFICE-ASSIGN-RATE-002.4` requires the gate to hold no state of its own.
|
||||
One repository read satisfies it:
|
||||
|
||||
```text
|
||||
SELECT COUNT(*) FROM runs
|
||||
WHERE reason = 'task_assigned'
|
||||
AND <json_extract payload task_id> = :taskID
|
||||
AND <json_extract payload actor_type> = 'agent'
|
||||
AND requested_at > :windowStart
|
||||
AND requested_at <= :evaluationInstant
|
||||
```
|
||||
|
||||
`windowStart` is `evaluationInstant - W`. The lower comparison is strictly
|
||||
greater-than and the upper comparison is inclusive, which is
|
||||
`AC-OFFICE-ASSIGN-RATE-001.11`'s half-open interval. `requested_at` is written
|
||||
by `queueRun` itself on every row, so it is the persisted request instant the
|
||||
window is measured against.
|
||||
|
||||
No `ORDER BY` appears here or anywhere else in this capability:
|
||||
`AC-OFFICE-ASSIGN-RATE-001.10` makes the decision explicitly order-independent,
|
||||
which is what a count needs and all it needs. That is not a simplification of a
|
||||
richer rule — the richer rule was not implementable. The assignment generation
|
||||
is not orderable from a run row: `runs` has no `assignment_generation` column,
|
||||
`RunContext` carries no such field, and the generation survives only inside the
|
||||
string `dedupkeys.AssignmentKey` formats, which a keyless wake does not have at
|
||||
all. Adding a payload field to carry it would *not* break coalescing —
|
||||
`CoalesceRun` matches on `agent_profile_id`, `reason`, status, the window, the
|
||||
comment-key exclusion and `task_id` alone, then overwrites the row's payload
|
||||
rather than comparing payloads for equality — but the surviving coalesced row
|
||||
would then carry the *last* merged wake's generation instead of the first, which
|
||||
makes a payload-borne generation an unreliable ordering key in precisely the
|
||||
case an ordering would be asked for.
|
||||
|
||||
Two portability notes carried over from `CoalesceRun`, which solved the same
|
||||
problem in the same table:
|
||||
|
||||
- Use `dialect.JSONExtract` / `dialect.JSONTypeIsString`, never a literal
|
||||
`json_extract`. The runs repository supports SQLite and PostgreSQL, and
|
||||
`runs_queue_postgres_test.go` exists precisely because a hand-written
|
||||
`json_extract` diverges between them.
|
||||
- Guard the string type. `CoalesceRun` learned that `{"task_id":42}` textually
|
||||
matches an incoming `{"task_id":"42"}`; the same guard applies here or a
|
||||
numeric `task_id` in an old payload silently joins another task's allowance.
|
||||
|
||||
Deriving the count from `runs` has a deliberate consequence worth stating: the
|
||||
history-retention capability deleting old run rows lowers an observed count.
|
||||
That cannot raise the allowance above `N` inside a live window, because
|
||||
retention operates on rows far older than ten minutes.
|
||||
|
||||
### The actor label on a stored row is not stable
|
||||
|
||||
The count reads `actor_type` off the **stored** payload, and `CoalesceRun`
|
||||
rewrites exactly that. It matches on `agent_profile_id`, `reason`, status, the
|
||||
window, the comment-key exclusion and `task_id` — never on `actor_type` — and
|
||||
then writes the incoming wake's payload over the surviving row while leaving
|
||||
that row's `requested_at` untouched. A queued row therefore carries the **last**
|
||||
merged wake's actor and the **first** wake's request instant.
|
||||
|
||||
Reachable, not theoretical. `runReactivityForAssigneeChange` stamps `agent` or
|
||||
`user` off `callerAgentID`, and one endpoint — `dashboard/handler.go`'s assignee
|
||||
route — produces both, so two wakes for one (agent, task) inside five seconds
|
||||
can carry different actors and merge. Per mixed-actor merge the window count
|
||||
runs one high (an agent wake relabels a queued user row, so a row
|
||||
`AC-OFFICE-ASSIGN-RATE-001.1` exempts starts counting) or one low (a user wake
|
||||
relabels a queued agent row, returning a slot).
|
||||
|
||||
Neither direction is an agent-only capability: the loosening one needs a human to
|
||||
assign the same task within five seconds of the agent's own wake, and the
|
||||
tightening one is conservative and bounded at one row per merge.
|
||||
|
||||
Accepted rather than fixed, and recorded in the requirement's
|
||||
`## Accepted consequence` so the build meets it as a stated consequence rather
|
||||
than as a surprise. Both available fixes are closed: matching `actor_type` inside
|
||||
`CoalesceRun` is the coalescing change the requirement's `## Out of scope` holds
|
||||
unchanged by repeated human decision, and a column or cache of this gate's own is
|
||||
what `AC-OFFICE-ASSIGN-RATE-002.4` forbids. What the build owes here is a test,
|
||||
not a mechanism — `## Test plan` carries the mixed-actor coalesce in both
|
||||
directions, so the behaviour is pinned at its measured value instead of assumed
|
||||
away.
|
||||
|
||||
## Constants
|
||||
|
||||
`N = 5` and `W = 10 * time.Minute`, declared once beside
|
||||
`CoalesceWindowSeconds` and `IdempotencyWindowHours` in
|
||||
`scheduler/run.go`, and referenced by the tests rather than restated —
|
||||
`AC-OFFICE-ASSIGN-RATE-002.6`. The duplicate `RunReason*` constant blocks in
|
||||
`office/scheduler` and `office/service` are a known wart that
|
||||
`run-dedup-generation.md` already excluded from its scope; do not add a second
|
||||
copy of these two.
|
||||
|
||||
## Failure direction, and why it differs from its neighbour
|
||||
|
||||
Three gates in this one function now fail three different ways, so the choice
|
||||
has to be explicit rather than inferred:
|
||||
|
||||
| Gate | On its own read failing | Why |
|
||||
|---|---|---|
|
||||
| `checkPauseGate` | **closed** (`ErrPauseGateUnavailable`) | a kill switch that fails open is not a kill switch |
|
||||
| `CheckIdempotencyKey` | **error** returned to the caller | a duplicate admitted twice is a correctness defect |
|
||||
| assignment allowance | **open**, counted and logged | a closed failure drops a legitimate wake, trading a bounded cost problem for an unbounded liveness one |
|
||||
|
||||
`AC-OFFICE-ASSIGN-RATE-002.2` fixes the third. The in-repo precedent is
|
||||
`checkIdleSkip` (`service/scheduler_integration.go`), which fails open on a
|
||||
database error and is documented as doing so in
|
||||
`apps/backend/internal/office/AGENTS.md`. The degraded admission is counted
|
||||
under its own reason so that a gate silently admitting everything because its
|
||||
read is broken is visible, rather than indistinguishable from a workspace that
|
||||
simply never exhausts its allowance.
|
||||
|
||||
`AC-OFFICE-ASSIGN-RATE-002.3`'s unattributed case is the same reasoning applied
|
||||
to a payload-shape change: if `task_id` stops resolving, every wake becomes
|
||||
unattributable and the gate stops binding. Counting that case separately turns
|
||||
a silent regression into a visible one.
|
||||
|
||||
## Concurrency
|
||||
|
||||
The count read and the subsequent `CreateRun` are not in one transaction, and
|
||||
`AC-OFFICE-ASSIGN-RATE-002.1` deliberately does not require them to be. Two
|
||||
concurrent agent-initiated assignment wakes for one task can both read `N-1`
|
||||
and both insert, yielding `N+1` admitted in the window.
|
||||
|
||||
That is accepted rather than fixed. Making it exact would need either a
|
||||
serialising lock on the task or a conditional insert, both of which put a
|
||||
contended write on the hot path of a queue shared by every workflow style, to
|
||||
buy exactness in a bound whose purpose is to turn "unbounded" into "about five
|
||||
per ten minutes". The asymmetry in `AC-OFFICE-ASSIGN-RATE-002.1` is the part
|
||||
that matters and is testable: over-admission is bounded by the number of
|
||||
concurrent evaluations, and under-admission — refusing before `N` are admitted
|
||||
— is never permitted.
|
||||
|
||||
## Observability
|
||||
|
||||
A new `expvar.Map`, published beside `office_run_dedup_total` and
|
||||
`office_run_dedup_keyless_total` in `internal/runs/service/metrics_vars.go`,
|
||||
counting refusals and degraded admissions by a closed three-value `reason`
|
||||
label: allowance exhausted, count read failed, task unattributed. Labels follow
|
||||
the existing `metricLabel("reason", …)` `k=v;k=v` convention.
|
||||
|
||||
`AC-OFFICE-ASSIGN-RATE-003.2`'s no-identifier rule is what keeps the map
|
||||
bounded, matching the reasoning behind `metricReasons`' `custom` fallback: a
|
||||
process-global `expvar.Map` must not be able to grow once per distinct input.
|
||||
`AC-OFFICE-ASSIGN-RATE-003.4` keeps admitted volume off these counters
|
||||
entirely, so the map's absolute values read as "how often did the guard act",
|
||||
not as traffic.
|
||||
|
||||
The Warn log's fields are fixed by the criteria rather than left to the
|
||||
implementation: `AC-OFFICE-ASSIGN-RATE-003.3` for a refusal,
|
||||
`AC-OFFICE-ASSIGN-RATE-002.2` for a failed count read (no observed count — the
|
||||
read did not produce one), and `AC-OFFICE-ASSIGN-RATE-002.3` for an
|
||||
unattributable task (no task identifier, for the same reason). The
|
||||
no-identifier rule above governs counter *labels* only; these are log fields.
|
||||
|
||||
The reporting helper belongs beside `ReportWindowedDedup` and
|
||||
`ReportDurableDedup` in `internal/runs/service/dedup.go` and follows their
|
||||
shape: increment, log, return an outcome.
|
||||
|
||||
## The outcome value
|
||||
|
||||
`AC-OFFICE-ASSIGN-RATE-003.5` needs an outcome distinct from
|
||||
`QueueOutcomeQueued`, `QueueOutcomeCoalesced`, `QueueOutcomeDeduped` and
|
||||
`QueueOutcomeNone`. Reusing `QueueOutcomeNone` would conflate a deliberate
|
||||
refusal with "no enqueue was attempted, or the attempt errored", which is what
|
||||
its own doc comment says it means.
|
||||
|
||||
So a new value — and one trap: `QueueOutcome` and `QueueOutcomeNone` are
|
||||
declared **twice**, in `internal/runs/service` and in
|
||||
`internal/workflow/engine/adapters.go`, with a comment on each stating the two
|
||||
declarations must match. A new member has to be added to both or the engine
|
||||
adapter cannot report it.
|
||||
|
||||
Callers treat the new value the way they treat `QueueOutcomeCoalesced`: not an
|
||||
error, and not a queued run. `ApplyTaskMutation`'s `queue` closure already
|
||||
appends to `res.Runs` only on `QueueOutcomeQueued`, so a refused wake correctly
|
||||
does not appear in `ApplyTaskMutationResult.Runs` with no change to that
|
||||
branch.
|
||||
|
||||
## What the mutation still does
|
||||
|
||||
`AC-OFFICE-ASSIGN-RATE-001.7` is mostly a statement that nothing needs to
|
||||
change: the gate lives downstream of the mutation, and
|
||||
`runReactivityForAssigneeChange` already ignores a non-queued outcome. So a
|
||||
refusal leaves intact the `UpdateTaskAssignee` write and its
|
||||
`assignment_generation` bump, the `publishTaskUpdated` broadcast, the
|
||||
`res.InterruptSessionID` hard-cancel of a displaced assignee's session, and the
|
||||
prev-assignee office session row transition. The criterion exists so a future
|
||||
refactor that moves the gate upstream has to break a written contract rather
|
||||
than a coincidence.
|
||||
|
||||
It does **not** include a comment or mention wake. `ApplyTaskMutation` calls
|
||||
`reactToComment` only when the mutation carries a comment, and the assignee
|
||||
mutator builds its `TaskReactivityChange` without one, so no such wake exists on
|
||||
this path to preserve. `AC-OFFICE-ASSIGN-RATE-001.7` says so explicitly, because
|
||||
a regression test that asserted one would fail against correct code.
|
||||
|
||||
## Prior art considered
|
||||
|
||||
**Searched:** `search_saas_docs` (`category: "ai_sdlc"`) on runaway-loop and
|
||||
self-trigger rate limiting, then the LangChain deepagents and Cloudflare Agents
|
||||
SDK documents. Each item is a vendor claim, not evidence the approach works.
|
||||
|
||||
- **Cloudflare Agents SDK** passes a `source` to `onStateChanged` and documents
|
||||
the best practice as *"Avoid infinite loops: be careful not to trigger state
|
||||
updates in response to your own updates"*. Same insight: gate on *who caused
|
||||
it*. **We cannot adopt its conclusion**, which is to suppress self-caused
|
||||
reactions outright — `AC-OFFICE-RUN-DEDUP-001.3` requires a same-agent repeat
|
||||
assignment to wake the agent. We keep the provenance distinction and bound the
|
||||
rate instead of suppressing the class.
|
||||
- **LangChain deepagents** classifies *"Excessive calls (runaway loops)"* as
|
||||
system-handled, with the remedy *"cap model and tool calls per run"* — a
|
||||
**count cap** over a scope, not a minimum interval. We follow that shape,
|
||||
because a cooldown long enough to matter also breaks a legitimate
|
||||
assign/hand-back/reassign burst.
|
||||
- **Concurrency caps are near-universal** and per-principal (Multica, Coder,
|
||||
Kiro). All bound *in-flight* work; none bounds the *arrival rate* of
|
||||
self-caused triggers, the axis here.
|
||||
|
||||
**In-repo precedents.** The requirement's `## Prior art` names three; their call
|
||||
sites live here. `internal/automation` caps a firing with `MaxConcurrentRuns` and
|
||||
records a `RunStatusSkipped` run rather than retrying — the better surface in the
|
||||
abstract, excluded here only because `runs` has no such status. The workspace
|
||||
pause gate (`checkPauseGateForAgent`) is the established refused-admission
|
||||
pattern for `runs`: typed error, no row written. `checkIdleSkip`
|
||||
(`service/scheduler_integration.go`) fails open on a database error, the
|
||||
precedent `AC-OFFICE-ASSIGN-RATE-002.2` follows and the deliberate opposite of
|
||||
the pause gate's choice; the table in `## Failure direction, and why it differs
|
||||
from its neighbour` is that contrast.
|
||||
|
||||
## Test plan
|
||||
|
||||
Backend, table-driven, in `apps/backend/internal/office/scheduler`:
|
||||
|
||||
- `N` agent-actor assignment wakes for one task admit; the `N+1`th refuses,
|
||||
inserts no row, and returns the new outcome (`001.2`, `001.6`).
|
||||
- A `user`-actor wake admits after the allowance is exhausted, and an
|
||||
empty-`callerAgentID` call classifies as `user` (`001.1`).
|
||||
- A wake carrying *no* `actor_type` admits after the allowance is exhausted,
|
||||
consumes no allowance, and moves no counter (`001.1`).
|
||||
- Alternating assignment between two agents shares one allowance (`001.3`).
|
||||
- A refused wake does not consume the allowance: after `N` admits and several
|
||||
refusals, exactly `N` wakes are admitted in the next window (`001.4`).
|
||||
- A wake inside the coalescing window merges and does not consume the
|
||||
allowance; a wake suppressed by the idempotency lookup likewise (`001.5`).
|
||||
- A keyless wake from the in-scope producer is still bounded (`001.8`).
|
||||
- The in-scope producer's payload carries `actor_type` `agent` for an
|
||||
agent-actor mutation (`001.13`). This is the test that makes losing the field
|
||||
a failure rather than a silent unbinding of the gate.
|
||||
- A wake from the event-subscriber producer is unaffected by an exhausted
|
||||
allowance, fixing the producer scope as a test rather than a comment
|
||||
(`001.12`).
|
||||
- Two admitted wakes sharing one persisted request instant both count, and the
|
||||
decision at `N` is the same whichever of them was enqueued first — the gate
|
||||
reads a count, not a sequence (`001.10`).
|
||||
- An unassignment neither refuses nor consumes (`001.9`).
|
||||
- A row whose `requested_at` is exactly `W` old does not count (`001.11`).
|
||||
- A failing count read admits and increments the degraded counter, and logs
|
||||
the fields `002.2` names without an observed count (`002.2`); an
|
||||
unattributable task likewise, without a task identifier (`002.3`).
|
||||
- A wake carrying neither an `actor_type` nor a task moves no counter at all,
|
||||
pinning the predicate's evaluation order (`001.1`, `002.3`).
|
||||
- A mixed-actor coalesce in both directions: an agent-actor wake merging into a
|
||||
queued user-actor row, and a user-actor wake merging into a queued agent-actor
|
||||
row. Assert the count the gate then observes — the accepted off-by-one of the
|
||||
requirement's `## Accepted consequence`, not an assumed exact count — so a
|
||||
later change to `CoalesceRun` breaks a test rather than silently moving the
|
||||
allowance (`001.4`, `001.5`).
|
||||
- Re-delivering a *keyed* wake does not consume the allowance twice;
|
||||
re-delivering a *keyless* one does, which is what `002.5` states and what
|
||||
`001.8` bounds. Two cases, not one.
|
||||
- The `task_assigned` producers are enumerated and the actor type each one emits
|
||||
is asserted, so a new producer setting `actor_type` `agent` fails this test
|
||||
instead of joining the gate unannounced (`001.12`).
|
||||
- An absent, null or empty `task_id` and a present-but-non-string `task_id` each
|
||||
take the unattributed path; an unparseable payload instead leaves scope at the
|
||||
actor step, admitted with no counter moved, so the two are distinguishable in
|
||||
the counters (`002.3`, `001.1`).
|
||||
- Refusal moves the counter with the expected `reason` label and no identifier
|
||||
label, and logs the five fields `003.3` names (`003.1`, `003.2`, `003.3`); an
|
||||
ordinary admit moves none (`003.4`).
|
||||
|
||||
Time is injected rather than slept — `synctest` per the repo's test discipline,
|
||||
never `time.Sleep`. PostgreSQL twins of the window-count query belong beside
|
||||
the existing `runs_queue_postgres_test.go` cases, for the same reason those
|
||||
exist.
|
||||
|
||||
No E2E coverage: `REQ-OFFICE-ASSIGN-RATE-001`'s `## Out of scope` excludes any
|
||||
frontend, so no user-visible surface changes and there is nothing for
|
||||
Playwright to drive.
|
||||
Reference in New Issue
Block a user