kandev/docs/plans/step-entry-dispatch-convergence/plan.md

---
created: 2026-09-02
status: in-progress
requirements:
  - REQ-OFFICE-STEP-ENTRY-DISPATCH-002
  - REQ-OFFICE-STEP-ENTRY-DISPATCH-003
  - REQ-OFFICE-STEP-ENTRY-DISPATCH-004
  - REQ-OFFICE-STEP-ENTRY-DISPATCH-005
  - REQ-OFFICE-STEP-ENTRY-DISPATCH-006
  - REQ-OFFICE-STEP-ENTRY-DISPATCH-007
  - REQ-OFFICE-STEP-ENTRY-DISPATCH-008
system_design:
  - ../../specs/office/system-design/step-entry-dispatch-convergence.md
legacy_specs:
  - ../../specs/workflow-on-enter-action-dispatch/spec.md
---

# Implementation Plan: Step Entry Dispatch Convergence

## Overview

Two dispatchers execute a step's declared entry sequence on the same committed
arrival, and `clear_decisions` and `queue_run_for_each_participant` are handled
by both. Because the paths derive idempotency keys from different identifiers,
the runs queue does not collapse the duplicate, so the shipped `office-default`
Review and Approval steps fan out twice per arrival and quorum can count two
decisions for one round.

The ledger dispatcher becomes the single owner of every session-independent
kind, and the marker machinery is moved onto it rather than deleted. The order
below is forced by that decision: the ownership declaration is the table both
dispatchers read, so it lands first; the allocated marker position set must be
persisted before terminality can be derived from it; the reclaim and skip
primitives must exist before the startup scan can consume them; and the
diagnostics work depends on the skip primitive because an unwired adapter at a
marker-bearing position must reach a terminal marker rather than strand its
entry.

Turn-completion re-keying and the coalescing exclusion do not depend on
convergence, but they share files with it and are sequenced after the work that
touches those files.

## Scope

### In scope

- One ownership declaration mapping each action kind to its owning dispatcher
  and to whether it is marker-bearing, read by both dispatchers.
- Ledger-path ownership of the five session-independent kinds, including the
  marker claim, the abandon-remainder stop rule for marker-bearing positions,
  and the two-branch idempotency key.
- Unconditional entry allocation for a step declaring a marker-bearing kind,
  with the allocated position set persisted on the entry row.
- A fourth terminal marker state `skipped` with a recorded reason, and a
  state-predicated reclaim path distinct from the live-dispatch claim.
- A single startup recovery scan that retires or resumes stranded entries.
- Task-scoped turn-completion serialization, retaining the session-keyed
  redelivery check.
- Coalescing exclusion for entry-triggered runs, carried in memory and on the
  run row.
- Per-participant fan-out reporting, declaration-fault handling, and the
  warning at every discard point.

### Out of scope

- Ordering guarantees across owners. No shipped workflow declares a step whose
  entry sequence mixes owners, and interleaving the synchronous ledger path
  with the marker path's goroutine is a larger change than this design carries.
- Extending durability to kinds that do not carry a marker today. Marker-bearing
  stays at `clear_decisions` and `queue_run_for_each_participant`.
- Cross-process serialization. Both turn-completion maps are in-process
  `sync.Map`, so REQ-003's guarantee is scoped to one backend process.
- A migration for runs enqueued under the pre-convergence idempotency key.
- Any change to `docs/specs/workflow-on-enter-action-dispatch/spec.md`.

## Technical approach

### Ownership declaration

One exported table in `apps/backend/internal/workflow/stepentry` maps action
kind to owning dispatcher and to marker-bearing, as two independent columns.
`Engine.DispatchStepEntry` (`internal/workflow/engine/entrydispatch.go`) and
`dispatchOnEnterActions` (`internal/orchestrator/event_handlers_workflow.go`)
both read it; neither keeps a private list. The ownership column is seeded from
`sessionIndependentActionKinds` and the marker-bearing column from
`IsEngineOwnedOnEnter`; the seeds are different functions and are not
interchangeable. `configure_session` is written in by hand because
`CompileOnEnterAction` emits no `ActionKind` for it. The table is wider than
either engine map, so `sessionShapedActionKinds` is not edited.

### Ledger ownership, claim, and stop rule

`Repository.dispatchStepEntry` claims through `ClaimStepEntryMarker` before
executing a marker-bearing kind. `idempotencyKey` keeps both branches: a step
declaring at least one marker-bearing kind keys from the
`workflow_step_entries` row id on every route; a step declaring none keys from
the step-transition row id. Which branch applies is read from the ownership
declaration, not from whether an entry row exists. The ledger dispatcher
abandons the remainder of an entry's sequence when a marker-bearing position
fails or loses its claim, narrowing `Engine.DispatchStepEntry`'s
record-and-continue contract to marker-bearing positions only.

### Entry allocation and the position set

`workflow_step_entries` gains `marker_positions TEXT NOT NULL DEFAULT ''` via
`r.migrate.Apply` in `internal/task/repository/sqlite/base_migrations.go`.
`BuildPendingAllocation` already computes the marker-bearing position set to
decide whether to allocate; it now returns it, and
`allocateStepEntryIfPending` persists it inside the transition transaction.
Terminality is derived from the stored set, never re-read from the declaration.

### Marker states and reclaim

`workflow_step_entry_markers` gains `skip_reason TEXT`.
`SkipStepEntryMarker` upserts `state='skipped'` guarded by `WHERE state <>
'done'`. `ReclaimStepEntryMarker` is a state-predicated compare-and-set over
`state IN ('in_progress','failed')`, reporting from the affected row count.
`ClaimStepEntryMarker` is unchanged and still refuses any existing row.

### Startup recovery scan

A single scan in `startSchedulingRuntime`
(`internal/backendapp/main.go`), after the repositories are constructed and
before the runs scheduler and engine dispatcher serve, joins
`workflow_step_entries` with its markers, orders by `(task_id, step_id,
entry_seq)`, and applies the skip rules. No workflow declaration is loaded for
selection. Every failure inside the scan is logged and skipped; the scan never
blocks startup.

### Coalescing exclusion

`QueueRunRequest` gains `EntryTriggered bool` in both `internal/runs/service`
and `internal/workflow/engine`. `shouldCoalesceRun` returns false when it is
set. Because `CoalesceRun`'s subquery reads persisted columns only, `runs`
gains `entry_triggered INTEGER NOT NULL DEFAULT 0` via `r.migrate.Apply` in
`internal/office/repository/sqlite/base.go`, the insert path writes it, and the
subquery gains `AND entry_triggered = 0`.

### Migrated column probes

The delivered `workflow_step_entries.marker_positions` ALTER uses the task
repository's required migration logger. `runMigrations` checks `Err()` after
that migration, so an unexpected failure stops repository startup before any
reader can issue a query against a missing column.

The deferred `runs.entry_triggered` and `workflow_step_entry_markers.skip_reason`
columns are each planned to be probed once at startup with the `columnExists`
shape `runs.outcome` uses. A failed probe emits one ERROR and takes the
degraded path rather than issuing a statement that cannot succeed. A probe
that errors for any other reason is treated as absent. These probe rules do not
change the required migration contract for `marker_positions`.

### Diagnostics and fan-out

The warning helper takes workflow id, step id, step name, and action type and
is called from all four discard points. The `ErrActionNotYetWired` callback is
replaced by an unwired check ahead of the claim, which for a marker-bearing
position writes a terminal `skipped` marker with reason `adapter_unwired`.
`queue_run_for_each_participant` gains a logger and emits per-participant ERROR
records plus one INFO summary. `roleSeatsForFanOut`'s `Position ASC,
AgentProfileID ASC` order is kept unchanged, because
`review-participant-seats.md` AC-OFFICE-REVIEW-SEATS-003.3 binds it.

## Tests

| Acceptance criteria | Evidence |
|---|---|
| 002.1, 002.2, 002.3, 002.6, 002.7 | `internal/workflow/stepentry/stepentry_test.go`, `internal/workflow/engine/entrydispatch_test.go` |
| 002.4, 002.10 | `internal/orchestrator/step_entry_dispatch_test.go` stop-rule case: fail `clear_decisions` at position 0 of the shipped Review sequence, assert the position-2 fan-out enqueues nothing |
| 002.5 | `internal/orchestrator/review_participant_seats_acceptance_test.go`, asserting an empty queue after the expected run |
| 002.8, 002.9, 008.2 | `internal/task/repository/sqlite/step_entries_test.go`, asserting two entries into one step yield different keys |
| 003.1, 003.2, 003.3, 003.4, 003.6 | `internal/orchestrator/step_entry_dispatch_concurrent_test.go` |
| 003.5 | new two-session-one-task case in `internal/orchestrator/step_entry_dispatch_concurrent_test.go` |
| 004.1, 004.3, 004.4, 004.5, 004.6, 004.12 | new `internal/office/service/step_entry_recovery_test.go` |
| 004.2, 004.11 | probe-failure case in the same file, asserting startup proceeds |
| 004.7, 004.8, 008.4 | `internal/task/repository/sqlite/step_entries_test.go`, distinguishing `ClaimStepEntryMarker` from `ReclaimStepEntryMarker` on the same row |
| 004.9, 004.10 | `internal/task/repository/sqlite/step_entries_test.go`, allocating at a step with a non-marker-bearing position and asserting `marker_positions` omits it |
| 005.1, 005.3, 005.4, 005.5 | `internal/runs/service/service_test.go` |
| 005.2, 005.6 | `internal/office/repository/sqlite/runs_test.go`, covering the target direction |
| 006.1, 006.2, 006.3, 006.5 | `internal/orchestrator/event_handlers_workflow_on_enter_warn_test.go` |
| 006.4 | same file: a marker-bearing position reaches terminal `skipped` with reason `adapter_unwired` and is not re-selected |
| 007.1 to 007.6, 007.8, 007.10, 007.11 | new `internal/workflow/engine/phase2_callbacks_test.go` |
| 007.7, 007.9 | same file, with a fixture whose agent-profile-id and row-creation orders disagree |
| 008.1 | `internal/orchestrator/step_entry_dispatch_atomic_test.go`, with the failure injected inside the repository method |
| 008.3 | `internal/task/repository/sqlite/step_entries_test.go`, asserting neither the entry row nor the step change survives |

## E2E tests

No Playwright coverage is added. Every criterion in this work package is a
backend dispatch, persistence, or logging guarantee with no UI surface; the
user-visible symptom, a duplicated reviewer fan-out, is asserted at the
production-shape boundary by
`internal/orchestrator/review_participant_seats_acceptance_test.go`, which wires
both dispatchers.

## Work orders

- [x] [Task 01: Ownership declaration for step-entry action kinds](task-01-ownership-declaration.md)
- [x] [Task 02: Persist the allocated marker position set](task-02-marker-positions.md)
- [x] [Task 03: Converge dispatch onto the ledger dispatcher](task-03-ledger-ownership.md)
- [ ] [Task 04: Terminal skipped marker state](task-04-skipped-marker-state.md) — deferred, see below
- [ ] [Task 05: Marker reclaim path](task-05-marker-reclaim.md) — deferred, see below
- [ ] [Task 06: Startup step-entry recovery scan](task-06-recovery-scan.md) — deferred, see below
- [ ] [Task 07: Task-scoped turn-completion serialization](task-07-task-scoped-turn-completion.md) — deferred, see below
- [ ] [Task 08: Entry-triggered runs bypass coalescing](task-08-coalescing-exclusion.md) — deferred, see below
- [ ] [Task 09: Discarded step-entry action diagnostics](task-09-discard-diagnostics.md) — deferred, see below
- [ ] [Task 10: Participant fan-out reporting](task-10-fanout-reporting.md) — deferred, see below

### Bounded-scope decision (2026-09-02)

This Build round completed Tasks 01-03 only — the ownership declaration, the
persisted marker-position set, and ledger convergence of `clear_decisions` /
`queue_run_for_each_participant` (including the AC-002.10 abandon-remainder
stop rule) — and stopped there rather than attempting all ten work orders in
one pass. Tasks 01-03 are the fix for the actual reported production defect
(office-default's Review/Approval steps double-dispatching per round); Tasks
04-10 add durability/observability/serialization hardening on top of a
now-correct dispatch path, each independently shippable and independently
tested against its own AC set. Attempting all ten with full TDD rigor,
migration coverage, and the full DoD gauntlet in one continuous session
was judged out of reasonable scope; splitting at the 01-03 boundary keeps the
delivered slice coherent (every AC in REQ-002 that's addressed is fully
addressed, nothing half-wired) and leaves 04-10 exactly as scoped by their
existing work-order files for a follow-up round. No frozen requirement or
system-design content changed to justify this boundary — it is a delivery
sequencing choice, not a scope reduction of the spec itself.

Within Task 02, "unconditional entry allocation" (extending allocation to
every write chokepoint, not just the existing `on_turn_complete`
ResultHolder-gated route in `workflow_store.go`/`applyTransition`) was also
narrowed: allocation stays on that one existing route. That route is exactly
where the production bug manifests (Review/Approval steps entered via
`on_turn_complete`), so the fix lands regardless. Other routes (manual step
move via `finalizeStepEnter`, WIP-promotion, workflow-switch) do not get an
allocated entry this round; a marker-bearing action reaching
`Engine.DispatchStepEntry` with `markerEntryID == 0` on those routes falls
back to the pre-convergence unprotected direct-execution path, unchanged from
today. Extending allocation to those routes is folded into Task 04's
remaining scope (terminal `skipped` state depends on knowing the full
declared position set on every route, which unconditional allocation would
provide).

Two pre-existing tests from an earlier build round
(`TestProcessOnEnter_ClearDecisionsFailure_BlocksSubsequentOnEnterActions` and
the renamed `TestProcessOnEnter_ClearDecisionsFailure_DoesNotBlockEarlierAutoStart`,
both in `step_entry_dispatch_test.go`) were updated during this round: the
first to match the ledger-side abort log line
(`dispatchEngineOwnedOnEnterAction`'s "DispatchStepEntry: marker-bearing
on_enter action failed") now that `clear_decisions` executes exclusively
through the ledger; the second inverted its assertion (auto-start now *does*
launch despite a same-entry `clear_decisions` failure) because the system
design's own "Out of scope: Ordering between actions owned by different
dispatchers" section states this cross-owner ordering was never a guarantee
this initiative provides — `auto_start_agent` is marker-owned and
`clear_decisions` is ledger-owned, and the two dispatch independently with no
barrier between them.

## Verification results

- `go build ./...`, `go vet ./...` — clean.
- `gofmt -l` on every changed file — clean.
- `golangci-lint run ./... --new-from-rev=646ff0063f5656ad2f0358fb9e01e39378ac113f` — 0 issues.
- `go test ./internal/orchestrator/... ./internal/backendapp/...` — all pass.
- `go test ./...` (full backend suite) — all pass except pre-existing,
  unrelated failures in `internal/worktree`, `internal/launcher`,
  `internal/agentctl/server/{process,config,api}`,
  `internal/agent/{managedruntime,runtime/lifecycle,settings/controller}`,
  `internal/common/config`, `internal/task/{service,handlers}`,
  `internal/system/storage/workspaces` — all traced to this macOS sandbox's
  `/var` -> `/private/var` symlink tripping a symlinked-path safety check in
  each package's `t.TempDir()`-based tests; confirmed unrelated to this
  change (none of those packages import `internal/task/repository/sqlite`,
  `internal/workflow/engine`, or `internal/orchestrator`'s step-entry code).

## Risks

- Moving ownership changes idempotency keys, so a deployment landing mid-round
  can duplicate one round's runs once. No migration is proposed.
- Allocation runs on the pre-existing `on_turn_complete` route only this round
  (Bounded-scope decision above), so the entry-row growth this PR ships is
  limited to a route that already allocated. Extending unconditional
  allocation to the other routes (manual move, WIP promotion, workflow
  switch) — which would add a row on routes that previously allocated
  none — is deferred to Task 04.
- The task lock widens a critical section that currently admits concurrent
  sessions of one task, making contention visible where it was previously
  silent corruption.
- Three `ALTER` migrations run on existing databases and `MigrateLogger.Apply`
  swallows failure at WARNING. A backend whose `marker_positions` ALTER failed
  runs indefinitely with no startup recovery and one ERROR as its only signal.
- `marker_positions` is a denormalized copy of the ownership declaration at
  allocation time, so an entry allocated before a kind's marker-bearing
  property changes keeps the old set and is retired as `digest_changed` rather
  than resumed.
- Seeding either ownership column from the other's function inverts the table.
  The ownership seed is `sessionIndependentActionKinds`; the marker-bearing
  seed is `IsEngineOwnedOnEnter`, whose name reads like an ownership predicate
  but reports what the marker system dispatches.