dreampiler/kandev
1
0
Fork 0
Code Issues 43 Pull Requests Actions 6 Packages Projects Releases Wiki Activity

fix: scope PR auto-fix outcome reporting (#3602)

Browse Source
This commit is contained in:
Carlos Florêncio
2026-09-12 16:24:56 +01:00
committed by GitHub
parent 140ef39cc6
commit 2b9d5f1a5d
13 changed files with 757 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/pr-auto-fix-outcome-scope/plan.md
+168
View File
@@ -0,0 +1,168 @@
---
created: 2026-09-11
status: complete
requirements:
- REQ-UI-CI-PR-AUTOMATION-001
system_design:
- ../../specs/ui/system-design/ci-pr-automation-02.md
legacy_specs: []
---
# Fix plan: PR auto-fix outcome scope
## Overview
Clarify when an agent must report a PR auto-fix outcome.
First correct the tool and dispatch guidance. Then explain rejected calls and
prove that ordinary turns cannot change auto-fix state.
The implementation applies the approved guidance and rejection-scope package
without changing automation eligibility or retry behavior.
## Diagnosis and evidence
Affected task: `93a74cfc-6135-452e-85d1-ccd6d7af78cd`, linked PR #3590.
Session: `2edf3b96-66b5-4409-9238-36c97f4d3a63`.
Turn: `71c201a0-4115-4ca3-bb61-bdc60dd48d6c`.
- The original user requested implementation, PR creation, a 15-minute wait,
and manual PR fixup. This request did not enable automatic PR repair.
- The affected turn started at `2026-09-11T09:28:21Z` from a sibling review
message, `9f209c9c-d7d3-54b6-bee4-f3b6a7501a2d`.
- At `2026-09-11T10:12:12Z`, message
`5865c632-ea68-4140-a507-6a4b6391180d` called
`report_pr_auto_fix_outcome_kandev` with `action_taken`.
Its summary described manual review fixes and head `46ce139`.
- The tool returned `VALIDATION_ERROR` with
`CI auto-fix attempt not found or no longer matches`.
- Read-only inspection of `/root/.kandev/data/kandev.db` found zero automation
option rows and zero attempt-state rows for this task. PR #3590 remained linked.
The legacy task settings row was absent too. `GetTaskPRAutomationOptions`
returns disabled defaults for a missing row.
- The recorded turn metadata had no auto-fix binding. The complete primary
conversation contained the original user prompt and sibling review prompt,
with no Kandev auto-fix dispatch message.
Current database state alone cannot prove every historical switch value.
The transcript, turn record, rejected call, and disabled defaults agree with the
user's report. No evidence indicates a scheduler dispatch with auto-fix disabled.
No settings, sessions, GitHub objects, or live records changed during diagnosis.
## Root cause
The agent used an automatic-attempt reporting tool to summarize manual fixup.
`registerPRAutomationTools` exposes the tool to GitHub-capable task sessions,
independent of automation settings or an active attempt. Its description names
an auto-fix turn but does not explicitly exclude ordinary PR work.
The existing dispatch protocol also lacks an explicit expiry across later turns.
That latter omission is a preventive concern, not a proven trigger in this incident.
The conditional store write correctly rejected the call. This repair addresses
agent guidance and error recovery, not an observed authorization bypass.
The exact internal reason for the model's choice is not independently provable.
## Requirement reconciliation
`AC-UI-CI-PR-AUTOMATION-001.10` already requires exact turn binding.
The implementation preserved that boundary in this incident.
Criteria `.16` and `.17` clarify guidance and rejection behavior that were missing.
The existing UI-owned automation pair remains authoritative during migration.
The integration index owns adjacent provider signals, not a second copy of this
attempt contract. No new ADR is needed because the existing outcome ADR already
separates ordinary turns from automatic attempts.
## Scope
### In scope
- Explicit current-turn eligibility in tool and immutable dispatch guidance.
- A specific explanation for an unmatched outcome report.
- Regression coverage for valid and invalid report contexts.
### Out of scope
- Dynamic catalog removal, a new eligibility endpoint, or session restarts.
- Automatic settings changes or conversion of manual work into an attempt.
- Changes to retry counts, provider progress, queue delivery, or GitLab behavior.
- UI layout, translated browser copy, and modification of repository skills.
- Repairing or changing the affected task or PR.
## Technical approach
Keep the existing provider-scoped catalog and backend identity checks.
Update the reporting tool description in `internal/mcp/server/server.go`.
Update `ciAutomationOutcomeProtocol` in
`internal/orchestrator/event_handlers_github_ci_automation.go`.
Map the missing-attempt sentinel separately in
`internal/mcp/handlers/task_pr_automation.go`.
Retain the sentinel, error code, schema, and all failure semantics.
Proposed rejection meaning: no matching unresolved auto-fix attempt exists for
this turn. Use this tool only for a current Kandev auto-fix dispatch.
For ordinary PR work, finish normally without retrying this report or enabling
auto-fix. Do not claim that the switch is off from this error alone.
## Tests
| Criteria | Planned evidence |
| --- | --- |
| `.16` | `TestReportPRAutoFixOutcomeToolDescriptionScopesCurrentTurn` in new `internal/mcp/server/pr_auto_fix_outcome_scope_test.go` |
| `.16` | `TestCIAutomationOutcomeProtocolScopesCurrentTurn` in new `internal/orchestrator/ci_automation_outcome_scope_test.go` |
| `.17` | `TestReportPRAutoFixOutcomeExplainsUnmatchedTurn` in new `internal/mcp/handlers/task_pr_automation_outcome_scope_test.go` |
| `.10`, `.17` | `TestStorePRAutoFixOutcomeOrdinaryTurnHasNoSideEffects` in new `internal/github/store_ci_outcome_scope_test.go` |
| `.10` | `TestStorePRAutoFixOutcomeAcceptsAllValidOutcomes` in new `internal/github/store_ci_outcome_scope_test.go` |
The first three tests must fail before the planned correction.
The store matrix protects the already-working rejection boundary.
Exercise a real temporary SQLite store for state assertions.
Cover absent and disabled settings, enabled-without-attempt, another turn's
attempt, an already-reported attempt, and all three valid outcomes.
Include two PRs with different settings and a foreign bound attempt.
## End-to-end evidence
The observable surface is the agent tool protocol. Use the existing in-process
MCP client pattern (`callTool`) to verify the returned error and caller binding.
Store tests prove persistent effects. No browser rendering changes require
Playwright, and deterministic tests do not claim to prove every model choice.
## Work orders
- [x] [Task 01: Scope agent reporting guidance](task-01-scope-reporting-guidance.md)
- [x] [Task 02: Explain unmatched outcome reports](task-02-explain-unmatched-reports.md)
Execute sequentially. The existing
[outcome retry package](../pr-auto-fix-outcome-retries/plan.md) is implemented.
This package adds clarification without reopening or rewriting its results.
## Verification results
Implementation: complete. Both work orders are done.
- `rtk python3 scripts/lint-spec-files.test.py`: passed, 36 tests.
- `rtk python3 scripts/lint-spec-files.py --all`: passed.
- `rtk git diff --check`: passed.
- `rtk go test ./internal/mcp/handlers -run 'Test.*Report.*PRAutoFix' -count=1`: passed, 7 tests.
- `rtk go test ./internal/mcp/server -run 'TestReportPRAutoFixOutcome|TestTaskPRAutomationTools' -count=1`: passed, 5 tests.
- `rtk go test ./internal/github -run 'TestStorePRAutoFixOutcomeOrdinaryTurnHasNoSideEffects|TestStoreTaskCIAutoFixAttempt' -count=1`: passed, 9 tests.
- `rtk go test ./internal/orchestrator -run 'TestCIAutomationOutcomeProtocol' -count=1`: passed, 4 tests.
- `rtk make -C apps/backend build`: passed; the environment reported only the existing missing codesign/rcodesign warning for Darwin artifacts.
- `rtk make -C apps/backend test`: the changed packages passed, but the full suite exited 2 in unrelated existing areas: process-tree probing, config/home discovery, launcher service configuration, and an Office SQLite migration.
- `rtk gofmt -l` on all changed Go files: passed.
- Work-order references and existing code symbols checked against the workspace.
## Risks
- Clear guidance reduces misuse but cannot guarantee a model never calls the tool.
The backend guard remains mandatory.
- A static catalog must remain usable when a later auto-fix turn reuses a session.
- Historical instructions can remain in model context. The protocol must state
its current-turn scope without weakening reporting for valid dispatches.
- A missing attempt can mean stale or completed work, not only disabled settings.
## Documentation impact
Implementation changes are limited to agent-facing backend guidance, rejection
classification, regression coverage, and the delivery record. Public guidance
remains accurate because automatic retries and controls do not change. The
implementation retains the distinction between manual fixup and
Kandev-dispatched auto-fix in agent-facing text.
docs/plans/pr-auto-fix-outcome-scope/task-01-scope-reporting-guidance.md
+93
View File
@@ -0,0 +1,93 @@
---
id: "01-scope-reporting-guidance"
title: "Scope agent reporting guidance"
status: done
wave: 1
depends_on: []
plan: "plan.md"
requirements:
- REQ-UI-CI-PR-AUTOMATION-001
acceptance_criteria:
- AC-UI-CI-PR-AUTOMATION-001.16
system_design:
- ../../specs/ui/system-design/ci-pr-automation-02.md
---
# Task 01: Scope agent reporting guidance
## Summary
Explain the current-turn reporting requirement at discovery and dispatch.
Keep the GitHub task catalog stable across manual and automatic turns.
## In scope
- Tool description exclusions for manual fixup, sibling review, and historical instructions.
- Explicit expiry of the immutable protocol at the end of its dispatched turn.
- Structured, passthrough, and customized-prompt coverage.
## Out of scope
- Catalog filtering, new tool arguments, setting changes, and retry logic.
## Acceptance
- Discovery explains that availability or enabled settings do not establish a reporting obligation.
- A valid current auto-fix dispatch still requires exactly one outcome.
- Prior-turn protocol text does not instruct later manual turns to report.
## TDD sequence
Add the two guidance tests named in the plan. Run them and record the expected failures.
Then change the two instruction surfaces and rerun the tests.
Retain existing provider membership and protocol visibility coverage.
## Verification
Run from the repository root:
```bash
(cd apps/backend && rtk go test ./internal/mcp/server -run 'TestReportPRAutoFixOutcome|TestTaskPRAutomationTools|TestServerModeTask_ProviderMembership' -count=1)
(cd apps/backend && rtk go test ./internal/orchestrator -run 'TestCIAutomationOutcomeProtocol' -count=1)
rtk python3 scripts/lint-spec-files.py --all
rtk git diff --check
```
## Files likely touched
- `apps/backend/internal/mcp/server/server.go`
- `apps/backend/internal/mcp/server/pr_auto_fix_outcome_scope_test.go` (new)
- `apps/backend/internal/orchestrator/event_handlers_github_ci_automation.go`
- `apps/backend/internal/orchestrator/ci_automation_outcome_scope_test.go` (new)
## Dependencies
None.
## Risks
Overly broad exclusions can suppress legitimate automatic reports.
Assert the positive reporting instruction as well as its scope.
## Parallelism
`sequential`
## Inputs
- Requirement `.16` and design section `Outcome reporting scope`.
- `TestCIAutomationOutcomeProtocolVisibility` and `newTaskModeServer`.
- [Incident evidence and test matrix](plan.md).
## Results
Done. The GitHub outcome tool description now scopes reporting to the current
Kandev-dispatched auto-fix turn and excludes manual fixup, sibling review, and
historical instructions. The immutable structured and passthrough dispatch
protocols state the same scope and expiry. Existing provider membership and
positive outcome visibility remain covered.
- `rtk go test ./internal/mcp/server -run 'TestReportPRAutoFixOutcome|TestTaskPRAutomationTools|TestServerModeTask_ProviderMembership' -count=1`: passed, 10 tests.
- `rtk go test ./internal/orchestrator -run 'TestCIAutomationOutcomeProtocol' -count=1`: passed, 4 tests.
- `rtk python3 scripts/lint-spec-files.py --all`: passed.
- `rtk git diff --check`: passed.
docs/plans/pr-auto-fix-outcome-scope/task-02-explain-unmatched-reports.md
+104
View File
@@ -0,0 +1,104 @@
---
id: "02-explain-unmatched-reports"
title: "Explain unmatched outcome reports"
status: done
wave: 2
depends_on:
- "01-scope-reporting-guidance"
plan: "plan.md"
requirements:
- REQ-UI-CI-PR-AUTOMATION-001
acceptance_criteria:
- AC-UI-CI-PR-AUTOMATION-001.10
- AC-UI-CI-PR-AUTOMATION-001.17
system_design:
- ../../specs/ui/system-design/ci-pr-automation-02.md
---
# Task 02: Explain unmatched outcome reports
## Summary
Return a scope explanation when the backend cannot match an unresolved attempt.
Prove that rejected reports leave automation settings and attempt state unchanged.
## In scope
- A separate error mapping for `ErrTaskCIAutoFixAttemptNotFound`.
- Handler and MCP-client coverage for rejection guidance and `VALIDATION_ERROR`.
- Real-store regression coverage for the context matrix in the plan.
## Out of scope
- Successful no-op responses, synthetic attempts, schema changes, and automatic retries.
- Changing the conditional store write or caller identity contract.
## Acceptance
- The error explains scope without asserting that auto-fix is disabled.
- Invalid reports create no attempt, consume no round, and change no settings.
- Valid bound reports retain all three outcomes and reject duplicate or foreign-turn reports.
## TDD sequence
First add `TestReportPRAutoFixOutcomeExplainsUnmatchedTurn` and record its expected failure.
Add the state matrix through the real temporary store.
Change only the missing-attempt error mapping, then run all commands below.
Extend the MCP client test to prove the explanation survives the tool transport.
## Verification
Run from the repository root:
```bash
(cd apps/backend && rtk go test ./internal/mcp/handlers -run 'Test.*Report.*PRAutoFix' -count=1)
(cd apps/backend && rtk go test ./internal/mcp/server -run 'TestReportPRAutoFixOutcome|TestTaskPRAutomationTools' -count=1)
(cd apps/backend && rtk go test ./internal/github -run 'TestStorePRAutoFixOutcome|TestStoreTaskCIAutoFixAttempt' -count=1)
rtk python3 scripts/lint-spec-files.test.py
rtk python3 scripts/lint-spec-files.py --all
rtk git diff --check
```
## Files likely touched
- `apps/backend/internal/mcp/handlers/task_pr_automation.go`
- `apps/backend/internal/mcp/handlers/task_pr_automation_outcome_scope_test.go` (new)
- `apps/backend/internal/mcp/server/pr_auto_fix_outcome_scope_test.go`
- `apps/backend/internal/github/store_ci_outcome_scope_test.go` (new)
## Dependencies
Task 01 owns the shared new MCP test file and the guidance terminology.
## Risks
The sentinel covers missing, stale, and completed attempts.
The explanation must not recommend changing settings to repair a reporting error.
State assertions must compare rows before and after rejected calls, including absent rows.
## Parallelism
`sequential`
## Inputs
- Criteria `.10` and `.17`, and design section `Outcome reporting scope`.
- `handleReportTaskPRAutoFixOutcome`, `ReportTaskCIAutoFixOutcome`, and existing handler tests.
- `TestStoreTaskCIAutoFixAttemptLifecycleUsesExactIdentity`.
- [Incident evidence and test matrix](plan.md).
## Results
Done. Unmatched auto-fix outcome reports now keep `VALIDATION_ERROR` while
returning a scope explanation that covers missing, stale, and completed turns.
The explanation directs ordinary work to continue without retrying the report
or changing automation settings. The MCP transport and temporary-SQLite tests
prove that the existing task/session/turn guard and all three valid outcomes
remain intact.
- `rtk go test ./internal/mcp/handlers -run 'Test.*Report.*PRAutoFix' -count=1`: passed, 7 tests.
- `rtk go test ./internal/mcp/server -run 'TestReportPRAutoFixOutcome|TestTaskPRAutomationTools' -count=1`: passed, 5 tests.
- `rtk go test ./internal/github -run 'TestStorePRAutoFixOutcomeOrdinaryTurnHasNoSideEffects|TestStoreTaskCIAutoFixAttempt' -count=1`: passed, 9 tests.
- `rtk python3 scripts/lint-spec-files.test.py`: passed, 36 tests.
- `rtk python3 scripts/lint-spec-files.py --all`: passed.
- `rtk git diff --check`: passed.
docs/specs/ui/requirements/ci-pr-automation.md
+15 -2
View File
@@ -2,7 +2,7 @@
status: active
system: ui
created: 2026-06-18
updated: 2026-09-06
updated: 2026-09-11
owners:
- tbd
---
@@ -64,6 +64,19 @@ Consequences section).
- **AC-UI-CI-PR-AUTOMATION-001.14:** Every retry shall consume the existing per-PR 10-round budget. Kandev shall never create an 11th accepted round, and disabling then re-enabling auto-fix shall reset the retry state with the existing checkpoint and round state.
- **AC-UI-CI-PR-AUTOMATION-001.15:** Existing installations whose built-in `ci-auto-fix` prompt still matches a shipped, untouched legacy revision shall receive the current default content on startup. Kandev shall preserve edited prompt rows, and the server-owned outcome instructions shall apply even when a task or global prompt is customized.
## System design
### Outcome scope clarification
The following criteria clarify the existing turn-bound outcome contract.
The [scope repair plan](../../../plans/pr-auto-fix-outcome-scope/plan.md) records the implementation and regression coverage.
- **AC-UI-CI-PR-AUTOMATION-001.16:** Agent guidance shall restrict outcome reporting to the current Kandev-dispatched auto-fix turn. Manual PR fixup, sibling review messages, and historical auto-fix instructions shall not establish that obligation. Tool availability and enabled automation settings alone shall not establish that obligation.
- **AC-UI-CI-PR-AUTOMATION-001.17:** When no matching unresolved attempt exists, Kandev shall reject the report with a scope explanation. The explanation shall direct the agent to finish ordinary work without retrying the report or enabling automation. The rejected report shall not create an attempt, consume a round, or change automation settings.
## Implementation plans
- [Outcome retries](../../../plans/pr-auto-fix-outcome-retries/plan.md): existing implementation.
- [Outcome scope repair](../../../plans/pr-auto-fix-outcome-scope/plan.md): implementation record.
## System-design references
The migrated technical source is split into [part 1](../system-design/ci-pr-automation-01.md), [part 2](../system-design/ci-pr-automation-02.md), [part 3](../system-design/ci-pr-automation-03.md).
docs/specs/ui/system-design/ci-pr-automation-02.md
+39 -2
View File
@@ -4,7 +4,7 @@ system: ui
requirements:
- REQ-UI-CI-PR-AUTOMATION-001
created: 2026-06-18
updated: 2026-09-06
updated: 2026-09-11
owners:
- tbd
---
@@ -144,7 +144,44 @@ Auto-merge cycle for one task/PR:
4. If the PR is ready and the readiness signature is new, Kandev calls the existing PR merge operation using the backend default merge-method selection.
5. Kandev records the merge attempt and refreshes PR state after a successful merge when practical.
## Permissions
## Outcome reporting scope
This clarification maps to `AC-UI-CI-PR-AUTOMATION-001.10`,
`AC-UI-CI-PR-AUTOMATION-001.16`, and `AC-UI-CI-PR-AUTOMATION-001.17`.
Its delivery record is the [scope repair plan](../../../plans/pr-auto-fix-outcome-scope/plan.md).
`registerPRAutomationTools` exposes the reporting tool through the GitHub task
catalog. Catalog membership describes capability, not an active auto-fix attempt.
The catalog remains stable across ordinary and automatic turns in one session.
The tool description must require a current Kandev auto-fix prompt with the
server-owned outcome protocol. It must explicitly exclude manual PR fixup and
sibling review work. A setting, a quoted marker, or an earlier turn's protocol
does not prove that the current turn owns an attempt.
`ciAutomationOutcomeProtocol` must state that its reporting instruction applies
only to the turn receiving that dispatch. Structured and passthrough sessions
retain their existing transport behavior. Custom prompts cannot remove this scope.
`Service.ReportTaskPRAutoFixOutcome` continues to resolve the active turn.
`Store.ReportTaskCIAutoFixOutcome` remains the authoritative conditional write.
The exact task, session, turn, running state, and empty outcome must match.
No caller-supplied identity, new attempt, or success fallback is introduced.
`handleReportTaskPRAutoFixOutcome` must map
`ErrTaskCIAutoFixAttemptNotFound` to a specific scope explanation while retaining
`VALIDATION_ERROR`. The explanation covers missing, stale, and completed attempts
without claiming that auto-fix is disabled. It directs the agent to finish ordinary
work without another report or a settings change. Other validation and internal
errors retain their existing classification.
Regression coverage must exercise missing settings, explicit disabled settings,
an enabled PR without an attempt, and a valid bound attempt. A mixed-PR case must
prove that an attempt on another session or turn cannot authorize this call.
No UI composition, persistence schema, retry policy, or provider catalog changes
are required.
## Access permissions
- Any user who can view and interact with the task chat can read and update the task CI automation options for that task.
- Any user who can edit prompts in Settings > Prompts can edit the default `ci-auto-fix` prompt.