kandev/docs/ci-merge-queue.md

Merge queue

The main ruleset enabled the merge queue and six required checks on 2026-08-16. This document records the workflow preparation, active rules, and the Stable release exception.

Claude execution budget

The automatic same-repository review, approved fork review, and interactive Claude jobs each have a 30-minute GitHub job timeout. The timeout bounds runner execution and a cancelled job cannot report a successful review. It does not bound approval or queue waits. An approval that expires before a job starts is an approval outcome, and must be measured separately from Claude execution.

Why

Before activation, main's ruleset (id 13341245) had deletion, non_fast_forward, required_linear_history, and pull_request rules. It had no required_status_checks rule.

That is how #2681 (38c94e7d0) landed red. Its E2E checks were fully green, but they had run 23 hours earlier against a merge ref that predated the retry loop added by #2569 (15788438b). Neither change is wrong on its own; together they put a roughly 4.5-second floor under content search, just over the 5-second default assertion in the test #2681 added. Both pull requests were green, and their merge was not.

Required status checks alone would not have caught it -- #2681's checks were green. A merge queue would: it builds the actual merge result and runs the checks against that, at merge time, in queue order.

What changed in the workflows

The merge_group trigger

Six workflows gained merge_group: types: [checks_requested]:

Workflow	Gate job	Check name to require
e2e-tests.yml	e2e-gate	E2E Tests Passed
backend-tests.yml	test	Run Backend Tests
frontend-tests.yml	frontend-gate	Frontend Tests Passed
architecture-lint.yml	lint	Architecture boundaries do not regress
lint-action-pinning.yml	lint	All action refs are SHA-pinned
lint-harness-files.yml	lint	Harness files pass lint

Deliberately excluded, because they cannot report in a queue and would stall it forever, or because their redness is not a property of the merge:

• pr-title.yml, claude-code-review.yml, opencode-code-review.yml, preview-env.yml, notify-docs.yml -- all pull_request or pull_request_target scoped. There is no pull request in a merge group, so they never run and never report.
• cargo-audit.yml -- also runs on a weekly schedule, by design: a newly published RUSTSEC advisory turns it red with no change to the code. In a queue that would block every merge in the repository until somebody bumped a crate. It stays a pull-request and scheduled check.
• ci-base-image.yml, universal-rebuild.yml -- these publish images rather than test a merge.
• release.yml and the plugin registry workflows -- out of scope.

The path-filter problem, and the approach chosen

Most of these workflows were heavily path-filtered (apps/web/**, !**/*.md, and so on). That is a problem for a queue in two independent ways.

1. A workflow skipped by a trigger-level paths: filter reports no conclusion at all. GitHub's documentation is explicit that "checks associated with that workflow will remain in a 'Pending' state". A required check that stays pending blocks a pull request from being added to the queue, and blocks a merge group from ever merging.
2. merge_group does not support paths: in the first place. It takes types: and branches: only. So the filters could not simply be mirrored onto the queue side even if we wanted them there.

Two approaches were available: mirror each filter onto an inverse trigger with a "skipped reports success" shim job, or make the gating jobs always run and short-circuit internally.

The shim was rejected, and the internal short-circuit implemented. The deciding factor is that paths with ! exclusions has no exact inverse. Take frontend-tests' old filter, apps/web/** followed by !**/*.md. A pull request that touches only apps/web/README.md is skipped by that filter, and is also skipped by the mirrored paths-ignore list, because every changed file appears in it. Both halves skip, no conclusion is ever reported, and the queue waits forever -- the exact failure the shim exists to prevent, reintroduced by the shim. Making the inversion correct would mean hand-deriving a second pattern list per workflow and keeping the two in sync by eye. Worse, the shim needs a job whose name matches the real check, which for e2e-tests.yml would mean reproducing fourteen shard names and six container shard names.

So: the trigger-level paths: filters are gone, and each expensive workflow grew a cheap changes job that applies the same list one level down.

changes ──> static_checks ─┐
        └─> test_shards ───┼──> test          <- the required check, if: always()
        └─> ...          ──┘

.github/scripts/changed-paths.py implements GitHub's filter-pattern semantics (*, **, ?, leading !, last match wins) and is unit-tested in changed-paths_test.py, which also asserts the pattern lists still live in the workflows and still select what they used to. It rejects +, [...], and {...} rather than quietly matching them as literals.

Two properties of this shape matter:

• The gate job always runs. It is if: always() and treats a skipped dependency as a pass and a cancelled one as a failure. This is on purpose: it means the required check never depends on how GitHub reports a conditionally-skipped job, only on a conclusion the workflow computes itself.
• The gate covers every job in its workflow. A queue gates on check names an admin picks by hand, so a job outside the gate's needs is a job whose failure would never block a merge, silently. backend-tests.yml's postgres-boot and test-windows were outside its aggregate before this change and are now inside it.

Two workflows -- lint-action-pinning.yml and lint-harness-files.yml -- have no changes job. They are a checkout plus a handful of short linters, so they simply always run: cheaper than the job that would decide whether to run them, and it closes the gap where a subject outside their hand-maintained lists changed and nothing checked it. Their contract tests were updated to assert the absence of a filter rather than the presence of specific entries.

External CI runner capacity

The normal Linux CI workflows have one activation switch, one percentage control, and two runner tiers. The current read-only snapshot on 2026-09-12 is shown below. The switch is off, so current jobs use GitHub-hosted runners; the retained percentage is the next pilot value.

Variable	Current value	Pilot value	Purpose
KANDEV_CI_EXTERNAL_ENABLED	false	true	Allow external assignments
KANDEV_CI_EXTERNAL_PERCENT	20	20	Assign 0 to 100 percent of eligible job instances
KANDEV_CI_RUNNER_LIGHT	ubicloud-standard-2-ubuntu-2404	same	Control and aggregate jobs
KANDEV_CI_RUNNER_STANDARD	ubicloud-standard-4-ubuntu-2404	same	Eligible browser and frontend test jobs

Set KANDEV_CI_EXTERNAL_ENABLED to the exact value true when the Actions queue needs extra capacity. Set it to false after the queue returns to its normal level. The two tier variables stay configured between bursts, so changing the instance type requires only an Actions variable update.

The 20% pilot sends a stable cohort of eligible instances to the configured tiers. For E2E this assigns two of fourteen normal shards per run because matrix allocation uses a floor. The frontend workflow has two unit-test instances, so the 20% pilot assigns zero of those instances; its singleton static and gate jobs still use stable hash cohorts. Backend checkout, static checks, service jobs, Windows, and the protected test shards remain on GitHub-hosted runners. Singleton jobs use a stable hash cohort, so their share approaches the percentage across workflow runs.

Set the percentage to 0 for an explicit GitHub-only allocation. Set it to 100 to send every eligible job with a non-empty tier label to external capacity. Malformed or out-of-range values fail closed to GitHub-hosted runners and emit a planner warning.

The reusable .github/actions/plan-external-runners composite action runs on the planner job's ubuntu-latest checkout. Each workflow declares its eligible families as JSON, and the action returns one JSON plan with resolved runner labels:

runs-on: ${{ fromJSON(needs.runner_plan.outputs.plan).frontend_runner }}
runs-on: ${{ matrix.runner }}

If a tier variable is empty, that tier uses ubuntu-latest. If a non-empty label is unavailable, GitHub leaves the job queued or fails it. Clear the invalid label and rerun the workflow. Jobs already queued or running keep their original runner. New jobs use the current variables.

The eligible families are the E2E normal-shard matrix and gate, the backend aggregate test gate, the frontend static verification job, the frontend unit test matrix, and their lightweight change-detection and gate jobs. The E2E build and report jobs, Docker/Kind shards, Kubernetes compatibility jobs, Playwright image job, desktop smoke job, backend checkout/static/test shards, Postgres service job, architecture and harness linters, and Windows job stay on GitHub-hosted runners. Release, publishing, signing, deployment, and credential-bearing jobs stay on their existing runners. The workflows do not add permissions, secrets, or persistent state.

The same switch and tier labels apply to eligible E2E, backend-gate, frontend static verification, frontend unit-test matrix, and frontend gate jobs. Architecture-lint, action-pinning, and harness-lint stay hosted and do not create planner jobs. The planner does not change job names, test selection, matrix values, artifacts, dependencies, timeouts, permissions, or required conclusions.

Pilot procedure. An operator must supply an approved provider budget and the provider's current rate before activation. Unknown rates, fees, or budget are an activation blocker.

1. Merge the workflow change while KANDEV_CI_EXTERNAL_ENABLED=false.

2. Install the Ubicloud GitHub App for this repository.

3. Confirm both tier variables match the approved labels. Leave Premium Runners disabled. Premium is an account-level setting, not a per-job tier.

4. Record the current variable snapshot and provider budget in the run record.

5. Activate the 20% pilot for newly dispatched workflows:

   gh variable set KANDEV_CI_EXTERNAL_ENABLED --repo kdlbs/kandev --body true
   gh variable set KANDEV_CI_EXTERNAL_PERCENT --repo kdlbs/kandev --body 20
   

6. Compare at least three representative frontend, E2E, or backend runs. For each eligible job record workflow run attempt, runner label, created time, started_at, completed_at, queue delay, execution duration, setup/cache steps, conclusion, and billed runner minutes. Keep protected hosted jobs in the comparison so queue relief is not confused with code speed.

7. Stop the pilot for newly dispatched workflows after the sample or when the budget is reached:

   gh variable set KANDEV_CI_EXTERNAL_ENABLED --repo kdlbs/kandev --body false
   gh variable set KANDEV_CI_EXTERNAL_PERCENT --repo kdlbs/kandev --body 0
   

   Jobs already queued or running keep their original runner assignment.

Cost calculation for each pilot sample:

external runner-minutes = sum(external job execution seconds) / 60
external cost = external runner-minutes / 60 * provider hourly rate
                + provider setup, storage, or transfer fees

Compare that cost with the GitHub-hosted minute accounting and record the provider rate, billing granularity, and any free allowance. The percentage controls assignment of job instances; it does not promise the same percentage of billed minutes. Do not expand the percentage until three comparable runs show queue relief without an unacceptable failure rate, execution regression, or budget overrun.

Use the Ubicloud runner type reference when changing a tier. Use GitHub's runner selection reference for runs-on expressions. Do not enable Ubicloud Premium Runners for this pilot.

PR context in a merge group

A merge_group run has no github.event.pull_request and no github.event.before. Every expression that read one has been made explicit about the event it is on, and given the merge-group equivalent, github.event.merge_group.base_sha -- the commit the queue branch was cut from, which is the merge group's base:

• architecture-lint.yml -- the two mutually exclusive pull_request / push lint steps were collapsed into one, behind a baseline-resolving step. Under the old shape a merge-group run would have executed the linter's self-tests and then linted nothing at all, reporting green having compared the branch against no baseline.
• backend-tests.yml -- Resolve base SHA.
• frontend-tests.yml -- Check formatting, the i18n new-code ratchet, and the E2E sleep ratchet.
• e2e-tests.yml -- the GHCR login guard reads github.event_name != 'pull_request' || <fork check>, which resolves to the first operand on a merge group. That is correct: the queue branch lives in this repository and its token is a base-repository token.

Concurrency groups were checked too. None of the six keyed on a PR number (preview-env.yml does, and it is not a gate). All six now set cancel-in-progress: ${{ github.event_name != 'merge_group' }}, because a queue reads a cancelled check as a failure and ejects the pull request.

Active ruleset configuration

The main ruleset contains a merge_queue rule and requires the six gate names in the table above. It does not require per-shard names.

Do not require lint from pr-title.yml, Validate public docs, review checks, or deployment checks. These checks do not run in every merge group.

PR documentation coverage rollout

pr-docs.yml also listens for merge_group: checks_requested and publishes the PR documentation coverage status on the synthetic merge-group head. It maps the queue entry boundaries back to the event base and evaluates every included pull request with its own changed files, work-order references, and labels. A member's no-docs-allow label cannot exempt another member.

All PR and merge-group events use the target branch as one non-cancelling concurrency key with queue: max. GitHub retains up to 100 pending runs in that key, so ordinary event bursts do not replace the one pending run; events that arrive after the bound is full can still be canceled. This keeps a queued label reevaluation from racing with a merge-group status write while the published coverage status remains attached to the evaluated PR or synthetic group revision. A label removal reevaluates every active queue-group prefix that contains the PR, not only the longest group.

The status is not in the active required-check list above until live merge-group evidence is collected. Roll out the requirement in two stages:

1. Observe ordinary, draft, fork, label-add, label-remove, manual-retry, and merge-group runs. Confirm that covered, missing, override, and incomplete responses are attached to the evaluated revision and that all affected queue group prefixes reevaluate after label removal.
2. Add PR documentation coverage as a separate required status through an administrator ruleset change. Preserve the existing six checks and bypass rules, then verify a covered PR, an uncovered PR, an override, and a mixed merge group.

Until stage two is complete, a failing result is visible but does not itself block merging. The workflow does not add or remove labels, dequeue PRs, or change rulesets.

Stable release exception

The Stable release workflow creates a mechanical release PR with GITHUB_TOKEN. GitHub holds the PR workflows for approval, but maintainers do not require CI for this generated commit.

The protected release environment contains RELEASE_PR_BYPASS_TOKEN. This fine-grained personal access token belongs to an organization administrator and selects only kdlbs/kandev with contents: write permission.

The workflow exposes this token only to gh pr merge --admin. It also binds the merge to the expected PR head. GITHUB_TOKEN performs all other PR work.

The administrator merge bypasses the queue and required checks for the release PR only. Ordinary PRs still use the queue and all six required checks.

Caveat: the ruleset can still be bypassed

The main ruleset lists one bypass actor:

{ "actor_type": "OrganizationAdmin", "bypass_mode": "always" }

bypass_mode: always means an organization admin can merge past whatever is configured here, queue included, without a review request or a temporary override. Configuring required checks constrains everybody else; it does not constrain an org admin. If the point of enabling the queue is that nothing red reaches main, that entry needs to change as well -- to pull_request mode, or removed -- and that is a separate decision about who is trusted to break glass.

What is verified, and what is not

Verified locally on this branch:

• actionlint v1.7.12 is clean on every file touched here. Its one repository finding is pre-existing and in release.yml (queue: max under concurrency, a key it does not know).
• zizmor v1.29.0, run over the six workflows before and after: one finding removed (a template-injection error in frontend-tests.yml, from moving github.base_ref out of an inline expression into env:), none added.
• Every .github/scripts/*_test.py contract test passes, as does lint-action-pinning.py over all 18 workflows.
• changed-paths_test.py passes, and fails when the guarantees are broken: restoring a paths: filter to a gating trigger, deleting a merge_group trigger, and reordering !**/*.md above the includes were each confirmed to fail it.

Not verified, and not verifiable until a queue is switched on:

• No merge_group event has ever fired against these workflows. Everything about the merge-group path -- that github.event.merge_group.base_sha is populated as expected, that the changes job resolves a usable base from it, that the gate jobs report under the names in the table -- is read from GitHub's documented payload, not observed.
• Whether GitHub blocks a pull request from entering the queue on a check that reports nothing. The design assumes it does, which is why the filters moved; if it turns out not to, the change is still correct, just more cautious than it needed to be.
• The cost. Every pull request now starts six workflows instead of however many its paths selected, and each merge group runs the full E2E suite. The changes jobs keep the expensive work filtered, but the floor per pull request is higher than it was.

The cheapest way to close the first gap is to enable the queue on a low-traffic day and watch the first merge group's check names against the table above.