kandev/docs/ci-merge-queue.md

# Merge queue

The `main` ruleset enabled the merge queue and six required checks on
2026-08-16. This document records the workflow preparation, active rules, and
the Stable release exception.

### Claude execution budget

The automatic same-repository review, approved fork review, and interactive
Claude jobs each have a 30-minute GitHub job timeout. The timeout bounds runner
execution and a cancelled job cannot report a successful review. It does not
bound approval or queue waits. An approval that expires before a job starts is
an approval outcome, and must be measured separately from Claude execution.

## Why

Before activation, `main`'s ruleset (id `13341245`) had `deletion`,
`non_fast_forward`, `required_linear_history`, and `pull_request` rules. It had
no `required_status_checks` rule.

That is how #2681 (`38c94e7d0`) landed red. Its E2E checks were fully green, but
they had run 23 hours earlier against a merge ref that predated the retry loop
added by #2569 (`15788438b`). Neither change is wrong on its own; together they
put a roughly 4.5-second floor under content search, just over the 5-second
default assertion in the test #2681 added. Both pull requests were green, and
their merge was not.

Required status checks alone would not have caught it -- #2681's checks were
green. A merge queue would: it builds the actual merge result and runs the
checks against that, at merge time, in queue order.

## What changed in the workflows

### The `merge_group` trigger

Six workflows gained `merge_group: types: [checks_requested]`:

| Workflow                 | Gate job       | Check name to require                   |
| ------------------------ | -------------- | --------------------------------------- |
| `e2e-tests.yml`          | `e2e-gate`     | `E2E Tests Passed`                      |
| `backend-tests.yml`      | `test`         | `Run Backend Tests`                     |
| `frontend-tests.yml`     | `frontend-gate`| `Frontend Tests Passed`                 |
| `architecture-lint.yml`  | `lint`         | `Architecture boundaries do not regress`|
| `lint-action-pinning.yml`| `lint`         | `All action refs are SHA-pinned`        |
| `lint-harness-files.yml` | `lint`         | `Harness files pass lint`               |

Deliberately excluded, because they cannot report in a queue and would stall it
forever, or because their redness is not a property of the merge:

- `pr-title.yml`, `claude-code-review.yml`, `opencode-code-review.yml`,
  `preview-env.yml`, `notify-docs.yml` -- all `pull_request` or
  `pull_request_target` scoped. There is no pull request in a merge group, so
  they never run and never report.
- `cargo-audit.yml` -- also runs on a weekly `schedule`, by design: a newly
  published RUSTSEC advisory turns it red with no change to the code. In a queue
  that would block every merge in the repository until somebody bumped a crate.
  It stays a pull-request and scheduled check.
- `ci-base-image.yml`, `universal-rebuild.yml` -- these publish images rather
  than test a merge.
- `release.yml` and the plugin registry workflows -- out of scope.

### The path-filter problem, and the approach chosen

Most of these workflows were heavily path-filtered (`apps/web/**`, `!**/*.md`,
and so on). That is a problem for a queue in two independent ways.

1. A workflow skipped by a trigger-level `paths:` filter reports **no
   conclusion at all**. GitHub's documentation is explicit that "checks
   associated with that workflow will remain in a 'Pending' state". A required
   check that stays pending blocks a pull request from being added to the queue,
   and blocks a merge group from ever merging.
2. `merge_group` does not support `paths:` in the first place. It takes `types:`
   and `branches:` only. So the filters could not simply be mirrored onto the
   queue side even if we wanted them there.

Two approaches were available: mirror each filter onto an inverse trigger with a
"skipped reports success" shim job, or make the gating jobs always run and
short-circuit internally.

**The shim was rejected, and the internal short-circuit implemented.** The
deciding factor is that `paths` with `!` exclusions has no exact inverse. Take
frontend-tests' old filter, `apps/web/**` followed by `!**/*.md`. A pull request
that touches only `apps/web/README.md` is skipped by that filter, and is *also*
skipped by the mirrored `paths-ignore` list, because every changed file appears
in it. Both halves skip, no conclusion is ever reported, and the queue waits
forever -- the exact failure the shim exists to prevent, reintroduced by the
shim. Making the inversion correct would mean hand-deriving a second pattern
list per workflow and keeping the two in sync by eye. Worse, the shim needs a
job whose *name* matches the real check, which for `e2e-tests.yml` would mean
reproducing fourteen shard names and six container shard names.

So: the trigger-level `paths:` filters are gone, and each expensive workflow
grew a cheap `changes` job that applies the same list one level down.

```
changes ──> static_checks ─┐
        └─> test_shards ───┼──> test          <- the required check, if: always()
        └─> ...          ──┘
```

`.github/scripts/changed-paths.py` implements GitHub's filter-pattern semantics
(`*`, `**`, `?`, leading `!`, last match wins) and is unit-tested in
`changed-paths_test.py`, which also asserts the pattern lists still live in the
workflows and still select what they used to. It rejects `+`, `[...]`, and
`{...}` rather than quietly matching them as literals.

Two properties of this shape matter:

- **The gate job always runs.** It is `if: always()` and treats a `skipped`
  dependency as a pass and a `cancelled` one as a failure. This is on purpose:
  it means the required check never depends on how GitHub reports a
  conditionally-skipped job, only on a conclusion the workflow computes itself.
- **The gate covers every job in its workflow.** A queue gates on check names an
  admin picks by hand, so a job outside the gate's `needs` is a job whose
  failure would never block a merge, silently. `backend-tests.yml`'s
  `postgres-boot` and `test-windows` were outside its aggregate before this
  change and are now inside it.

Two workflows -- `lint-action-pinning.yml` and `lint-harness-files.yml` -- have
no `changes` job. They are a checkout plus a handful of short linters, so they
simply always run: cheaper than the job that would decide whether to run them,
and it closes the gap where a subject outside their hand-maintained lists
changed and nothing checked it. Their contract tests were updated to assert the
absence of a filter rather than the presence of specific entries.

### External CI runner capacity

The normal Linux CI workflows have one activation switch, one percentage
control, and two runner tiers. The current read-only snapshot on 2026-09-12
is shown below. The switch is off, so current jobs use GitHub-hosted runners;
the retained percentage is the next pilot value.

| Variable | Current value | Pilot value | Purpose |
| -------- | ------------ | ----------- | ------- |
| `KANDEV_CI_EXTERNAL_ENABLED` | `false` | `true` | Allow external assignments |
| `KANDEV_CI_EXTERNAL_PERCENT` | `20` | `20` | Assign 0 to 100 percent of eligible job instances |
| `KANDEV_CI_RUNNER_LIGHT` | `ubicloud-standard-2-ubuntu-2404` | same | Control and aggregate jobs |
| `KANDEV_CI_RUNNER_STANDARD` | `ubicloud-standard-4-ubuntu-2404` | same | Eligible browser and frontend test jobs |

Set `KANDEV_CI_EXTERNAL_ENABLED` to the exact value `true` when the Actions queue
needs extra capacity. Set it to `false` after the queue returns to its normal
level. The two tier variables stay configured between bursts, so changing the
instance type requires only an Actions variable update.

The 20% pilot sends a stable cohort of eligible instances to the configured
tiers. For E2E this assigns two of fourteen normal shards per run because matrix
allocation uses a floor. The frontend workflow has two unit-test instances, so
the 20% pilot assigns zero of those instances; its singleton static and gate
jobs still use stable hash cohorts. Backend checkout, static checks,
service jobs, Windows, and the protected test shards remain on GitHub-hosted
runners. Singleton jobs use a stable hash cohort, so their share approaches the
percentage across workflow runs.

Set the percentage to `0` for an explicit GitHub-only allocation. Set it to
`100` to send every eligible job with a non-empty tier label to external
capacity. Malformed or out-of-range values fail closed to GitHub-hosted runners
and emit a planner warning.

The reusable `.github/actions/plan-external-runners` composite action runs on
the planner job's `ubuntu-latest` checkout. Each workflow declares its eligible
families as JSON, and the action returns one JSON plan with resolved runner
labels:

```yaml
runs-on: ${{ fromJSON(needs.runner_plan.outputs.plan).frontend_runner }}
runs-on: ${{ matrix.runner }}
```

If a tier variable is empty, that tier uses `ubuntu-latest`. If a non-empty
label is unavailable, GitHub leaves the job queued or fails it. Clear the
invalid label and rerun the workflow. Jobs already queued or running keep their
original runner. New jobs use the current variables.

The eligible families are the E2E normal-shard matrix and gate, the backend
aggregate test gate, the frontend static verification job, the frontend unit
test matrix, and their lightweight change-detection and gate jobs. The E2E build
and report jobs, Docker/Kind shards, Kubernetes compatibility jobs, Playwright
image job, desktop smoke job, backend checkout/static/test shards, Postgres
service job, architecture and harness linters, and Windows job stay on
GitHub-hosted runners. Release, publishing, signing, deployment, and
credential-bearing jobs stay on their existing runners. The workflows do not
add permissions, secrets, or persistent state.

The same switch and tier labels apply to eligible E2E, backend-gate, frontend
static verification, frontend unit-test matrix, and frontend gate jobs.
Architecture-lint, action-pinning, and harness-lint stay hosted and do not
create planner jobs. The planner does not change job names, test selection,
matrix values, artifacts, dependencies, timeouts, permissions, or required
conclusions.

Pilot procedure. An operator must supply an approved provider budget and the
provider's current rate before activation. Unknown rates, fees, or budget are
an activation blocker.

1. Merge the workflow change while `KANDEV_CI_EXTERNAL_ENABLED=false`.
2. Install the [Ubicloud GitHub App](https://www.ubicloud.com/docs/github-actions-integration) for this repository.
3. Confirm both tier variables match the approved labels. Leave Premium Runners
   disabled. Premium is an account-level setting, not a per-job tier.
4. Record the current variable snapshot and provider budget in the run record.
5. Activate the 20% pilot for newly dispatched workflows:

   ```bash
   gh variable set KANDEV_CI_EXTERNAL_ENABLED --repo kdlbs/kandev --body true
   gh variable set KANDEV_CI_EXTERNAL_PERCENT --repo kdlbs/kandev --body 20
   ```

6. Compare at least three representative frontend, E2E, or backend runs. For
   each eligible job record workflow run attempt, runner label, created time,
   `started_at`, `completed_at`, queue delay, execution duration, setup/cache
   steps, conclusion, and billed runner minutes. Keep protected hosted jobs in
   the comparison so queue relief is not confused with code speed.
7. Stop the pilot for newly dispatched workflows after the sample or when the
   budget is reached:

   ```bash
   gh variable set KANDEV_CI_EXTERNAL_ENABLED --repo kdlbs/kandev --body false
   gh variable set KANDEV_CI_EXTERNAL_PERCENT --repo kdlbs/kandev --body 0
   ```

   Jobs already queued or running keep their original runner assignment.

Cost calculation for each pilot sample:

```text
external runner-minutes = sum(external job execution seconds) / 60
external cost = external runner-minutes / 60 * provider hourly rate
                + provider setup, storage, or transfer fees
```

Compare that cost with the GitHub-hosted minute accounting and record the
provider rate, billing granularity, and any free allowance. The percentage
controls assignment of job instances; it does not promise the same percentage
of billed minutes. Do not expand the percentage until three comparable runs
show queue relief without an unacceptable failure rate, execution regression,
or budget overrun.

Use the [Ubicloud runner type reference](https://www.ubicloud.com/docs/github-actions-integration/runner-types) when changing a tier. Use GitHub's [runner selection reference](https://docs.github.com/en/actions/how-tos/write-workflows/choose-where-workflows-run/choose-the-runner-for-a-job) for `runs-on` expressions. Do not enable [Ubicloud Premium Runners](https://www.ubicloud.com/docs/github-actions-integration/use-premium-runners) for this pilot.

### PR context in a merge group

A `merge_group` run has no `github.event.pull_request` and no
`github.event.before`. Every expression that read one has been made explicit
about the event it is on, and given the merge-group equivalent,
`github.event.merge_group.base_sha` -- the commit the queue branch was cut from,
which is the merge group's base:

- `architecture-lint.yml` -- the two mutually exclusive `pull_request` / `push`
  lint steps were collapsed into one, behind a baseline-resolving step. Under
  the old shape a merge-group run would have executed the linter's self-tests
  and then linted nothing at all, reporting green having compared the branch
  against no baseline.
- `backend-tests.yml` -- `Resolve base SHA`.
- `frontend-tests.yml` -- `Check formatting`, the i18n new-code ratchet, and the
  E2E sleep ratchet.
- `e2e-tests.yml` -- the GHCR login guard reads
  `github.event_name != 'pull_request' || <fork check>`, which resolves to the
  first operand on a merge group. That is correct: the queue branch lives in
  this repository and its token is a base-repository token.

Concurrency groups were checked too. None of the six keyed on a PR number
(`preview-env.yml` does, and it is not a gate). All six now set
`cancel-in-progress: ${{ github.event_name != 'merge_group' }}`, because a queue
reads a cancelled check as a failure and ejects the pull request.

## Active ruleset configuration

The `main` ruleset contains a `merge_queue` rule and requires the six gate names
in the table above. It does not require per-shard names.

Do not require `lint` from `pr-title.yml`, `Validate public docs`, review checks,
or deployment checks. These checks do not run in every merge group.

### PR documentation coverage rollout

`pr-docs.yml` also listens for `merge_group: checks_requested` and publishes the
`PR documentation coverage` status on the synthetic merge-group head. It maps
the queue entry boundaries back to the event base and evaluates every included
pull request with its own changed files, work-order references, and labels. A
member's `no-docs-allow` label cannot exempt another member.

All PR and merge-group events use the target branch as one non-cancelling
concurrency key with `queue: max`. GitHub retains up to 100 pending runs in
that key, so ordinary event bursts do not replace the one pending run; events
that arrive after the bound is full can still be canceled. This keeps a queued
label reevaluation from racing with a merge-group status write while the
published coverage status remains attached to the evaluated PR or synthetic
group revision. A label removal reevaluates every active queue-group prefix
that contains the PR, not only the longest group.

The status is not in the active required-check list above until live merge-group
evidence is collected. Roll out the requirement in two stages:

1. Observe ordinary, draft, fork, label-add, label-remove, manual-retry, and
   merge-group runs. Confirm that covered, missing, override, and incomplete
   responses are attached to the evaluated revision and that all affected queue
   group prefixes reevaluate after label removal.
2. Add `PR documentation coverage` as a separate required status through an
   administrator ruleset change. Preserve the existing six checks and bypass
   rules, then verify a covered PR, an uncovered PR, an override, and a mixed
   merge group.

Until stage two is complete, a failing result is visible but does not itself
block merging. The workflow does not add or remove labels, dequeue PRs, or
change rulesets.

### Stable release exception

The Stable release workflow creates a mechanical release PR with `GITHUB_TOKEN`.
GitHub holds the PR workflows for approval, but maintainers do not require CI
for this generated commit.

The protected `release` environment contains `RELEASE_PR_BYPASS_TOKEN`. This
fine-grained personal access token belongs to an organization administrator and
selects only `kdlbs/kandev` with `contents: write` permission.

The workflow exposes this token only to `gh pr merge --admin`. It also binds
the merge to the expected PR head. `GITHUB_TOKEN` performs all other PR work.

The administrator merge bypasses the queue and required checks for the release
PR only. Ordinary PRs still use the queue and all six required checks.

### Caveat: the ruleset can still be bypassed

The `main` ruleset lists one bypass actor:

```json
{ "actor_type": "OrganizationAdmin", "bypass_mode": "always" }
```

`bypass_mode: always` means an organization admin can merge past whatever is
configured here, queue included, without a review request or a temporary
override. Configuring required checks constrains everybody else; it does not
constrain an org admin. If the point of enabling the queue is that nothing red
reaches `main`, that entry needs to change as well -- to `pull_request` mode, or
removed -- and that is a separate decision about who is trusted to break glass.

## What is verified, and what is not

Verified locally on this branch:

- `actionlint` v1.7.12 is clean on every file touched here. Its one repository
  finding is pre-existing and in `release.yml` (`queue: max` under
  `concurrency`, a key it does not know).
- `zizmor` v1.29.0, run over the six workflows before and after: one finding
  removed (a `template-injection` error in `frontend-tests.yml`, from moving
  `github.base_ref` out of an inline expression into `env:`), none added.
- Every `.github/scripts/*_test.py` contract test passes, as does
  `lint-action-pinning.py` over all 18 workflows.
- `changed-paths_test.py` passes, and fails when the guarantees are broken:
  restoring a `paths:` filter to a gating trigger, deleting a `merge_group`
  trigger, and reordering `!**/*.md` above the includes were each confirmed to
  fail it.

Not verified, and not verifiable until a queue is switched on:

- **No `merge_group` event has ever fired against these workflows.** Everything
  about the merge-group path -- that `github.event.merge_group.base_sha` is
  populated as expected, that the `changes` job resolves a usable base from it,
  that the gate jobs report under the names in the table -- is read from
  GitHub's documented payload, not observed.
- Whether GitHub blocks a pull request from *entering* the queue on a check that
  reports nothing. The design assumes it does, which is why the filters moved;
  if it turns out not to, the change is still correct, just more cautious than
  it needed to be.
- The cost. Every pull request now starts six workflows instead of however many
  its paths selected, and each merge group runs the full E2E suite. The
  `changes` jobs keep the expensive work filtered, but the floor per pull
  request is higher than it was.

The cheapest way to close the first gap is to enable the queue on a low-traffic
day and watch the first merge group's check names against the table above.