# Merge queue

The `main` ruleset enabled the merge queue and six required checks on
2026-08-16. This document records the workflow preparation, active rules, and
the Stable release exception.

## Why

Before activation, `main`'s ruleset (id `13341245`) had `deletion`,
`non_fast_forward`, `required_linear_history`, and `pull_request` rules. It had
no `required_status_checks` rule.

That is how #2681 (`38c94e7d0`) landed red. Its E2E checks were fully green, but
they had run 23 hours earlier against a merge ref that predated the retry loop
added by #2569 (`15788438b`). Neither change is wrong on its own; together they
put a roughly 4.5-second floor under content search, just over the 5-second
default assertion in the test #2681 added. Both pull requests were green, and
their merge was not.

Required status checks alone would not have caught it -- #2681's checks were
green. A merge queue would: it builds the actual merge result and runs the
checks against that, at merge time, in queue order.

## What changed in the workflows

### The `merge_group` trigger

Six workflows gained `merge_group: types: [checks_requested]`:

| Workflow                 | Gate job       | Check name to require                   |
| ------------------------ | -------------- | --------------------------------------- |
| `e2e-tests.yml`          | `e2e-gate`     | `E2E Tests Passed`                      |
| `backend-tests.yml`      | `test`         | `Run Backend Tests`                     |
| `frontend-tests.yml`     | `frontend-gate`| `Frontend Tests Passed`                 |
| `architecture-lint.yml`  | `lint`         | `Architecture boundaries do not regress`|
| `lint-action-pinning.yml`| `lint`         | `All action refs are SHA-pinned`        |
| `lint-harness-files.yml` | `lint`         | `Harness files pass lint`               |

Deliberately excluded, because they cannot report in a queue and would stall it
forever, or because their redness is not a property of the merge:

- `pr-title.yml`, `claude-code-review.yml`, `opencode-code-review.yml`,
  `preview-env.yml`, `notify-docs.yml` -- all `pull_request` or
  `pull_request_target` scoped. There is no pull request in a merge group, so
  they never run and never report.
- `cargo-audit.yml` -- also runs on a weekly `schedule`, by design: a newly
  published RUSTSEC advisory turns it red with no change to the code. In a queue
  that would block every merge in the repository until somebody bumped a crate.
  It stays a pull-request and scheduled check.
- `ci-base-image.yml`, `universal-rebuild.yml` -- these publish images rather
  than test a merge.
- `release.yml` and the plugin registry workflows -- out of scope.

### The path-filter problem, and the approach chosen

Most of these workflows were heavily path-filtered (`apps/web/**`, `!**/*.md`,
and so on). That is a problem for a queue in two independent ways.

1. A workflow skipped by a trigger-level `paths:` filter reports **no
   conclusion at all**. GitHub's documentation is explicit that "checks
   associated with that workflow will remain in a 'Pending' state". A required
   check that stays pending blocks a pull request from being added to the queue,
   and blocks a merge group from ever merging.
2. `merge_group` does not support `paths:` in the first place. It takes `types:`
   and `branches:` only. So the filters could not simply be mirrored onto the
   queue side even if we wanted them there.

Two approaches were available: mirror each filter onto an inverse trigger with a
"skipped reports success" shim job, or make the gating jobs always run and
short-circuit internally.

**The shim was rejected, and the internal short-circuit implemented.** The
deciding factor is that `paths` with `!` exclusions has no exact inverse. Take
frontend-tests' old filter, `apps/web/**` followed by `!**/*.md`. A pull request
that touches only `apps/web/README.md` is skipped by that filter, and is *also*
skipped by the mirrored `paths-ignore` list, because every changed file appears
in it. Both halves skip, no conclusion is ever reported, and the queue waits
forever -- the exact failure the shim exists to prevent, reintroduced by the
shim. Making the inversion correct would mean hand-deriving a second pattern
list per workflow and keeping the two in sync by eye. Worse, the shim needs a
job whose *name* matches the real check, which for `e2e-tests.yml` would mean
reproducing fourteen shard names and six container shard names.

So: the trigger-level `paths:` filters are gone, and each expensive workflow
grew a cheap `changes` job that applies the same list one level down.

```
changes ──> static_checks ─┐
        └─> test_shards ───┼──> test          <- the required check, if: always()
        └─> ...          ──┘
```

`.github/scripts/changed-paths.py` implements GitHub's filter-pattern semantics
(`*`, `**`, `?`, leading `!`, last match wins) and is unit-tested in
`changed-paths_test.py`, which also asserts the pattern lists still live in the
workflows and still select what they used to. It rejects `+`, `[...]`, and
`{...}` rather than quietly matching them as literals.

Two properties of this shape matter:

- **The gate job always runs.** It is `if: always()` and treats a `skipped`
  dependency as a pass and a `cancelled` one as a failure. This is on purpose:
  it means the required check never depends on how GitHub reports a
  conditionally-skipped job, only on a conclusion the workflow computes itself.
- **The gate covers every job in its workflow.** A queue gates on check names an
  admin picks by hand, so a job outside the gate's `needs` is a job whose
  failure would never block a merge, silently. `backend-tests.yml`'s
  `postgres-boot` and `test-windows` were outside its aggregate before this
  change and are now inside it.

Two workflows -- `lint-action-pinning.yml` and `lint-harness-files.yml` -- have
no `changes` job. They are a checkout plus a handful of short linters, so they
simply always run: cheaper than the job that would decide whether to run them,
and it closes the gap where a subject outside their hand-maintained lists
changed and nothing checked it. Their contract tests were updated to assert the
absence of a filter rather than the presence of specific entries.

### External CI runner capacity

The normal Linux CI workflows have one activation switch, one percentage
control, and two runner tiers. The switch and percentage are unset by default,
so the workflows use GitHub-hosted runners.

| Variable | Pilot value | Purpose |
| -------- | ----------- | ------- |
| `KANDEV_CI_EXTERNAL_ENABLED` | unset | Set to `true` to allow external assignments |
| `KANDEV_CI_EXTERNAL_PERCENT` | unset or `0` | Assign 0 to 100 percent of eligible jobs |
| `KANDEV_CI_RUNNER_LIGHT` | `ubicloud-standard-2-ubuntu-2404` | Control and aggregate jobs |
| `KANDEV_CI_RUNNER_STANDARD` | `ubicloud-standard-4-ubuntu-2404` | Eligible browser and frontend test jobs |

Set `KANDEV_CI_EXTERNAL_ENABLED` to the exact value `true` when the Actions queue
needs extra capacity. Set it to `false`, or remove it, after the queue returns
to its normal level. The two tier variables stay configured between bursts, so
changing the instance type requires only an Actions variable update.

Set `KANDEV_CI_EXTERNAL_PERCENT` to `50` for a pilot. The E2E matrix then sends
seven of fourteen normal shards to the standard tier. Backend checkout and
report-token jobs remain on GitHub-hosted runners; the aggregate backend gate
and frontend jobs are eligible. Singleton jobs use a stable hash cohort, so
their share approaches the percentage across workflow runs.

Leave the percentage unset for the default GitHub-only mode. Set it to `100`
to send every eligible job with a non-empty tier label to external capacity.
Malformed or out-of-range values fail closed to GitHub-hosted runners and emit
a planner warning.

The reusable `.github/actions/plan-external-runners` composite action runs on
the planner job's `ubuntu-latest` checkout. Each workflow declares its eligible
families as JSON, and the action returns one JSON plan with resolved runner
labels:

```yaml
runs-on: ${{ fromJSON(needs.runner_plan.outputs.plan).frontend_runner }}
runs-on: ${{ matrix.runner }}
```

If a tier variable is empty, that tier uses `ubuntu-latest`. If a non-empty
label is unavailable, GitHub leaves the job queued or fails it. Clear the
invalid label and rerun the workflow. Jobs already queued or running keep their
original runner. New jobs use the current variables.

The E2E build and report jobs, Docker/Kind shards, Kubernetes compatibility
jobs, Playwright image job, desktop smoke job, backend checkout/static/test
shards, Postgres service job, architecture and harness linters, and Windows job
stay on GitHub-hosted runners. Release, publishing, signing, deployment, and
credential-bearing jobs stay on their existing runners. The workflows do not
add permissions, secrets, or persistent state.

The same switch and tier labels apply to eligible E2E, backend-gate, and
frontend jobs. Architecture-lint, action-pinning, and harness-lint stay hosted
and do not create planner jobs. The planner does not change job names, test
selection, matrix values, artifacts, dependencies, timeouts, permissions, or
required conclusions.

Pilot procedure:

1. Merge the workflow change while `KANDEV_CI_EXTERNAL_ENABLED` and
   `KANDEV_CI_EXTERNAL_PERCENT` are unset.
2. Install the [Ubicloud GitHub App](https://www.ubicloud.com/docs/github-actions-integration) for this repository.
3. Set both tier variables to the pilot labels in the table. Leave Premium
   Runners disabled. Premium is an account-level setting, not a per-job tier.
4. Set `KANDEV_CI_EXTERNAL_PERCENT=50` for the first pilot.
5. Set `KANDEV_CI_EXTERNAL_ENABLED=true` while the queue is full.
6. Compare at least three runs for queue delay, job duration, runner label,
   image setup, and failures.
7. Set the switch to `false` after the queue recovers. Keep the labels and
   percentage for the next burst.

Use the [Ubicloud runner type reference](https://www.ubicloud.com/docs/github-actions-integration/runner-types) when changing a tier. Use GitHub's [runner selection reference](https://docs.github.com/en/actions/how-tos/write-workflows/choose-where-workflows-run/choose-the-runner-for-a-job) for `runs-on` expressions. Do not enable [Ubicloud Premium Runners](https://www.ubicloud.com/docs/github-actions-integration/use-premium-runners) for this pilot.

### PR context in a merge group

A `merge_group` run has no `github.event.pull_request` and no
`github.event.before`. Every expression that read one has been made explicit
about the event it is on, and given the merge-group equivalent,
`github.event.merge_group.base_sha` -- the commit the queue branch was cut from,
which is the merge group's base:

- `architecture-lint.yml` -- the two mutually exclusive `pull_request` / `push`
  lint steps were collapsed into one, behind a baseline-resolving step. Under
  the old shape a merge-group run would have executed the linter's self-tests
  and then linted nothing at all, reporting green having compared the branch
  against no baseline.
- `backend-tests.yml` -- `Resolve base SHA`.
- `frontend-tests.yml` -- `Check formatting`, the i18n new-code ratchet, and the
  E2E sleep ratchet.
- `e2e-tests.yml` -- the GHCR login guard reads
  `github.event_name != 'pull_request' || <fork check>`, which resolves to the
  first operand on a merge group. That is correct: the queue branch lives in
  this repository and its token is a base-repository token.

Concurrency groups were checked too. None of the six keyed on a PR number
(`preview-env.yml` does, and it is not a gate). All six now set
`cancel-in-progress: ${{ github.event_name != 'merge_group' }}`, because a queue
reads a cancelled check as a failure and ejects the pull request.

## Active ruleset configuration

The `main` ruleset contains a `merge_queue` rule and requires the six gate names
in the table above. It does not require per-shard names.

Do not require `lint` from `pr-title.yml`, `Validate public docs`, review checks,
or deployment checks. These checks do not run in every merge group.

### PR documentation coverage rollout

`pr-docs.yml` also listens for `merge_group: checks_requested` and publishes the
`PR documentation coverage` status on the synthetic merge-group head. It maps
the queue entry boundaries back to the event base and evaluates every included
pull request with its own changed files, work-order references, and labels. A
member's `no-docs-allow` label cannot exempt another member.

All PR and merge-group events use the target branch as one non-cancelling
concurrency key with `queue: max`. GitHub retains up to 100 pending runs in
that key, so ordinary event bursts do not replace the one pending run; events
that arrive after the bound is full can still be canceled. This keeps a queued
label reevaluation from racing with a merge-group status write while the
published coverage status remains attached to the evaluated PR or synthetic
group revision. A label removal reevaluates every active queue-group prefix
that contains the PR, not only the longest group.

The status is not in the active required-check list above until live merge-group
evidence is collected. Roll out the requirement in two stages:

1. Observe ordinary, draft, fork, label-add, label-remove, manual-retry, and
   merge-group runs. Confirm that covered, missing, override, and incomplete
   responses are attached to the evaluated revision and that all affected queue
   group prefixes reevaluate after label removal.
2. Add `PR documentation coverage` as a separate required status through an
   administrator ruleset change. Preserve the existing six checks and bypass
   rules, then verify a covered PR, an uncovered PR, an override, and a mixed
   merge group.

Until stage two is complete, a failing result is visible but does not itself
block merging. The workflow does not add or remove labels, dequeue PRs, or
change rulesets.

### Stable release exception

The Stable release workflow creates a mechanical release PR with `GITHUB_TOKEN`.
GitHub holds the PR workflows for approval, but maintainers do not require CI
for this generated commit.

The protected `release` environment contains `RELEASE_PR_BYPASS_TOKEN`. This
fine-grained personal access token belongs to an organization administrator and
selects only `kdlbs/kandev` with `contents: write` permission.

The workflow exposes this token only to `gh pr merge --admin`. It also binds
the merge to the expected PR head. `GITHUB_TOKEN` performs all other PR work.

The administrator merge bypasses the queue and required checks for the release
PR only. Ordinary PRs still use the queue and all six required checks.

### Caveat: the ruleset can still be bypassed

The `main` ruleset lists one bypass actor:

```json
{ "actor_type": "OrganizationAdmin", "bypass_mode": "always" }
```

`bypass_mode: always` means an organization admin can merge past whatever is
configured here, queue included, without a review request or a temporary
override. Configuring required checks constrains everybody else; it does not
constrain an org admin. If the point of enabling the queue is that nothing red
reaches `main`, that entry needs to change as well -- to `pull_request` mode, or
removed -- and that is a separate decision about who is trusted to break glass.

## What is verified, and what is not

Verified locally on this branch:

- `actionlint` v1.7.12 is clean on every file touched here. Its one repository
  finding is pre-existing and in `release.yml` (`queue: max` under
  `concurrency`, a key it does not know).
- `zizmor` v1.29.0, run over the six workflows before and after: one finding
  removed (a `template-injection` error in `frontend-tests.yml`, from moving
  `github.base_ref` out of an inline expression into `env:`), none added.
- Every `.github/scripts/*_test.py` contract test passes, as does
  `lint-action-pinning.py` over all 18 workflows.
- `changed-paths_test.py` passes, and fails when the guarantees are broken:
  restoring a `paths:` filter to a gating trigger, deleting a `merge_group`
  trigger, and reordering `!**/*.md` above the includes were each confirmed to
  fail it.

Not verified, and not verifiable until a queue is switched on:

- **No `merge_group` event has ever fired against these workflows.** Everything
  about the merge-group path -- that `github.event.merge_group.base_sha` is
  populated as expected, that the `changes` job resolves a usable base from it,
  that the gate jobs report under the names in the table -- is read from
  GitHub's documented payload, not observed.
- Whether GitHub blocks a pull request from *entering* the queue on a check that
  reports nothing. The design assumes it does, which is why the filters moved;
  if it turns out not to, the change is still correct, just more cautious than
  it needed to be.
- The cost. Every pull request now starts six workflows instead of however many
  its paths selected, and each merge group runs the full E2E suite. The
  `changes` jobs keep the expensive work filtered, but the floor per pull
  request is higher than it was.

The cheapest way to close the first gap is to enable the queue on a low-traffic
day and watch the first merge group's check names against the table above.