kandev/docs/ci-merge-queue.md

Merge queue

The main ruleset enabled the merge queue and six required checks on 2026-08-16. This document records the workflow preparation, active rules, and the Stable release exception.

Why

Before activation, main's ruleset (id 13341245) had deletion, non_fast_forward, required_linear_history, and pull_request rules. It had no required_status_checks rule.

That is how #2681 (38c94e7d0) landed red. Its E2E checks were fully green, but they had run 23 hours earlier against a merge ref that predated the retry loop added by #2569 (15788438b). Neither change is wrong on its own; together they put a roughly 4.5-second floor under content search, just over the 5-second default assertion in the test #2681 added. Both pull requests were green, and their merge was not.

Required status checks alone would not have caught it -- #2681's checks were green. A merge queue would: it builds the actual merge result and runs the checks against that, at merge time, in queue order.

What changed in the workflows

The merge_group trigger

Six workflows gained merge_group: types: [checks_requested]:

Workflow	Gate job	Check name to require
e2e-tests.yml	e2e-gate	E2E Tests Passed
backend-tests.yml	test	Run Backend Tests
frontend-tests.yml	frontend-gate	Frontend Tests Passed
architecture-lint.yml	lint	Architecture boundaries do not regress
lint-action-pinning.yml	lint	All action refs are SHA-pinned
lint-harness-files.yml	lint	Harness files pass lint

Deliberately excluded, because they cannot report in a queue and would stall it forever, or because their redness is not a property of the merge:

• pr-title.yml, claude-code-review.yml, opencode-code-review.yml, preview-env.yml, notify-docs.yml -- all pull_request or pull_request_target scoped. There is no pull request in a merge group, so they never run and never report.
• cargo-audit.yml -- also runs on a weekly schedule, by design: a newly published RUSTSEC advisory turns it red with no change to the code. In a queue that would block every merge in the repository until somebody bumped a crate. It stays a pull-request and scheduled check.
• ci-base-image.yml, universal-rebuild.yml -- these publish images rather than test a merge.
• release.yml and the plugin registry workflows -- out of scope.

The path-filter problem, and the approach chosen

Most of these workflows were heavily path-filtered (apps/web/**, !**/*.md, and so on). That is a problem for a queue in two independent ways.

1. A workflow skipped by a trigger-level paths: filter reports no conclusion at all. GitHub's documentation is explicit that "checks associated with that workflow will remain in a 'Pending' state". A required check that stays pending blocks a pull request from being added to the queue, and blocks a merge group from ever merging.
2. merge_group does not support paths: in the first place. It takes types: and branches: only. So the filters could not simply be mirrored onto the queue side even if we wanted them there.

Two approaches were available: mirror each filter onto an inverse trigger with a "skipped reports success" shim job, or make the gating jobs always run and short-circuit internally.

The shim was rejected, and the internal short-circuit implemented. The deciding factor is that paths with ! exclusions has no exact inverse. Take frontend-tests' old filter, apps/web/** followed by !**/*.md. A pull request that touches only apps/web/README.md is skipped by that filter, and is also skipped by the mirrored paths-ignore list, because every changed file appears in it. Both halves skip, no conclusion is ever reported, and the queue waits forever -- the exact failure the shim exists to prevent, reintroduced by the shim. Making the inversion correct would mean hand-deriving a second pattern list per workflow and keeping the two in sync by eye. Worse, the shim needs a job whose name matches the real check, which for e2e-tests.yml would mean reproducing fourteen shard names and six container shard names.

So: the trigger-level paths: filters are gone, and each expensive workflow grew a cheap changes job that applies the same list one level down.

changes ──> static_checks ─┐
        └─> test_shards ───┼──> test          <- the required check, if: always()
        └─> ...          ──┘

.github/scripts/changed-paths.py implements GitHub's filter-pattern semantics (*, **, ?, leading !, last match wins) and is unit-tested in changed-paths_test.py, which also asserts the pattern lists still live in the workflows and still select what they used to. It rejects +, [...], and {...} rather than quietly matching them as literals.

Two properties of this shape matter:

• The gate job always runs. It is if: always() and treats a skipped dependency as a pass and a cancelled one as a failure. This is on purpose: it means the required check never depends on how GitHub reports a conditionally-skipped job, only on a conclusion the workflow computes itself.
• The gate covers every job in its workflow. A queue gates on check names an admin picks by hand, so a job outside the gate's needs is a job whose failure would never block a merge, silently. backend-tests.yml's postgres-boot and test-windows were outside its aggregate before this change and are now inside it.

Two workflows -- lint-action-pinning.yml and lint-harness-files.yml -- have no changes job. They are a checkout plus a handful of short linters, so they simply always run: cheaper than the job that would decide whether to run them, and it closes the gap where a subject outside their hand-maintained lists changed and nothing checked it. Their contract tests were updated to assert the absence of a filter rather than the presence of specific entries.

External CI runner capacity

The normal Linux CI workflows have one activation switch, one percentage control, and two runner tiers. The switch and percentage are unset by default, so the workflows use GitHub-hosted runners.

Variable	Pilot value	Purpose
KANDEV_CI_EXTERNAL_ENABLED	unset	Set to true to allow external assignments
KANDEV_CI_EXTERNAL_PERCENT	unset or 0	Assign 0 to 100 percent of eligible jobs
KANDEV_CI_RUNNER_LIGHT	ubicloud-standard-2-ubuntu-2404	Control and aggregate jobs
KANDEV_CI_RUNNER_STANDARD	ubicloud-standard-4-ubuntu-2404	Eligible browser and frontend test jobs

Set KANDEV_CI_EXTERNAL_ENABLED to the exact value true when the Actions queue needs extra capacity. Set it to false, or remove it, after the queue returns to its normal level. The two tier variables stay configured between bursts, so changing the instance type requires only an Actions variable update.

Set KANDEV_CI_EXTERNAL_PERCENT to 50 for a pilot. The E2E matrix then sends seven of fourteen normal shards to the standard tier. Backend checkout and report-token jobs remain on GitHub-hosted runners; the aggregate backend gate and frontend jobs are eligible. Singleton jobs use a stable hash cohort, so their share approaches the percentage across workflow runs.

Leave the percentage unset for the default GitHub-only mode. Set it to 100 to send every eligible job with a non-empty tier label to external capacity. Malformed or out-of-range values fail closed to GitHub-hosted runners and emit a planner warning.

The reusable .github/actions/plan-external-runners composite action runs on the planner job's ubuntu-latest checkout. Each workflow declares its eligible families as JSON, and the action returns one JSON plan with resolved runner labels:

runs-on: ${{ fromJSON(needs.runner_plan.outputs.plan).frontend_runner }}
runs-on: ${{ matrix.runner }}

If a tier variable is empty, that tier uses ubuntu-latest. If a non-empty label is unavailable, GitHub leaves the job queued or fails it. Clear the invalid label and rerun the workflow. Jobs already queued or running keep their original runner. New jobs use the current variables.

The E2E build and report jobs, Docker/Kind shards, Kubernetes compatibility jobs, Playwright image job, desktop smoke job, backend checkout/static/test shards, Postgres service job, architecture and harness linters, and Windows job stay on GitHub-hosted runners. Release, publishing, signing, deployment, and credential-bearing jobs stay on their existing runners. The workflows do not add permissions, secrets, or persistent state.

The same switch and tier labels apply to eligible E2E, backend-gate, and frontend jobs. Architecture-lint, action-pinning, and harness-lint stay hosted and do not create planner jobs. The planner does not change job names, test selection, matrix values, artifacts, dependencies, timeouts, permissions, or required conclusions.

Pilot procedure:

1. Merge the workflow change while KANDEV_CI_EXTERNAL_ENABLED and KANDEV_CI_EXTERNAL_PERCENT are unset.
2. Install the Ubicloud GitHub App for this repository.
3. Set both tier variables to the pilot labels in the table. Leave Premium Runners disabled. Premium is an account-level setting, not a per-job tier.
4. Set KANDEV_CI_EXTERNAL_PERCENT=50 for the first pilot.
5. Set KANDEV_CI_EXTERNAL_ENABLED=true while the queue is full.
6. Compare at least three runs for queue delay, job duration, runner label, image setup, and failures.
7. Set the switch to false after the queue recovers. Keep the labels and percentage for the next burst.

Use the Ubicloud runner type reference when changing a tier. Use GitHub's runner selection reference for runs-on expressions. Do not enable Ubicloud Premium Runners for this pilot.

PR context in a merge group

A merge_group run has no github.event.pull_request and no github.event.before. Every expression that read one has been made explicit about the event it is on, and given the merge-group equivalent, github.event.merge_group.base_sha -- the commit the queue branch was cut from, which is the merge group's base:

• architecture-lint.yml -- the two mutually exclusive pull_request / push lint steps were collapsed into one, behind a baseline-resolving step. Under the old shape a merge-group run would have executed the linter's self-tests and then linted nothing at all, reporting green having compared the branch against no baseline.
• backend-tests.yml -- Resolve base SHA.
• frontend-tests.yml -- Check formatting, the i18n new-code ratchet, and the E2E sleep ratchet.
• e2e-tests.yml -- the GHCR login guard reads github.event_name != 'pull_request' || <fork check>, which resolves to the first operand on a merge group. That is correct: the queue branch lives in this repository and its token is a base-repository token.

Concurrency groups were checked too. None of the six keyed on a PR number (preview-env.yml does, and it is not a gate). All six now set cancel-in-progress: ${{ github.event_name != 'merge_group' }}, because a queue reads a cancelled check as a failure and ejects the pull request.

Active ruleset configuration

The main ruleset contains a merge_queue rule and requires the six gate names in the table above. It does not require per-shard names.

Do not require lint from pr-title.yml, Validate public docs, review checks, or deployment checks. These checks do not run in every merge group.

PR documentation coverage rollout

pr-docs.yml also listens for merge_group: checks_requested and publishes the PR documentation coverage status on the synthetic merge-group head. It maps the queue entry boundaries back to the event base and evaluates every included pull request with its own changed files, work-order references, and labels. A member's no-docs-allow label cannot exempt another member.

All PR and merge-group events use the target branch as one non-cancelling concurrency key with queue: max. GitHub retains up to 100 pending runs in that key, so ordinary event bursts do not replace the one pending run; events that arrive after the bound is full can still be canceled. This keeps a queued label reevaluation from racing with a merge-group status write while the published coverage status remains attached to the evaluated PR or synthetic group revision. A label removal reevaluates every active queue-group prefix that contains the PR, not only the longest group.

The status is not in the active required-check list above until live merge-group evidence is collected. Roll out the requirement in two stages:

1. Observe ordinary, draft, fork, label-add, label-remove, manual-retry, and merge-group runs. Confirm that covered, missing, override, and incomplete responses are attached to the evaluated revision and that all affected queue group prefixes reevaluate after label removal.
2. Add PR documentation coverage as a separate required status through an administrator ruleset change. Preserve the existing six checks and bypass rules, then verify a covered PR, an uncovered PR, an override, and a mixed merge group.

Until stage two is complete, a failing result is visible but does not itself block merging. The workflow does not add or remove labels, dequeue PRs, or change rulesets.

Stable release exception

The Stable release workflow creates a mechanical release PR with GITHUB_TOKEN. GitHub holds the PR workflows for approval, but maintainers do not require CI for this generated commit.

The protected release environment contains RELEASE_PR_BYPASS_TOKEN. This fine-grained personal access token belongs to an organization administrator and selects only kdlbs/kandev with contents: write permission.

The workflow exposes this token only to gh pr merge --admin. It also binds the merge to the expected PR head. GITHUB_TOKEN performs all other PR work.

The administrator merge bypasses the queue and required checks for the release PR only. Ordinary PRs still use the queue and all six required checks.

Caveat: the ruleset can still be bypassed

The main ruleset lists one bypass actor:

{ "actor_type": "OrganizationAdmin", "bypass_mode": "always" }

bypass_mode: always means an organization admin can merge past whatever is configured here, queue included, without a review request or a temporary override. Configuring required checks constrains everybody else; it does not constrain an org admin. If the point of enabling the queue is that nothing red reaches main, that entry needs to change as well -- to pull_request mode, or removed -- and that is a separate decision about who is trusted to break glass.

What is verified, and what is not

Verified locally on this branch:

• actionlint v1.7.12 is clean on every file touched here. Its one repository finding is pre-existing and in release.yml (queue: max under concurrency, a key it does not know).
• zizmor v1.29.0, run over the six workflows before and after: one finding removed (a template-injection error in frontend-tests.yml, from moving github.base_ref out of an inline expression into env:), none added.
• Every .github/scripts/*_test.py contract test passes, as does lint-action-pinning.py over all 18 workflows.
• changed-paths_test.py passes, and fails when the guarantees are broken: restoring a paths: filter to a gating trigger, deleting a merge_group trigger, and reordering !**/*.md above the includes were each confirmed to fail it.

Not verified, and not verifiable until a queue is switched on:

• No merge_group event has ever fired against these workflows. Everything about the merge-group path -- that github.event.merge_group.base_sha is populated as expected, that the changes job resolves a usable base from it, that the gate jobs report under the names in the table -- is read from GitHub's documented payload, not observed.
• Whether GitHub blocks a pull request from entering the queue on a check that reports nothing. The design assumes it does, which is why the filters moved; if it turns out not to, the change is still correct, just more cautious than it needed to be.
• The cost. Every pull request now starts six workflows instead of however many its paths selected, and each merge group runs the full E2E suite. The changes jobs keep the expensive work filtered, but the floor per pull request is higher than it was.

The cheapest way to close the first gap is to enable the queue on a low-traffic day and watch the first merge group's check names against the table above.