dreampiler/kandev
1
0
Fork 0
Code Issues 43 Pull Requests Actions 2 Packages Projects Releases Wiki Activity
Files
b1b28d72a3fe94462002fa3618ad7f547c8fd79d
kandev/docs/plans/external-e2e-runner-capacity/plan.md
T
Carlos Florêncio 89cddc2239 ci: add selectable external CI runners (#3451)
2026-09-06 14:45:59 +01:00

7.9 KiB
Raw Blame History

created, status, requirements, system_design, legacy_specs
created status requirements system_design legacy_specs
2026-09-06 complete
REQ-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-001
REQ-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-002
REQ-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-003
../../specs/platform/system-design/external-e2e-runner-capacity.md

Implementation Plan: External CI Runner Capacity

Overview

Add two provider-neutral runner tiers, one burst-mode switch, and an optional deterministic percentage rollout to the normal Linux CI workflows. Protect runner placement with workflow and planner contract tests. Update the merge-queue runbook with the Ubicloud configuration, activation, rollback, and measurement procedure.

Scope

In scope

  • Route eligible E2E and frontend jobs, plus the backend aggregate gate, through the light or standard runner tier.
  • Keep token-bearing E2E build/report, backend checkout/test, architecture, action-pinning, and harness jobs on GitHub-hosted runners.
  • Use KANDEV_CI_EXTERNAL_ENABLED as the single activation switch for both tiers.
  • Use KANDEV_CI_EXTERNAL_PERCENT to allocate 0 to 100 percent of eligible job instances when burst mode is active.
  • Keep configured tier labels unchanged while burst mode is inactive.
  • Keep playwright_image, e2e-containers, e2e-kubernetes-compatibility, and desktop-e2e explicitly on ubuntu-latest.
  • Keep backend Postgres/service, Windows, and other protected jobs on ubuntu-latest.
  • Add workflow and planner contract coverage for eligible and protected jobs.
  • Document the Ubicloud standard-2 and standard-4 pilot, manual rollback, and trust boundary in the merge-queue runbook.

Out of scope

  • Installing the Ubicloud GitHub App or changing live repository Actions variables from repository code.
  • Moving Docker/Kind, compatibility, image-resolution, desktop, Postgres service, Windows, release, publishing, signing, deployment, or pull_request_target jobs.
  • Adding a heavy tier before the first pilot supplies CPU or memory evidence.
  • Weighted or adaptive allocation based on historical runtime or queue metrics.
  • Guaranteeing an exact percentage of total compute across unrelated workflows.
  • Changing shard counts, test selection, worker counts, timeouts, artifact flow, permissions, required checks, or merge-queue rules.

Technical approach

Workflow placement

Eligible singleton jobs read a resolved runner label from the planner's JSON output:

runs-on: ${{ fromJSON(needs.runner_plan.outputs.plan).changes_runner }}

Matrix jobs read their matrix from that plan and use its runner field. Protected jobs keep an explicit ubuntu-latest value and do not depend on the planner.

For percentage rollout, add a read-only runner-plan.py planner job on ubuntu-latest. It validates the percentage and emits runner assignments for each eligible job family. Matrix families receive exactly floor(N * percentage / 100) external assignments. Singleton families use a stable hash bucket. Workflows declare their family data through a reusable composite action. Downstream jobs consume one JSON plan containing resolved runner labels and matrices.

If burst mode is inactive, eligible jobs use GitHub-hosted capacity. If burst mode is active, each job uses its configured tier. An empty tier uses ubuntu-latest.

Contract tests

Extend the E2E, backend, and frontend workflow contract tests with focused assertions for every affected job block. Add equivalent assertions that the architecture-lint, action-pinning, and harness-lint workflows stay hosted and do not create planner jobs.

Assert runs-on: ubuntu-latest in every protected block. Also assert that protected jobs do not reference the burst, percentage, or tier variables. Keep Ubicloud labels out of the workflow and contract tests.

Add .github/scripts/runner-plan_test.py for percentage validation, matrix floor allocation, singleton stability, rerun stability, empty labels, and fail-closed malformed input.

Operator runbook

Add an external-runner section to docs/ci-merge-queue.md. Explain that the workflow change is merged while burst mode is inactive. List both tier variables, their initial Ubicloud values, and the protected job categories.

Explain that one variable activates or deactivates paid capacity. State that variable changes affect only new jobs. Define the comparison and rollback signals. Link the GitHub and Ubicloud source documentation.

Tests

  • AC-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-001.1 through .4: the workflow contract test verifies the toggle, two tier variables, and GitHub fallback.
  • AC-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-001.5: existing E2E workflow contract tests remain unchanged and pass, protecting shard, cache, report, and gate behavior.
  • AC-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-001.6: the design and runbook document visible failure plus manual clear-and-rerun recovery. GitHub owns label scheduling behavior.
  • AC-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-002.1 and .2: the focused contract test verifies that protected jobs retain ubuntu-latest. Action pinning and zizmor verify that the workflow adds no unpinned action or new permission path.
  • AC-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-002.3: post-merge pilot evidence records selected labels, queue delay, job duration, and reliability for at least three representative runs.
  • AC-PLATFORM-EXTERNAL-E2E-RUNNER-CAPACITY-003.1 through .6: runner-plan_test.py and workflow contract tests verify 0/50/100 behavior, matrix floor allocation, stable singleton allocation, rerun stability, and fail-closed malformed input.

Work orders

Verification results

  • python3 .github/scripts/e2e-tests-workflow-contract_test.py: 14 tests passed.
  • python3 .github/scripts/external-runner-workflow-contract_test.py: 5 tests passed.
  • python3 .github/scripts/runner-plan_test.py: 8 tests passed.
  • python3 .github/scripts/frontend-tests-workflow-contract_test.py: 4 tests passed.
  • python3 .github/scripts/lint-action-pinning_test.py: 9 tests passed.
  • python3 .github/scripts/lint-action-pinning.py: all 21 workflow files passed.
  • actionlint v1.7.12: passed for all six touched workflows.
  • zizmor v1.25.2: no findings for all six touched workflows.
  • python3 scripts/lint-spec-files.py --all: passed.
  • git diff --check: passed.

The full zizmor .github/workflows scan can still report findings in existing workflows outside this change. The six touched workflows report no findings.

The Ubicloud pilot remains an operator action after merge. It needs at least three representative runs before a heavier tier or protected job moves.

Tasks 01, 02, and 03 are complete. The Ubicloud pilot remains an operator action after merge.

Risks

  • A misspelled or unavailable tier label can leave an eligible job queued until an operator clears the variable and reruns it.
  • An active burst with an empty tier label silently uses ubuntu-latest for that class. The runbook must include a pre-activation configuration check.
  • An external runner image can differ from GitHub's host image even when the normal build and shard jobs use repository-owned containers.
  • External transparent caching can change cache performance and poisoning risk. Branch protection must remain enabled for the public repository.
  • Moving the report and gate removes a small amount of GitHub-hosted demand but also makes their prompt completion depend on external capacity.
  • Runner capacity can reduce queue delay without improving merge-group serialization or slow test execution.
  • A planner job adds a small GitHub-hosted dependency before eligible jobs can start. Singleton percentage allocation is a cohort approximation, not an exact per-run share.
Reference in New Issue View Git Blame Copy Permalink