159 lines
7.6 KiB
Markdown
159 lines
7.6 KiB
Markdown
---
|
|
spec: ../../specs/platform/requirements/e2e-duration-aware-sharding.md
|
|
created: 2026-08-30
|
|
status: in_progress
|
|
---
|
|
# Implementation Plan: E2E CI setup and flake efficiency
|
|
|
|
## Objective
|
|
|
|
Reduce the E2E critical path after the duration-aware planner has already
|
|
removed most test-work imbalance. This package reuses verified Playwright
|
|
browsers on host-runner container jobs and removes one recurring false flake
|
|
from the review-diff regression test. It preserves the fourteen normal shards,
|
|
the six container shards, one worker per shard, existing retries, and the
|
|
required aggregate gate.
|
|
|
|
## Evidence and diagnosis
|
|
|
|
The recent merged E2E stabilization changes are relevant inputs:
|
|
|
|
- [PR #3150](https://github.com/kdlbs/kandev/pull/3150) added a causal layout
|
|
wait for the PR status badge hover. [PR #3151](https://github.com/kdlbs/kandev/pull/3151)
|
|
then narrowed an automation-trigger notification assertion. Both are recent
|
|
examples of synchronization fixes that should be preferred over longer
|
|
waits.
|
|
- [PR #3133](https://github.com/kdlbs/kandev/pull/3133) removed shared-worker
|
|
cleanup races and reinforced task-owned test state.
|
|
- A fair successful-main run, [run 33250014029](https://github.com/kdlbs/kandev/actions/runs/33250014029),
|
|
had a roughly 16m28s normal-shard tail after the build completed. Its six
|
|
container jobs repeated browser provisioning.
|
|
- In [run 33308951784](https://github.com/kdlbs/kandev/actions/runs/33308951784),
|
|
the predicted-versus-actual normal shards stayed within the existing 1.25x
|
|
cohort-median target and the container cohort was also balanced. The
|
|
workflow's 75m44s end-to-end wall time was dominated by runner queue delay,
|
|
so queue capacity is tracked separately from code-level setup savings.
|
|
- Container job logs show about 54 seconds to pull/copy Playwright browsers
|
|
into `/tmp/ms-playwright` on each host runner.
|
|
- The same latest-main run recorded one passed-after-retry result in
|
|
`tests/review/review-file-status.spec.ts`. Its final assertion counted two
|
|
unrelated `Loading diff...` placeholders in lazy sections while checking a
|
|
moved file that had already rendered. The test has also appeared in earlier
|
|
flake history, so this is a reproducible synchronization defect rather than
|
|
a reason to increase retries.
|
|
|
|
## Scope
|
|
|
|
### In scope
|
|
|
|
- Best-effort browser-cache restore and save in the existing host-runner
|
|
container cohort, keyed to the exact checked-in browser source.
|
|
- A browser smoke verification that decides whether the restored path is safe;
|
|
the current pinned-image extraction remains the fallback.
|
|
- Step-summary diagnostics for cache state, browser verification, and setup
|
|
duration so future runs can measure the gain.
|
|
- A selection-scoped readiness assertion in the review-file-status E2E test.
|
|
- Workflow contract coverage, focused E2E coverage, and rollout evidence.
|
|
|
|
### Out of scope
|
|
|
|
- Changing the duration-aware planner, timing-profile schema, 14/6 matrix, or
|
|
`workers: 1` default.
|
|
- Splitting mobile Chrome into a separate matrix.
|
|
- Increasing retries or timeouts, adding fixed sleeps, or weakening the E2E
|
|
gate.
|
|
- Replacing the runtime image, moving browsers to an external service, or
|
|
changing application behavior.
|
|
- Treating GitHub runner queue time as a test or sharding defect.
|
|
|
|
## Technical approach
|
|
|
|
The container jobs resolve the `runtime-latest` convenience tag to an immutable
|
|
sha256 digest, then restore `/tmp/ms-playwright` before the digest-pinned image
|
|
pull and copy step. The cache key includes the runner OS, browser source,
|
|
resolved image digest, and a run-specific primary-key suffix with a stable
|
|
digest-scoped restore prefix. A verified exact or prefix hit skips the
|
|
pull/copy step. A miss, stale, cache-service failure, or failed Chromium
|
|
verification runs the existing digest-pinned extraction path and can populate
|
|
the cache under a new primary key after successful setup. Cache actions are
|
|
best-effort and cannot turn a healthy fallback into a failed E2E job.
|
|
|
|
The review test anchors the final moved-file assertions to the selected file's
|
|
existing review header/section boundary. Other lazy sections may still show a
|
|
loading marker and do not affect the selected file's readiness.
|
|
|
|
## Work orders
|
|
|
|
### Wave 1
|
|
|
|
- [ ] [task-01-cache-host-runner-browsers](task-01-cache-host-runner-browsers.md)
|
|
- [x] [task-02-scope-review-diff-readiness](task-02-scope-review-diff-readiness.md)
|
|
|
|
The slices are independently verifiable, but execution remains sequential in
|
|
the primary session so workflow and test changes can be reviewed together.
|
|
|
|
## Rollout and success criteria
|
|
|
|
1. Run the workflow contract and action-pinning checks locally.
|
|
2. Run the focused review E2E test repeatedly with retries disabled.
|
|
3. Merge the workflow change and compare at least three successful `main`
|
|
runs with the preceding setup and execution measurements.
|
|
4. Confirm warm container jobs report a verified cache hit and omit browser
|
|
extraction. Confirm a forced key miss or unavailable cache uses the pinned
|
|
image fallback and still passes.
|
|
5. Retain the change only if the container setup component decreases without
|
|
new browser-verification failures, retries, or test failures. Continue to
|
|
report queue delay separately.
|
|
|
|
The existing shard-tail target remains the acceptance threshold for test-work
|
|
balance. A cache change is not credited with a sharding improvement, and a
|
|
shorter queued workflow is not credited as a test-speed improvement.
|
|
|
|
## Verification strategy
|
|
|
|
The work orders own focused verification. The implementation handoff should
|
|
also run the repository specification lint after changing the design package:
|
|
|
|
```bash
|
|
python3 scripts/lint-spec-files.py --all
|
|
git diff --check
|
|
```
|
|
|
|
Remote rollout evidence belongs in this plan after the branch workflow runs.
|
|
Until then, the cache hit rate and setup timings remain open operational
|
|
measurements.
|
|
|
|
## Implementation results
|
|
|
|
- The container workflow now resolves an immutable runtime image digest,
|
|
restores and saves a Playwright browser cache keyed by runner OS, browser
|
|
source, digest, and a run-specific suffix, and uses a stable digest-scoped
|
|
restore prefix. Cache verification gates the digest-pinned GHCR fallback,
|
|
and setup state is written to the job summary. Cache errors remain
|
|
non-blocking.
|
|
- The review-file-status regression now scopes the moved-file assertions to its
|
|
selected diff section. A 30-repeat host run with retries disabled passed all
|
|
30 repetitions.
|
|
- PR run [#3163](https://github.com/kdlbs/kandev/pull/3163) passed all 20 E2E
|
|
shards and the full required check set. A container shard restored a
|
|
digest-scoped prefix cache, verified Chromium, skipped image extraction, and
|
|
reported about 9 seconds for browser setup. The preceding PR run exposed a
|
|
transient GHCR metadata lookup failure; the workflow now retries that lookup
|
|
three times before failing safely.
|
|
- Post-merge comparison across three successful `main` runs and a forced-miss
|
|
fallback measurement remain open rollout work.
|
|
|
|
## Risks
|
|
|
|
- GitHub Actions cache transfer can cost enough time to erase the browser-copy
|
|
saving. Step summaries must report restore and verification duration before
|
|
declaring success.
|
|
- Concurrent first-run cache saves can race. The fallback remains valid and
|
|
each run has a distinct primary key; the restore prefix selects the newest
|
|
compatible successful entry.
|
|
- The runtime image may change without a browser-version change. Including the
|
|
resolved image digest in the key is a conservative invalidation rule and
|
|
binds fallback extraction to the same image identity.
|
|
- Hosted-runner queue capacity can dominate total workflow wall time. The
|
|
rollout must compare job execution intervals and queue intervals separately.
|