kandev/docs/plans/e2e-duration-aware-sharding/task-08-performance-experiments.md

---
id: "08-performance-experiments"
title: "Run concurrency and setup experiments"
status: completed
wave: 4
depends_on:
  - "03-ci-manifest-lifecycle"
  - "06-container-isolation"
plan: "plan.md"
spec: "../../specs/platform/requirements/e2e-duration-aware-sharding.md"
---

# Task 08: Run concurrency and setup experiments

## Acceptance

- A repeatable comparison records `workers: 1` versus `workers: 2` on the
  known heavy normal shards, including wall time, CPU/memory pressure, backend
  isolation failures, and retry counts; the default remains one worker unless
  evidence supports a later change.
- A repeatable comparison measures the unified normal matrix against a
  separately balanced mobile matrix, including project/setup overhead and
  actual shard tail; no matrix split is enabled without the report.
- Build, pnpm install, runtime-image startup, and Playwright browser extraction
  are measured separately, and any cache or pre-baked-image change is kept
  behind evidence of lower wall time without a reliability regression.

## Verification

```sh
cd apps/web
E2E_SHARD=2 /usr/bin/time -v bash e2e/scripts/run-planned-shard.sh \
  e2e/manifests/normal/2.json -- --workers=2
E2E_SHARD=2 /usr/bin/time -v bash e2e/scripts/run-planned-shard.sh \
  e2e/manifests/normal/2.json -- --workers=1
```

Generate `e2e/manifests/normal/2.json` with the Task 03 planner first. Run the
paired commands on the same runner class and record results in the experiment
report. The runner forwards the worker override after `--`, so both commands
execute the exact same duration-aware file selection. Do not compare unrelated
ordinal assignments.

## Files likely touched

- `.github/workflows/e2e-tests.yml` (diagnostic/manual experiment hooks only)
- `apps/web/e2e/README.md` (experiment procedure and result format)
- CI image/cache configuration if a measured setup optimization is approved

## Dependencies

Depends on Task 03 for reproducible manifests and Task 06 for safe concurrent
container behavior.

## Parallelism

Sequential after the default manifest workflow is stable. Experiments must be
isolated from the blocking PR lane.

## Inputs

- Current workers-one invariant and worker-scoped backend fixture.
- Heavy normal and container shard timing evidence from the investigation.
- Build and setup timings from the PR #2471 workflow run.

## Output contract

Report paired measurements, runner/environment details, flake/resource
observations, recommendation, and explicit statement of which defaults remain
unchanged.

## Experiment report (2026-08-11)

The README contains the controlled experiment procedure and result format, and
the runner forwards worker overrides after the manifest separator. The local
experiment used the post-merge checkout at `d150554a5`, a rebuilt backend/web
artifact, and the heaviest predicted normal shard (`normal/10.json`, 44 files,
191 tests). The runner image does not provide GNU `/usr/bin/time`, so Bash
`time -p` supplied wall/user/system time and a process-group sampler supplied
peak RSS and CPU for the monitored runs.

### Environment

- Node `v24.16.0`; pnpm `9.15.9`.
- AMD Ryzen 5 7640HS, 6 cores / 12 threads; 10 online CPUs reported to the
  process.
- 18 GiB RAM, 15 GiB available at measurement start, 8 GiB swap.
- Docker `26.1.5`, Linux `amd64`.
- Local build artifacts were rebuilt at the measured commit before E2E runs.

### CI comparison

The latest successful main baseline, [run 31423650879](https://github.com/kdlbs/kandev/actions/runs/31423650879), took 28m30s from workflow creation to completion. The post-conflict PR run, [run 31469837346](https://github.com/kdlbs/kandev/actions/runs/31469837346), took 22m39s and passed all 42 checks: a 5m51s (20.5%) full-workflow reduction.

The PR run's normal shard cohort ran from 07:45:35Z to the slowest job finish at 08:00:41Z (15m06s); the slowest normal job was 12m47s. The baseline's slowest normal job was 17m41s. The container tail improved from 9m26s to 5m14s. The PR retry summary recorded 2 tests that passed after one retry, 0 final failures, and 0 timeouts.

Both the PR run and the latest successful main run used the deterministic
count-fallback path: the latest main run, [31469080336](https://github.com/kdlbs/kandev/actions/runs/31469080336), has no `e2e-timing-profile` artifact. The first successful main run after this change lands will bootstrap the rolling profile; until then, the planner is implemented but the CI timing-profile path is not yet exercised.

### Worker experiment

| workers | repetitions | wall time | result | sampled peak RSS | decision |
| ---: | ---: | --- | --- | ---: | --- |
| 1 | 3 | 458.110s / 458.237s / 458.252s; median 458.237s; CV 0.014% | 191/191 passed, no retries | 874–948 MiB on 2 monitored runs | keep default |
| 2 | 1 | 268.715s | 189 passed; 2 flaky tests passed after retry; exit 1 with `E2E_FAIL_ON_FLAKY=1` | 1.32 GiB | reject |

The two-worker run failed the reliability gate with a passthrough-terminal
assertion and a `fetch failed` error. It is faster by 41.4%, but it does not
meet the zero-retry/zero-isolation-error requirement. The checked-in default
remains `workers: 1`; no candidate worker change was applied.

### Matrix, shard-shape, and setup measurements

- The current unified 14-shard profile simulation predicts a 611.1s maximum
  shard. The best separate matrix allocation was 11 Chromium + 3 mobile
  shards at 653.7s (+7.0%) before extra setup, backend, and report overhead.
  The split was not executed because it failed the 10% improvement threshold.
- Normal shard simulation produced maxima of 855.3s at 10 shards, 611.1s at
  14, and 534.8s at 16. Container maxima were 210.9s at 4, 154.8s at 6, and
  remained 154.8s at 8 because `docker-launch.spec.ts` is the dominant
  indivisible unit. Keep 14 normal and 6 container shards.
- On the CI runner, dependency install took 13s, backend/web build took
  4m44s, and the complete Build job took 6m18s. Container setup extracted
  Playwright browsers in 33s. Warm local measurements were pnpm install
  0.93s, browser install verification 0.57s, and a local runtime-image
  startup probe 0.45s. Cold cache misses and image pulls still need a runner
  measurement before changing setup or image strategy.

### Decisions

- **Adopt now:** keep the generated-manifest workflow, 14/6 matrix, unified
  normal/mobile projects, and `workers: 1`; retain retry and shard diagnostics.
- **Needs CI confirmation:** consume the first post-merge main timing profile,
  measure cold setup/cache behavior, and repeat any future worker candidate on
  the CI runner class.
- **Reject:** workers=2 for the current shared-backend fixture, and a separate
  mobile matrix based on the profile simulation.