kandev/docs/plans/e2e-duration-aware-sharding/task-08-performance-experiments.md

id, title, status, wave, depends_on, plan, spec

id	title	status	wave	depends_on	plan	spec
08-performance-experiments	Run concurrency and setup experiments	completed	4	03-ci-manifest-lifecycle 06-container-isolation	plan.md	../../specs/platform/requirements/e2e-duration-aware-sharding.md

Task 08: Run concurrency and setup experiments

Acceptance

• A repeatable comparison records workers: 1 versus workers: 2 on the known heavy normal shards, including wall time, CPU/memory pressure, backend isolation failures, and retry counts; the default remains one worker unless evidence supports a later change.
• A repeatable comparison measures the unified normal matrix against a separately balanced mobile matrix, including project/setup overhead and actual shard tail; no matrix split is enabled without the report.
• Build, pnpm install, runtime-image startup, and Playwright browser extraction are measured separately, and any cache or pre-baked-image change is kept behind evidence of lower wall time without a reliability regression.

Verification

cd apps/web
E2E_SHARD=2 /usr/bin/time -v bash e2e/scripts/run-planned-shard.sh \
  e2e/manifests/normal/2.json -- --workers=2
E2E_SHARD=2 /usr/bin/time -v bash e2e/scripts/run-planned-shard.sh \
  e2e/manifests/normal/2.json -- --workers=1

Generate e2e/manifests/normal/2.json with the Task 03 planner first. Run the paired commands on the same runner class and record results in the experiment report. The runner forwards the worker override after --, so both commands execute the exact same duration-aware file selection. Do not compare unrelated ordinal assignments.

Files likely touched

• .github/workflows/e2e-tests.yml (diagnostic/manual experiment hooks only)
• apps/web/e2e/README.md (experiment procedure and result format)
• CI image/cache configuration if a measured setup optimization is approved

Dependencies

Depends on Task 03 for reproducible manifests and Task 06 for safe concurrent container behavior.

Parallelism

Sequential after the default manifest workflow is stable. Experiments must be isolated from the blocking PR lane.

Inputs

• Current workers-one invariant and worker-scoped backend fixture.
• Heavy normal and container shard timing evidence from the investigation.
• Build and setup timings from the PR #2471 workflow run.

Output contract

Report paired measurements, runner/environment details, flake/resource observations, recommendation, and explicit statement of which defaults remain unchanged.

Experiment report (2026-08-11)

The README contains the controlled experiment procedure and result format, and the runner forwards worker overrides after the manifest separator. The local experiment used the post-merge checkout at d150554a5, a rebuilt backend/web artifact, and the heaviest predicted normal shard (normal/10.json, 44 files, 191 tests). The runner image does not provide GNU /usr/bin/time, so Bash time -p supplied wall/user/system time and a process-group sampler supplied peak RSS and CPU for the monitored runs.

Environment

• Node v24.16.0; pnpm 9.15.9.
• AMD Ryzen 5 7640HS, 6 cores / 12 threads; 10 online CPUs reported to the process.
• 18 GiB RAM, 15 GiB available at measurement start, 8 GiB swap.
• Docker 26.1.5, Linux amd64.
• Local build artifacts were rebuilt at the measured commit before E2E runs.

CI comparison

The latest successful main baseline, run 31423650879, took 28m30s from workflow creation to completion. The post-conflict PR run, run 31469837346, took 22m39s and passed all 42 checks: a 5m51s (20.5%) full-workflow reduction.

The PR run's normal shard cohort ran from 07:45:35Z to the slowest job finish at 08:00:41Z (15m06s); the slowest normal job was 12m47s. The baseline's slowest normal job was 17m41s. The container tail improved from 9m26s to 5m14s. The PR retry summary recorded 2 tests that passed after one retry, 0 final failures, and 0 timeouts.

Both the PR run and the latest successful main run used the deterministic count-fallback path: the latest main run, 31469080336, has no e2e-timing-profile artifact. The first successful main run after this change lands will bootstrap the rolling profile; until then, the planner is implemented but the CI timing-profile path is not yet exercised.

Worker experiment

workers	repetitions	wall time	result	sampled peak RSS	decision
1	3	458.110s / 458.237s / 458.252s; median 458.237s; CV 0.014%	191/191 passed, no retries	874–948 MiB on 2 monitored runs	keep default
2	1	268.715s	189 passed; 2 flaky tests passed after retry; exit 1 with E2E_FAIL_ON_FLAKY=1	1.32 GiB	reject

The two-worker run failed the reliability gate with a passthrough-terminal assertion and a fetch failed error. It is faster by 41.4%, but it does not meet the zero-retry/zero-isolation-error requirement. The checked-in default remains workers: 1; no candidate worker change was applied.

Matrix, shard-shape, and setup measurements

• The current unified 14-shard profile simulation predicts a 611.1s maximum shard. The best separate matrix allocation was 11 Chromium + 3 mobile shards at 653.7s (+7.0%) before extra setup, backend, and report overhead. The split was not executed because it failed the 10% improvement threshold.
• Normal shard simulation produced maxima of 855.3s at 10 shards, 611.1s at 14, and 534.8s at 16. Container maxima were 210.9s at 4, 154.8s at 6, and remained 154.8s at 8 because docker-launch.spec.ts is the dominant indivisible unit. Keep 14 normal and 6 container shards.
• On the CI runner, dependency install took 13s, backend/web build took 4m44s, and the complete Build job took 6m18s. Container setup extracted Playwright browsers in 33s. Warm local measurements were pnpm install 0.93s, browser install verification 0.57s, and a local runtime-image startup probe 0.45s. Cold cache misses and image pulls still need a runner measurement before changing setup or image strategy.

Decisions

• Adopt now: keep the generated-manifest workflow, 14/6 matrix, unified normal/mobile projects, and workers: 1; retain retry and shard diagnostics.
• Needs CI confirmation: consume the first post-merge main timing profile, measure cold setup/cache behavior, and repeat any future worker candidate on the CI runner class.
• Reject: workers=2 for the current shared-backend fixture, and a separate mobile matrix based on the profile simulation.