kandev/docs/specs/executors/system-design/kubernetes-startup-timing.md

status, system, requirements

status	system	requirements
current	executors	REQ-EXECUTORS-K8S-TIMING-001 REQ-EXECUTORS-K8S-TIMING-002

Kubernetes Startup Timing System Design

Purpose and boundaries

The requirements cover only KubernetesExecutor.createFresh in internal/agent/runtime/lifecycle. KubernetesPreparer.Prepare currently emits a short progress step before actual resource creation. Its duration is not an environment-start measurement.

Requirement mapping

Requirement	Design section
REQ-EXECUTORS-K8S-TIMING-001	Measurement contract
REQ-EXECUTORS-K8S-TIMING-002	Passive collection and failures

Measurement contract

One local recorder belongs to one createFresh call. A new opaque attempt ID distinguishes retries that reuse a session or instance ID. Existing task, session, and instance IDs provide correlation with lifecycle evidence.

Stage	Start	End
storage	Before launch.provisionWorkspace	After workspace verification/provisioning and its checkpoint
pod_ready	Before launch.createRunningPod	After admitted main-container-running observation and checkpoint
bootstrap	Before nonce generation and binary resolution	After bootstrap uploads, remote preparation, and entrypoint signaling return
agentctl_connect	Before connectNewAgentctl	After its health/nonce handshake and forward setup return

pod_ready includes scheduling, image pulls, init work, and storage binding waits that occur inside that call. It does not measure those substeps independently. storage can complete before a deferred-binding PVC becomes Bound. bootstrap includes the bounded administrator preparation script executed through Pod exec. A preparation failure therefore ends this stage with an error before the managed entrypoint is signaled.

The total begins at createFresh entry and ends at its final return. It includes launch.complete, bookkeeping, and failure rollback. Stage sums therefore need not equal the total. Queue wait, instance-lock wait, client construction, manager secret persistence, and agent-process startup are outside this total. The summary means environment-call completion, not durable session readiness.

Proposed Info-level records are kubernetes.launch.stage and kubernetes.launch.completed. Both use attempt_id, instance_id, task_id, session_id, mode: fresh, duration_ms, and outcome. Stage records add stage. The summary adds failed_stage only when an entered stage failed. A post-stage launch.complete failure uses failure_site: finalize. An identity/setup failure before stages uses failure_site: initialize. These fields are a bounded diagnostic vocabulary, not a public API.

Outcomes are success, error, canceled, and timeout. Context outcomes use errors.Is on the returned error. A later context cancellation does not relabel a successful result. No record contains err.Error() or serialized request metadata.

Passive collection and failures

Use the existing logger and Go monotonic durations (time.Now/time.Since). A narrow clock seam or testing/synctest supports deterministic duration tests. The helper stays local to the Kubernetes lifecycle package. No shared executor interface, global registry, timer goroutine, or telemetry queue is needed.

A stage emits once on return, including errors. Skipped stages emit nothing. The final summary defer is registered before rollback so it observes the final returned error after rollback. It preserves the original failed-stage result. Rollback errors can change the summary outcome but do not overwrite stage evidence. The helper never intercepts or replaces the original return error.

Existing reconnect and replacement paths bypass this recorder. Their successful behavior does not emit a misleading fresh-start measurement. Each attempt emits at most four stage records and one summary. Existing log filtering and retention govern storage. Info records are available in backend file logs under normal configuration, not necessarily kubectl logs.

Verification

executor_kubernetes_timing_test.go drives real fresh-launch wiring with the existing fake clients and a Zap observer. Assertions cover phase order, bounded record counts, concurrent attempts, and failures at every stage and finalization. Sentinel credentials, nonces, scripts, and hostile error text must not appear. Fake-client action counts and existing lifecycle tests prove no new API operations. Reconnect and missing-Pod replacement must emit no mode: fresh records.

The existing Kind launch scenario supplies a real backend-log smoke assertion. It checks correlation and entered-stage completeness without timing thresholds. Performance comparisons remain observations on a named machine and image.

Persistence and related contracts

No timing fields enter executors_running, task state, or browser payloads. No schema migration or new runtime configuration is required. The resource ownership ADR continues to govern checkpoint and rollback behavior.

Delivery

See the startup timing plan.