feat: improve Kubernetes worker lifecycle visibility (#3553)

2026-09-10 11:05:05 +01:00
parent 796bf5853f
commit 2858109308
59 changed files with 4364 additions and 262 deletions
@@ -259,6 +259,43 @@ A custom image does not need `agentctl`; Kandev injects the platform-matched hel

 The effective runtime user must be able to write `/opt/kandev`, `/run/kandev/home`, and `/workspace`. Configure a compatible image user and volume ownership, or set an appropriate Pod `fsGroup` and verify the CSI driver's behavior. If an agent install uses npm globals, point its prefix and `PATH` at a writable location rather than assuming `/usr/local` or an image-specific home is writable. A read-only root filesystem can work only if the template and image leave every non-volume path used by installed tooling writable elsewhere. Test agent installation, Git clone, npm global installation, a terminal, and the repository's real build before promoting the image.

+### Use prepared worker image recipes
+
+The repository includes copyable worker image recipes in
+[`k8s/worker-images`](../../k8s/worker-images/README.md) and strict PodTemplate
+examples in [`k8s/presets`](../../k8s/presets/). They are inputs to the existing
+Kubernetes executor. They do not create Pods or start sessions by themselves.
+Kandev still injects `agentctl`, the launch command, credentials, runtime files,
+and the selected workspace mount.
+
+The recipes provide these starting targets:
+
+| Target | Extra runtime |
+|---|---|
+| `minimal` | Published Kandev base tools, Node, npm, Python, venv, and Git |
+| `node-pnpm` | The pinned pnpm version in `k8s/worker-images/pins.env` |
+| `python` | The published base Python and venv support |
+
+Build and smoke-test them with the pinned base image:
+
+~~~bash
+bash scripts/test-kubernetes-worker-images.sh --check
+bash scripts/test-kubernetes-worker-images.sh --smoke --platform linux/amd64
+~~~
+
+The current recipe and Kind lifecycle evidence covers Linux `amd64` and the
+Kubernetes version named in the experimental matrix. It does not establish
+support for another architecture or cluster version. Build and push a selected
+target to an operator-controlled registry, then replace the preset image marker
+with that image's immutable digest. Keep `imagePullPolicy` appropriate for the
+registry. The local `imagePullPolicy: Never` and image tags used by Kind tests
+are fixture-only values.
+
+The examples use UID 1000, writable `/workspace` paths for npm and Python
+artifacts, no service-account token automount, dropped capabilities, and no
+privilege escalation. Review resource requests, Pod Security admission, storage
+ownership, registry access, and repository tool requirements before use.
+
 ### Choose workspace storage

 | Mode | Profile fields | Lifecycle |
@@ -286,6 +323,53 @@ Kandev blocks deleting an executor, or changing an executor into or out of Kuber

 Do not manually delete by name or label alone. If manual incident cleanup is unavoidable, compare the Kandev runtime inventory with namespace, name, UID, and all ownership labels first, preserve required workspace data, and record the out-of-band action.

+### Read retained resource status
+
+The **Active sessions** card shows the sanitized session inventory for the
+selected executor. It includes sessions that Kandev can resume after ordinary
+Stop, not only sessions whose agent process is currently running. The card
+shows separate session state, Pod state, retention state, workspace mode, and
+main-container CPU and memory requests.
+
+Desktop rows use two status lines. The first line shows session and retention
+badges. The second line shows Pod phase and main-container state. Use the row
+disclosure to see the full labeled details. Phone cards show the same facts in
+a compact group and keep the full card as the task link.
+
+`active` means the recorded session is in an active lifecycle state.
+`retained` means the verified Pending or Running Pod remains after a stopped,
+completed, failed, or idle session. `terminating`, `terminal`, `missing`, and
+`unknown` describe other verified or incomplete observations. These labels do
+not prove agent connectivity, process activity, actual CPU or memory use, or
+cost. Requests describe only the main container. They exclude sidecars and
+Pod-wide resources.
+
+Use **Stop** when the session must remain resumable. Archive or Delete can
+remove a Kandev-managed Pod and workspace during terminal cleanup. An
+operator-owned `existing_claim` is never deleted by Kandev. Refresh the card
+after a lifecycle change because session and Pod observations can change
+independently.
+
+### Inspect Kubernetes launch timings
+
+Kandev writes bounded Info-level records for fresh Kubernetes environment
+creation. Set `KANDEV_LOG_LEVEL=info` when starting the backend, then inspect
+the active backend log under `KANDEV_HOME_DIR/logs/backend-logs.log` or the
+configured container log sink.
+
+The `kubernetes.launch.stage` record reports one of `storage`, `pod_ready`,
+`bootstrap`, or `agentctl_connect`. The `kubernetes.launch.completed` record
+reports the total environment-call duration. Both records include an opaque
+attempt ID, task/session/instance IDs, `mode: fresh`, `duration_ms`, and an
+outcome of `success`, `error`, `canceled`, or `timeout`. A failed stage or
+finalization site is included when known.
+
+The total covers fresh environment creation and its rollback. It does not
+measure queue wait, instance-lock wait, client construction, credential
+persistence, or agent-process startup. Reconnect and replacement paths do not
+emit fresh-start records. Logs never include raw requests, scripts, errors,
+tokens, nonces, or credentials.
+
 ## Persistence and filesystem permissions

 With `KANDEV_HOME_DIR=/data`, the PVC includes: