feat: add Kubernetes executor (#3261)

2026-09-02 13:50:03 +00:00
parent 65e6759d7d
commit 0064b9fb5a
376 changed files with 40671 additions and 1221 deletions
@@ -1,12 +1,12 @@
 ---
 title: "Kubernetes"
-description: "Deploy a single Kandev control plane to Kubernetes with explicit persistence, security, and lifecycle constraints."
+description: "Deploy a single Kandev control plane and configure Kubernetes-backed task sessions with explicit security and lifecycle constraints."
 status: experimental
 ---

 # Kubernetes

-Kandev ships example Kubernetes YAML in `k8s/`. It is a single-replica, persistent deployment example, not a Helm chart, operator, or supported high-availability topology. The release workflow publishes container images but does not apply these manifests to a cluster. Review and adapt every manifest before production use.
+Kandev ships example Kubernetes YAML in `k8s/`. The control-plane files are a single-replica, persistent deployment example, not a Helm chart, operator, or supported high-availability topology. A separate opt-in RBAC file supports the Kubernetes executor and is never applied by the documented deployment path. The release workflow publishes container images but does not apply manifests to a cluster. Review and adapt every file before production use.

 ## Quick path

@@ -15,6 +15,8 @@ Kandev ships example Kubernetes YAML in `k8s/`. It is a single-replica, persiste
 3. Port-forward first; add TLS and authentication before exposing an Ingress.
 4. Back up the database and PVC before upgrades or deletion.

+To run task sessions as Pods, follow [Configure the Kubernetes executor](#configure-the-kubernetes-executor) after the control plane is reachable. That is a separate cluster-authority decision.
+
 ## Architecture and limitations

 The example creates:
@@ -26,6 +28,7 @@ The example creates:
 | `k8s/deployment.yaml` | `Deployment/kandev` | one replica, `Recreate`, example resources and probes |
 | `k8s/service.yaml` | `Service/kandev` | `ClusterIP`, TCP 38429 |
 | `k8s/ingress.yaml` | `Ingress/kandev` | ingress-nginx-oriented example for `kandev.example.com` |
+| `k8s/executor-rbac.yaml` | `ServiceAccount`, `Role`, and `RoleBinding` | Opt-in access for task-session Pods in `kandev-agents`; not part of the deployment quick path |

 One pod serves the SPA, API, WebSocket, external MCP endpoint, and `/health` on port 38429. With SQLite, the same PVC holds the database, workspaces, CLI installs, and authentication files.

@@ -69,6 +72,8 @@ Replace `X.Y.Z` with a real release. Add `-n <namespace>` consistently if deploy

 Do not apply `k8s/ingress.yaml` yet. It contains a placeholder host, no TLS section, no Kandev authentication, and controller-specific annotations.

+The commands above also omit `k8s/executor-rbac.yaml` intentionally. Deploying the Kandev control plane must not silently grant it permission to create or enter workload Pods.
+
 For private initial access:

 ```bash
@@ -77,6 +82,202 @@ kubectl port-forward service/kandev 38429:38429

 Open `http://localhost:38429`.

+## Configure the Kubernetes executor
+
+The Kubernetes executor is independent of where the Kandev control plane runs. A host process can use a kubeconfig, while a Kandev Pod can use either a mounted kubeconfig or its in-cluster service account. One executor connects to one API server context and one namespace; every selected task session gets one Pod in that namespace.
+
+Only administrators can create, edit, delete, or test Kubernetes executors and profiles. Members can list and use administrator-configured profiles, resume their authorized sessions, and view the sanitized status rows available to them. When Kandev authentication is disabled, requests use the synthetic single-user administrator identity.
+
+The experimental E2E matrix validates API and `agentctl` connectivity against Kubernetes 1.34.8 and 1.36.1, with the full launch, reconnect, storage, failure, and cleanup lifecycle exercised on 1.36.1. Other Kubernetes server versions are currently unvalidated rather than known incompatible.
+
+### Executor prerequisites
+
+- A pre-created namespace dedicated to an appropriate task trust level. Kandev does not create or discover namespaces.
+- Linux nodes matching the profile's fixed `linux/amd64` or `linux/arm64` platform.
+- Kubernetes API connectivity that supports Pod exec and port-forward over WebSocket or SPDY upgrades. A proxy that allows normal REST calls can still block these streaming subresources.
+- Namespaced Pod permissions and storage permissions for the selected workspace mode.
+- API discovery and `SelfSubjectAccessReview` access for **Test Kubernetes**. Standard clusters normally grant these basic self-inspection operations to authenticated identities; hardened clusters may require a separate cluster administrator decision.
+- An admitted main-container image that satisfies the [image contract](#choose-and-pin-the-main-container-image).
+
+Kandev does not require `list` or `get` access to `Namespace` resources. The configured namespace is a validated DNS label, not a namespace browser.
+
+### Grant only the namespaced operations
+
+[`k8s/executor-rbac.yaml`](../../k8s/executor-rbac.yaml) is an opt-in `ServiceAccount`/`Role`/`RoleBinding` example. It targets `kandev-agents` and is deliberately absent from the control-plane apply commands above. Review and change every namespace occurrence before applying it.
+
+Create the namespace with a cluster-administrator identity, then apply the reviewed file explicitly:
+
+```bash
+kubectl create namespace kandev-agents --dry-run=client -o yaml | kubectl apply -f -
+kubectl apply -f k8s/executor-rbac.yaml
+```
+
+The Role grants exactly:
+
+- `get`, `create`, `delete`, and `watch` on Pods;
+- `get` and `create` on `pods/exec` and `pods/portforward`; and
+- `get`, `create`, and `delete` on PVCs for the starter `managed_pvc` mode.
+
+Tailor the PVC rule to the profiles that use this identity:
+
+- `managed_pvc`: keep `get`, `create`, and `delete`;
+- `existing_claim`: reduce the rule to `get`; or
+- `empty_dir` only: remove the PVC rule.
+
+The Role grants no Namespace, Pod list, Secret, ConfigMap, Event, Pod log, node, workload-controller, or cluster-wide access. Kandev reads exact recorded Pods and claims by name. Add unrelated diagnostic permissions only to a separate operator identity.
+
+The connection test creates `SelfSubjectAccessReview` objects to verify each permission and uses API discovery to obtain the server version. Those are cluster-scoped API operations and cannot be granted by a namespaced Role. If a hardened cluster removed the usual basic-user/discovery grants, decide whether to restore that self-inspection access or accept that **Test Kubernetes** will fail even though direct runtime operations might be allowed.
+
+The example binds the Role to `ServiceAccount/kandev-executor` in the same namespace. For in-cluster authentication, run the Kandev control-plane Pod with that service account:
+
+```yaml
+spec:
+  template:
+    spec:
+      serviceAccountName: kandev-executor
+```
+
+If the control plane runs in another namespace, keep the Role and RoleBinding in the executor namespace, create the ServiceAccount in the control-plane namespace, and change the RoleBinding subject's `namespace` accordingly. If a kubeconfig authenticates as another user or service account, bind the Role to that exact subject instead. Do not give workload Pods this executor identity.
+
+Before saving, inspect effective access as the intended service account:
+
+```bash
+kubectl auth can-i get pods \
+  --as=system:serviceaccount:kandev-agents:kandev-executor \
+  -n kandev-agents
+kubectl auth can-i create pods/exec \
+  --as=system:serviceaccount:kandev-agents:kandev-executor \
+  -n kandev-agents
+kubectl auth can-i create persistentvolumeclaims \
+  --as=system:serviceaccount:kandev-agents:kandev-executor \
+  -n kandev-agents
+```
+
+The impersonation checks themselves require permission to impersonate that service account. Run them as a cluster administrator, not as the executor identity.
+
+### Choose kubeconfig or in-cluster authentication
+
+Open **Settings > Executors > Kubernetes** and set one connection mode:
+
+- **Kubeconfig**: enter an absolute path on the machine or container running the Kandev backend. A path on the browser's computer is irrelevant. An optional context overrides the file's current context. Mount the file read-only into a containerized control plane and protect it with owner-only filesystem permissions.
+- **In-cluster**: Kandev uses the service-account CA, token, and API endpoint mounted into its own Pod. Kubeconfig path and context must remain empty. This mode fails outside a Kubernetes Pod or when service-account credentials are not mounted.
+
+> **Trust kubeconfig content like executable code.** `client-go` can run credential helpers declared in `users[].user.exec` and initialize configured auth-provider plugins inside the Kandev backend, with the backend's OS privileges. Use a root-owned, read-only kubeconfig, or one owned by a dedicated Kandev service account and not writable by agents. Accept kubeconfigs only from trusted administrators, and review every exec plugin command and referenced binary.
+
+Set the exact executor namespace and a request timeout from 1 to 300 seconds; the UI defaults to `default` and 30 seconds. Kubeconfig bytes and resolved credentials are never copied into executor JSON or a session Pod. The saved path, context, namespace, auth mode, and timeout are connection configuration, so anyone who can mutate them can redirect the backend's cluster access.
+
+### Create and test a profile
+
+The create flow saves one executor plus its first profile. A profile sets:
+
+- main container, default `kandev-agent`;
+- platform, `linux/amd64` or `linux/arm64`;
+- one strict `apiVersion: v1`, `kind: PodTemplate` YAML document, limited to 256 KiB; and
+- workspace mode plus only the fields used by that mode.
+
+Run **Test Kubernetes** before Save and after changing cluster, namespace, template, image, or storage policy. The test reports causal steps for configuration, discovery, namespace selection, RBAC, storage/admission, and streaming:
+
+1. It parses the unsaved connection and optional profile.
+2. It discovers the API server version. The namespace step reports the configured namespace; it does not list or get a Namespace object.
+3. It submits self-access reviews for the exact Pod, streaming, and conditional PVC permissions.
+4. With a profile, it dry-runs the composed Pod and, for `managed_pvc`, the PVC. For `existing_claim`, it reads the exact named claim.
+5. It creates one disposable `emptyDir` probe Pod from the profile, waits for the main container, verifies exec and port-forward transport, and deletes that exact Pod with a UID precondition.
+
+No diagnostic PVC is retained. The streaming probe is a real temporary Pod, so quota, scheduling, image pulls, admission, and Pod billing can apply. A cleanup failure makes the test fail; inspect `kandev-stream-probe-*` in the configured namespace before retrying. Kandev returns sanitized errors and template-policy warnings, but do not place secrets in a template or error-producing field.
+
+After saving, each Kubernetes profile opens with the shared **Cluster connection** editor, **Test Kubernetes**, and executor-wide **Active sessions** before workload, workspace, credential, and script settings. The test combines the current unsaved connection and profile values. Administrators edit both resources through the page's one Save/Reset flow; members see the same hierarchy read-only. A connection edit is shared by every profile on that executor and can change how active sessions reconnect, while a profile edit affects only new workload snapshots. If both are dirty, Kandev confirms active-session impact once and saves the connection before the profile. The raw PodTemplate field grows and shrinks with its YAML; long unwrapped lines scroll inside the field without widening the settings page.
+
+Configured executor rows expand to their profile pages, and task settings links open the exact selected profile. The standalone `/settings/executors/k8s/<executor-id>` connection editor is only a recovery surface for an executor that has no profiles; a bookmarked URL for a configured executor redirects to its first profile.
+
+The **Active sessions** card refreshes every 90 seconds and reads only Kandev's runtime inventory. For each authorized task row, the backend gets the exact recorded Pod and checks namespace, name, UID, and ownership labels. It does not list arbitrary namespace workloads or return Pod specs, environment, logs, or credentials. Administrator change and deletion confirmations use a separate admin-only global count of every runtime row that refers to the executor, including rows whose detailed metadata is malformed; that endpoint returns only the count and never cross-user task or session identities.
+
+On a Kubernetes task, use the Pod control in the task header to inspect the exact authorized Pod's live phase, container state, restart count, workspace mode, creation time, and sanitized failure reason. Desktop opens this disclosure on hover; touch and coarse-pointer layouts provide a named 44 px button that opens a Drawer with the same structured Pod summary. Quiet Refresh and Settings icons sit in the summary header. Refresh visibly stays busy until the immediate status read settles, and Settings opens the exact selected profile. Kubernetes does not show the generic **Reset environment** action because Pod and PVC lifecycle remains owned by Stop, Resume, Archive, and Delete.
+
+The smaller Pod glyphs on Kanban cards and in task lists request their exact session status as soon as they appear, so their healthy or failed color does not depend on a first hover. On a fine pointer, hover or keyboard focus opens a compact summary with the Pod identity and aligned rows for state, restarts, workspace mode, creation time, last check, and any sanitized failure. On touch layouts, the same glyph has an expanded 44 px hit target and opens that summary in a bottom Drawer without selecting the task row.
+
+### Understand template ownership
+
+Kandev preserves compatible administrator fields such as images, resources, security contexts, tolerations, affinity, sidecars, init containers, image-pull secrets, and workload service accounts. It rejects a profile before an API write when it conflicts with runtime invariants, uses unknown strict-YAML fields, contains multiple documents, omits the named main container or its image, or exceeds the size limit.
+
+Kandev owns generated resource names, namespace, `restartPolicy: Always`, Linux OS/architecture scheduling, the main-container command/arguments/working directory, port 8765, reserved environment, and three volumes/mounts:
+
+| Volume | Main-container path | Purpose |
+|---|---|---|
+| `kandev-runtime` | `/opt/kandev` | Injected helper, runtime environment, prepare script, and restart marker |
+| `kandev-auth` | `/run/kandev` | Memory-backed auth environment and runtime home |
+| `kandev-workspace` | `/workspace` | Selected `emptyDir`, managed PVC, or existing claim |
+
+The template cannot set the main container's command, args, or working directory; use reserved volume names or mount paths; use container port 8765 or the `kandev-agentctl` port name; or define `HOME`, `AGENTCTL_*`, or `KANDEV_*` environment keys. It also cannot set Pod name/namespace/UID, owner references, finalizers, restart policy, node name, OS, or Linux OS/architecture selectors. Kandev defaults `automountServiceAccountToken: false` only when the template leaves it unset.
+
+Every Kandev-created Pod and managed PVC has this complete custom identity:
+
+| Label | Recorded value |
+|---|---|
+| `kandev.ai/executor-id` | Executor record ID |
+| `kandev.ai/profile-id` | Executor-profile record ID |
+| `kandev.ai/instance-id` | Runtime instance ID |
+| `kandev.ai/task-id` | Task ID |
+| `kandev.ai/session-id` | Task-session ID |
+| `kandev.ai/environment-id` | Task-environment ID |
+
+It also owns `app.kubernetes.io/name=kandev-agent`, `app.kubernetes.io/component=agent-session`, `app.kubernetes.io/managed-by=kandev`, and `app.kubernetes.io/instance=<runtime-instance-id>`. All `kandev.ai/*` label and annotation keys are reserved, including keys not listed above. Kandev rejects an admission response that changes or adds one of those reserved identities.
+
+Every Kandev-created Pod and managed PVC also receives a fresh 256-bit `kandev.ai/create-nonce` annotation for that individual create request. If a create response is lost or ambiguous, Kandev adopts, checkpoints, bootstraps, or deletes the resulting object only when the API object preserves that exact nonce and the complete recorded identity.
+
+Structurally valid high-risk policy remains administrator-owned. Kandev warns about privileged containers, host networking, host PID/IPC, `hostPath`, host ports, and explicit service-account token automount, but does not strip them. Enforce the real boundary with Pod Security admission, policy controls, quotas, network policy, workload identity, and storage policy. See [Security and Trust](security.md#separate-kubernetes-control-and-workload-identities).
+
+### Choose and pin the main-container image
+
+The UI's starter template uses:
+
+```yaml
+apiVersion: v1
+kind: PodTemplate
+template:
+  spec:
+    containers:
+      - name: kandev-agent
+        image: ghcr.io/kdlbs/kandev:latest
+```
+
+`latest` is convenient for a first disposable test but is a moving tag. For a controlled environment, replace it with the matching stable release, such as `ghcr.io/kdlbs/kandev:X.Y.Z`, or an immutable digest. Validate both supported architectures before reusing one profile across heterogeneous clusters.
+
+A custom image does not need `agentctl`; Kandev injects the platform-matched helper. It must provide:
+
+- Linux `amd64` or `arm64` matching the selected platform;
+- `sh`, `sleep`, `git`, CA trust, and ordinary shell/core utilities used by bootstrap and repository setup, including `cat`, `chmod`, `cp`, `dd`, `find`, `grep`, `mkdir`, and `rm`;
+- Node.js and npm for the supported agent-install flow, plus the selected agent CLI or everything its install script needs; and
+- compilers, package managers, shells, certificates, and repository-specific tools needed by the task.
+
+The effective runtime user must be able to write `/opt/kandev`, `/run/kandev/home`, and `/workspace`. Configure a compatible image user and volume ownership, or set an appropriate Pod `fsGroup` and verify the CSI driver's behavior. If an agent install uses npm globals, point its prefix and `PATH` at a writable location rather than assuming `/usr/local` or an image-specific home is writable. A read-only root filesystem can work only if the template and image leave every non-volume path used by installed tooling writable elsewhere. Test agent installation, Git clone, npm global installation, a terminal, and the repository's real build before promoting the image.
+
+### Choose workspace storage
+
+| Mode | Profile fields | Lifecycle |
+|---|---|---|
+| `managed_pvc` | Positive Kubernetes quantity, optional StorageClass, and one or more access modes. Defaults are `10Gi` and `ReadWriteOnce`. | One filesystem PVC per session, without a Pod owner reference. It survives ordinary stop, backend restart, main-container restart, and Pod replacement. Terminal/forced cleanup deletes it only after exact identity verification. |
+| `empty_dir` | No PVC fields. | Pod-scoped storage. It survives a container restart but disappears with the Pod. A missing Pod makes the workspace unrecoverable. |
+| `existing_claim` | Exact claim name in the executor namespace. | Kandev gets and mounts the claim but never creates, relabels, or deletes it. Data retention, access modes, concurrent mounts, and backup remain the operator's responsibility. |
+
+A managed claim may remain Pending because of StorageClass, topology, quota, access-mode, or binding policy even after dry-run admission succeeds. An existing claim can expose data from other applications or sessions to the agent. Use a dedicated claim unless deliberate sharing and its concurrency model have been reviewed.
+
+### Recovery, snapshots, and cleanup
+
+Kandev's `executors_running` inventory is authoritative. It records exact Pod and PVC namespace/name/UID, full resource identity, workspace ownership, platform, main container, remote port, hashes, and the validated workload launch snapshot. The snapshot contains Pod template, platform, main container, and storage configuration only. It excludes Kandev-resolved credentials, resolved profile environment values, injected files, and scripts; literal values written by an administrator into the Pod template remain in the snapshot.
+
+Current connection configuration and recorded workload configuration serve different purposes:
+
+- Saved kubeconfig/in-cluster mode, kubeconfig path/context, and timeout are used for later status, reconnect, replacement, and cleanup. Changing credentials or context can restore or break access to an existing session. Retained sessions continue to target their recorded namespace; changing the saved namespace affects new sessions only.
+- Saved profile edits affect new sessions. An existing Pod is never live-mutated, and a missing-Pod replacement uses the recorded workload snapshot rather than the profile's current image, template, platform, main container, or storage fields.
+
+Ordinary Stop and backend shutdown close local clients and forwards but preserve the Pod and workspace. Agent or main-container restart keeps the Pod volumes; Kandev performs a new nonce handshake and local port-forward. If a Pod disappears, managed or existing PVC storage can support a replacement Pod after identity checks; `emptyDir` cannot.
+
+Archive/delete terminal cleanup and explicit force cleanup are destructive. Before deletion, Kandev verifies the recorded namespace, name, UID, and complete standard plus `kandev.ai/*` ownership-label set. It then deletes the exact Pod with UID/resource-version preconditions and deletes a PVC only when inventory proves Kandev created that managed claim. Existing claims are never deleted. A missing object is idempotent; an inventory read error, same-name replacement, UID mismatch, missing label, changed label, extra `kandev.ai/*` label, or mismatched create nonce fails closed without deleting the ambiguous object.
+
+Kandev blocks deleting an executor, or changing an executor into or out of Kubernetes, while runtime inventory still refers to it. Finish normal session cleanup first. Profile deletion does not rewrite or destroy a retained workload because recovery owns the recorded profile ID and snapshot.
+
+Do not manually delete by name or label alone. If manual incident cleanup is unavoidable, compare the Kandev runtime inventory with namespace, name, UID, and all ownership labels first, preserve required workspace data, and record the out-of-band action.
+
 ## Persistence and filesystem permissions

 With `KANDEV_HOME_DIR=/data`, the PVC includes:
@@ -173,6 +374,8 @@ Use the SSL mode and trust material required by your database. PostgreSQL moves

 > **Docker boundary:** Do not add only a Docker socket mount. The runtime also needs helper, credential-session, and local-clone bind sources at matching daemon-host paths. Privileged Docker-in-Docker has a separate security and persistence model and no supplied manifest.

+The [Kubernetes executor](#configure-the-kubernetes-executor) creates separate task-session Pods through the Kubernetes API and does not require a Docker socket. The Local and Worktree behavior below instead runs agent processes inside the control-plane Pod itself.
+
 <details>
 <summary>Agent execution, resources, and probes</summary>