dreampiler/kandev
1
0
Fork 0
Code Issues 43 Pull Requests Actions 6 Packages Projects Releases Wiki Activity

fix(k8s): share one pod across task sessions (#3865)

Browse Source
This commit is contained in:
José Almeida
2026-09-24 07:37:17 +01:00
committed by GitHub
parent 9885fd40ba
commit 25271b6e7f
65 changed files with 3627 additions and 220 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/public/k8s.md
+21 -16
View File
@@ -17,13 +17,13 @@ Kandev ships example Kubernetes YAML in `k8s/`. The control-plane files are a si
To run task sessions as Pods, follow [Configure the Kubernetes executor](#configure-the-kubernetes-executor) after the control plane is reachable. That is a separate cluster-authority decision.
![Kubernetes placement showing one persistent Kandev control-plane Pod, its PVC and Service, optional task-session Pods, and external repositories and providers.](../screenshots/k8s.svg)
![Kubernetes placement showing one persistent Kandev control-plane Pod, its PVC and Service, optional task Pods, and external repositories and providers.](../screenshots/k8s.svg)
[Open full-size SVG diagram][k8s-diagram]
[k8s-diagram]: ../../docs/screenshots/k8s.svg
The control-plane Pod owns the persistent home and API identity. Task-session Pods are a separate, opt-in workload path with their own service-account permissions.
The control-plane Pod owns the persistent home and API identity. Task Pods are a separate, opt-in workload path with their own service-account permissions.
## Architecture and limitations
@@ -36,7 +36,7 @@ The example creates:
| `k8s/deployment.yaml` | `Deployment/kandev` | one replica, `Recreate`, example resources and probes |
| `k8s/service.yaml` | `Service/kandev` | `ClusterIP`, TCP 38429 |
| `k8s/ingress.yaml` | `Ingress/kandev` | ingress-nginx-oriented example for `kandev.example.com` |
| `k8s/executor-rbac.yaml` | `ServiceAccount`, `Role`, and `RoleBinding` | Opt-in access for task-session Pods in `kandev-agents`; not part of the deployment quick path |
| `k8s/executor-rbac.yaml` | `ServiceAccount`, `Role`, and `RoleBinding` | Opt-in access for task Pods in `kandev-agents`; not part of the deployment quick path |
One pod serves the SPA, API, WebSocket, external MCP endpoint, and `/health` on port 38429. With SQLite, the same PVC holds the database, workspaces, CLI installs, and authentication files.
@@ -92,7 +92,9 @@ Open `http://localhost:38429`.
## Configure the Kubernetes executor
The Kubernetes executor is independent of where the Kandev control plane runs. A host process can use a kubeconfig, while a Kandev Pod can use either a mounted kubeconfig or its in-cluster service account. One executor connects to one API server context and one namespace; every selected task session gets one Pod in that namespace.
The Kubernetes executor is independent of where the Kandev control plane runs. A host process can use a kubeconfig, while a Kandev Pod can use either a mounted kubeconfig or its in-cluster service account. One executor connects to one API server context and one namespace; each task gets one Pod in that namespace. Additional sessions on the same task share that Pod and workspace, with independent agentctl instances and session configuration.
**A task is one credential trust boundary.** All agents in its Pod run as the same OS user and can read sibling credentials, including credentials from different agent profiles. Separate session HOME directories and configuration prevent accidental mixing; they do not isolate secrets. Attach only mutually trusted agents and credentials. If any sibling is compromised, stop the task's agents and revoke or rotate exposed credentials with their providers. Stopping a session alone does not revoke credentials or undo exposure. Use separate tasks with appropriately isolated executor policies for agents that must not share trust.
Only administrators can create, edit, delete, or test Kubernetes executors and profiles. Members can list and use administrator-configured profiles, resume their authorized sessions, and view the sanitized status rows available to them. When Kandev authentication is disabled, requests use the synthetic single-user administrator identity.
@@ -241,12 +243,13 @@ Every Kandev-created Pod and managed PVC has this complete custom identity:
|---|---|
| `kandev.ai/executor-id` | Executor record ID |
| `kandev.ai/profile-id` | Executor-profile record ID |
| `kandev.ai/instance-id` | Runtime instance ID |
| `kandev.ai/instance-id` | Stable task-environment resource instance ID |
| `kandev.ai/task-id` | Task ID |
| `kandev.ai/session-id` | Task-session ID |
| `kandev.ai/session-id` | Original creating session ID, retained as an immutable provenance label |
| `kandev.ai/environment-id` | Task-environment ID |
| `kandev.ai/ownership-version` | `task-v1` for new task-owned resources |
It also owns `app.kubernetes.io/name=kandev-agent`, `app.kubernetes.io/component=agent-session`, `app.kubernetes.io/managed-by=kandev`, and `app.kubernetes.io/instance=<runtime-instance-id>`. All `kandev.ai/*` label and annotation keys are reserved, including keys not listed above. Kandev rejects an admission response that changes or adds one of those reserved identities.
It also owns `app.kubernetes.io/name=kandev-agent`, `app.kubernetes.io/component=task-environment`, `app.kubernetes.io/managed-by=kandev`, and `app.kubernetes.io/instance=<task-environment-resource-instance-id>`. All `kandev.ai/*` label and annotation keys are reserved, including keys not listed above. Kandev rejects an admission response that changes or adds one of those reserved identities.
Every Kandev-created Pod and managed PVC also receives a fresh 256-bit `kandev.ai/create-nonce` annotation for that individual create request. If a create response is lost or ambiguous, Kandev adopts, checkpoints, bootstraps, or deletes the resulting object only when the API object preserves that exact nonce and the complete recorded identity.
@@ -337,7 +340,7 @@ before use. No production deployment or universal cgroup compatibility is implie
| Mode | Profile fields | Lifecycle |
|---|---|---|
| `managed_pvc` | Positive Kubernetes quantity, optional StorageClass, and one or more access modes. Defaults are `10Gi` and `ReadWriteOnce`. | One filesystem PVC per session, without a Pod owner reference. It survives ordinary stop, backend restart, main-container restart, and Pod replacement. Terminal/forced cleanup deletes it only after exact identity verification. |
| `managed_pvc` | Positive Kubernetes quantity, optional StorageClass, and one or more access modes. Defaults are `10Gi` and `ReadWriteOnce`. | One filesystem PVC per task, without a Pod owner reference. It survives ordinary stop, backend restart, main-container restart, and Pod replacement. Task archive/delete cleanup deletes it only after exact identity verification. |
| `empty_dir` | No PVC fields. | Pod-scoped storage. It survives a container restart but disappears with the Pod. A missing Pod makes the workspace unrecoverable. |
| `existing_claim` | Exact claim name in the executor namespace. | Kandev gets and mounts the claim but never creates, relabels, or deletes it. Data retention, access modes, concurrent mounts, and backup remain the operator's responsibility. |
@@ -345,16 +348,16 @@ A managed claim may remain Pending because of StorageClass, topology, quota, acc
### Recovery, snapshots, and cleanup
Kandev's `executors_running` inventory is authoritative. It records exact Pod and PVC namespace/name/UID, full resource identity, workspace ownership, platform, main container, remote port, hashes, and the validated workload launch snapshot. The snapshot contains Pod template, platform, main container, and storage configuration only. It excludes Kandev-resolved credentials, resolved profile environment values, injected files, and scripts; literal values written by an administrator into the Pod template remain in the snapshot.
Kandev's `task_environment_kubernetes` inventory is authoritative for task-owned physical resources; `executors_running` records per-session executions and legacy resources. The physical inventory records exact Pod and PVC namespace/name/UID, full resource identity, workspace ownership, platform, main container, remote port, hashes, and the validated workload launch snapshot. The snapshot contains Pod template, platform, main container, and storage configuration only. It excludes Kandev-resolved credentials, resolved profile environment values, injected files, and scripts; literal values written by an administrator into the Pod template remain in the snapshot.
Current connection configuration and recorded workload configuration serve different purposes:
- Saved kubeconfig/in-cluster mode, kubeconfig path/context, and timeout are used for later status, reconnect, replacement, and cleanup. Changing credentials or context can restore or break access to an existing session. Retained sessions continue to target their recorded namespace; changing the saved namespace affects new sessions only.
- Saved workload profile edits affect new sessions. An existing Pod is never live-mutated, and a missing-Pod replacement uses the recorded workload snapshot rather than the profile's current image, template, platform, main container, or storage fields.
- Saved kubeconfig/in-cluster mode, kubeconfig path/context, and timeout are used for later status, reconnect, replacement, and cleanup. Changing credentials or context can restore or break access to an existing session. Retained sessions continue to target their recorded namespace; changing the saved namespace affects new task Pods only.
- Saved workload profile edits affect newly provisioned task Pods. Additional sessions reuse the existing task workload snapshot. An existing Pod is never live-mutated, and a missing-Pod replacement uses the recorded workload snapshot rather than the profile's current image, template, platform, main container, or storage fields.
On resume, Kandev reloads environment definitions from the recorded executor profile and resolves secret references at the launch checkpoint. Changes to that profile's environment can therefore affect a resumed process; they do not rewrite the Pod or storage snapshot. If the profile has been deleted, its environment definitions are unavailable. Profile lookup failures and executor ownership mismatches block resume.
Ordinary Stop and backend shutdown close local clients and forwards but preserve the Pod and workspace. Agent or main-container restart keeps the Pod volumes; Kandev performs a new nonce handshake and local port-forward. If a Pod disappears, managed or existing PVC storage can support a replacement Pod after identity checks; `emptyDir` cannot.
Stop, including force-stop, terminates only the selected session’s agentctl instance and closes its local clients and forwards. Sibling sessions, the Pod, and the workspace remain available. Backend shutdown closes local connections and preserves the remote resources. Agent or main-container restart keeps the Pod volumes; Kandev performs a new nonce handshake and local port-forward. If a Pod disappears, managed or existing PVC storage can support a replacement Pod after identity checks; `emptyDir` cannot.
A recoverable agent error also preserves the established Pod, workspace, and
recovery credentials. Use **Resume** to continue that session. If its retained
@@ -362,17 +365,19 @@ credentials or storage are missing, recovery fails rather than silently creating
an empty replacement workspace. A software update cannot restore resources that
were already deleted.
Archive/delete terminal cleanup and explicit force cleanup are destructive. Before deletion, Kandev verifies the recorded namespace, name, UID, and complete standard plus `kandev.ai/*` ownership-label set. It then deletes the exact Pod with UID/resource-version preconditions and deletes a PVC only when inventory proves Kandev created that managed claim. Existing claims are never deleted. A missing object is idempotent; an inventory read error, same-name replacement, UID mismatch, missing label, changed label, extra `kandev.ai/*` label, or mismatched create nonce fails closed without deleting the ambiguous object.
Task archive/delete cleanup is destructive after all attached sessions are settled. Deleting or stopping an individual session does not delete the task Pod or its managed claim. Before deletion, Kandev verifies the recorded namespace, name, UID, and complete standard plus `kandev.ai/*` ownership-label set. It then deletes the exact Pod with UID/resource-version preconditions and deletes a PVC only when inventory proves Kandev created that managed claim. Existing claims are never deleted. A missing object is idempotent; an inventory read error, same-name replacement, UID mismatch, missing label, changed label, extra `kandev.ai/*` label, or mismatched create nonce fails closed without deleting the ambiguous object.
Kandev blocks deleting an executor, or changing an executor into or out of Kubernetes, while runtime inventory still refers to it. Finish normal session cleanup first. Profile deletion does not rewrite or destroy a retained workload because recovery owns the recorded profile ID and snapshot.
Kandev blocks deleting an executor, or changing an executor into or out of Kubernetes, while runtime inventory still refers to it. Archive or delete the owning tasks to finish resource cleanup first. Removing the last session alone retains the task Pod. Profile deletion does not rewrite or destroy a retained workload because recovery owns the recorded profile ID and snapshot.
Do not manually delete by name or label alone. If manual incident cleanup is unavoidable, compare the Kandev runtime inventory with namespace, name, UID, and all ownership labels first, preserve required workspace data, and record the out-of-band action.
Older installations may have several session Pods for one task. Kandev does not merge or delete those Pods automatically. Additional attachment requires an unambiguous recorded Pod and claim with matching task ownership and recoverable credentials; conflicting inventory blocks attachment. Adopted legacy resources keep their original labels.
### Read retained resource status
The **Active sessions** card shows the sanitized session inventory for the
selected executor. It includes sessions that Kandev can resume after ordinary
Stop, not only sessions whose agent process is currently running. The card
Stop, not only sessions whose agent process is currently running. A task whose last session has been removed remains listed with its retained Pod and no session ID. Multiple sessions on a task show the same current Pod identity. The card
shows separate session state, Pod state, retention state, workspace mode, and
main-container CPU and memory requests.
@@ -511,7 +516,7 @@ Use the SSL mode and trust material required by your database. PostgreSQL moves
> **Docker boundary:** Do not add only a Docker socket mount. The runtime also needs helper, credential-session, and local-clone bind sources at matching daemon-host paths. Privileged Docker-in-Docker has a separate security and persistence model and no supplied manifest.
The [Kubernetes executor](#configure-the-kubernetes-executor) creates separate task-session Pods through the Kubernetes API and does not require a Docker socket. The Local and Worktree behavior below instead runs agent processes inside the control-plane Pod itself.
The [Kubernetes executor](#configure-the-kubernetes-executor) creates separate task Pods through the Kubernetes API and does not require a Docker socket. The Local and Worktree behavior below instead runs agent processes inside the control-plane Pod itself.
<details>
<summary>Agent execution, resources, and probes</summary>