kandev/docs/decisions/2026-08-24-kubernetes-executor-resource-ownership.md

# ADR-2026-08-24-kubernetes-executor-resource-ownership: Kubernetes Executor Resource Ownership

**Status:** accepted
**Date:** 2026-08-24
**Area:** backend, frontend, infra, security

## Context

The Kubernetes executor must combine an administrator-authored Pod template
with Kandev-owned control-plane fields, survive backend and container restarts,
and delete remote resources safely. Kubernetes object names and labels alone
are reusable or forgeable, Pod owner references would delete persistent
workspace storage after an unexpected Pod loss, and template fields can grant
node-level privileges.

## Decision

A lifecycle instance maps to one namespaced Pod and optionally one per-session
managed PVC. `executors_running` remains the authoritative inventory.

Kandev owns and enforces:

- the namespace and generated resource names;
- standard `app.kubernetes.io/*` ownership labels and custom `kandev.ai/*`
  executor/profile/instance/task/session/environment identity labels;
- a fresh, cryptographically unpredictable 32-byte lowercase-hex
  `kandev.ai/create-nonce` annotation on every managed PVC, lifecycle Pod, and
  disposable streaming-probe Pod immediately before its API create request;
- the designated main-container command, args, working directory, agentctl
  port, reserved environment keys, and runtime/auth/workspace mounts;
- Linux OS and declared architecture scheduling constraints;
- `restartPolicy: Always` and default-disabled service-account token automount;
- exact Pod/PVC names and UIDs recorded after API creation.

The administrator owns compatible template policy, including images,
resources, security contexts, scheduling preferences, sidecars, init
containers, image-pull secrets, and workload service accounts. Structurally
compatible privileged/host settings are warned about and left to administrator
trust plus cluster admission policy; fields that collide with Kandev invariants
are rejected.

An ordinary non-main container may explicitly request one RO/RW workspace
mount at exactly `/workspace`. Kandev retains the volume definition; redirected,
overlapping, subpath, propagation and recursive grants remain prohibited.
Admission compares grants in both directions by named recipient. Runtime/auth
mounts and init/ephemeral recipients remain excluded. The recorded launch
snapshot governs replacements. See the [workspace grant design](../specs/executors/system-design/kubernetes-docker-workloads.md).

Managed PVCs do not receive a Pod owner reference. Kandev preserves them on
ordinary stop, backend shutdown, and unexpected Pod loss, then explicitly
deletes them after a terminal cleanup proves the stored UID and full ownership
label set. Existing claims are never deleted. Names alone never authorize
reconnect or deletion, and inventory/identity errors fail closed.

Create-error reconciliation is bound to one exact API request. When a create
result is ambiguous, Kandev may accept an exact-name GET only when the admitted
object preserves that request's create nonce exactly, in addition to passing
the existing owned-field admission validation. A missing or different nonce
fails closed before inventory checkpointing, Pod bootstrap, or deletion. A
successful create response must preserve the nonce under the same validator.

Kandev persists the validated workload launch snapshot (Pod template,
platform, main container, and storage configuration only) with the exact
runtime inventory. Kandev-resolved credentials, resolved profile environment
values, injected files, and scripts are excluded; administrators remain
responsible for literal values in the Pod template. Existing-Pod reconnect
uses only the recorded identity; replacement of a missing Pod uses that launch
snapshot, so a later profile edit cannot mutate a retained session.

Kandev injects agentctl and sensitive runtime material through `pods/exec` into
reserved Pod-scoped volumes, then reaches agentctl through a local
`127.0.0.1` port-forward. It does not put task credentials or kubeconfig
material in the Pod spec and does not expose agentctl through a cluster Service.

## Consequences

- Resume and cleanup can distinguish a retained Kandev session from a
  same-name replacement or foreign workload.
- A same-name object with copied ownership labels cannot be adopted after an
  ambiguous create unless it also carries the unpredictable nonce from that
  exact create request.
- A managed workspace survives Pod replacement, but terminal cleanup must
  delete Pod and PVC explicitly and idempotently.
- The raw-template surface remains powerful and requires administrator-only
  mutation, fixed namespace scope, prominent warnings, and cluster admission
  policy.
- Port-forward and exec streaming are required capabilities; API discovery
  alone is insufficient for connection validation.
- Workload images must contain the documented POSIX bootstrap tools and allow
  Kandev's reserved mounts.
- Profile edits do not alter existing or replacement Pods for retained
  sessions; new sessions receive the new template.

## Alternatives Considered

- **Kubernetes Job or Deployment per session:** rejected because these
  controllers add replacement/ownership semantics that conflict with the
  existing one-instance lifecycle and exact cleanup inventory.
- **Pod name or label-only recovery:** rejected because names can be reused and
  labels can be forged or copied.
- **Adopting a same-name object after owned-label validation alone:** rejected
  because an attacker can pre-create an object with copied stable identity
  labels during an ambiguous API outcome; reconciliation must prove continuity
  with the exact create request.
- **Pod owner reference on managed PVC:** rejected because unexpected Pod
  deletion would garbage-collect resumable workspace data.
- **A Kubernetes Secret per session:** rejected for the first version because
  exec delivery into a memory-backed volume avoids another credential-bearing
  API object and its cleanup lifecycle.
- **Kandev Service/Ingress for agentctl:** rejected because a loopback local
  port-forward keeps the control endpoint out of the cluster network surface.
- **Silently strip privileged template fields:** rejected because it obscures
  administrator intent and cannot replace Kubernetes admission controls.