kandev/docs/specs/executors/system-design/kubernetes-docker-workloads.md

---
status: draft
system: executors
requirements:
  - REQ-EXECUTORS-K8S-DOCKER-001
  - REQ-EXECUTORS-K8S-DOCKER-002
  - REQ-EXECUTORS-K8S-DOCKER-003
---

# Kubernetes Docker Workloads System Design

## Purpose and boundaries

The executor owns workspace composition, workload snapshots and exact resource
cleanup. The [requirements](../requirements/kubernetes-docker-workloads.md)
introduce explicit companion workspace access plus an opt-in Docker workload
recipe. Operators supply deployment-specific placement, credentials and capacity.
The [implementation plan](../../../plans/kubernetes-docker-workloads/plan.md)
contains only the reusable executor change, recipe and integration evidence.

The existing [resource-ownership decision](../../../decisions/2026-08-24-kubernetes-executor-resource-ownership.md)
continues to govern names, UIDs, nonces, storage identity and cleanup. The new
mount exception must be reconciled with the foundation and public documentation
when implemented. Existing release behavior still rejects the exception until
that change ships. This design contains its rationale; no independent ADR is
needed for a parallel copy of the same contract.

## Requirement mapping

| Requirement | Design section |
| --- | --- |
| REQ-EXECUTORS-K8S-DOCKER-001 | Workspace grant and admission |
| REQ-EXECUTORS-K8S-DOCKER-002 | Worker and daemon configuration |
| REQ-EXECUTORS-K8S-DOCKER-003 | Lifecycle and evidence |

## Workspace grant and admission

Use existing `pod_template_yaml`; add no profile/API field, migration or UI.
An ordinary container other than `ProfileConfig.MainContainer` may explicitly
reference `kandev-workspace` at exactly `/workspace` once. `readOnly` selects
access mode. Reject subpaths, subpath expressions, mount propagation, recursive
mount options, alternate paths/devices and additional overlapping mounts. The
template still cannot define the volume itself. Main, init and ephemeral
container rules remain unchanged, including all runtime/auth mount prohibitions.

`ValidatePodTemplate` and `validateContainerReservedFields` in
`internal/agent/kubernetes/template.go` need container-role context for this
narrow exception. `ComposePod` preserves the requested sidecar reference and
continues to inject/own the actual workspace volume and the main-container
mount. All three workspace modes use the same grant rule.

`ValidateAdmittedPod` in `internal/agent/kubernetes/admission.go` must carry the
desired Pod's ordinary-container identities into companion mount validation.
Compare grants by container name and full permitted mount shape, not by array
position, volume name alone or the main container's mount. Verify both directions:
every admitted grant must have been requested, and every requested grant must
remain exactly once on the same named ordinary container. Reject added/removed
grants, missing/renamed recipients, duplicate grants, changed access modes and
attempts to move a grant into init or ephemeral containers.

Normalize only documented neutral API defaults before comparison; real API
integration must prove any such normalization. Continue accepting unrelated
labels and compatible additions that carry no reserved access. Do not weaken
existing main-container, control-port, env, volume, nonce or metadata validation.
Existing Kandev-owned volumes retain exact-source validation.

The grant is an administrative data-access decision, not a security barrier
against a privileged companion. A writable companion can alter repository
contents. Runtime/auth mounts are not explicitly shared with it.

## Worker and daemon configuration

Add a separate recipe under `k8s/worker-images/full/`; preserve the existing
minimal/node-pnpm/python recipe and its checks. Derive from an immutable
universal digest and pin all added inputs in the new recipe's `pins.env`.
Resolve actual registry digests during implementation; source examples may use
clearly marked substitutions, but deployable configuration must reject them.

Use `mise.toml`, the Go module and pnpm lockfile for repository-compatible tools.
The inspected universal defaults need golangci-lint v2 instead of v1.62.0 and
Playwright libraries/browser matching the resolved current package. Include
Go, Rust, Node, pnpm, Python/venv, native compilers, Docker CLI, Buildx and Compose.
Baked tools/browser binaries live outside mounted workspace/auth paths.

Set writable npm, pnpm, Python and Go caches under `/workspace`, but create
them after initial repository clone: the default Kubernetes prepare script
correctly refuses a fresh nonempty non-repository workspace. Kandev owns HOME
and injects it as `/run/kandev/home`; do not freeze host-home paths into config.
Agent CLIs continue through Kandev's selected-agent installation path.

The advanced template has ordinary containers `kandev-agent` and `docker-engine`.
The agent uses UID/GID 1000, dropped capabilities, no escalation and default
seccomp. Pod fsGroup is 1000. Put non-root constraints on the main container;
a Pod-wide `runAsNonRoot` would incorrectly conflict with the chosen daemon.
The daemon is an explicitly selected pinned DinD image, privileged and root,
without hostPath, host namespaces or any service-account token.

Use a shared Pod-scoped `emptyDir` mounted at `/run/docker` for the Unix socket;
run the daemon with an explicit socket group compatible with GID 1000. The
client's `DOCKER_HOST` targets that socket. Do not expose a TCP daemon listener,
even through the sidecar's inherited entrypoint defaults. Both containers mount
the managed workspace at `/workspace`; only the agent receives runtime/auth
mounts. Docker data uses its own `emptyDir` at `/var/lib/docker` with an explicit
size budget, separate from the durable checkout.

Docker bridges run inside an already reduced-MTU Pod network. Set their MTU no
higher than the actual Pod interface MTU,
and test HTTPS/large transfers from nested containers. Do not change Flannel or
the cluster API endpoint to fix a nested bridge problem.

An administrator prepare script prefixes a bounded `docker info` wait before
the existing Kubernetes preparation flow. Preserve clone/origin verification,
repository setup, agent installation and Kandev's branch-selection postlude;
do not use a wait-only replacement script. A startup probe or running main
container is not a substitute for this preparation gate.

Nested bind sources must exist in the daemon's mount namespace. `/workspace`
is deliberately shared; arbitrary agent paths such as its private runtime home
are not. Repository tests requiring callbacks, port forwarding, temporary
directories or SSH fixtures must either use supported shared workspace paths
or be reported as incompatible. Do not describe a basic Compose smoke as full
Kandev Docker-executor parity.

## Lifecycle and evidence

`executors_running` remains authoritative. The raw template already persists
in the workload snapshot, so grants need no new persistence schema. Replacement
through `executor_kubernetes_reconnect.go` composes the recorded template with
the recorded PVC. Tests must prove that edited profiles cannot add grants to
old snapshots and that existing identity-based cleanup remains intact.

Ordinary Stop retains the daemon and nested containers because it retains the
Pod. Source tests clean their own disposable Compose project. Terminal Pod
deletion must remove the nested processes; managed-PVC deletion remains Kandev's
existing UID/nonce-checked operation. Docker cache loss on replacement is expected.
Never mount a retained Docker cache across separate sessions.

Provide a bounded smoke script that tests source tools, browser launch, image
build/run, workspace bind mounts and Compose cleanup. Use real filesystem
assertions and emitted results, not method-call counts. Automated Kubernetes
coverage uses the existing Kind fixture and mock agent, then verifies visible
terminal/results and lifecycle through the existing UI. This changes no rendered
UI layout; desktop/mobile settings retain their current shared raw-template flow.

Deployment acceptance additionally uses an operator-selected agent credential,
tests each selected worker, and verifies nested cgroup accounting, cross-session
daemon isolation, results, retained compute and exact final cleanup. Sidecar resource
fields alone are not proof of nested resource enforcement. Refuse acceptance if
nested processes escape the measured Pod budget; record the result before any
design revision. Privileged operation does not promise adversarial isolation.

## Deployment and compatibility

This feature is opt-in through a privileged administrator-authored template,
not a rollout to existing users. Only a separately reviewed namespace policy
may permit the chosen companion privilege. Operators still configure placement,
tolerations, quotas, network access, workload identity and image distribution.
The backend uses a restricted kubeconfig; it does not install cluster RBAC or
copy cluster credentials into task Pods.

The existing service must run a binary containing the workspace-grant change
before it can accept the advanced template. Building the branch alone does not
update the active service. Stage a compatible build and backup, then schedule
the existing service update explicitly around active sessions; no second control
plane is a production fallback. Development/Kind fixtures remain disposable.

Public docs must distinguish experimental evidence from universal runtime support
and remove the conflicting bulk-apply guidance that could deploy another control
plane. The ordinary executor's existing behavior and image presets remain valid.

## Alternatives

Running a daemon in the main container would require changing the agent's
privilege/process model. A shared host socket would merge session trust and
daemon lifecycle. An existing claim duplicated under another mount path would
couple profiles to fixed storage and bypass the intended per-session ownership.
An explicit sidecar grant is the smallest durable change that preserves the
existing managed-workspace model. Rootless BuildKit alone does not run general
Docker/Compose workloads; a different runtime would require separate evidence.

## Delivery

See [implementation work orders](../../../plans/kubernetes-docker-workloads/plan.md).
All named new recipe, smoke and test files are implementation outputs, not
claims that they already exist.