José Almeida
11 KiB
status, system, requirements
| status | system | requirements | |||
|---|---|---|---|---|---|
| draft | executors |
|
Kubernetes Docker Workloads System Design
Purpose and boundaries
The executor owns workspace composition, workload snapshots and exact resource cleanup. The requirements introduce explicit companion workspace access plus an opt-in Docker workload recipe. Operators supply deployment-specific placement, credentials and capacity. The implementation plan contains only the reusable executor change, recipe and integration evidence.
The existing resource-ownership decision continues to govern names, UIDs, nonces, storage identity and cleanup. The new mount exception must be reconciled with the foundation and public documentation when implemented. Existing release behavior still rejects the exception until that change ships. This design contains its rationale; no independent ADR is needed for a parallel copy of the same contract.
Requirement mapping
| Requirement | Design section |
|---|---|
| REQ-EXECUTORS-K8S-DOCKER-001 | Workspace grant and admission |
| REQ-EXECUTORS-K8S-DOCKER-002 | Worker and daemon configuration |
| REQ-EXECUTORS-K8S-DOCKER-003 | Lifecycle and evidence |
Workspace grant and admission
Use existing pod_template_yaml; add no profile/API field, migration or UI.
An ordinary container other than ProfileConfig.MainContainer may explicitly
reference kandev-workspace at exactly /workspace once. readOnly selects
access mode. Reject subpaths, subpath expressions, mount propagation, recursive
mount options, alternate paths/devices and additional overlapping mounts. The
template still cannot define the volume itself. Main, init and ephemeral
container rules remain unchanged, including all runtime/auth mount prohibitions.
ValidatePodTemplate and validateContainerReservedFields in
internal/agent/kubernetes/template.go need container-role context for this
narrow exception. ComposePod preserves the requested sidecar reference and
continues to inject/own the actual workspace volume and the main-container
mount. All three workspace modes use the same grant rule.
ValidateAdmittedPod in internal/agent/kubernetes/admission.go must carry the
desired Pod's ordinary-container identities into companion mount validation.
Compare grants by container name and full permitted mount shape, not by array
position, volume name alone or the main container's mount. Verify both directions:
every admitted grant must have been requested, and every requested grant must
remain exactly once on the same named ordinary container. Reject added/removed
grants, missing/renamed recipients, duplicate grants, changed access modes and
attempts to move a grant into init or ephemeral containers.
Normalize only documented neutral API defaults before comparison; real API integration must prove any such normalization. Continue accepting unrelated labels and compatible additions that carry no reserved access. Do not weaken existing main-container, control-port, env, volume, nonce or metadata validation. Existing Kandev-owned volumes retain exact-source validation.
The grant is an administrative data-access decision, not a security barrier against a privileged companion. A writable companion can alter repository contents. Runtime/auth mounts are not explicitly shared with it.
Worker and daemon configuration
Add a separate recipe under k8s/worker-images/full/; preserve the existing
minimal/node-pnpm/python recipe and its checks. Derive from an immutable
universal digest and pin all added inputs in the new recipe's pins.env.
Resolve actual registry digests during implementation; source examples may use
clearly marked substitutions, but deployable configuration must reject them.
Use mise.toml, the Go module and pnpm lockfile for repository-compatible tools.
The inspected universal defaults need golangci-lint v2 instead of v1.62.0 and
Playwright libraries/browser matching the resolved current package. Include
Go, Rust, Node, pnpm, Python/venv, native compilers, Docker CLI, Buildx and Compose.
Baked tools/browser binaries live outside mounted workspace/auth paths.
Set writable npm, pnpm, Python and Go caches under /workspace, but create
them after initial repository clone: the default Kubernetes prepare script
correctly refuses a fresh nonempty non-repository workspace. Kandev owns HOME
and injects it as /run/kandev/home; do not freeze host-home paths into config.
Agent CLIs continue through Kandev's selected-agent installation path.
The advanced template has ordinary containers kandev-agent and docker-engine.
The agent uses UID/GID 1000, dropped capabilities, no escalation and default
seccomp. Pod fsGroup is 1000. Put non-root constraints on the main container;
a Pod-wide runAsNonRoot would incorrectly conflict with the chosen daemon.
The daemon is an explicitly selected pinned DinD image, privileged and root,
without hostPath, host namespaces or any service-account token.
Use a shared Pod-scoped emptyDir mounted at /run/docker for the Unix socket;
run the daemon with an explicit socket group compatible with GID 1000. The
client's DOCKER_HOST targets that socket. Do not expose a TCP daemon listener,
even through the sidecar's inherited entrypoint defaults. Both containers mount
the managed workspace at /workspace; only the agent receives runtime/auth
mounts. Docker data uses its own emptyDir at /var/lib/docker with an explicit
size budget, separate from the durable checkout.
On cgroup v2, configure Docker's cgroupfs driver with a relative cgroup parent. This keeps nested scopes beneath the daemon container's cgroup on compatible runtimes instead of placing them in a host-root systemd slice. Treat measured ancestry and accounting as runtime acceptance evidence; configuration alone is not proof of containment.
Docker bridges run inside an already reduced-MTU Pod network. Set their MTU no higher than the actual Pod interface MTU, and test HTTPS/large transfers from nested containers. Do not change Flannel or the cluster API endpoint to fix a nested bridge problem.
An administrator prepare script prefixes a bounded docker info wait before
the existing Kubernetes preparation flow. Preserve clone/origin verification,
repository setup, agent installation and Kandev's branch-selection postlude;
do not use a wait-only replacement script. The backend uploads and executes the
resolved script through the existing Pod exec channel within the setup budget,
captures a bounded sanitized diagnostic, and creates the preparation marker
only on success. It then signals the waiting managed entrypoint. A failed
prepare command must return without releasing agentctl; a startup probe or a
running main container is not a substitute for this preparation gate.
Nested bind sources must exist in the daemon's mount namespace. /workspace
is deliberately shared; arbitrary agent paths such as its private runtime home
are not. Repository tests requiring callbacks, port forwarding, temporary
directories or SSH fixtures must either use supported shared workspace paths
or be reported as incompatible. Do not describe a basic Compose smoke as full
Kandev Docker-executor parity.
Lifecycle and evidence
executors_running remains authoritative. The raw template already persists
in the workload snapshot, so grants need no new persistence schema. Replacement
through executor_kubernetes_reconnect.go composes the recorded template with
the recorded PVC. Tests must prove that edited profiles cannot add grants to
old snapshots and that existing identity-based cleanup remains intact.
Ordinary Stop retains the daemon and nested containers because it retains the Pod. Source tests clean their own disposable Compose project. Terminal Pod deletion must remove the nested processes; managed-PVC deletion remains Kandev's existing UID/nonce-checked operation. Docker cache loss on replacement is expected. Never mount a retained Docker cache across separate sessions.
Provide a bounded smoke script that tests source tools, browser launch, image build/run, workspace bind mounts and Compose cleanup. Use real filesystem assertions and emitted results, not method-call counts. Automated Kubernetes coverage uses the existing Kind fixture and mock agent, then verifies visible terminal/results and lifecycle through the existing UI. This changes no rendered UI layout; desktop/mobile settings retain their current shared raw-template flow.
Deployment acceptance additionally uses an operator-selected agent credential, tests each selected worker, and verifies nested cgroup accounting, cross-session daemon isolation, results, retained compute and exact final cleanup. Sidecar resource fields alone are not proof of nested resource enforcement. Refuse acceptance if nested processes escape the measured Pod budget; record the result before any design revision. Privileged operation does not promise adversarial isolation.
Deployment and compatibility
This feature is opt-in through a privileged administrator-authored template, not a rollout to existing users. Only a separately reviewed namespace policy may permit the chosen companion privilege. Operators still configure placement, tolerations, quotas, network access, workload identity and image distribution. The backend uses a restricted kubeconfig; it does not install cluster RBAC or copy cluster credentials into task Pods.
The existing service must run a binary containing the workspace-grant change before it can accept the advanced template. Building the branch alone does not update the active service. Stage a compatible build and backup, then schedule the existing service update explicitly around active sessions; no second control plane is a production fallback. Development/Kind fixtures remain disposable.
Public docs must distinguish experimental evidence from universal runtime support and remove the conflicting bulk-apply guidance that could deploy another control plane. The ordinary executor's existing behavior and image presets remain valid.
Alternatives
Running a daemon in the main container would require changing the agent's privilege/process model. A shared host socket would merge session trust and daemon lifecycle. An existing claim duplicated under another mount path would couple profiles to fixed storage and bypass the intended per-session ownership. An explicit sidecar grant is the smallest durable change that preserves the existing managed-workspace model. Rootless BuildKit alone does not run general Docker/Compose workloads; a different runtime would require separate evidence.
Delivery
See implementation work orders. All named new recipe, smoke and test files are implementation outputs, not claims that they already exist.