kandev/docs/decisions/2026-08-18-executor-userns-security-options.md

# Executor Container Security Options for User Namespace Support

| Metadata | Value |
|---|---|
| **Date** | 2026-08-18 |
| **Status** | accepted |
| **Tags** | backend, security, docker |
| **Replaces** | (none — no prior ADR covers this boundary) |

## Context

Kandev's Local Docker executor launches every task container with no `SecurityOpt` set, so Docker applies its defaults: the default seccomp profile plus the `docker-default` AppArmor profile. Both deny `CLONE_NEWUSER` to processes without `CAP_SYS_ADMIN` — which is every process in a Kandev task container, since Kandev never adds that capability.

Any agent runtime that sandboxes its own file edits via user namespaces therefore fails. The observed symptom is Codex's `apply_patch` → `bwrap: No permissions to create a new namespace`. This stalled four board tasks on 2026-08-17/18. The interim mitigation was pinning agent profiles away from the affected runtime; this ADR records the durable fix, after which pinning can be reverted.

Empirical probe results on the target host confirmed two independent blocking layers:

1. **seccomp** — Docker's default profile gates `clone` on a `SCMP_CMP_MASKED_EQ` check over the namespace flags and puts `unshare`, `mount`, `umount2`, and `setns` in a `CAP_SYS_ADMIN`-gated group. It does not name `pivot_root`, so the default action denies it.
2. **AppArmor** — the host runs `apparmor_restrict_unprivileged_userns=1` (Ubuntu 24.04 kernel), and `docker-default` carries no `userns` allow rule plus an explicit `deny mount,`.

Relaxing only one is not enough: seccomp alone fails because AppArmor blocks mount operations inside user namespaces, and AppArmor alone fails because seccomp blocks namespace creation.

## Decision

Add a per-executor-profile boolean `allow_user_namespaces` that, when enabled, emits two Docker `SecurityOpt` entries at container creation time:

- `seccomp=<Kandev-tailored seccomp profile>`
- `apparmor=unconfined`

The profile is a modified copy of Docker's default seccomp profile (vendored from the moby/profiles repository at v29.7.1). The modifications are:

- `clone`: the `SCMP_CMP_MASKED_EQ` namespace-flag restriction for non-CAP_SYS_ADMIN processes is removed.
- `clone3`: the `SCMP_ACT_ERRNO` (ENOSYS) fallback for non-CAP_SYS_ADMIN processes is removed.
- `mount`, `mount_setattr`, `move_mount`, `open_tree`, `setns`, `umount`, `umount2`, `unshare`: moved from the `CAP_SYS_ADMIN`-gated allow list into the unconditional allow list.
- `pivot_root`: added to the unconditional allow list. Docker's default profile does not name it in any rule, so it falls through to the profile's `SCMP_ACT_ERRNO` default. `bwrap` calls `pivot_root` immediately after creating its namespace and aborts with `bwrap: pivot_root: Operation not permitted` when it is denied, so relaxing `clone`/`unshare` alone leaves the motivating use case broken. Verified against a live daemon: with `pivot_root` denied, `unshare -U true` succeeds but `bwrap --unshare-user --dev-bind / / true` exits 1; with it allowed, both exit 0.
- Compatibility: nine optional syscall allows that older Docker/libseccomp daemons cannot load are removed. These syscalls remain denied by the profile default action.

Apart from that compatibility filter, every other syscall restriction in Docker's default profile is preserved. `kexec_load`, `bpf`, `perf_event_open`, `add_key`, and ~50 other syscalls Docker blocks for good reason remain blocked.

The setting is:
- **Off by default** — every existing profile produces byte-identical launch config.
- **Per-profile** — stored as a `config` entry (`allow_user_namespaces: "true"`), not a typed DB column.
- **Authoritative** — the profile value unconditionally overwrites any task-supplied metadata value, preventing self-escalation. Launch-config resolution clears the key before applying the profile, so the guarantee also holds for launches that attach no executor profile, carry a stale profile ID, or hit a profile lookup error; task metadata is caller-writable through the task REST/WS API and is not key-filtered there.
- **Gated from the MCP surface** — the `create_executor_profile` and `update_executor_profile` MCP handlers reject the key with a validation error. The operator HTTP/WS API remains unguarded; this is accepted because the same operator surface already exposes `prepare_script` (arbitrary shell at every launch), a strictly more powerful primitive.
- **Create-time only** — `SecurityOpt` cannot be changed on an existing container. Operators must reset task environments after enabling it.

## Rationale

### Why not free-form `security_opt` list?
Maximum flexibility, but it hands every profile editor `seccomp=unconfined`, `apparmor=unconfined`, and `privileged` — the setting becomes an arbitrary container-escape switch rather than a single, reviewed relaxation.

### Why not `seccomp=unconfined`?
It would restore `kexec_load`, `bpf`, `perf_event_open`, `add_key`, `open_by_handle_at`, and ~50 other syscalls. The tailored profile is strictly better: it only relaxes namespace-related syscalls and removes optional names for older daemon compatibility.

### Why not `--privileged` or `CapAdd: SYS_ADMIN`?
These grant real host privilege. User namespaces grant privilege only within the new namespace. Strictly worse for the same outcome.

### Why not daemon-wide `userns-remap` / `HostConfig.UsernsMode`?
A different feature (remapping container root to host subuids); does not enable nested namespace creation, and is daemon-global rather than per-profile.

### Why not a Kandev-authored AppArmor profile instead of `apparmor=unconfined`?
Strictly better security, but loading it needs host root and `apparmor_parser`, and Kandev's control plane is frequently itself a container. Left as a documented future escape hatch.

### Why `apparmor=unconfined` for the AppArmor half?
`apparmor_restrict_unprivileged_unconfined=0` on the target hosts means an unconfined process is permitted to create user namespaces. The `apparmor=unconfined` SecurityOpt is what makes this work. On hosts without AppArmor, `apparmor=unconfined` is a no-op.

## Consequences

### Positive

- Agent runtimes that use user namespaces (Codex's `apply_patch`, bwrap-based sandboxing) work inside Kandev container executors without `CAP_SYS_ADMIN` or `--privileged`.
- Default-off means zero risk for profiles that don't need this.
- The tailored seccomp profile preserves all other Docker security hardening and keeps compatibility-filtered calls denied by default.
- The profile is vendored and version-pinned; tests assert the namespace additions and compatibility removals.

### Negative

- `apparmor=unconfined` is the coarse half of the fix. It removes `deny mount,` and all other docker-default AppArmor restrictions — a broader relaxation than the user namespace itself needs. This is mitigated by default-off + explicit per-profile opt-in.
- User namespaces are historically the single richest source of Linux LPE CVEs. The exposure is kernel attack surface, not a direct grant: privilege obtained inside the namespace does not map to host privilege. Capabilities (still no `CAP_SYS_ADMIN`), cgroups, and namespace isolation remain in force.
- The setting can still be set through the operator HTTP/WS API by anything that can reach the backend port. This is accepted because the same surface already exposes `prepare_script`.

### Risks

- Create-time-only: toggling the profile affects only newly created containers. Operators must be aware of this (stated in UI copy and docs).
- The seccomp base profile is vendored from moby/profiles. A periodic manual refresh on moby upgrades is needed; automating it is out of scope.

## Future Work

- A Kandev-authored AppArmor profile (and a `docker_apparmor_profile` key to name a site-loaded one instead of `unconfined`).
- Installing `bwrap` / `newuidmap` into Kandev-provided agent images.