kandev/docs/tracing.md

Backend Tracing

Kandev includes optional OpenTelemetry (OTel) tracing for debugging agent communication. Two tracing layers are available:

1. Agent protocol (kandev-agentctl) — traces protocol events between agentctl and the agent subprocess (ACP, StreamJSON, Codex, OpenCode, Amp, Copilot)
2. Transport (kandev-transport) — traces HTTP, WebSocket, and session lifecycle between the backend and agentctl

Both layers use a shared OTel initialization in internal/agentctl/tracing/ but have different activation requirements.

Enabling Tracing

Transport layer (backend <> agentctl)

Only requires the OTLP endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

Agent protocol layer (agentctl <> agent subprocess)

Requires both the OTLP endpoint and the debug flag:

export KANDEV_DEBUG_AGENT_MESSAGES=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

KANDEV_DEBUG_AGENT_MESSAGES=true also enables JSONL debug log files (see below).

Without OTEL_EXPORTER_OTLP_ENDPOINT, all tracers are no-op with zero runtime overhead.

Running a Trace Collector

Any OTLP-compatible collector works. Point OTEL_EXPORTER_OTLP_ENDPOINT to its HTTP receiver (typically port 4318).

Jaeger (quickstart)

docker run --rm --name jaeger \
        -p 16686:16686 \
        -p 4317:4317 \
        -p 4318:4318 \
        -p 5778:5778 \
        -p 9411:9411 \
        cr.jaegertracing.io/jaegertracing/jaeger:2.15.0

• UI: http://localhost:16686
• OTLP HTTP receiver: http://localhost:4318

SigNoz / Grafana Tempo

Point OTEL_EXPORTER_OTLP_ENDPOINT to the OTLP HTTP receiver (e.g. http://localhost:4318).

What Gets Traced

Session lifecycle

Each agent session gets a long-lived root span (session or session.recovered for recovered sessions after backend restart). This span groups all activity for the session and is exported when the session ends.

A short-lived session.init child span covers the initialization phase (stream connection, ACP handshake, session creation). This span ends immediately after init, making init-phase operations visible in the trace backend while the session is still active.

Note: The root session span appears as "Missing Span" in trace UIs while the session is active. This is expected — the OTel batch exporter only exports completed spans. Once the session ends (stop, completion, or deletion), the root span is exported and the full trace tree becomes visible.

Transport layer (kandev-transport)

HTTP calls — all backend-to-agentctl HTTP requests are traced with method, path, status code, and execution ID.

Agent WebSocket stream — outgoing requests (initialize, session/new, session/load, prompt, cancel, permission responses) are traced as round-trip spans. Incoming agent events (agent.event.<type>) are traced as instant spans with the full raw JSON event attached as a span event. MCP request relays are also traced.

Workspace WebSocket stream — only low-volume events are traced to avoid noise (git operations, file changes, process status, connection/error events). High-volume events (shell I/O, process output, ping/pong) are skipped.

Turn lifecycle — a turn_end span is created when an agent turn completes, recording the stop reason and error state.

All transport spans include execution_id and session_id attributes for filtering.

Agent protocol layer (kandev-agentctl)

Traces protocol events between agentctl and the agent subprocess. Each event span includes both the raw protocol JSON and the normalized AgentEvent as span events, enabling side-by-side comparison. Prompt spans include session_id and prompt length.

Debug Log Files

When KANDEV_DEBUG_AGENT_MESSAGES=true is set, raw and normalized protocol events are also written to JSONL files in the working directory (or KANDEV_DEBUG_LOG_DIR):

• raw-{protocol}-{agentId}.jsonl — raw protocol JSON
• normalized-{protocol}-{agentId}.jsonl — normalized AgentEvent

Architecture

internal/agentctl/tracing/
├── otel.go           # Shared OTel TracerProvider init (lazy, sync.Once)
└── transport.go      # Transport-layer span helpers (session, HTTP, WS, events)

internal/agentctl/server/adapter/transport/shared/
├── tracing.go        # Protocol-layer tracing (delegates OTel init to tracing/)
└── debug.go          # JSONL file logging + protocol constants

internal/agentctl/client/
├── client.go         # HTTP call tracing
├── client_stream.go  # WS request/response tracing
├── agent.go          # Agent event + MCP dispatch tracing
└── workspace_stream.go  # Workspace event tracing

internal/agent/lifecycle/
├── session.go        # Session span creation (session.init)
├── manager_lifecycle.go  # Recovery span creation (session.recovered)
└── types.go          # AgentExecution.SessionTraceContext / EndSessionSpan

The executionID (instance ID) is passed to each Client via WithExecutionID() and appears on every span, making it easy to filter traces for a specific agent session.

Example: Debugging a Failed Resume

1. Start your trace collector and set the env vars
2. Trigger a session resume in the UI
3. Search for service kandev-agentctl or filter by execution_id=<instance-id>
4. Look for the session.init span — init-phase operations (initialize, session/load, prompt) will be nested under it
5. If a span has an error status, expand it to see the recorded error