Files
kandev/apps/backend/AGENTS.md
T
João Salavisa de92557f46 test(backend): wire goleak goroutine-leak detection across 8 packages (#1139)
* chore(backend): add go.uber.org/goleak test dependency

Promotes the previously-indirect goleak v1.3.0 to a direct require so
per-package TestMain calls can use it for goroutine leak detection.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(backend/lifecycle): drain workspace-stream backoff on Manager.Stop

StreamManager.connectWorkspaceStream slept the full retry backoff in
time.Sleep between failed attempts. SessionTraceContext is deliberately
uncancellable (it carries a long-lived span), so if the Manager began
shutting down mid-backoff the goroutine outlived Stop() and stranded
until the timer fired.

Thread Manager.stopCh through the StreamManager constructor and add a
sleepOrStop helper that selects between the timer and the shutdown
signal. ReconnectAll's startup sleep gets the same treatment.

TestConnectWorkspaceStream_BackoffDrainsOnStop closes stopCh while a
failing-port retry is in flight and asserts the goroutine exits within
500ms — fails without the fix, passes with it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(backend/github): close sqlite DB in newTestStore to drain connectionOpener

newTestStore opened *sqlx.DB via db.OpenSQLite but never registered a
Close on cleanup, leaving database/sql.(*DB).connectionOpener goroutines
parked across test boundaries. Add t.Cleanup(sqlxDB.Close); goleak's
per-package TestMain (added in a follow-up commit) is the regression
check.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(backend): wire goleak.VerifyTestMain across leak-prone packages

Per-package TestMain in:
- internal/integrations/healthpoll (auth-health ticker loop)
- internal/jira, internal/linear (auth + issue-watch loops)
- internal/github (PR / review / issue poller loops)
- internal/gateway/websocket (per-conn pumps + terminal bridge)
- internal/orchestrator (detached event-handler goroutines)
- internal/agent/runtime/lifecycle (remote-status loop, StreamManager
  helpers; ignores OTel batch-span processor and OTLP retry helper —
  initialised early in the process and only exit on shutdown)
- internal/agentctl/server/process (extends the existing fixture-binary
  TestMain so subprocess + PTY pump leaks surface there)

Catches regressions where a Start/Stop forgets to drain on WaitGroup,
or a detached goroutine forgets to honour ctx/stopCh cancellation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(backend): document goroutine ownership and leak-testing convention

New "Goroutine ownership and leak testing" subsection under Conventions
in apps/backend/AGENTS.md (CLAUDE.md symlinks to it). Covers:

- Lifecycle: Start/Stop + WaitGroup, both idempotent.
- Cancellation: every long wait selects on ctx.Done() or stopCh; never
  time.Sleep in retry loops (reference lifecycle.StreamManager.sleepOrStop).
- Detached helpers: must respect the owning type's shutdown signal.
- Leak testing: new packages spawning goroutines add goleak.VerifyTestMain;
  IgnoreTopFunction is reserved for genuine third-party background
  goroutines and must carry a comment explaining the suppression.

Lists the packages currently instrumented so future contributors can
mirror the pattern.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(backend/lifecycle): drop dead OTel IgnoreTopFunction from goleak setup

Review follow-up. agentctl/tracing.initTracing only spawns the OTel SDK
provider when OTEL_EXPORTER_OTLP_ENDPOINT is set; in tests it isn't, so
the noop provider is used and no batchSpanProcessor / OTLP retry
goroutines ever exist. The IgnoreTopFunction calls were dead
suppressions and the comment ("called by NewManager via the agent
execution trace path") misdescribed when the SDK init runs. Drop the
ignores and correct AGENTS.md to stop citing OTel as the canonical
suppression example.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(backend/lifecycle): drop unneeded 50ms sleep from backoff-drain test

QA follow-up. The 50ms time.Sleep before close(stopCh) in
TestConnectWorkspaceStream_BackoffDrainsOnStop was meant to ensure the
goroutine had entered the backoff sleep before shutdown — but that
ordering doesn't affect the test outcome:

  - With the fix: sleepOrStop sees stopCh already closed and returns
    false on the first call, so the goroutine drains within one attempt
    regardless of when stopCh closes.
  - Without the fix: time.Sleep ignores stopCh, the goroutine burns
    its full 5-retry backoff (~31s), and the 500ms timeout fires.

Removing the sleep makes the test ~12x faster (15ms vs 200ms total) and
removes the only time.Sleep-based wait from the new tests, aligning
with the codebase's "channel-based sync, no sleep waits" convention.
Verified by 50x -race and 5x revert-and-fail.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(backend/github): drain HTTP idle conns before goleak inspects

CI surfaced a leaked goroutine on the Linux runner that local runs
missed: a single net/http.(*http2ClientConn).readLoop parked on a
TLS read, created by net/http.(*http2Transport).newClientConn. The
package's anonymousHTTPClient and PATClient's rewriteTransport both
route through http.DefaultTransport's connection pool, so an idle
HTTP/2 connection in that pool keeps the reader goroutine alive past
m.Run() — invisible on sandboxed local runs where outbound network
is blocked.

Replace github/goleak_test.go's plain VerifyTestMain with an
expanded TestMain that drains http.DefaultTransport's idle
connections before goleak.Find runs. This is the stdlib's
documented teardown for the connection pool, keeps the assertion
strict (no IgnoreTopFunction), and doesn't leak into production
code. Other in-scope packages don't need this — they don't use
http.DefaultTransport for HTTPS calls.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(backend): drain ws.Done on stopCh; always run goleak in github tests

- lifecycle/streams.go: replace bare <-ws.Done() with a select that also
  watches sm.stopCh so a connected goroutine drains when Manager.Stop()
  fires before the remote WS closes. A nil stopCh (isolated tests) blocks
  the nil channel case forever, degrading to plain <-ws.Done() semantics.
- github/goleak_test.go: remove the exitCode==0 guard around goleak.Find
  so leaks are always surfaced even when a test has already failed; original
  exit code is preserved (leaks only promote it to 1).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(backend/lifecycle): close ws on stopCh so agentctl WS loops drain

cubic flagged that the previous shutdown path observed stopCh but never
called ws.Close(), so the underlying WebSocket and its agentctl-side
read/write goroutines (workspace_stream.go:67-68) kept running past
Manager.Stop(). ClearWorkspaceStream only removes the WS from the
execution's tracking — it doesn't close the connection.

Call ws.Close() on the stopCh path. ws.Close is idempotent via
closeOnce, so it's safe even if the remote end already closed.

TestConnectWorkspaceStream_ClosesWSOnStop connects to a mock workspace
server that stays open until the client closes its side; after stopCh
fires, the test asserts ws.Done() closes within 500ms — which is only
true if connectWorkspaceStream invoked ws.Close(). Verified to fail
without the fix and pass with it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(backend/lifecycle): use channel sync instead of sleep polling in ws-close test

cubic flagged that TestConnectWorkspaceStream_ClosesWSOnStop polled
execution.GetWorkspaceStream() in a 500ms sleep loop, which is brittle
under CI load. connectWorkspaceStream already accepts a ready
channel that closes after SetWorkspaceStream — pass it in and wait on
the channel instead.

synctest isn't viable here (real network I/O), so channel-based sync is
the testing convention per AGENTS.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Kandev Agent <agent@kandev.dev>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 10:23:18 +01:00

17 KiB

Backend (Go) — architecture and conventions

Scoped guidance for apps/backend/. Repo-wide rules (commit format, code-quality limits, etc.) live in the root AGENTS.md.

Package Structure

apps/backend/
├── cmd/
│   ├── kandev/           # Main backend binary entry point
│   ├── agentctl/         # Agentctl binary (runs inside containers or standalone)
│   └── mock-agent/       # Mock agent for testing
├── internal/
│   ├── agent/
│   │   ├── runtime/      # Agent runtime: single seam for Launch/Resume/Stop/observe
│   │   │   ├── lifecycle/    # Agent instance management (moved from agent/lifecycle)
│   │   │   ├── agentctl/     # HTTP client for talking to agentctl (moved from agentctl/client)
│   │   │   └── routingerr/   # Provider error classifier + sanitizer + ProviderProber registry
│   │   ├── agents/       # Agent type implementations
│   │   ├── controller/   # Agent control operations
│   │   ├── credentials/  # Agent credential management
│   │   ├── discovery/    # Agent discovery
│   │   ├── docker/       # Docker-specific agent logic
│   │   ├── dto/          # Agent data transfer objects
│   │   ├── executor/     # Executor types, checks, and service
│   │   ├── handlers/     # Agent event handlers
│   │   ├── registry/     # Agent type registry and defaults
│   │   ├── settings/     # Agent settings
│   │   ├── mcpconfig/    # MCP server configuration
│   │   └── remoteauth/   # Remote auth catalog and method IDs for remote executors/UI
│   ├── agentctl/
│   │   └── server/       # agentctl HTTP server
│   │       ├── acp/      # ACP protocol implementation
│   │       ├── adapter/  # Protocol adapters + transport/ (ACP, Codex, OpenCode, Copilot, Amp)
│   │       ├── api/      # HTTP endpoints
│   │       ├── config/   # agentctl configuration
│   │       ├── instance/ # Multi-instance management
│   │       ├── mcp/      # MCP server integration
│   │       ├── process/  # Agent subprocess management
│   │       ├── shell/    # Shell session management
│   │       └── utility/  # agentctl utilities
│   ├── orchestrator/     # Task execution coordination
│   │   ├── dto/          # Orchestrator data transfer objects
│   │   ├── executor/     # Launches agents via lifecycle manager
│   │   ├── handlers/     # Orchestrator event handlers
│   │   ├── messagequeue/ # Message queue for agent prompts
│   │   ├── queue/        # Task queue
│   │   ├── scheduler/    # Task scheduling
│   │   └── watcher/      # Event handlers
│   ├── task/
│   │   ├── controller/   # Task HTTP/WS controllers
│   │   ├── dto/          # Task data transfer objects
│   │   ├── events/       # Task event types
│   │   ├── handlers/     # Task event handlers
│   │   ├── models/       # Task, Session, Executor, Message models
│   │   ├── repository/   # Database access (SQLite)
│   │   └── service/      # Task business logic
│   ├── office/           # Autonomous agent management (Office feature)
│   │   ├── agents/       # Agent instance CRUD + auth guards
│   │   ├── approvals/    # Approval requests and decisions
│   │   ├── channels/     # External integration channels (webhooks)
│   │   ├── config/       # Config sync (DB ↔ filesystem)
│   │   ├── configloader/ # Filesystem config reader/writer
│   │   ├── costs/        # Cost tracking and budget policies
│   │   ├── dashboard/    # Dashboard API, issues, activity, live runs
│   │   ├── infra/        # GC, reconciliation
│   │   ├── labels/       # Task labels
│   │   ├── onboarding/   # Workspace onboarding wizard API
│   │   ├── projects/     # Project management
│   │   ├── repository/   # Office SQLite persistence
│   │   ├── runtime/      # Agent run context, capabilities, and runtime action surface
│   │   ├── routines/     # Scheduled recurring tasks
│   │   ├── routing/      # Provider routing: resolver, validators, catalogue, backoff, agent-overrides
│   │   ├── scheduler/    # Wakeup scheduler (duplicate of service scheduler features)
│   │   ├── service/      # Core office service (wakeups, event subscribers, execution policy)
│   │   ├── shared/       # Shared interfaces and activity logging
│   │   ├── skills/       # Skill injection and materialization
│   │   └── workspaces/   # Workspace deletion handler
│   ├── events/           # Event bus for internal pub/sub
│   ├── gateway/          # WebSocket gateway
│   ├── github/           # GitHub API integration (PRs, reviews, webhooks)
│   ├── common/           # Shared utilities, config, logger
│   ├── integration/      # External integrations
│   ├── integrations/     # Shared shapes for third-party integrations
│   │   ├── healthpoll/   # Reusable 90s auth-health Poller (used by jira, linear)
│   │   └── secretadapter/ # Upsert-style adapter over secrets.SecretStore
│   ├── jira/             # Jira/Atlassian Cloud integration (config, REST client, poller)
│   ├── linear/           # Linear integration (config, GraphQL client, poller)
│   ├── lsp/              # LSP server
│   ├── mcp/              # MCP protocol support
│   ├── health/           # Health check endpoints
│   ├── notifications/    # Notification system
│   ├── persistence/      # Persistence layer
│   ├── prompts/          # Prompt management
│   ├── repoclone/        # Repository cloning for remote executors
│   ├── scriptengine/     # Script placeholder resolution and interpolation
│   ├── secrets/          # Secret management
│   ├── sprites/          # Sprites AI integration
│   ├── sysprompt/        # System prompt injection
│   ├── tools/            # Tool integrations
│   ├── user/             # User management
│   ├── utility/          # Shared utility functions
│   ├── workflow/         # Workflow engine
│   │   ├── engine/       # Typed state-machine engine
│   │   ├── models/       # Workflow step, template, and history models
│   │   ├── repository/   # Workflow persistence (SQLite)
│   │   └── service/      # Workflow CRUD and step resolution
│   └── worktree/         # Git worktree management for workspace isolation

Key Concepts

Orchestrator coordinates task execution:

  • Receives task start/stop/resume requests via WebSocket
  • Delegates to lifecycle manager for agent operations
  • Handles event-driven state transitions via workflow engine
  • Located in internal/orchestrator/

Watcher Dispatch Coordinator (internal/orchestrator/watcher_dispatch.go) is the single pipeline that turns a freshly-observed external issue (Linear, Jira, future) into a Kandev task. Bus subscribers for each integration forward the event to WatcherDispatchCoordinator.Dispatch with a per-integration WatcherSource implementation (source_linear.go, source_jira.go). Source methods carry the integration-specific bits (reserve dedup, build task request, attach task ID, release, auto-start params); the coordinator owns the cross-cutting pipeline (create task, decide auto-start, error/release handling). Add a new watcher = implement WatcherSource + register a one-line bus subscriber. Do NOT add another createXIssueTask mirror.

Workflow Engine (internal/workflow/engine/) provides typed state-machine evaluation:

  • Engine.HandleTrigger() evaluates step actions for triggers (on_enter, on_turn_start, on_turn_complete, on_exit)
  • TransitionStore interface abstracts persistence (implemented by orchestrator.workflowStore)
  • CallbackRegistry maps action kinds to callbacks (plan mode, auto-start, context reset)
  • First-transition-wins: multiple transition actions in one trigger, first eligible wins
  • EvaluateOnly mode: engine evaluates without persisting, caller orchestrates on_exit → DB → on_enter
  • RequiresApproval on actions: transitions requiring review gating are skipped
  • Idempotent by OperationID; session-scoped data bag via MachineState.Data

Agent Runtime (internal/agent/runtime/) is the single seam for launching, resuming, stopping, and observing agent executions. ADR 0004 introduced this in Phase 1 of task-model-unification. The public surface is runtime.Runtime (runtime.go); a thin facade (facade.go) delegates to a Backend (satisfied by *lifecycle.Manager).

Convention: only internal/agent/runtime/ (and code that pre-dates Phase 1 migration) may import runtime/lifecycle or runtime/agentctl directly. New consumers — workflow engine actions, cron-driven trigger handlers, future task-tier callers — should depend on runtime.Runtime. Existing call sites are migrated through later phases of task-model-unification.

Lifecycle Manager (internal/agent/runtime/lifecycle/) manages agent instances under the runtime:

  • Manager (manager.go, manager_*.go) - central coordinator for agent lifecycle
  • ExecutorBackend interface (executor_backend.go) - abstracts execution environment (Docker, Standalone, Sprites, Remote Docker)
  • ExecutionStore (execution_store.go) - thread-safe in-memory execution tracking
  • session.go - ACP session initialization and resume
  • streams.go - WebSocket stream connections to agentctl
  • process_runner.go - agent process launch and management
  • profile_resolver.go - resolves agent profiles/settings

agentctl client (internal/agent/runtime/agentctl/) is the HTTP/WS client used by the lifecycle manager to talk to a running agentctl instance. It is a runtime-tier package and should not be imported outside internal/agent/runtime/.

agentctl is an HTTP server that:

  • Runs inside Docker containers or as standalone process
  • Manages agent subprocess via stdin/stdout (ACP protocol)
  • Exposes workspace operations (shell, git, files)
  • Supports multiple concurrent instances on different ports

Executor Types (database model):

  • local_pc - Standalone process on host
  • local_docker - Docker container on host
  • sprites - Sprites cloud environment
  • remote_docker, remote_vps, k8s - Planned

Execution Flow

Client (WS) → Orchestrator → Lifecycle Manager → ExecutorBackend (container/process) → agentctl
                                                                                          ↓
Client (WS) ← Orchestrator ← Lifecycle Manager ←──── stream updates (WS) ──────── agent subprocess
  1. Orchestrator receives session.launch via WS
  2. Lifecycle Manager creates executor instance (container or process)
  3. agentctl starts inside the instance, agent subprocess is configured and started
  4. Agent events stream back via WS through the chain

Session Resume: TaskSession.ACPSessionID stored for resume; ExecutorRunning tracks active state; on restart RecoverInstances() reconnects.

Provider Pattern: Packages expose Provide(cfg, log) (*impl, cleanup, error) for DI. Returns implementation, cleanup function, and error. Cleanup called during graceful shutdown.

Worktrees: internal/worktree/Manager provides workspace isolation. Each session can have its own worktree (branch) to prevent conflicts between concurrent agents.

Executor default scripts: Default prepare scripts are in internal/agent/runtime/lifecycle/default_scripts.go; internal/scriptengine/ handles placeholder resolution.

Conventions

  • Provider pattern for DI; stderr for logs, stdout for ACP only.
  • Pass context through chains; event bus for cross-component comm.
  • Execution access: Workspace-oriented handlers (files, shell, inference, ports, vscode, LSP) MUST use GetOrEnsureExecution(ctx, sessionID) — it recovers from backend restarts by creating executions on-demand. Only use GetExecutionBySessionID for operations that require a running agent process (prompt, cancel, mode).
  • Task lifecycle events: Any code path that mutates a task row must publish via the event bus (task.created / task.updated / task.deleted) — either by going through Service.CreateTask / UpdateTask / DeleteTask / ArchiveTask, or by calling publishTaskEvent (or one of the Publish* helpers in service_events.go) directly. Walking repository.TaskRepository straight bypasses event publishing and breaks WS-driven UI like the All-Workflows kanban view. HandoffService's cascade methods learned this the hard way — they now require a TaskEventPublisher wired via SetTaskEventPublisher. New cascade / bulk / cleanup paths must follow the same pattern.
  • Testing: Prefer testing/synctest (Go 1.24+) over time.Sleep for time-dependent tests. Use synctest.Test to wrap tests with tickers or timeouts — it advances fake time instantly when all goroutines are idle. When synctest is not feasible (e.g., tests spawning external processes like git), use channel-based synchronization (<-started, non-blocking select) instead of sleep-based waits. Reserve time.Sleep only for integration tests that need real subprocess execution time.

Goroutine ownership and leak testing

Every long-running goroutine must have a single owner with explicit start and stop semantics:

  • Lifecycle: the type that spawns the goroutine also exposes Start(ctx) / Stop() (or equivalent). Start registers on a sync.WaitGroup; Stop cancels the goroutine's context (or closes a stopCh) and wg.Wait()s for drain. Idempotent on both ends. internal/integrations/healthpoll, internal/jira, internal/linear, and internal/github pollers are the canonical shape.
  • Cancellation: the goroutine selects on ctx.Done() (or stopCh) in every long wait. Never use time.Sleep in a retry/backoff loop — use time.NewTimer inside a select that also watches the shutdown signal (see lifecycle.StreamManager.sleepOrStop).
  • Detached helpers: event handlers and short-lived go func() calls in internal/orchestrator/ and internal/agent/runtime/lifecycle/ must accept a cancellable context (or check the owning type's shutdown signal) and return promptly when it fires.
  • Leak testing: packages that spawn goroutines add goleak.VerifyTestMain(m) in a per-package TestMain. New packages of this kind must follow suit. When a third-party background goroutine genuinely can't be drained, suppress it with goleak.IgnoreTopFunction(...) and leave a comment explaining why. Currently instrumented: internal/gateway/websocket/, internal/agent/runtime/lifecycle/, internal/agentctl/server/process/, internal/orchestrator/, internal/github/, internal/jira/, internal/linear/, internal/integrations/healthpoll/.

Backups

  • On every SQLite boot, persistence.Provide reads kandev_meta.kandev_version. If the stored version differs from the binary version (or any user tables exist but no version is recorded), it takes a VACUUM INTO snapshot into <data-dir>/backups/ before running migrations.
  • Retention: 2 backups kept (newest two by mtime); older ones are pruned after the snapshot succeeds.
  • Postgres: backup is skipped with a log line. Use pg_dump for Postgres backups.
  • Boot aborts if the backup fails — the pool is closed and Provide returns an error.
  • After all repos complete initSchema, cmd/kandev/storage.go:recordSchemaVersion writes the current binary version into kandev_meta (non-fatal; a failure just means the next boot will take a fresh snapshot).
  • Migration logging: db.MigrateLogger.Apply(name, stmt) — success logs Info, "already exists" / "duplicate column name" is silently swallowed, anything else logs Warn but never returns an error (preserving the existing swallow-error contract).

Code-quality limits

Enforced by apps/backend/.golangci.yml (errors on new code only):

  • Functions: ≤80 lines, ≤50 statements
  • Cyclomatic complexity: ≤15 · Cognitive complexity: ≤30
  • Nesting depth: ≤5 · Naked returns only in functions ≤30 lines
  • No duplicated blocks (≥150 tokens) · Repeated strings → constants (≥3 occurrences)

When you hit a limit, extract a helper function. Prefer composition over growing a single function.

Further scoped notes

  • internal/agentctl/AGENTS.md — agentctl server route groups, adapter model, ACP protocol
  • internal/agentctl/server/api/AGENTS.md — reverse-proxy body rewriting (Accept-Encoding), iframe-blocking header stripping
  • internal/integrations/AGENTS.md — playbook for adding a new third-party integration (Jira/Linear pattern)