test(backend): wire goleak goroutine-leak detection across 8 packages (#1139)
* chore(backend): add go.uber.org/goleak test dependency Promotes the previously-indirect goleak v1.3.0 to a direct require so per-package TestMain calls can use it for goroutine leak detection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(backend/lifecycle): drain workspace-stream backoff on Manager.Stop StreamManager.connectWorkspaceStream slept the full retry backoff in time.Sleep between failed attempts. SessionTraceContext is deliberately uncancellable (it carries a long-lived span), so if the Manager began shutting down mid-backoff the goroutine outlived Stop() and stranded until the timer fired. Thread Manager.stopCh through the StreamManager constructor and add a sleepOrStop helper that selects between the timer and the shutdown signal. ReconnectAll's startup sleep gets the same treatment. TestConnectWorkspaceStream_BackoffDrainsOnStop closes stopCh while a failing-port retry is in flight and asserts the goroutine exits within 500ms — fails without the fix, passes with it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(backend/github): close sqlite DB in newTestStore to drain connectionOpener newTestStore opened *sqlx.DB via db.OpenSQLite but never registered a Close on cleanup, leaving database/sql.(*DB).connectionOpener goroutines parked across test boundaries. Add t.Cleanup(sqlxDB.Close); goleak's per-package TestMain (added in a follow-up commit) is the regression check. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(backend): wire goleak.VerifyTestMain across leak-prone packages Per-package TestMain in: - internal/integrations/healthpoll (auth-health ticker loop) - internal/jira, internal/linear (auth + issue-watch loops) - internal/github (PR / review / issue poller loops) - internal/gateway/websocket (per-conn pumps + terminal bridge) - internal/orchestrator (detached event-handler goroutines) - internal/agent/runtime/lifecycle (remote-status loop, StreamManager helpers; ignores OTel batch-span processor and OTLP retry helper — initialised early in the process and only exit on shutdown) - internal/agentctl/server/process (extends the existing fixture-binary TestMain so subprocess + PTY pump leaks surface there) Catches regressions where a Start/Stop forgets to drain on WaitGroup, or a detached goroutine forgets to honour ctx/stopCh cancellation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(backend): document goroutine ownership and leak-testing convention New "Goroutine ownership and leak testing" subsection under Conventions in apps/backend/AGENTS.md (CLAUDE.md symlinks to it). Covers: - Lifecycle: Start/Stop + WaitGroup, both idempotent. - Cancellation: every long wait selects on ctx.Done() or stopCh; never time.Sleep in retry loops (reference lifecycle.StreamManager.sleepOrStop). - Detached helpers: must respect the owning type's shutdown signal. - Leak testing: new packages spawning goroutines add goleak.VerifyTestMain; IgnoreTopFunction is reserved for genuine third-party background goroutines and must carry a comment explaining the suppression. Lists the packages currently instrumented so future contributors can mirror the pattern. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(backend/lifecycle): drop dead OTel IgnoreTopFunction from goleak setup Review follow-up. agentctl/tracing.initTracing only spawns the OTel SDK provider when OTEL_EXPORTER_OTLP_ENDPOINT is set; in tests it isn't, so the noop provider is used and no batchSpanProcessor / OTLP retry goroutines ever exist. The IgnoreTopFunction calls were dead suppressions and the comment ("called by NewManager via the agent execution trace path") misdescribed when the SDK init runs. Drop the ignores and correct AGENTS.md to stop citing OTel as the canonical suppression example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(backend/lifecycle): drop unneeded 50ms sleep from backoff-drain test QA follow-up. The 50ms time.Sleep before close(stopCh) in TestConnectWorkspaceStream_BackoffDrainsOnStop was meant to ensure the goroutine had entered the backoff sleep before shutdown — but that ordering doesn't affect the test outcome: - With the fix: sleepOrStop sees stopCh already closed and returns false on the first call, so the goroutine drains within one attempt regardless of when stopCh closes. - Without the fix: time.Sleep ignores stopCh, the goroutine burns its full 5-retry backoff (~31s), and the 500ms timeout fires. Removing the sleep makes the test ~12x faster (15ms vs 200ms total) and removes the only time.Sleep-based wait from the new tests, aligning with the codebase's "channel-based sync, no sleep waits" convention. Verified by 50x -race and 5x revert-and-fail. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(backend/github): drain HTTP idle conns before goleak inspects CI surfaced a leaked goroutine on the Linux runner that local runs missed: a single net/http.(*http2ClientConn).readLoop parked on a TLS read, created by net/http.(*http2Transport).newClientConn. The package's anonymousHTTPClient and PATClient's rewriteTransport both route through http.DefaultTransport's connection pool, so an idle HTTP/2 connection in that pool keeps the reader goroutine alive past m.Run() — invisible on sandboxed local runs where outbound network is blocked. Replace github/goleak_test.go's plain VerifyTestMain with an expanded TestMain that drains http.DefaultTransport's idle connections before goleak.Find runs. This is the stdlib's documented teardown for the connection pool, keeps the assertion strict (no IgnoreTopFunction), and doesn't leak into production code. Other in-scope packages don't need this — they don't use http.DefaultTransport for HTTPS calls. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(backend): drain ws.Done on stopCh; always run goleak in github tests - lifecycle/streams.go: replace bare <-ws.Done() with a select that also watches sm.stopCh so a connected goroutine drains when Manager.Stop() fires before the remote WS closes. A nil stopCh (isolated tests) blocks the nil channel case forever, degrading to plain <-ws.Done() semantics. - github/goleak_test.go: remove the exitCode==0 guard around goleak.Find so leaks are always surfaced even when a test has already failed; original exit code is preserved (leaks only promote it to 1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(backend/lifecycle): close ws on stopCh so agentctl WS loops drain cubic flagged that the previous shutdown path observed stopCh but never called ws.Close(), so the underlying WebSocket and its agentctl-side read/write goroutines (workspace_stream.go:67-68) kept running past Manager.Stop(). ClearWorkspaceStream only removes the WS from the execution's tracking — it doesn't close the connection. Call ws.Close() on the stopCh path. ws.Close is idempotent via closeOnce, so it's safe even if the remote end already closed. TestConnectWorkspaceStream_ClosesWSOnStop connects to a mock workspace server that stays open until the client closes its side; after stopCh fires, the test asserts ws.Done() closes within 500ms — which is only true if connectWorkspaceStream invoked ws.Close(). Verified to fail without the fix and pass with it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(backend/lifecycle): use channel sync instead of sleep polling in ws-close test cubic flagged that TestConnectWorkspaceStream_ClosesWSOnStop polled execution.GetWorkspaceStream() in a 500ms sleep loop, which is brittle under CI load. connectWorkspaceStream already accepts a ready channel that closes after SetWorkspaceStream — pass it in and wait on the channel instead. synctest isn't viable here (real network I/O), so channel-based sync is the testing convention per AGENTS.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Kandev Agent <agent@kandev.dev> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -183,6 +183,15 @@ Client (WS) ← Orchestrator ← Lifecycle Manager ←──── stream update
|
||||
- **Task lifecycle events:** Any code path that mutates a task row must publish via the event bus (`task.created` / `task.updated` / `task.deleted`) — either by going through `Service.CreateTask` / `UpdateTask` / `DeleteTask` / `ArchiveTask`, or by calling `publishTaskEvent` (or one of the `Publish*` helpers in `service_events.go`) directly. Walking `repository.TaskRepository` straight bypasses event publishing and breaks WS-driven UI like the All-Workflows kanban view. `HandoffService`'s cascade methods learned this the hard way — they now require a `TaskEventPublisher` wired via `SetTaskEventPublisher`. New cascade / bulk / cleanup paths must follow the same pattern.
|
||||
- **Testing:** Prefer `testing/synctest` (Go 1.24+) over `time.Sleep` for time-dependent tests. Use `synctest.Test` to wrap tests with tickers or timeouts — it advances fake time instantly when all goroutines are idle. When `synctest` is not feasible (e.g., tests spawning external processes like `git`), use channel-based synchronization (`<-started`, non-blocking `select`) instead of sleep-based waits. Reserve `time.Sleep` only for integration tests that need real subprocess execution time.
|
||||
|
||||
### Goroutine ownership and leak testing
|
||||
|
||||
Every long-running goroutine must have a single owner with explicit start and stop semantics:
|
||||
|
||||
- **Lifecycle:** the type that spawns the goroutine also exposes `Start(ctx)` / `Stop()` (or equivalent). `Start` registers on a `sync.WaitGroup`; `Stop` cancels the goroutine's context (or closes a `stopCh`) and `wg.Wait()`s for drain. Idempotent on both ends. `internal/integrations/healthpoll`, `internal/jira`, `internal/linear`, and `internal/github` pollers are the canonical shape.
|
||||
- **Cancellation:** the goroutine selects on `ctx.Done()` (or `stopCh`) in every long wait. Never use `time.Sleep` in a retry/backoff loop — use `time.NewTimer` inside a `select` that also watches the shutdown signal (see `lifecycle.StreamManager.sleepOrStop`).
|
||||
- **Detached helpers:** event handlers and short-lived `go func()` calls in `internal/orchestrator/` and `internal/agent/runtime/lifecycle/` must accept a cancellable context (or check the owning type's shutdown signal) and return promptly when it fires.
|
||||
- **Leak testing:** packages that spawn goroutines add `goleak.VerifyTestMain(m)` in a per-package `TestMain`. New packages of this kind must follow suit. When a third-party background goroutine genuinely can't be drained, suppress it with `goleak.IgnoreTopFunction(...)` and leave a comment explaining why. Currently instrumented: `internal/gateway/websocket/`, `internal/agent/runtime/lifecycle/`, `internal/agentctl/server/process/`, `internal/orchestrator/`, `internal/github/`, `internal/jira/`, `internal/linear/`, `internal/integrations/healthpoll/`.
|
||||
|
||||
## Backups
|
||||
|
||||
- On every SQLite boot, `persistence.Provide` reads `kandev_meta.kandev_version`. If the stored version differs from the binary version (or any user tables exist but no version is recorded), it takes a `VACUUM INTO` snapshot into `<data-dir>/backups/` before running migrations.
|
||||
|
||||
+2
-1
@@ -8,7 +8,6 @@ require (
|
||||
github.com/creack/pty v1.1.21
|
||||
github.com/docker/docker v28.5.2+incompatible
|
||||
github.com/docker/go-connections v0.6.0
|
||||
github.com/fsnotify/fsnotify v1.6.0
|
||||
github.com/gin-gonic/gin v1.9.1
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/gorilla/websocket v1.5.3
|
||||
@@ -27,6 +26,7 @@ require (
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0
|
||||
go.opentelemetry.io/otel/sdk v1.43.0
|
||||
go.opentelemetry.io/otel/trace v1.43.0
|
||||
go.uber.org/goleak v1.3.0
|
||||
go.uber.org/zap v1.26.0
|
||||
golang.org/x/crypto v0.51.0
|
||||
golang.org/x/net v0.53.0
|
||||
@@ -52,6 +52,7 @@ require (
|
||||
github.com/distribution/reference v0.6.0 // indirect
|
||||
github.com/docker/go-units v0.5.0 // indirect
|
||||
github.com/felixge/httpsnoop v1.0.4 // indirect
|
||||
github.com/fsnotify/fsnotify v1.6.0 // indirect
|
||||
github.com/gabriel-vasile/mimetype v1.4.2 // indirect
|
||||
github.com/gin-contrib/sse v0.1.0 // indirect
|
||||
github.com/go-logr/logr v1.4.3 // indirect
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
package lifecycle
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// TestMain enforces no goroutine leaks across the lifecycle package. The
|
||||
// Manager owns a remote-status polling loop registered on m.wg plus
|
||||
// detached helpers (stream connectors, agentctl-ready watcher, passthrough
|
||||
// stderr poller) spawned across manager_*.go and session.go. Regressions
|
||||
// where a Stop path returns before background work drains surface here.
|
||||
//
|
||||
// OTel note: `agentctl/tracing.initTracing` lazily inits the SDK provider
|
||||
// only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set — in tests it isn't, so
|
||||
// the noop provider is used and no batchSpanProcessor / OTLP retry
|
||||
// goroutines are spawned. No IgnoreTopFunction is needed here.
|
||||
func TestMain(m *testing.M) {
|
||||
goleak.VerifyTestMain(m)
|
||||
}
|
||||
@@ -184,7 +184,8 @@ func NewManager(
|
||||
}
|
||||
|
||||
// Initialize stream manager with callbacks that delegate to manager methods
|
||||
// mcpHandler will be set later via SetMCPHandler
|
||||
// mcpHandler will be set later via SetMCPHandler.
|
||||
// stopCh is shared with the manager so workspace-stream backoff drains on Stop.
|
||||
mgr.streamManager = NewStreamManager(log, StreamCallbacks{
|
||||
OnAgentEvent: mgr.handleAgentEvent,
|
||||
OnStreamDisconnect: mgr.handleStreamDisconnect,
|
||||
@@ -197,7 +198,7 @@ func NewManager(
|
||||
OnShellExit: mgr.handleShellExit,
|
||||
OnProcessOutput: mgr.handleProcessOutput,
|
||||
OnProcessStatus: mgr.handleProcessStatus,
|
||||
}, nil)
|
||||
}, nil, stopCh)
|
||||
|
||||
// Set session manager dependencies for full orchestration
|
||||
sessionManager.SetDependencies(eventPublisher, mgr.streamManager, executionStore, historyManager)
|
||||
|
||||
@@ -28,6 +28,23 @@ func newSessionTestLogger() *logger.Logger {
|
||||
return log
|
||||
}
|
||||
|
||||
// newTestStopCh returns a stopCh shared with t.Cleanup so any background
|
||||
// StreamManager retry loop that races past the test body's return drains
|
||||
// cleanly. Closing on cleanup is what lets goleak.VerifyTestMain stay green
|
||||
// for tests that exercise connectWorkspaceStream's backoff.
|
||||
func newTestStopCh(t *testing.T) chan struct{} {
|
||||
t.Helper()
|
||||
stopCh := make(chan struct{})
|
||||
t.Cleanup(func() {
|
||||
select {
|
||||
case <-stopCh:
|
||||
default:
|
||||
close(stopCh)
|
||||
}
|
||||
})
|
||||
return stopCh
|
||||
}
|
||||
|
||||
// mockAgentServer creates a test WebSocket server simulating agentctl.
|
||||
// It responds to agent stream requests and tracks which actions were called and in what order.
|
||||
type mockAgentServer struct {
|
||||
@@ -196,12 +213,13 @@ func TestInitializeAndPrompt_StreamBeforeInitialize(t *testing.T) {
|
||||
defer mock.Close()
|
||||
|
||||
log := newSessionTestLogger()
|
||||
sm := NewSessionManager(log, make(chan struct{}))
|
||||
stopCh := newTestStopCh(t)
|
||||
sm := NewSessionManager(log, stopCh)
|
||||
|
||||
// Set up real stream manager with callbacks
|
||||
streamMgr := NewStreamManager(log, StreamCallbacks{
|
||||
OnAgentEvent: func(execution *AgentExecution, event agentctl.AgentEvent) {},
|
||||
}, nil)
|
||||
}, nil, stopCh)
|
||||
sm.SetDependencies(nil, streamMgr, nil, nil)
|
||||
|
||||
client := createTestClient(t, mock.server.URL)
|
||||
@@ -269,12 +287,13 @@ func TestInitializeAndPrompt_StreamTimeout(t *testing.T) {
|
||||
// This test verifies that InitializeAndPrompt returns an error if
|
||||
// the stream fails to connect within the timeout.
|
||||
log := newSessionTestLogger()
|
||||
sm := NewSessionManager(log, make(chan struct{}))
|
||||
stopCh := newTestStopCh(t)
|
||||
sm := NewSessionManager(log, stopCh)
|
||||
|
||||
// Create a stream manager that will try to connect to a server that doesn't exist
|
||||
streamMgr := NewStreamManager(log, StreamCallbacks{
|
||||
OnAgentEvent: func(execution *AgentExecution, event agentctl.AgentEvent) {},
|
||||
}, nil)
|
||||
}, nil, stopCh)
|
||||
sm.SetDependencies(nil, streamMgr, nil, nil)
|
||||
|
||||
// Point client at a port that doesn't exist
|
||||
@@ -322,11 +341,12 @@ func TestInitializeAndPrompt_WithTaskDescription(t *testing.T) {
|
||||
defer mock.Close()
|
||||
|
||||
log := newSessionTestLogger()
|
||||
sm := NewSessionManager(log, make(chan struct{}))
|
||||
stopCh := newTestStopCh(t)
|
||||
sm := NewSessionManager(log, stopCh)
|
||||
|
||||
streamMgr := NewStreamManager(log, StreamCallbacks{
|
||||
OnAgentEvent: func(execution *AgentExecution, event agentctl.AgentEvent) {},
|
||||
}, nil)
|
||||
}, nil, stopCh)
|
||||
sm.SetDependencies(nil, streamMgr, nil, nil)
|
||||
|
||||
client := createTestClient(t, mock.server.URL)
|
||||
|
||||
@@ -30,14 +30,25 @@ type StreamManager struct {
|
||||
logger *logger.Logger
|
||||
callbacks StreamCallbacks
|
||||
mcpHandler agentctl.MCPHandler
|
||||
// stopCh is the Manager-owned shutdown signal. SessionTraceContext is
|
||||
// deliberately uncancellable (it carries a long-lived span), so the
|
||||
// reconnect/backoff loops below select on stopCh to drain on Manager.Stop.
|
||||
// nil is treated as "no shutdown signal" and falls back to the prior
|
||||
// uncancellable behaviour (compat with constructors used by isolated tests).
|
||||
stopCh <-chan struct{}
|
||||
}
|
||||
|
||||
// NewStreamManager creates a new StreamManager
|
||||
func NewStreamManager(log *logger.Logger, callbacks StreamCallbacks, mcpHandler agentctl.MCPHandler) *StreamManager {
|
||||
// NewStreamManager creates a new StreamManager.
|
||||
//
|
||||
// stopCh is the Manager-owned shutdown signal used by the workspace-stream
|
||||
// retry backoff to drain cleanly. Pass nil from tests that exercise the
|
||||
// manager in isolation; production callers wire it from Manager.stopCh.
|
||||
func NewStreamManager(log *logger.Logger, callbacks StreamCallbacks, mcpHandler agentctl.MCPHandler, stopCh <-chan struct{}) *StreamManager {
|
||||
return &StreamManager{
|
||||
logger: log.WithFields(zap.String("component", "stream-manager")),
|
||||
callbacks: callbacks,
|
||||
mcpHandler: mcpHandler,
|
||||
stopCh: stopCh,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,8 +68,12 @@ func (sm *StreamManager) ReconnectAll(execution *AgentExecution) {
|
||||
zap.String("instance_id", execution.ID),
|
||||
zap.String("task_id", execution.TaskID))
|
||||
|
||||
// Wait a moment for any startup operations to settle
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
// Wait a moment for any startup operations to settle. Selecting on
|
||||
// stopCh lets shutdown drain this goroutine without burning the full
|
||||
// 500ms when the manager is already stopping.
|
||||
if !sm.sleepOrStop(500 * time.Millisecond) {
|
||||
return
|
||||
}
|
||||
|
||||
// Check if agentctl is responsive
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
@@ -79,6 +94,24 @@ func (sm *StreamManager) ReconnectAll(execution *AgentExecution) {
|
||||
zap.String("task_id", execution.TaskID))
|
||||
}
|
||||
|
||||
// sleepOrStop blocks for d or until the Manager begins shutting down.
|
||||
// Returns true when the timer fires, false when stopCh closes first.
|
||||
// A nil stopCh degrades to time.Sleep semantics (always returns true).
|
||||
func (sm *StreamManager) sleepOrStop(d time.Duration) bool {
|
||||
if sm.stopCh == nil {
|
||||
time.Sleep(d)
|
||||
return true
|
||||
}
|
||||
timer := time.NewTimer(d)
|
||||
defer timer.Stop()
|
||||
select {
|
||||
case <-timer.C:
|
||||
return true
|
||||
case <-sm.stopCh:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// connectUpdatesStream handles the updates WebSocket stream with ready signaling
|
||||
func (sm *StreamManager) connectUpdatesStream(execution *AgentExecution, ready chan<- struct{}) {
|
||||
ctx := execution.SessionTraceContext()
|
||||
@@ -229,7 +262,11 @@ func (sm *StreamManager) connectWorkspaceStream(execution *AgentExecution, ready
|
||||
zap.Error(err))
|
||||
|
||||
if attempt < maxRetries {
|
||||
time.Sleep(backoff)
|
||||
// Exit early on Manager shutdown so the backoff doesn't
|
||||
// strand a goroutine after Stop() returns.
|
||||
if !sm.sleepOrStop(backoff) {
|
||||
return
|
||||
}
|
||||
backoff *= 2 // Exponential backoff
|
||||
}
|
||||
continue
|
||||
@@ -243,8 +280,17 @@ func (sm *StreamManager) connectWorkspaceStream(execution *AgentExecution, ready
|
||||
// Signal that workspace stream is ready
|
||||
signalReady()
|
||||
|
||||
// Wait for the stream to close (it stays open until disconnected)
|
||||
<-ws.Done()
|
||||
// Wait for the stream to close. Also exits on Manager shutdown so the
|
||||
// goroutine drains when the remote end keeps the connection open —
|
||||
// in that case we close ws ourselves so the underlying WS read/write
|
||||
// loops in agentctl.WorkspaceStream also exit. ws.Close is idempotent
|
||||
// via closeOnce. A nil stopCh (isolated tests) blocks on the nil
|
||||
// channel forever, which degrades to plain <-ws.Done() semantics.
|
||||
select {
|
||||
case <-ws.Done():
|
||||
case <-sm.stopCh:
|
||||
ws.Close()
|
||||
}
|
||||
execution.ClearWorkspaceStream(ws)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
// "workspace stream already connected" and burned 5 retries before logging
|
||||
// a terminal ERROR. The fix short-circuits when a stream is already attached.
|
||||
func TestConnectWorkspaceStream_IdempotentWhenAlreadyAttached(t *testing.T) {
|
||||
sm := NewStreamManager(newTestLogger(), StreamCallbacks{}, nil)
|
||||
sm := NewStreamManager(newTestLogger(), StreamCallbacks{}, nil, nil)
|
||||
|
||||
execution := &AgentExecution{ID: "exec-1", SessionID: "sess-1"}
|
||||
// Pre-attach a non-nil workspace stream — simulates another goroutine
|
||||
@@ -44,3 +44,107 @@ func TestConnectWorkspaceStream_IdempotentWhenAlreadyAttached(t *testing.T) {
|
||||
t.Error("ready channel was not closed on early-exit path")
|
||||
}
|
||||
}
|
||||
|
||||
// TestConnectWorkspaceStream_BackoffDrainsOnStop is the regression test for
|
||||
// the workspace-stream retry-backoff leak. Before the fix, connectWorkspaceStream
|
||||
// slept the full backoff in time.Sleep between retries; if the test (or
|
||||
// production Manager) tore down while a retry was in flight, the goroutine
|
||||
// stranded until the backoff fired, surviving the Manager's Stop(). The fix
|
||||
// selects on stopCh so the backoff drains immediately on shutdown.
|
||||
//
|
||||
// We trigger the failing path by pointing the client at a closed port —
|
||||
// StreamWorkspace returns "connection refused" on every attempt, sending the
|
||||
// loop into its retry backoff. Closing stopCh must release the goroutine in
|
||||
// well under the (uncapped) backoff window.
|
||||
func TestConnectWorkspaceStream_BackoffDrainsOnStop(t *testing.T) {
|
||||
stopCh := make(chan struct{})
|
||||
sm := NewStreamManager(newTestLogger(), StreamCallbacks{}, nil, stopCh)
|
||||
|
||||
log := newTestLogger()
|
||||
badClient := agentctl.NewClient("127.0.0.1", 1, log) // port 1 is reserved
|
||||
defer badClient.Close()
|
||||
|
||||
execution := &AgentExecution{
|
||||
ID: "exec-backoff",
|
||||
SessionID: "sess-backoff",
|
||||
agentctl: badClient,
|
||||
}
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
sm.connectWorkspaceStream(execution, nil)
|
||||
close(done)
|
||||
}()
|
||||
|
||||
close(stopCh)
|
||||
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(500 * time.Millisecond):
|
||||
t.Fatal("connectWorkspaceStream did not drain on stopCh close — backoff is leaking")
|
||||
}
|
||||
}
|
||||
|
||||
// TestConnectWorkspaceStream_ClosesWSOnStop is the regression test for the
|
||||
// connected-stream shutdown leak. Once StreamWorkspace returns successfully
|
||||
// the goroutine parks on `<-ws.Done()`. If the manager shuts down while the
|
||||
// remote WS is still alive, simply observing stopCh isn't enough — the
|
||||
// underlying WS connection (and the agentctl-side read/write loops it owns)
|
||||
// keep running. The fix calls ws.Close() on the stopCh path so the WS
|
||||
// actually tears down, and ws.Done() fires for any other observer.
|
||||
//
|
||||
// We exercise this by pointing a real agentctl client at a mock workspace
|
||||
// stream server that stays open until the client closes the WS. After the
|
||||
// goroutine connects, we close stopCh and assert ws.Done() fires within
|
||||
// a short window — which is only true if connectWorkspaceStream called
|
||||
// ws.Close() on its way out.
|
||||
func TestConnectWorkspaceStream_ClosesWSOnStop(t *testing.T) {
|
||||
mock := newMockAgentServer(t)
|
||||
defer mock.Close()
|
||||
|
||||
stopCh := make(chan struct{})
|
||||
sm := NewStreamManager(newSessionTestLogger(), StreamCallbacks{}, nil, stopCh)
|
||||
|
||||
client := createTestClient(t, mock.server.URL)
|
||||
defer client.Close()
|
||||
|
||||
execution := &AgentExecution{
|
||||
ID: "exec-close",
|
||||
SessionID: "sess-close",
|
||||
agentctl: client,
|
||||
}
|
||||
|
||||
// ready closes after SetWorkspaceStream returns and signalReady fires —
|
||||
// channel-based sync per the testing convention, no sleep polling.
|
||||
ready := make(chan struct{})
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
sm.connectWorkspaceStream(execution, ready)
|
||||
close(done)
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-ready:
|
||||
case <-time.After(500 * time.Millisecond):
|
||||
t.Fatal("workspace stream never attached (ready channel did not close)")
|
||||
}
|
||||
|
||||
ws := execution.GetWorkspaceStream()
|
||||
if ws == nil {
|
||||
t.Fatal("workspace stream attached but execution lost the reference")
|
||||
}
|
||||
|
||||
close(stopCh)
|
||||
|
||||
select {
|
||||
case <-ws.Done():
|
||||
case <-time.After(500 * time.Millisecond):
|
||||
t.Fatal("ws.Done() did not fire on stopCh close — ws.Close() was not called")
|
||||
}
|
||||
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(500 * time.Millisecond):
|
||||
t.Fatal("connectWorkspaceStream goroutine did not drain on stopCh close")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -10,6 +10,8 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// kandevTestFixtureEnv activates fixture-binary mode in this test binary.
|
||||
@@ -22,13 +24,15 @@ import (
|
||||
const kandevTestFixtureEnv = "KANDEV_TEST_FIXTURE"
|
||||
|
||||
// TestMain branches into fixture-binary mode when the activation env var
|
||||
// is set; otherwise it runs the test suite normally.
|
||||
// is set; otherwise it runs the test suite normally — wrapped in goleak so
|
||||
// the per-process subprocess managers (workspace tracker, PTY pumps, poll
|
||||
// loops) surface goroutine leaks here.
|
||||
func TestMain(m *testing.M) {
|
||||
if spec := os.Getenv(kandevTestFixtureEnv); spec != "" {
|
||||
runFixture(spec)
|
||||
return
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
goleak.VerifyTestMain(m)
|
||||
}
|
||||
|
||||
// runFixture executes a whitespace-separated command spec and exits.
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
package websocket
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// TestMain enforces no goroutine leaks across the gateway/websocket package.
|
||||
// Per-connection read/write pumps in client.go and the terminal bridge in
|
||||
// terminal_pumps.go are the primary leak vectors — both must exit cleanly
|
||||
// when the underlying connection closes.
|
||||
func TestMain(m *testing.M) {
|
||||
goleak.VerifyTestMain(m)
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package github
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// TestMain enforces no goroutine leaks across the github package — the Poller
|
||||
// owns three loops (prMonitor, reviewQueue, issueWatch) gated on a single
|
||||
// context via Stop. Regressions where Stop forgets to cancel the context
|
||||
// or drain the WaitGroup surface here.
|
||||
//
|
||||
// HTTP/2 pool drain: anonymousHTTPClient and PATClient's rewriteTransport both
|
||||
// route through http.DefaultTransport. On runners with outbound network (CI),
|
||||
// one or more tests end up keeping an HTTP/2 client connection (readLoop
|
||||
// goroutine) idle in the default pool — invisible locally where the network
|
||||
// is sandboxed. CloseIdleConnections is the stdlib's documented way to drain
|
||||
// these readers before checking for leaks; doing it here keeps the assertion
|
||||
// strict (no broad IgnoreTopFunction suppression) without polluting prod code.
|
||||
func TestMain(m *testing.M) {
|
||||
exitCode := m.Run()
|
||||
if tr, ok := http.DefaultTransport.(*http.Transport); ok {
|
||||
tr.CloseIdleConnections()
|
||||
}
|
||||
if err := goleak.Find(); err != nil {
|
||||
println("goleak:", err.Error())
|
||||
if exitCode == 0 {
|
||||
exitCode = 1
|
||||
}
|
||||
}
|
||||
os.Exit(exitCode)
|
||||
}
|
||||
@@ -20,6 +20,7 @@ func newTestStore(t *testing.T) *Store {
|
||||
t.Fatalf("open db: %v", err)
|
||||
}
|
||||
sqlxDB := sqlx.NewDb(dbConn, "sqlite3")
|
||||
t.Cleanup(func() { _ = sqlxDB.Close() })
|
||||
if _, err := sqlxDB.Exec(`CREATE TABLE tasks (id TEXT PRIMARY KEY, archived_at DATETIME)`); err != nil {
|
||||
t.Fatalf("create tasks table: %v", err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
package healthpoll
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// TestMain asserts that no goroutines from this package outlive the test
|
||||
// process. The Poller owns a single ticker loop guarded by Start/Stop —
|
||||
// goleak catches regressions where a Start path forgets to register on the
|
||||
// WaitGroup, or a Stop path returns before the loop drains.
|
||||
func TestMain(m *testing.M) {
|
||||
goleak.VerifyTestMain(m)
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
package jira
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// TestMain enforces no goroutine leaks across the jira package — the Poller
|
||||
// owns an auth-health loop and an issue-watch loop, both lifecycle-managed
|
||||
// via Start/Stop. Regressions where Stop forgets to cancel either context
|
||||
// or drain the WaitGroup surface here.
|
||||
func TestMain(m *testing.M) {
|
||||
goleak.VerifyTestMain(m)
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
package linear
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// TestMain enforces no goroutine leaks across the linear package — the Poller
|
||||
// owns an auth-health loop and an issue-watch loop, both lifecycle-managed
|
||||
// via Start/Stop. Regressions where Stop forgets to cancel either context
|
||||
// or drain the WaitGroup surface here.
|
||||
func TestMain(m *testing.M) {
|
||||
goleak.VerifyTestMain(m)
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
package orchestrator
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// TestMain enforces no goroutine leaks across the orchestrator package.
|
||||
// Event handlers in event_handlers_*.go spawn detached goroutines for
|
||||
// queued message execution, cleanup, watchdogs, and step-exit/enter
|
||||
// processing — each must terminate on its own (context cancellation or
|
||||
// natural completion). Regressions where a handler forgets a return on
|
||||
// ctx.Done() or hangs on a buffered channel surface here.
|
||||
func TestMain(m *testing.M) {
|
||||
goleak.VerifyTestMain(m)
|
||||
}
|
||||
Reference in New Issue
Block a user