golang.
golang8 min read

golang goroutine leak detection

Practical patterns to detect and prevent goroutine leaks using runtime.NumGoroutine, goleak, and context cancellation

golang goroutine leak detection

Goroutine Leak Detection in Go: Patterns That Catch Leaks Before Production

A goroutine leak is the Go equivalent of forgetting to close a file handle, except it costs you stack memory, scheduler overhead, and sometimes the entire process when you eventually OOM. A leaked goroutine never returns. It sits parked on a channel send, a sync.Cond.Wait, or a network read that no one will ever feed. The runtime cannot reclaim it. Each leak is small (around 2 KB of stack to start), but a service handling 1000 requests per second that leaks one goroutine per request will accumulate 86 million stuck goroutines per day. The process dies long before that.

Detection is the entire problem. A leaked goroutine produces no error and no panic. It does not show up in your test pass count. You have to look for it deliberately, with the right tools wired into the right places.

Why goroutine leaks happen

Leaks almost always come from one of three shapes:

  1. A send on an unbuffered channel with no receiver. The producer goroutine blocks forever.
  2. A receive on a channel that will never be closed. The consumer parks indefinitely.
  3. A blocking call (network read, mutex, ticker) inside a goroutine that has no cancellation path.

The third pattern is the most common in modern Go code because context.Context is so easy to ignore. You spawn a goroutine, pass it a context, and then forget to actually select on ctx.Done() inside the loop. The goroutine survives the parent's cancellation and stays alive until the process exits.

// Leak: ticker fires forever, no ctx.Done() branch
func poller(ctx context.Context, ch chan<- Result) {
    ticker := time.NewTicker(5 * time.Second)
    for range ticker.C {
        ch <- fetch()
    }
}

The fix is two lines, but you have to write a test that catches the missing version before it ships.

Tool 1: runtime.NumGoroutine for sanity checks

The simplest detection is counting. runtime.NumGoroutine() returns the current goroutine count for the entire process. In a test, you snapshot before and after, and any positive delta is suspicious.

func TestPollerCleansUp(t *testing.T) {
    before := runtime.NumGoroutine()

    ctx, cancel := context.WithCancel(context.Background())
    ch := make(chan Result, 1)
    go poller(ctx, ch)

    time.Sleep(50 * time.Millisecond) // let poller start
    cancel()
    time.Sleep(50 * time.Millisecond) // let poller exit

    after := runtime.NumGoroutine()
    if after > before {
        t.Fatalf("leaked %d goroutines", after-before)
    }
}

This works, sort of. It catches the obvious case where you forgot to cancel. It does not tell you which goroutine leaked, where it was spawned, or what stack frame it is parked on. It is also racy: any background goroutine in the test binary (the testing package, garbage collector helpers, runtime monitoring) can shift the count. You will see flaky tests that pass locally and fail in CI on a slower machine.

Use runtime.NumGoroutine for ad-hoc debugging during a development session. Do not commit it as your only leak detector.

Tool 2: goleak for deterministic test-level detection

Uber's goleak library is the standard answer. It runs at test teardown, captures every goroutine still running, filters out the well-known background ones (testing infrastructure, runtime), and fails the test if anything else remains.

The library publishes a single helper, goleak.VerifyNone(t), that you call from the end of your test. Even better, you can call goleak.VerifyTestMain(m) once per package and every test in the package gets leak verification for free.

// in a file named main_test.go at the package root
func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)
}

That single line turns every test in the package into a leak-checking test. When a test leaks, the failure includes the full goroutine stack trace, so you can see exactly which function spawned the orphan and what it is parked on.

Compared to manual runtime.NumGoroutine snapshots, goleak gives you three concrete wins. It is deterministic because it polls until counts stabilize. It shows you the stack so triage takes seconds instead of minutes. It works at the package level, so you do not have to remember to instrument every test.

The cost is exactly one import and one TestMain function. If you write production Go and you are not using goleak, this is the single highest-leverage change you can make today.

Tool 3: context cancellation as a prevention pattern

Detection catches leaks; prevention stops them at the source. The cancellation pattern is the convention that almost all leak-free Go code follows: every long-running goroutine takes a context.Context and selects on ctx.Done() inside its main loop.

The fixed poller looks like this:

func poller(ctx context.Context, ch chan<- Result) {
    ticker := time.NewTicker(5 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            select {
            case ch <- fetch():
            case <-ctx.Done():
                return
            }
        }
    }
}

Three rules are baked in. The outer select checks cancellation on every loop iteration. The inner select makes the channel send itself cancellable, because if the consumer disappears the send would otherwise block forever. The defer ticker.Stop() releases the ticker's internal goroutine, which is itself a leak source many developers miss.

When you review Go code for leaks, scan for any for {} loop inside a goroutine and check that the loop has a path that exits on context cancellation. Loops with for range ch are a special case: they exit when the channel is closed, so the question becomes "who closes this channel, and are they guaranteed to do so?"

Tool 4: pprof for production diagnosis

Once code is in production, you do not have the luxury of a test harness. You need a way to ask a running process "what are all your goroutines doing right now?" The answer is net/http/pprof, which the standard library ships in the box.

Wire it into your service:

import _ "net/http/pprof"

func main() {
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()
    // ... rest of the service
}

The blank import registers handlers under /debug/pprof/. The endpoint you want for leak hunting is /debug/pprof/goroutine?debug=2, which dumps every goroutine in the process with its full stack trace. You can hit it from your laptop:

curl http://localhost:6060/debug/pprof/goroutine?debug=2 > goroutines.txt

The output is grouped by stack signature. If 47,000 goroutines are all parked at the same line of fetch() waiting on a channel send, you have found your leak and its exact location. The official runtime/pprof documentation covers the heavier go tool pprof workflow for when you need flame graphs and allocation tracking on top of goroutine counts.

A trick worth knowing: take two snapshots 60 seconds apart and diff them. Stable goroutines (HTTP server workers, GC helpers) appear in both. Leak suspects grow. The diff is usually obvious by eye.

Wiring detection into CI

The pattern that catches the most leaks for the least effort:

  1. Add goleak as a test dependency. Put goleak.VerifyTestMain(m) in every package that runs concurrent code.
  2. Run go test ./... on every PR. Leaks fail the build.
  3. Expose pprof on a non-public port in production. Snapshot when memory or goroutine count alerts fire.
  4. For long-running services, add a Prometheus metric for runtime.NumGoroutine() and alert when it grows monotonically over a 30-minute window.

Step 1 alone will catch around 80 percent of leaks before they ship. Step 4 catches the rest, the slow leaks that take days to manifest because they only happen on a specific code path that hits production once an hour.

Common false positives and how to handle them

Goleak is strict, sometimes more strict than you want. A few real-world cases:

  • Database driver background goroutines. Some drivers (older database/sql implementations, certain Redis clients) leave a connection-pool goroutine running until process exit. Goleak flags these. The fix is goleak.IgnoreTopFunction("github.com/your/driver.(*Pool).maintain") in your TestMain options.
  • time.AfterFunc and time.Sleep. A goroutine sitting inside time.Sleep is technically alive. If your test creates a sleeper deliberately, either wait for it or filter it.
  • The testing package itself. Goleak knows about these and filters by default, but if you see a stack trace mentioning testing.tRunner.func1, it is the testing harness and you can ignore it.

The general principle is: every ignore is a documented exception with an explanation. Do not paper over leaks with broad filters. If you find yourself ignoring three or more goroutine signatures, you have a real architectural problem (probably a global connection pool you never close), and the right fix is in the code, not the test config.

A practical checklist

When you write or review Go code that spawns goroutines, run through this mentally:

  1. Does the goroutine take a context.Context?
  2. Does its main loop have a select that includes <-ctx.Done()?
  3. If it sends to a channel, is the send wrapped in a select with the same cancellation branch?
  4. If it owns a time.Ticker, is there a defer ticker.Stop()?
  5. If it holds a sync.WaitGroup.Add, does the caller Wait on it before exiting?
  6. Does the package have goleak.VerifyTestMain so the test suite enforces all of the above?

Six questions, ninety seconds of review, and you have eliminated the entire class of bugs that historically takes Go services down at 3 AM.

When to reach for what

For local debugging: runtime.NumGoroutine plus a few fmt.Println calls. Cheap, fast, no dependencies.

For unit and integration tests: goleak, full stop. The 10-minute setup pays back the first time it catches a leak in CI instead of in production.

For production diagnosis: net/http/pprof on a private port, with a runbook entry that says "when goroutine count alert fires, curl /debug/pprof/goroutine?debug=2 and look for stacks that account for the bulk of the count."

The combination of goleak in CI and pprof in production catches roughly 95 percent of goroutine leaks before they cause an outage, at the cost of one import and one HTTP handler. The remaining 5 percent are the architectural leaks that need design changes, and those you find by watching the metric trend over weeks.

References: