Go pprof: CPU Profiling in Production with Flamegraphs
Production latency spikes that vanish the moment you SSH in are the worst kind of bug. A handler that runs in two milliseconds on your laptop suddenly chews 400ms of CPU under real traffic, your p99 graph goes vertical, and the only signal you have is a flat "container CPU at 85%" line in your dashboard. Adding more log lines does not help, because the cost is not in any single line you wrote; it is smeared across a hot path that allocates too much, hashes too often, or calls into a regex compile inside a loop somebody forgot about. Without a CPU profile taken on the actual production process under actual production load, you are guessing.
This walkthrough builds a small Go HTTP service with a deliberately expensive handler, wires net/http/pprof onto a separate admin listener so the profiling surface is never exposed alongside business traffic, captures a 30-second CPU profile with go tool pprof, and renders it as an interactive flamegraph you can actually navigate. You will use the standard library plus runtime/pprof, drive synthetic load with hey, and finish with a small shell-and-Go automation that uploads periodic profiles to object storage. By the last step you will have a working repository, a real flamegraph PNG, and a hardened admin endpoint pinned to loopback behind basic auth.
This is written for backend engineers who already ship Go services in production and know what a goroutine and an HTTP handler are, but have never opened pprof in anger. By the end you will be able to attach to a running service, capture a profile without disturbing user traffic, read the flamegraph well enough to point at the offending function, and bake profile capture into a routine you can trust on call.
Step 1: Scaffolding a CPU-Bound Prime-Counting HTTP Service in Go
Before we can flamegraph anything, we need a service whose CPU profile is worth looking at. The whole article series is about reading pprof output from a real workload, so step 1 builds a small Go HTTP server with a deliberately inefficient hot path: a trial-division prime counter wrapped in a /work endpoint.
The design goal here is honesty, not cleverness. The handler must be slow in a predictable, profileable way — no goroutine pools, no caches, no hidden concurrency — so that when we attach net/http/pprof in step 2, the flamegraph points at code we can actually reason about.
Setup
We start a fresh Go module at the repository root and lay out four source files plus their tests. No third-party dependencies are required — everything we need lives in the standard library.
go mod init github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph
The resulting go.mod pins the minimum toolchain we rely on:
module github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph
go 1.21
The package layout for this step is intentionally flat. Four files live next to go.mod — main.go (entrypoint), server.go (HTTP wiring), workload.go (the CPU-bound primitive), and their _test.go siblings. A flat layout keeps the main package importable by the test files without any internal-package gymnastics, which matters once we start instrumenting it.
Implementation
The entrypoint is a thin wrapper around http.ListenAndServe. It reads APP_ADDR from the environment so a container or local shell can override the bind address without recompiling:
package main
import (
"log"
"net/http"
"os"
)
func main() {
addr := os.Getenv("APP_ADDR")
if addr == "" {
addr = ":8080"
}
log.Printf("listening on %s", addr)
if err := http.ListenAndServe(addr, newServer()); err != nil {
log.Fatal(err)
}
}
Keeping main this small is deliberate. The handler graph is constructed in newServer(), which makes it cheap to spin up an in-process server inside httptest without touching main at all. That separation is what lets the test file exercise the real router instead of a mock.
The routing layer registers two endpoints on a plain http.ServeMux. Both handlers are package-level functions so a single test can install them into a fresh mux per case:
package main
import (
"fmt"
"net/http"
"strconv"
)
const defaultUpperBound = 100_000
func newServer() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("/healthz", healthHandler)
mux.HandleFunc("/work", workHandler)
return mux
}
/healthz is the trivial liveness probe — we will reuse it later to confirm the server stays responsive while a profile is being captured. /work is the interesting one. It parses an upper bound from the n query parameter, runs the prime counter, and writes the result as a one-line JSON object so a load generator can pipe responses through jq without buffering:
func healthHandler(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
fmt.Fprintln(w, "ok")
}
func workHandler(w http.ResponseWriter, r *http.Request) {
upper, err := parseUpperBound(r.URL.Query().Get("n"))
if err != nil {
http.Error(w, "invalid n: "+err.Error(), http.StatusBadRequest)
return
}
count := CountPrimes(upper)
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"upper":%d,"primes":%d}`+"\n", upper, count)
}
Parsing lives in its own function for two reasons. First, it caps workHandler at one level of branching, satisfying the workspace's "max two if levels" rule. Second, it makes the default-vs-explicit case directly testable without spinning up a ResponseRecorder:
func parseUpperBound(raw string) (int, error) {
if raw == "" {
return defaultUpperBound, nil
}
n, err := strconv.Atoi(raw)
if err != nil {
return 0, err
}
if n < 0 {
return 0, fmt.Errorf("n must be >= 0")
}
return n, nil
}
The CPU-bound workload itself lives in workload.go. The naive O(n·√n) trial-division loop is the entire point of this article series — it is the function our future flamegraphs will accuse:
package main
func CountPrimes(upper int) int {
if upper < 2 {
return 0
}
count := 0
for n := 2; n <= upper; n++ {
if isPrime(n) {
count++
}
}
return count
}
func isPrime(n int) bool {
if n < 2 {
return false
}
if n == 2 {
return true
}
if n%2 == 0 {
return false
}
for i := 3; i*i <= n; i += 2 {
if n%i == 0 {
return false
}
}
return true
}
isPrime skips even candidates and stops at √n, but it deliberately keeps the loop allocation-free and goroutine-free. Allocation-heavy workloads obscure CPU profiles because runtime.mallocgc shows up everywhere; a tight arithmetic loop will produce the cleanest possible flamegraph in step 2.
Two test files pin down the contract. workload_test.go table-tests the prime counter against the published prime-counting function π(n) for the bounds we care about — 4 primes ≤ 10, 25 primes ≤ 100, 168 primes ≤ 1000. server_test.go walks every branch of workHandler: the happy path, the non-numeric rejection, the negative rejection, the default fallback, and the health probe. Together they give us a regression net to lean on once we start mutating the code to add profiling.
Verification
Run the standard test suite from the repository root:
go test ./...
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph 0.412s
A quick smoke test confirms the server boots and the prime count for n=100 agrees with the table test:
go run . &
curl -s 'http://localhost:8080/work?n=100'
{"upper":100,"primes":25}
What we built
We now have a working Go HTTP service whose only interesting behaviour is being slow at one specific, named function. The wiring is so thin that future steps can hang a /debug/pprof/* router off the same mux without rewriting the handler graph.
The test suite locks down the contract we care about — health probe returns ok, /work returns a JSON object whose primes field matches π(n), and the parser rejects garbage with a 400 status. That gives us a safety net to lean on the moment we start instrumenting the binary.
The choice of trial division over a sieve, of synchronous handling over a worker pool, and of zero allocations inside the hot loop is intentional. Each of those choices is a knob we can twiddle in later steps to watch the flamegraph rearrange itself — but only because step 1 fixed every other variable first.
What this unlocks for step 2 is direct: a single import of net/http/pprof plus a side-effect blank import will expose runtime profiles against a binary whose hot path we already understand. The flamegraph we capture there is going to point straight at isPrime, and we will know we have read it correctly because we already wrote and tested that function ourselves.
Repository
The state of the code after this step: 90f55bd
Step 2: Isolating net/http/pprof on a Private Admin Mux
Step 1 left us with a working HTTP service whose hot path is a trial-division prime counter. The flamegraph we want next requires runtime profiles, and the standard way to surface those in Go is net/http/pprof. The catch is that the package's init() blank-import side effect registers its handlers on http.DefaultServeMux — which is exactly the wrong mux to expose to the public internet.
This step splits the binary into two listeners: a public app server on :8080 (unchanged from step 1) and a private admin server on 127.0.0.1:6060 that owns every /debug/pprof/* path. The split is the one architectural decision that makes production profiling safe by default.
Setup
No new dependencies are added; everything we need still lives in the standard library. We add one new source file plus its tests next to the existing flat layout:
codebase/
├── go.mod
├── main.go # updated: two listeners
├── server.go # unchanged
├── workload.go # unchanged
├── admin.go # NEW: pprof handlers on an isolated mux
├── admin_test.go # NEW: isolation guarantees
├── server_test.go # unchanged
└── workload_test.go # unchanged
The admin listener defaults to 127.0.0.1:6060 so it is unreachable from outside the host unless an operator explicitly forwards a port or tunnels through SSH. Both addresses are overridable through environment variables — APP_ADDR for the public mux, ADMIN_ADDR for the private one.
Implementation
The new admin.go file builds a dedicated mux and registers each pprof endpoint by name. We deliberately avoid the _ "net/http/pprof" blank import in any file that calls http.ListenAndServe(addr, nil), because that combination would silently leak the profilers onto the default mux:
package main
import (
"net/http"
"net/http/pprof"
)
func newAdminMux() *http.ServeMux {
mux := http.NewServeMux()
mux.HandleFunc("/debug/pprof/", pprof.Index)
mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
return mux
}
Three design choices are worth naming. First, the registrations are explicit handler-by-handler rather than mux.Handle("/debug/pprof/", http.DefaultServeMux), because reusing DefaultServeMux here would re-expose every other package that ever registers on it. Second, pprof.Index is mounted on the trailing-slash prefix path /debug/pprof/ — that single prefix handler is what dispatches the named runtime profiles (heap, goroutine, allocs, block, mutex, threadcreate). Third, we keep the function returning a concrete *http.ServeMux so tests can assert against it without an interface dance.
The entrypoint becomes two listeners, one goroutine and one foreground:
func main() {
appAddr := envOr("APP_ADDR", ":8080")
adminAddr := envOr("ADMIN_ADDR", "127.0.0.1:6060")
go runAdmin(adminAddr)
runApp(appAddr)
}
func runApp(addr string) {
log.Printf("app listening on %s", addr)
if err := http.ListenAndServe(addr, newServer()); err != nil {
log.Fatalf("app server: %v", err)
}
}
func runAdmin(addr string) {
log.Printf("admin (pprof) listening on %s", addr)
if err := http.ListenAndServe(addr, newAdminMux()); err != nil {
log.Fatalf("admin server: %v", err)
}
}
Running the admin server in a goroutine and the app server in the foreground means a fatal admin-listener error will still kill the process — Go's log.Fatalf calls os.Exit(1). That is the behaviour we want: profiling endpoints failing to bind is operator-visible, not silently degraded.
The new tests do two complementary jobs. The positive cases hit the admin mux and assert the index lists the standard profile names and that each named profile returns 200. The negative case hits the app mux on the same paths and asserts a 404 — the regression net that catches any future refactor accidentally collapsing the two muxes back together:
func TestAppMuxDoesNotExposePprof(t *testing.T) {
srv := newServer()
for _, path := range []string{"/debug/pprof/", "/debug/pprof/heap", "/debug/pprof/profile"} {
req := httptest.NewRequest(http.MethodGet, path, nil)
rec := httptest.NewRecorder()
srv.ServeHTTP(rec, req)
if rec.Code != http.StatusNotFound {
t.Errorf("public mux exposed %s (status %d); pprof MUST stay on the admin mux", path, rec.Code)
}
}
}
A second guard confirms newServer() does not return http.DefaultServeMux itself. Even though net/http/pprof's init() unavoidably pollutes the default mux as soon as we import the package, we can still guarantee that the public listener is wired to a different mux entirely.
Verification
Run the test suite from the codebase root:
go test ./...
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph 0.587s
A live smoke test exercises both listeners simultaneously. The public address should serve /work and refuse /debug/pprof/; the admin address should do the opposite:
go run . &
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/work?n=100
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/debug/pprof/
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6060/debug/pprof/
200
404
200
A pprof-aware client confirms the admin endpoint is the real thing — go tool pprof can attach to it and dump the named profile list:
go tool pprof -top -seconds 1 http://127.0.0.1:6060/debug/pprof/profile
Fetching profile over HTTP from http://127.0.0.1:6060/debug/pprof/profile?seconds=1
Saved profile in /tmp/pprof/pprof.samples.cpu.001.pb.gz
What we built
We now run two HTTP listeners out of the same Go process. The public one is the same flat mux from step 1 and still only knows about /healthz and /work. The private one owns every /debug/pprof/* endpoint and binds to loopback by default.
The isolation is enforced by tests, not by convention. TestAppMuxDoesNotExposePprof will fail the moment anyone accidentally wires pprof onto the public mux — for example by adding _ "net/http/pprof" to a file that then calls http.ListenAndServe(":8080", nil). TestAppMuxIsNotDefaultServeMux catches the subtler form of that mistake, where the public listener gets handed http.DefaultServeMux directly.
There is a deliberate piece of plumbing we did NOT do: we did not add authentication on the admin port. The threat model we are committing to is "private interface, accessed via SSH tunnel or kubectl port-forward" — the same model the Go runtime team uses in its own tutorials. If you need authenticated profiling instead, this is the exact mux to wrap in http.HandlerFunc middleware, and the tests above will still pass unchanged.
What this unlocks for step 3 is the part we have been building toward — capturing a CPU profile against a binary under real load. With the admin mux in place we can drive synthetic traffic at /work while go tool pprof collects a /debug/pprof/profile sample, then render the result as a flamegraph and see isPrime light up exactly where step 1 predicted.
Repository
The state of the code after this step: 842a7d4
Step 3: Capturing a 30-Second CPU Profile Under Synthetic Load
Step 2 left us with two listeners — a public app server on :8080 serving /work and /healthz, and a private admin server on 127.0.0.1:6060 serving every /debug/pprof/* endpoint. The binary is now profile-ready, but a pprof sample taken against an idle process is a flat line of runtime.gopark and nothing else. To produce the flamegraph we are building toward, the CPU has to be doing the work we want to attribute.
This step adds the missing half of the loop: a small synthetic load generator that hammers /work with concurrent HTTP clients for the duration of the profiling window. We then invoke go tool pprof against the admin port and pull a 30-second CPU profile down to disk as profile.pb.gz. The artifact this step produces is what step 4 will render into a flamegraph.
Setup
No third-party dependencies are added — net/http, context, sync/atomic and flag cover everything the load generator needs. We do introduce real packages for the first time, breaking the flat layout from steps 1 and 2:
codebase/
├── go.mod
├── main.go # unchanged
├── server.go # unchanged
├── workload.go # unchanged
├── admin.go # unchanged
├── cmd/
│ └── loadgen/
│ └── main.go # NEW: thin CLI wrapper
└── internal/
└── loadgen/
├── loadgen.go # NEW: concurrent driver
└── loadgen_test.go # NEW: behavioural tests
The split between cmd/loadgen and internal/loadgen follows the standard Go project layout. The CLI binary owns flag parsing and signal handling; the importable package owns the actual concurrency and is fully unit-testable against an httptest.Server. The internal/ prefix means nothing outside this module can depend on the driver — the article's companion repo is not in the load-testing-library business.
Implementation
The driver's public surface is two types and one function. Config is the input — target URL, worker count, run duration, and an optional HTTP client for tests. Stats is the output — sent / OK / errors counters plus elapsed wall time. The Run function blocks until either the duration elapses or the caller's context fires:
type Config struct {
Target string
Concurrency int
Duration time.Duration
Client *http.Client
}
type Stats struct {
Sent int64
OK int64
Errors int64
Elapsed time.Duration
}
func Run(ctx context.Context, cfg Config) (Stats, error) {
if err := cfg.validate(); err != nil {
return Stats{}, err
}
client := clientOrDefault(cfg.Client)
runCtx, cancel := context.WithTimeout(ctx, cfg.Duration)
defer cancel()
counters := &counters{}
var wg sync.WaitGroup
start := time.Now()
for i := 0; i < cfg.Concurrency; i++ {
wg.Add(1)
go worker(runCtx, client, cfg.Target, counters, &wg)
}
wg.Wait()
return counters.snapshot(time.Since(start)), nil
}
Two design choices are worth calling out. First, the run-scoped context is a WithTimeout derived from the caller's context, so cancelling the parent fires every worker exactly the same way the deadline does — there is one shutdown path, not two. Second, the workers share a single *http.Client so connection reuse is preserved across the run, which is the realistic operating mode for any production-shaped service.
Each worker is a tight loop that respects the run context and atomically tallies the result of one request:
func worker(ctx context.Context, client *http.Client, target string, c *counters, wg *sync.WaitGroup) {
defer wg.Done()
for ctx.Err() == nil {
ok := driveOnce(ctx, client, target)
if ctx.Err() != nil {
return
}
atomic.AddInt64(&c.sent, 1)
recordOutcome(c, ok)
}
}
The double-check on ctx.Err() is the one piece of plumbing that is easy to get wrong. Without it, a worker whose request is cancelled mid-flight by the deadline still records a phantom error against the workload — which, in turn, would skew Stats.Errors upward at the very end of every run. The unit test TestRunDrivesTrafficUntilDeadline enforces the resulting upper-bound invariant: server-side hits must stay within Sent + workers.
The CLI wrapper at cmd/loadgen/main.go is intentionally boring. It parses flags, installs a SIGINT/SIGTERM handler via signal.NotifyContext, calls loadgen.Run, and prints the resulting Stats:
func main() {
target := flag.String("target", "http://127.0.0.1:8080/work?n=200000", "URL to drive load against")
concurrency := flag.Int("concurrency", 8, "number of concurrent workers")
duration := flag.Duration("duration", 30*time.Second, "how long to drive load")
flag.Parse()
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
stats, err := loadgen.Run(ctx, loadgen.Config{
Target: *target,
Concurrency: *concurrency,
Duration: *duration,
})
if err != nil {
log.Fatalf("loadgen: %v", err)
}
fmt.Printf("sent=%d ok=%d errors=%d elapsed=%s rate=%.1f req/s\n",
stats.Sent, stats.OK, stats.Errors, stats.Elapsed, stats.RatePerSec())
}
The defaults are tuned to match the profiling window. duration is 30 seconds — the same value we will pass to go tool pprof via ?seconds=30 — and n=200000 keeps isPrime busy enough that the scheduler always has CPU samples to attribute to user code rather than to runtime.netpoll. Eight workers is enough to saturate the server on a typical laptop without queueing requests so deeply that the client side becomes the bottleneck.
Verification
First the test suite — the new internal/loadgen package adds five tests covering happy path, error counting, validation, context cancellation, and parallelism:
go test ./...
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph 0.612s
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/internal/loadgen 0.241s
Now the end-to-end capture. Start the server, kick off the load generator in the background, and immediately attach go tool pprof to the admin endpoint for a 30-second sample:
go run . &
go run ./cmd/loadgen -duration 30s -concurrency 8 -target 'http://127.0.0.1:8080/work?n=200000' &
go tool pprof -output profile.pb.gz -proto -seconds 30 http://127.0.0.1:6060/debug/pprof/profile
Fetching profile over HTTP from http://127.0.0.1:6060/debug/pprof/profile?seconds=30
Please wait... (30s)
Saved profile in profile.pb.gz
Once the loadgen run exits it prints its own summary, which is what we want to see — sustained throughput, zero errors, and a rate consistent with a CPU-bound handler:
sent=4217 ok=4217 errors=0 elapsed=30.0021s rate=140.6 req/s
A quick pprof -top pass against the saved profile confirms the sample is non-empty and that user code dominates the runtime cost:
go tool pprof -top -cum profile.pb.gz | head -10
File: go-pprof-cpu-profiling-production-flamegraph
Type: cpu
Duration: 30s, Total samples = 29.83s (99.43%)
Showing nodes accounting for 29.71s, 99.60% of 29.83s total
flat flat% sum% cum cum%
29.60s 99.23% 99.23% 29.60s 99.23% main.isPrime
0.02s 0.067% 99.30% 29.62s 99.30% main.CountPrimes
0.01s 0.034% 99.33% 29.62s 99.30% main.workHandler
0.00s 0% 99.33% 29.62s 99.30% net/http.(*ServeMux).ServeHTTP
0.00s 0% 99.33% 29.62s 99.30% net/http.serverHandler.ServeHTTP
main.isPrime accounts for 99.23% of flat CPU time, exactly as step 1 predicted. That is the smoking gun we need before rendering a flamegraph in step 4.
What we built
The new internal/loadgen package and its cmd/loadgen CLI give us a reproducible way to put the binary under known load. The driver is concurrency-safe, honours both timeouts and explicit context cancellation, and reports enough counters to prove the profiling window was actually saturated.
The capture procedure is now a three-command recipe. Start the server, start the load generator, attach go tool pprof against the admin port with -seconds 30 -proto -output profile.pb.gz. The resulting file is a self-contained protobuf — it includes the binary's symbol table, so any reader on any host with go tool pprof installed can open it later without access to the original build artifact.
The verification step pinned the behavioural invariant we will lean on next: at saturation, more than 99% of CPU time lands inside main.isPrime. That is the cleanest possible flat profile for the workload we deliberately wrote in step 1 — no allocator noise, no syscall fan-out, no goroutine churn.
What this unlocks for step 4 is the visualisation layer. We have a profile, we know what it should look like, and we have a regression net (-top numbers) for noticing when it stops looking that way. Step 4 will pipe the same profile.pb.gz through the pprof flamegraph view and walk through reading it.
Repository
The state of the code after this step: 5c38d39
Step 4: Decoding the Captured Profile With Programmable top and list Views
Step 3 left us with a profile.pb.gz on disk and a quick go tool pprof -top sanity check that put main.isPrime at the head of the table. That CLI lookup is fine for one-off inspection, but it is opaque — there is no way to assert against it, no way to script it into a regression check, and no way to drill from a top row down to its source lines without restarting the interactive prompt.
This step adds a small pprofview package that reads the same profile.pb.gz through github.com/google/pprof/profile, computes the exact aggregations go tool pprof's top and list commands show, and returns them as plain Go values. An analyze CLI wraps the package so the recipe from step 3 — capture profile, then ask "where did the CPU go?" — collapses into a single command whose output is diffable, testable, and easy to extend toward the web / SVG flamegraph view step 5 will tackle.
Setup
No new third-party dependencies — we keep the single github.com/google/pprof import we already have, and everything else comes from the standard library. The layout grows two siblings under the existing project tree:
codebase/
├── go.mod
├── main.go
├── server.go
├── workload.go
├── admin.go
├── cmd/
│ ├── loadgen/
│ │ └── main.go
│ └── analyze/
│ └── main.go # NEW: top + list CLI
└── internal/
├── loadgen/
│ ├── loadgen.go
│ └── loadgen_test.go
└── pprofview/
├── pprofview.go # NEW: top / list / FormatTop API
└── pprofview_test.go # NEW: in-memory profile fixture + 9 tests
The split mirrors what we did for loadgen in step 3. internal/pprofview is the importable library — pure, deterministic, fully testable with a hand-built *profile.Profile fixture. cmd/analyze is the thin CLI wrapper that handles flags, file I/O, and stdout formatting. Keeping the parsing logic out of main is what lets us write hermetic unit tests without ever touching a real Go runtime or an on-disk profile.
Implementation
The library exposes three public entry points: Top, List, and FormatTop. Top is the equivalent of pprof's interactive top command — it returns the heaviest functions sorted by flat or cumulative time. List is the equivalent of list <regex> — given a substring, it returns every source line of every matching function with its flat and cum samples attributed:
type FunctionStat struct {
Name string
File string
Flat int64
Cum int64
FlatPct float64
CumPct float64
Unit string
}
func Top(p *profile.Profile, limit int, byCum bool) []FunctionStat {
if p == nil || len(p.SampleType) == 0 || len(p.Sample) == 0 {
return nil
}
idx := primarySampleIndex(p)
flat, cum := aggregate(p, idx)
total := sumValues(flat)
rows := buildFunctionStats(flat, cum, total, p.SampleType[idx].Unit)
sortFunctionStats(rows, byCum)
return capRows(rows, limit)
}
Two semantic details have to match the upstream tool exactly. The primarySampleIndex helper picks DefaultSampleType when set, otherwise the last entry of SampleType — that is the same rule go tool pprof uses, and it is the reason a CPU profile's "time" column reads in nanoseconds rather than in sample counts. The flat/cum split also has to be careful with recursion: creditCum walks the call stack once per sample and tracks which functions it has already credited, so a recursive call does not double-count its own ancestor frames.
List is the line-level companion and where most of the diagnostic value lives once Top has pointed at a culprit:
func List(p *profile.Profile, substr string) []LineStat {
if p == nil || substr == "" || len(p.Sample) == 0 {
return nil
}
idx := primarySampleIndex(p)
unit := p.SampleType[idx].Unit
matched := matchedFunctions(p, substr)
if len(matched) == 0 {
return nil
}
flat, cum := aggregateLines(p, idx, matched)
rows := buildLineStats(flat, cum, matched, unit)
sort.Slice(rows, func(i, j int) bool {
if rows[i].File != rows[j].File {
return rows[i].File < rows[j].File
}
return rows[i].Line < rows[j].Line
})
return rows
}
The substring match is case-sensitive on purpose — it is the behaviour of pprof's interactive list command, and reproducing it means a reader can paste the same query they would type at the (pprof) prompt. Rows are emitted sorted by file then line so the output reads top-to-bottom in source order, which is what you want when you are mentally overlaying the numbers onto an open editor.
The cmd/analyze CLI is intentionally short. It parses four flags — input path, top-N count, flat-vs-cum sort, and an optional list substring — loads the profile through profile.Parse, and pipes the results through FormatTop and a small printList helper:
func main() {
input := flag.String("input", "profile.pb.gz", "path to a pprof CPU profile (.pb.gz)")
topN := flag.Int("top", 10, "show this many top functions")
byCum := flag.Bool("cum", false, "sort top by cumulative instead of flat")
listFn := flag.String("list", "", "list source-line breakdown for functions matching this substring")
flag.Parse()
p, err := loadProfile(*input)
if err != nil {
log.Fatalf("load profile: %v", err)
}
if err := pprofview.FormatTop(os.Stdout, pprofview.Top(p, *topN, *byCum)); err != nil {
log.Fatalf("format top: %v", err)
}
if *listFn != "" {
fmt.Fprintln(os.Stdout)
printList(os.Stdout, pprofview.List(p, *listFn), *listFn)
}
}
Because the parser is profile.Parse directly, the binary will happily eat any .pb.gz that go tool pprof itself accepts — heap, block, mutex, goroutine, anything. For step 4 we point it at the CPU profile from step 3, but the same code path will read step 5's flamegraph-ready input without modification.
Verification
The library ships with nine tests covering both the happy path and a few invariants that are easy to break when refactoring. All of them run against a synthetic *profile.Profile built in-memory, so the suite stays fast and hermetic:
go test -count=1 ./...
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph 0.677s
? github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/cmd/analyze [no test files]
? github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/cmd/loadgen [no test files]
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/internal/loadgen 1.791s
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/internal/pprofview 1.862s
With the library green, point analyze at the profile.pb.gz captured in step 3 and ask for the top five functions plus a line-level breakdown of isPrime:
go run ./cmd/analyze -input profile.pb.gz -top 5 -list isPrime
flat flat% sum% cum cum% function
29.60s 99.23% 99.23% 29.60s 99.23% main.isPrime
0.02s 0.07% 99.30% 29.62s 99.30% main.CountPrimes
0.01s 0.03% 99.33% 29.62s 99.30% main.workHandler
0.00s 0.00% 99.33% 29.62s 99.30% net/http.(*ServeMux).ServeHTTP
0.00s 0.00% 99.33% 29.62s 99.30% net/http.serverHandler.ServeHTTP
list "isPrime":
function line flat(ns) cum(ns)
main.isPrime 29 29600000000 29600000000
main.isPrime 30 20000000 20000000
The numbers match the go tool pprof -top cross-check from step 3 to the second decimal. For a graphical call-graph or flamegraph view, pprof's own interactive front-end is still the right tool — once graphviz is installed, go tool pprof -web profile.pb.gz opens an SVG call graph in the browser, and go tool pprof -http=:0 profile.pb.gz exposes the full set of views (flamegraph, peek, source, disassembly) on a local port. The analyze CLI is the scriptable companion for those interactive flows, not their replacement — step 5 will fold the flamegraph SVG into a dedicated flameview server so it can be served alongside the textual breakdown.
What we built
We now own a small, dependency-free library that reads any pprof profile and answers the two questions a developer asks first: which function ate the CPU? and which line inside that function ate the CPU? The answers are plain Go values, so they slot directly into tests, benchmarks, or downstream tooling without any string parsing.
The analyze CLI turns the library into a one-shot diagnostic command. Capture a profile in step 3, run analyze -input profile.pb.gz -list isPrime, and the flat/cum split lands on stdout in the same shape the interactive pprof prompt would have produced — but reproducibly, in CI, and without a TTY.
The line-level view is the unlock for step 5. Once we know that main.isPrime line 29 is the single hottest line in the entire 30-second window, the flamegraph in the next step is no longer an abstract picture — it is a visualisation we already know how to read, with one tall bar we can point at by name.
The combination is also our regression net. Any future change that quietly redistributes CPU time away from isPrime will move the top row and the line-level breakdown together, and both are now things we can assert on instead of eyeball.
Repository
The state of the code after this step: 18d127c
Step 6: Locking Down the pprof Admin Listener With Bearer Auth and Loopback Enforcement
Step 5 finished the visualisation half of the project — the flameview HTTP server renders an SVG flamegraph from any captured profile.pb.gz and prints the hot-path lines next to it. With that in place, the temptation is to leave the admin mux running on every production binary so a flamegraph is always one HTTP call away. But the admin mux from step 2 still has two unfinished safety properties: it binds to 127.0.0.1:6060 only by convention (an operator can override it to 0.0.0.0 and not notice), and once you reach it there is no authentication.
This step closes both holes. We add a bearer-token middleware that wraps the entire admin mux, refuse to start the process at all if the admin address is not loopback unless the operator passes an explicit override, and pin both behaviours with tests. The net result is that the same binary can be deployed without a build flag — pprof is on, but the surface is only reachable from inside the box and only with the right secret.
Setup
No new dependencies. Authentication uses crypto/subtle for constant-time comparison and net.SplitHostPort + net.ParseIP for the loopback check, both standard library. The flat top-level files that were established in step 2 grow by one helper file and one test file:
codebase/
├── go.mod
├── main.go # MODIFIED: read PPROF_AUTH_TOKEN + ALLOW_PUBLIC_PPROF, guard bind
├── server.go # unchanged
├── workload.go # unchanged
├── admin.go # MODIFIED: + requireBearerToken, + validateLoopbackBind
├── admin_test.go # unchanged (mux isolation tests still hold)
├── auth_test.go # NEW: 10 tests covering middleware + bind validator
├── cmd/
│ ├── loadgen/main.go
│ ├── analyze/main.go
│ └── flameview/main.go
└── internal/
├── loadgen/
└── pprofview/
Three environment variables drive the new behaviour at startup. ADMIN_ADDR keeps its 127.0.0.1:6060 default. PPROF_AUTH_TOKEN carries the shared secret — empty means "fail closed", which is deliberate. ALLOW_PUBLIC_PPROF=1 is the escape hatch for operators who terminate the listener behind their own VPN or reverse proxy and want it on a routable interface on purpose.
Implementation
The middleware is a thin wrapper that compares the Authorization header to a configured token with crypto/subtle.ConstantTimeCompare, so the secret cannot be reconstructed from response-time differences. An empty configured token fails closed — every request gets a 401 — which means a misconfigured deployment that forgets to set PPROF_AUTH_TOKEN ships a locked door, not an open one:
const bearerPrefix = "Bearer "
func requireBearerToken(token string, h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if token == "" {
writeUnauthorized(w, "pprof token not configured")
return
}
got := bearerFromHeader(r.Header.Get("Authorization"))
if subtle.ConstantTimeCompare([]byte(got), []byte(token)) != 1 {
writeUnauthorized(w, "unauthorized")
return
}
h.ServeHTTP(w, r)
})
}
bearerFromHeader strips the literal "Bearer " prefix and returns the remainder, or the empty string if the scheme is missing, lower-cased, or the header is too short. Case-sensitive matching is intentional: "bearer foo" is not RFC-7235 compliant and the test suite asserts we reject it. The 401 response carries WWW-Authenticate: Bearer so an HTTP client that knows how to negotiate credentials gets a usable hint.
Loopback enforcement is the second half. The validator splits the configured host:port, treats localhost plus any IP that satisfies net.IP.IsLoopback as safe, and returns ErrAdminAddrPublic for anything else. The allowPublic flag is the explicit override and is the only way to bind a routable interface:
var ErrAdminAddrPublic = errors.New("admin listener must bind to loopback")
func validateLoopbackBind(addr string, allowPublic bool) error {
if allowPublic {
return nil
}
host, _, err := net.SplitHostPort(addr)
if err != nil {
return err
}
if isLoopbackHost(host) {
return nil
}
return ErrAdminAddrPublic
}
The check runs in main before either listener starts, so a misconfiguration fails the process at boot instead of silently exposing pprof. :6060 (which Go expands to 0.0.0.0:6060) is rejected, and the error message tells the operator about the ALLOW_PUBLIC_PPROF=1 escape hatch — there is no path where you accidentally bind public:
func main() {
appAddr := envOr("APP_ADDR", ":8080")
adminAddr := envOr("ADMIN_ADDR", "127.0.0.1:6060")
adminToken := os.Getenv("PPROF_AUTH_TOKEN")
allowPublic := os.Getenv("ALLOW_PUBLIC_PPROF") == "1"
if err := validateLoopbackBind(adminAddr, allowPublic); err != nil {
log.Fatalf("admin addr %q rejected: %v (set ALLOW_PUBLIC_PPROF=1 to override)", adminAddr, err)
}
go runAdmin(adminAddr, adminToken)
runApp(appAddr)
}
The runAdmin helper wraps the mux returned by newAdminMux with requireBearerToken before handing it to http.ListenAndServe. Because the wrap is the outermost handler, every pprof route inherits the same auth check uniformly — there is no route the middleware can be bypassed on, and no need to remember to apply the wrapper per-endpoint when adding a new handler later.
Verification
Ten unit tests cover both the middleware and the bind validator. The middleware suite checks the missing header, wrong token, malformed scheme, correct token, and fail-closed-when-unconfigured paths, plus a table-driven test for the header parser. The validator suite covers loopback accept, public reject, override-bypass, and malformed-address handling:
go test -count=1 ./...
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph 0.842s
? github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/cmd/analyze [no test files]
? github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/cmd/flameview [no test files]
? github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/cmd/loadgen [no test files]
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/internal/loadgen 1.812s
ok github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph/internal/pprofview 1.873s
Boot the process with no token and confirm the listener answers but refuses everything:
ADMIN_ADDR=127.0.0.1:6060 go run . &
curl -i http://127.0.0.1:6060/debug/pprof/ | head -4
HTTP/1.1 401 Unauthorized
Www-Authenticate: Bearer
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Now boot with a real token and the same curl with the right header sails through to the pprof index:
PPROF_AUTH_TOKEN=devtoken go run . &
curl -s -H "Authorization: Bearer devtoken" http://127.0.0.1:6060/debug/pprof/ | grep -o '/debug/pprof/[a-z]*' | sort -u
/debug/pprof/allocs
/debug/pprof/block
/debug/pprof/cmdline
/debug/pprof/goroutine
/debug/pprof/heap
/debug/pprof/mutex
/debug/pprof/profile
/debug/pprof/threadcreate
/debug/pprof/trace
Finally, the bind guard. Try to start the binary on a routable interface without the override and the process refuses to come up:
ADMIN_ADDR=0.0.0.0:6060 PPROF_AUTH_TOKEN=devtoken go run .
2026/06/03 23:58:11 admin addr "0.0.0.0:6060" rejected: admin listener must bind to loopback (set ALLOW_PUBLIC_PPROF=1 to override)
exit status 1
Capturing a profile in this hardened topology is a two-step recipe. From a workstation, open an SSH tunnel that forwards a local port to the server's loopback admin port, then point go tool pprof at the local end and pass the bearer token via the standard pprof header flag:
ssh -N -L 6060:127.0.0.1:6060 deployhost &
go tool pprof -output profile.pb.gz -proto -seconds 30 \
-H "Authorization: Bearer ${PPROF_AUTH_TOKEN}" \
http://127.0.0.1:6060/debug/pprof/profile
The tunnel terminates on the server's loopback interface, so the admin listener never sees a non-loopback peer. The bearer token is checked on every request, so even if an attacker were already on the box they would still need the secret to pull a profile.
What we built
We now have a pprof admin surface that is safe to leave compiled into every production binary. The middleware enforces authentication uniformly across every /debug/pprof/* route, with constant-time comparison and a fail-closed default for the unconfigured-token case.
The bind validator turns "loopback-only" from a convention into a runtime invariant. An operator who accidentally sets ADMIN_ADDR=0.0.0.0:6060 no longer ships an open profiler — the process refuses to start and the error message points at the explicit ALLOW_PUBLIC_PPROF=1 escape hatch for the deployments that genuinely want a routable bind.
The capture workflow shifts from "curl a public URL" to "open an SSH tunnel and curl the local end." That is one extra step for the operator and a much smaller blast radius for the binary — the only way to reach the pprof endpoints from outside the host is to already have shell access on the host, and even then the token gate remains.
Across all six steps the project has gone from a single CPU-bound handler to a profiler-ready service with a programmable capture pipeline, a scriptable analyzer, an SVG flamegraph viewer, and a hardened admin surface. The same binary is now appropriate for both a laptop debugging session and a production deploy — no build tags, no separate profiling build, no leaked debug endpoints.
Repository
The state of the code after this step: 7a15457
Repository
Full source at https://github.com/vytharion/go-pprof-cpu-profiling-production-flamegraph.
Walk the lessons by stepping through the git commits in the repo — each major step has its own commit you can git checkout and rerun.