Go Profiling in Production: pprof, Flamegraphs and Memory Leaks

Your Go service handles thousands of requests per minute — until it doesn't. Memory climbs steadily, p99 latency spikes, and the first instinct is to add more instances. Before doing that, you need to understand what the code is actually doing. Go's built-in profiling tooling makes this tractable without third-party frameworks or expensive APM subscriptions. pprof is part of the standard library. Adding it to an existing HTTP service takes four lines: import ( "net/http" _ "net/http/pprof" ) func main() { // Expose pprof on a separate internal port — never public go func() { http.ListenAndServe("localhost:6060", nil) }() // ... rest of your server setup } The blank import registers the pprof HTTP handlers automatically. Never expose this on a public port — the CPU profile endpoint can be used to degrade your service deliberately. Bind to localhost or an internal network interface only. Once the service is running: go tool pprof http://localhost:6060/debug/pprof/heap This downloads a heap snapshot and opens an interactive prompt. Type top10 to see the ten functions allocating the most memory. Type web to open a flame graph in your browser (requires Graphviz). Flamegraphs look intimidating until you know one rule: width equals time or allocations spent. The x-axis is not time order — it is sorted alphabetically within each stack frame. A wide box means that function consumes a lot of the profiled resource. The y-axis is the call stack — top is where execution is at sample time, bottom is main. When investigating a memory leak: Take a heap profile under normal load: curl http://localhost:6060/debug/pprof/heap > heap1.prof Wait 10–15 minutes Take another snapshot: curl http://localhost:6060/debug/pprof/heap > heap2.prof Compare: go tool pprof -diff_base heap1.prof heap2.prof The diff shows exactly what grew between the two snapshots. If the same function keeps appearing in inuse_objects across successive profiles, that is your leak candidate. Go's garbage collector handles most memory automatically, but several patterns slip past it. Goroutine leaks are the most frequent issue. A goroutine blocked forever on a channel that nobody writes to is unreachable in the traditional sense — but the GC won't collect it because it is still alive on the scheduler. Check goroutine count: package main import ( "context" "fmt" "net/http" "runtime" "time" _ "net/http/pprof" ) func worker(ctx context.Context, ch <-chan struct{}) { select { case <-ch: fmt.Println("got work") case <-ctx.Done(): return // goroutine exits cleanly on cancellation } } func main() { go func() { http.ListenAndServe("localhost:6060", nil) }() ctx, cancel := context.WithCancel(context.Background()) defer cancel() for i := 0; i < 100; i++ { ch := make(chan struct{}) go worker(ctx, ch) // When cancel() is called, all workers exit } for { fmt.Printf("goroutines: %d\n", runtime.NumGoroutine()) time.Sleep(5 * time.Second) } } Check the goroutine profile: go tool pprof http://localhost:6060/debug/pprof/goroutine. The top output will show goroutines stuck on chan receive. The fix is to always use context.Context for lifecycle management so goroutines have a guaranteed exit path. Slice header retention is subtler. When you sub-slice a large byte array, the slice holds a reference to the entire underlying array. The whole allocation stays in memory even if you only need a small piece: // Keeps the full 10MB buffer alive in memory func extractHeader(data []byte) []byte { return data[:16] } // Copies only what is needed — the 10MB buffer is eligible for GC func extractHeaderSafe(data []byte) []byte { out := make([]byte, 16) copy(out, data[:16]) return out } The heap profile will point to the original make([]byte, 10<<20) allocation site rather than extractHeader, which makes this easy to miss on first inspection. For CPU-bound latency issues, collect a 30-second CPU profile: go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30 In the flame graph, look for unexpected width in functions that should be fast. Common findings: encoding/json.Marshal on hot paths — switch to sonic or jsoniter for 3–5x throughput fmt.Sprintf called thousands of times per second in logging code sync.Mutex.Lock contention showing up as runtime.lock Reflection-heavy code that could be replaced with code generation For mutex contention specifically, enable the mutex profile fraction before your service starts: import "runtime" func init() { runtime.SetMutexProfileFraction(1) // record every contention event } Then: go tool pprof http://localhost:6060/debug/pprof/mutex. This shows which mutexes are hot and the exact call sites where they are acquired. If your service runs in Docker or Kubernetes, the localhost binding becomes a problem. Two clean options: Kubernetes port-forward: kubectl port-forward pod/my-service-abc123 6060:6060 Profile from your local machine at localhost:6060. Conditional binding via environment variable: gate the pprof listener behind ENABLE_PPROF=1, and only set that variable in staging or when actively investigating production. Never route it through the public ingress. A useful pattern for performance-critical services is to keep a short operational runbook — including the specific pprof commands for your service — alongside your infrastructure security hardening checklists. Both degrade without maintenance, and both are most valuable when you are under pressure at 2am. Go's profiling tools are production-ready and require zero external dependencies. The practical workflow: Add the pprof import to an internal HTTP listener at startup Collect heap profiles before and after a suspected leak window Use diff_base to isolate allocation growth between snapshots Check goroutine count growth with the goroutine profile Capture a 30-second CPU profile during a latency event — not after The hardest part is having the profiles at the right moment. Consider setting up an alert that automatically collects a heap snapshot when RSS memory crosses a threshold. By the time you notice the problem manually and log in, the evidence is often already gone. I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.