Worker pools in Go: limiting goroutines, collecting results, safe cancellation

1. Overview

Goroutines in Go are cheap, so calling go for each task is common practice. When the number of tasks grows to tens of thousands and each task hits a database or HTTP endpoint, this approach easily exhausts the connection pool or exceeds the rate limit of external services.

This article guides you through building a worker pool—a fixed number of goroutines that pull work from a single channel. By the end, you’ll know how to write a basic pool, collect results and errors, cancel with context, simplify using errgroup.SetLimit, and test that the pool truly limits concurrency.

2. The Problem: Unbounded Goroutines

Consider this code that processes a list of orders:

for _, o := range orders {
	go chargeOrder(o) // 50,000 orders → 50,000 goroutines calling payment API at once
}

Each goroutine uses only a few KB of memory, so RAM is rarely the bottleneck. The issue is downstream resources:

• The database connection pool has only 20–50 connections.
• Payment gateways limit requests per second.
• The process has a limit on file descriptors.

A worker pool solves this by separating two concepts: the number of tasks can be huge, but the number of tasks running concurrently is capped by the number of workers.

3. Basic Worker Pool

The pattern has three parts:

1. A channel jobs containing work to do.
2. N worker goroutines, each looping over the channel with for range.
3. A sync.WaitGroup to wait for all workers to finish.

package main

import (
	"fmt"
	"sync"
	"time"
)

type Order struct {
	ID     int
	Amount float64
}

func worker(id int, orders <-chan Order, wg *sync.WaitGroup) {
	defer wg.Done()
	for o := range orders {
		time.Sleep(100 * time.Millisecond) // simulate payment API call
		fmt.Printf("worker %d processing order %d (%.0f)\n", id, o.ID, o.Amount)
	}
}

func main() {
	orders := make(chan Order)
	var wg sync.WaitGroup

	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go worker(i, orders, &wg)
	}

	start := time.Now()
	for i := 1; i <= 9; i++ {
		orders <- Order{ID: i, Amount: float64(i) * 10}
	}
	close(orders)
	wg.Wait()
	fmt.Println("done in", time.Since(start).Round(100*time.Millisecond))
}

Here’s how it works:

• orders <-chan Order is a receive-only channel on the worker side. The compiler prevents the worker from accidentally sending or closing it.
• for o := range orders exits automatically when the channel is closed and drained.
• close(orders) signals “no more work.” If you forget this line, workers wait forever and wg.Wait() deadlocks.
• wg.Add(1) must be called before go worker(...), not inside the goroutine.

Output (line order may vary):

worker 3 processing order 1 (10)
worker 1 processing order 2 (20)
worker 2 processing order 3 (30)
worker 2 processing order 4 (40)
...
done in 300ms

Nine orders divided by three workers, each taking 100ms, gives roughly 300ms total instead of 900ms sequentially.

Note: From Go 1.25, you can write wg.Go(func() { worker(i, orders) }). This method auto-calls Add and Done, avoiding the easy mistake of forgetting either.

4. Collecting Results and Errors

In real code, you usually need to know which orders succeeded and which failed. Add a results channel:

package main

import (
	"errors"
	"fmt"
	"sync"
)

type Order struct {
	ID     int
	Amount float64
}

type Result struct {
	OrderID int
	Fee     float64
	Err     error
}

func chargeOrder(o Order) (float64, error) {
	if o.Amount <= 0 {
		return 0, errors.New("invalid amount")
	}
	return o.Amount * 0.02, nil // 2% fee
}

func main() {
	orders := []Order{{1, 100}, {2, 0}, {3, 250}, {4, 80}}

	jobs := make(chan Order)
	results := make(chan Result)
	var wg sync.WaitGroup

	for range 3 {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for o := range jobs {
				fee, err := chargeOrder(o)
				results <- Result{OrderID: o.ID, Fee: fee, Err: err}
			}
		}()
	}

	go func() {
		for _, o := range orders {
			jobs <- o
		}
		close(jobs)
	}()

	go func() {
		wg.Wait()
		close(results)
	}()

	for r := range results {
		if r.Err != nil {
			fmt.Printf("order %d error: %v\n", r.OrderID, r.Err)
			continue
		}
		fmt.Printf("order %d fee %.2f\n", r.OrderID, r.Fee)
	}
}

Output looks like:

order 1 fee 2.00
order 3 fee 5.00
order 2 error: invalid amount
order 4 fee 1.60

Three important points:

• Sending jobs runs in a separate goroutine. If main sent jobs and read results on the same goroutine, the worker would block on results <- while main blocked on jobs <-, causing deadlock.
• A third goroutine waits on wg.Wait() before close(results). Only when all workers finish can you guarantee no one is sending to results. Rule of thumb: only the sender closes the channel.
• Errors are wrapped in Result rather than stopping the entire pool. One order’s failure shouldn’t block others.

Result order does not match input order. If order matters, carry an index in the job or use the approach in section 6.

5. Canceling the Pool with Context

When an HTTP request is canceled or times out, the pool should stop instead of continuing to call the payment API. Do this by having workers listen to ctx.Done():

func worker(ctx context.Context, jobs <-chan Order, results chan<- Result) {
	for {
		select {
		case <-ctx.Done():
			return
		case o, ok := <-jobs:
			if !ok {
				return
			}
			fee, err := chargeOrder(o)
			select {
			case results <- Result{OrderID: o.ID, Fee: fee, Err: err}:
			case <-ctx.Done():
				return
			}
		}
	}
}

The producer side also needs a similar select:

go func() {
	defer close(jobs)
	for _, o := range orders {
		select {
		case jobs <- o:
		case <-ctx.Done():
			return
		}
	}
}()

And create a context with a timeout at the call site:

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

Why select on both send directions: if the worker only checks ctx when receiving a job, it could still block forever on results <- if the reader has already given up. This is a common source of goroutine leaks.

Note: select only helps workers exit between jobs. If chargeOrder makes an HTTP call, you must pass ctx into it (http.NewRequestWithContext) to cancel the in-flight request itself.

6. Simplifying with errgroup.SetLimit

Writing channels, WaitGroup, and context by hand is error-prone. The errgroup package provides SetLimit, letting you create a bounded pool in just a few lines. First, add the dependency:

go get golang.org/x/sync/errgroup

This generic function processes a slice with at most limit goroutines, preserves result order, and stops on the first error:

package pool

import (
	"context"

	"golang.org/x/sync/errgroup"
)

func Process[T, R any](
	ctx context.Context,
	limit int,
	items []T,
	fn func(context.Context, T) (R, error),
) ([]R, error) {
	results := make([]R, len(items))
	g, ctx := errgroup.WithContext(ctx)
	g.SetLimit(limit)

	for i, item := range items {
		g.Go(func() error {
			r, err := fn(ctx, item)
			if err != nil {
				return err
			}
			results[i] = r
			return nil
		})
	}

	if err := g.Wait(); err != nil {
		return nil, err
	}
	return results, nil
}

Here’s how it works:

• g.SetLimit(limit) makes g.Go block when there are already limit goroutines running. In effect, it’s equivalent to a worker pool.
• errgroup.WithContext returns a ctx that gets canceled as soon as any function returns an error. Other tasks using ctx receive the stop signal.
• results[i] = r is safe to do concurrently because each goroutine writes to a unique index. No mutex needed here.
• From Go 1.22, loop variables i and item are recreated each iteration, so closures don’t share variables like older versions.

Usage:

fees, err := pool.Process(ctx, 5, orders, func(ctx context.Context, o Order) (float64, error) {
	return paymentService.Charge(ctx, o)
})

Difference from a hand-written pool: errgroup spawns a new goroutine for each item instead of reusing N workers. For most applications, this overhead is negligible. Use a hand-written pool from sections 3–5 if workers need private state, like each holding a database connection.

7. Testing a Worker Pool

You need to test two main behaviors: the pool never exceeds limit, and errors come back correctly. Measure concurrency using atomic.Int32:

package pool

import (
	"context"
	"errors"
	"sync/atomic"
	"testing"
	"time"
)

func TestProcess_LimitsConcurrencyAndKeepsOrder(t *testing.T) {
	var running, maxSeen atomic.Int32
	items := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

	got, err := Process(context.Background(), 3, items, func(_ context.Context, n int) (int, error) {
		cur := running.Add(1)
		defer running.Add(-1)
		for {
			m := maxSeen.Load()
			if cur <= m || maxSeen.CompareAndSwap(m, cur) {
				break
			}
		}
		time.Sleep(10 * time.Millisecond)
		return n * 2, nil
	})

	if err != nil {
		t.Fatalf("unexpected error: %v", err)
	}
	if maxSeen.Load() > 3 {
		t.Errorf("ran %d concurrently, limit is 3", maxSeen.Load())
	}
	want := []int{2, 4, 6, 8, 10, 12, 14, 16, 18, 20}
	for i := range want {
		if got[i] != want[i] {
			t.Fatalf("got[%d] = %d, want %d", i, got[i], want[i])
		}
	}
}

func TestProcess_ReturnsFirstError(t *testing.T) {
	errDeclined := errors.New("card declined")

	_, err := Process(context.Background(), 2, []int{1, 2, 3}, func(_ context.Context, n int) (int, error) {
		if n == 2 {
			return 0, errDeclined
		}
		return n, nil
	})

	if !errors.Is(err, errDeclined) {
		t.Fatalf("want errDeclined, got %v", err)
	}
}

Run tests with the race detector:

go test -race ./...

ok  	example.com/shop/pool	0.061s

The CompareAndSwap loop updates maxSeen without locks. The -race flag is mandatory with concurrent code—it catches data races your eyes might miss.

8. Common Mistakes and When Not to Use a Pool

Common mistakes:

• Forgetting close(jobs): workers never exit and the program reports fatal error: all goroutines are asleep - deadlock!.
• Closing a channel on the receive side, or twice: causes panic: close of closed channel.
• Calling wg.Add inside a goroutine: Wait might return before the goroutine has added itself.
• Sending jobs and reading results on the same goroutine with an unbuffered channel: causes deadlock as explained in section 4.
• Not selecting on ctx.Done() when sending results: leads to goroutine leaks if the reader exits early.
• Picking a worker count randomly: for CPU-bound work, use runtime.NumCPU(). For I/O-bound work, base it on downstream limits, like connection pool size.

When not to use a worker pool:

• Few, known-in-advance tasks, like calling three services in parallel. Plain errgroup without SetLimit is enough.
• Extremely short tasks, only microseconds. Channel overhead might exceed the actual work, so sequential or batched processing is faster.
• You need to limit throughput (requests/second) not concurrency. This requires a rate limiter like golang.org/x/time/rate. A worker pool can’t replace that.

9. Conclusion

A worker pool lets you control parallelism in Go: the number of tasks can be huge but downstream resources stay protected. This article covered:

• Basic pools with channels and sync.WaitGroup.
• Collecting results and errors via a results channel, closing at the right time.
• Safe cancellation with context on both send sides.
• Simplifying with errgroup.SetLimit and a generic Process function.
• Testing concurrency limits and error handling with go test -race.

For most use cases, errgroup.SetLimit is the simplest and least error-prone choice. Write your own pool only if workers need private state or you need fine-grained control over job and result flow.