Go Load Balancer: Circuit Breaker Lessons

DEV Community

My circuit breaker let 100 requests through: lessons from building a small load balancer in Go

I built loadbear, a small HTTP load balancer / reverse proxy in Go, mainly to see how concurrency and resilience patterns fit together in one program. It has four balancing strategies, active health checks, a circuit breaker, retries, per-IP rate limiting and an admin port with Prometheus-style metrics. It is not meant to replace nginx or HAProxy. After the first release, a reviewer asked a question that turned into the most useful week of the project: "How is half-open admission bounded?" Here is what we found, what was wrong, and what the numbers say. Repo: loadbear 1. The half-open state let everything through A circuit breaker has three states. When the cooldown ends, it moves from open to half-open and should let one probe request through to test the backend. If 100 requests arrive at that moment, all 100 shouldn't hit a backend that just recovered. I wrote a test with 100 concurrent goroutines. All 100 were admitted. The check looked at the state, but nothing limited how many requests could pass it. The fix is to make admission a single atomic operation. A simplified sketch of the idea (not the exact code): type breaker struct {
state atomic.Int32 // closed, open, halfOpen
probing atomic.Bool // is a probe in flight?
}

func (b breaker) allow() bool {
switch b.state.Load() {
case closed:
return true
case halfOpen:
// only the goroutine that flips false -> true gets to probe
return b.probing.CompareAndSwap(false, true)
default:
return false
}
}
The less obvious part is the other direction: the probe slot has to be released on every exit path, including timeouts, client cancellation and panics. Otherwise a cancelled probe could leave the breaker stuck in half-open. I added a regression test for that and a deferred release. 2. Retries were less safe than they looked I had limited retries to GET, HEAD and OPTIONS, which felt safe. Review and tests found more cases: A GET with a body that cannot be replayed was still retried. A retry could start after the client's deadline had already passed, because backoff did not look at the request context. Once response headers have been sent to the client, retrying is impossible, and the code needed to say so explicitly. Retries now require a replayable body, an uncommitted response and the original request context. I also added fault modes to the fake backend (hang, close, drop-mid, slow), so these cases can be reproduced on demand. 3. WebSocket "didn't work" and SIGHUP was broken The README said WebSocket upgrades were unverified. When I wrote a test, it failed immediately with a 400: The proxy stripped Connection and Upgrade from every request as hop-by-hop headers. My access-log response wrapper did not implement http.Hijacker or http.Flusher, so ReverseProxy could not set up the tunnel. Wrapping a ResponseWriter is an easy way to silently lose interfaces. After the fix, an HTTP/1.1 upgrade handshake and a bidirectional byte echo work through the proxy. I only verified that, not full RFC 6455 behavior, and the README says so. While adding config reload, I found that SIGHUP was wired into the shutdown signals. In v0.1.x, sending SIGHUP killed the process instead of reloading it. It now reloads the backend list and strategy, keeps state for unchanged backends, rejects invalid configs and lets in-flight requests finish. Only that part reloads; other settings still need a restart. 4. Benchmarks, and a measurement bug in my own tool I wrote a small load generator (standard library only) so the benchmark works on Windows too. The first run reported errors for roughly one request per worker. The proxy was fine: my generator cancelled in-flight requests at the end of the measured window and counted them as failures. The fix was to stop starting requests at the cutoff and let in-flight ones drain. Final results on a Ryzen 5 7600 (Windows 11, Go 1.27.4, 10 s measured, concurrency 64, rate limiting off, access logs off, clien...

Powered by Readybot.io*