Fix Docker TLS Handshake Timeouts in CI

DEV Community

Docker net/http: TLS handshake timeout in CI

A Docker TLS handshake timeout in CI means the pull opened a connection to the registry and then ran out its deadline before the certificate exchange finished. It is a path problem rather than an image problem, which is why the same docker pull works on the next run, and why the first thing worth changing is how many pulls your job starts at once. The message names the stage the pull died in. Each stage has a different set of causes, and only one of them is worth pointing at the registry. What this error means The pull fails in seconds rather than minutes, with a single line from the daemon quoting the URL it was trying and a Go networking error. Three wordings mean the same thing at three different points: a handshake that started and did not finish, a request canceled while waiting for headers, and a context deadline exceeded. All three are the client giving up rather than the registry refusing you, which is the difference between this and a rate limit or an authentication failure. This page carries no recorded run. The reproduction available to us is a listener that accepts the connection and never completes the handshake, which produces the message but models a registry that is permanently unreachable rather than the transient this failure actually is, so the block below is the sample text the Latchkey pattern library matches on, labeled as such, and the causes come from the daemon issues in the references. ```Sample log from the Latchkey pattern library Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: TLS handshake timeout Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Error response from daemon: failed to resolve reference "docker.io/library/node:lts-slim": failed to do request: Head "https://registry-1.docker.io/v2/library/node/manifests/lts-slim": net/http: TLS handshake timeout

Common causes

The runner was pulling several things at once

The most common and the most fixable. Handshakes are latency sensitive and small; a link busy moving layer blobs will delay them enough to trip a deadline measured in seconds. A job that fails in the step where compose starts six containers, and passes when it pulls one image, has said what the problem is.

The registry edge was having a bad minute

Public registries are shared infrastructure and they have slow periods. The moby issue tracker carries this one continuously, with the same shape each time: intermittent, not reproducible on demand, gone on the next attempt. Nothing in your workflow caused it and nothing in your workflow will prevent it, so the answer is a retry and a mirror.

Something is inspecting TLS between the runner and the registry

A corporate proxy, an egress appliance or a service mesh sidecar terminating TLS adds work and latency to every handshake, and sometimes rewrites the certificate chain. If the failures are constant rather than occasional, and only from one network, this is the first place to look rather than the last.

The path itself is lossy, or IPv6 is a black hole

Packet loss and a broken path MTU both show up as handshakes that stall, because the handshake needs several round trips before anything useful has happened. An environment that advertises IPv6 without a working route is the same failure with a cleaner cause: the daemon tries the address it was given, waits, and reports a timeout.

How to fix it

Retry the pull, with a bounded loop

1. Pull explicitly before the step that needs the image, so a failure is attributed to the pull rather than to a build or a test.
2. Retry three times with a short backoff. A handshake that fails twice in a row is usually a path problem rather than a busy minute.
3. Fail the step on the last attempt rather than continuing into a build that has no image.

```Terminal
for attempt in 1 2 3; do
docker pull "$IMAGE" && break
...

Powered by Readybot.io