Why a Java HPA Got Stuck at 20 Kubernetes Pods

Краткое содержание

Pixel shares how a Spring Boot pricing service autoscaled to 20 Kubernetes pods and stayed there because the JVM retained committed heap memory after traffic dropped. The post explains why memory-based HPA misread that usage as ongoing load and how switching to CPU and requests per second, with a fixed heap size, improved scaling.

DEV Community

We autoscaled our Java pricing service on memory and it grew to twenty pods and stayed there

Our quote pricing service is a Spring Boot application on Kubernetes. In August we gave it a horizontal pod autoscaler with two targets: seventy percent CPU and seventy percent memory, between four and twenty replicas. Memory was there because the service had once run out of it, and scaling on it felt like a sensible safety net. On the first busy Monday it scaled out to twenty, which was right. It was still at twenty on Tuesday, and on Saturday, and three weeks later, with CPU around fifteen percent and traffic at overnight levels. The cluster autoscaler had kept the nodes to match. The autoscaler computes the replicas each metric wants as the current replicas times the current utilisation divided by the target, rounded up, and when there are several metrics it takes the largest answer. Memory utilisation is the pod's working set divided by its memory request. CPU was asking for four replicas. Memory was asking for twenty or more, every time it was asked. The JVM sizes its heap for the work it has seen. Under Monday's load each pod's heap grew towards its maximum, and the garbage collector keeps that memory committed afterwards rather than handing it back to the operating system. It rarely gives committed heap back without settings we had never touched. So from the container's side, a pod handling two requests a second used almost the same memory as one handling two hundred. The autoscaler read that as load, and memory alone was enough to hold every replica in place. Nothing had malfunctioned. The autoscaler was scaling on a number that only ever went up, and doing it correctly. We removed the memory target. The service now scales on CPU and on requests per second per pod, which it exports and which falls when the traffic does. We set the initial and maximum heap to the same value, so memory use is flat from start to finish and nobody is tempted to read meaning into it. A fixed heap that never changes is a capacity setting, not a load signal. Memory pressure is now covered by an alert on heap use after garbage collection, which is the number that actually predicts running out. The rule we wrote down: before you scale on a metric, find out whether it can come back down. – Sergey Shinder
Why a Java HPA Got Stuck at 20 Kubernetes Pods
Powered by Readybot.io

Последнее от Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …