Will a Local LLM Run on Your Laptop? Memory Math
ملخص
Pixel shares a practical guide to checking whether a local LLM can run on your laptop before downloading it. It explains how model weights, KV cache, memory capacity, and bandwidth affect fit and tokens per second, with real model configs and llama.cpp flags.
DEV Community
Will this LLM run on my laptop? The memory math to do before you download
The usual way people pick a local model goes like this. A model is trending, you pull 18 GB of weights over a slow connection, you start it, and it types four words a second while your fans spin up. You decide local LLMs are for people with a 4090 and go back to the API. I have done exactly that on my own laptop, an M1 Pro with 16 GB of RAM. What I do now is a few minutes of arithmetic first. Whether a model runs on a machine, and roughly how fast, is not a matter of opinion. It falls out of two numbers: how many bytes the model needs to keep in memory, and how many bytes it has to read for every token it writes. This post is that arithmetic, worked through with real model configs, plus the llama.cpp flags that act on each part of it. Every number below comes from a small Python script you can copy and point at your own hardware. The two numbers that decide everything Memory decides whether the model loads at all. It is the weights, plus the KV cache for your context, plus a little working space for the runtime. If that total fits in the memory your GPU can use, everything runs on the GPU. If it doesn't, some of it moves somewhere slower, and the speed drops off a cliff. Bandwidth decides how fast it can generate. To produce one token, the model reads every weight it uses out of memory. Every single token. The math inside is cheap next to that read, so generation speed has a hard ceiling: the rule tokens per second (ceiling) = memory bandwidth / bytes read per token
Real speed lands under that ceiling, never over it. But the ceiling is enough to rule things in or out before you download anything. Part one: the weights A model file is its parameter count times the bytes per parameter. Quantization is what changes the second number. These are the effective bits per weight for the GGUF formats llama.cpp, Ollama and LM Studio load, including the small per-block scale factors each format stores: bits per weight F16 16.0 bits/weight ~2.0 GB per billion params
Q8_0 8.5 bits/weight ~1.06 GB per billion
Q6_K 6.56 bits/weight ~0.82 GB per billion
Q4_K_M 4.85 bits/weight ~0.61 GB per billion
MXFP4 4.25 bits/weight ~0.53 GB per billion (gpt-oss experts)
So Qwen3-8B at Q4_K_M is about 8.2 x 0.61 = 5.0 GB, and Qwen3-14B is about 9.0 GB. Q4_K_M is the default for a reason: it keeps most of the quality at under a third of the F16 size. Go to Q5 or Q6 when you have room and the work is exact (code, JSON, maths). Below Q4 is for squeezing a model in, and I wouldn't trust it for tool calls, where one broken quote in the JSON is a failed request. Part two: the KV cache, which nobody budgets for While a model reads your prompt, every layer stores a key and a value vector for every token, so it doesn't have to recompute them for the next token. That store is the KV cache, and it grows linearly with context length. On a long context it can be as big as the weights. The size comes straight from the model's config.json: kv cache KV bytes = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_number x context_tokens
You can read those fields without downloading the model: terminal $ curl -sL https://huggingface.co/Qwen/Qwen3-14B/resolve/main/config.json \
| python3 -c "import json,sys; c=json.load(sys.stdin); print({k: c[k] for k in ['num_hidden_layers','num_key_value_heads','head_dim']})"
{'num_hidden_layers': 40, 'num_key_value_heads': 8, 'head_dim': 128}
For Qwen3-14B with an F16 cache, that is 2 x 40 x 8 x 128 x 2 bytes = 163,840 bytes, or 160 KB, for every token of context. At 32k tokens the cache alone is 5.4 GB, more than half the size of the weights. Two details in the config change this a lot. The first is num_key_value_heads. Qwen3-14B has 40 attention heads but only 8 KV heads (grouped-query attention), so its cache is five times smaller than it would be with one KV head per attention head. The second is sliding-window attention. gpt-oss-20b alternates layers: 12 use full attention, and 12 on...

Powered by Readybot.io