DEV Community

I Cut Dedicated GPU Memory to 4GB, and a Local LLM Still Ran at 90% Speed

On a Surface Laptop Ultra (RTX Spark, unified memory), I measured local LLM speed and Windows' free memory with the GPU's dedicated memory set to 4, 64, 82 and 96GB. Speed barely moved; memory left for Windows did. This is an English adaptation of an article I originally published in Japanese on Zenn: 専用 GPU メモリーを 4GB にしても、ローカル LLM は 9 割の速さで動いた. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original. Short version: on this laptop, setting the GPU's dedicated memory to the minimum (4GB) still left a local LLM at about 90% of its speed, while the memory left for Windows changed from 122.5GB to 30.5GB across the settings. A 71-second video summary (narration is synthetic, English captions available): TL;DR This laptop's CPU and GPU share one pool of physical memory. A setting decides how much is dedicated to the GPU, from 4GB to 96GB. The rest is main memory for Windows and apps. With dedicated GPU memory at the minimum 4GB, gpt-oss-120b (63.4GB) still generated 58.8 tokens/s, about 92% of the 63.9–64.2 at 64GB and up. Most of the model sat outside the dedicated part, in memory the GPU borrows from main memory. Memory left for Windows went from 122.5GB (4GB setting) to 30.5GB (96GB setting). At 82GB and 96GB, with the model loaded and a browser open, committed memory reached 99% of the commit limit. Moving MoE experts to the CPU (-ncmoe) was slower than leaving everything on the GPU, even at 4GB: about 2.1–2.6× slower for gpt-oss-120b and 1.8× for the other model. I lowered my everyday setting from 82GB to 64GB. One laptop, two models, one run per setting. On a PC with a separate graphics card (its own VRAM), I'd expect very different results. Setup PC Surface Laptop Ultra (Windows 11 Pro, NVIDIA RTX Spark N1X, 128GB unified memory) How the dedicated amount was changed NVIDIA App, "Dedicated GPU memory (GB)", with a reboot each time. Options ran from 4GB (minimum) to 96GB (maximum) Settings tested 4GB (minimum), 64GB (about the size of gpt-oss-120b's 63.4GB file), 82GB (my setting until then), 96GB (maximum) Driver 621.24 llama.cpp b12110 (CUDA), the same build as my 8-model benchmark Models gpt-oss-120b (MXFP4, 63.4GB), Nemotron 3.5 Bolt (74B total, about 9B active per token, 2-bit, about 22GB) Stopped during the test WSL, ComfyUI and other LLMs. Only at 82GB, an Ollama instance with no model loaded was running What I call each part of memory There is one pool of physical memory, split in two by the setting: Name What it is Dedicated GPU memory The part reserved for the GPU. Set in the NVIDIA App. This is what nvidia-smi reports Main memory The rest. Used by Windows and apps. This is what Task Manager shows as "Memory" Shared GPU memory The part of main memory the GPU borrows when it needs more Dedicated plus main came to 126.5GB at every setting. How I measured At each setting, in the same order: 3 minutes idle, logging free memory and more every 5 seconds. llama-bench, average of 3 runs: pp512 (prompt processing): how fast the model reads a 512-token prompt before answering. tg128 (generation): how fast it writes 128 tokens. Closest to how fast text appears in a chat. A 5-minute load with gpt-oss-120b loaded in llama-server: 10 browser tabs, plus a script grabbing main memory 4GB at a time up to 16GB. Still logging every 5 seconds. I logged Windows performance counters (Available MBytes, Committed Bytes, Commit Limit, page file usage, disk writes) and dedicated GPU memory use from nvidia-smi. Memory sizes from the counters are converted at 1GB = 10^9 bytes. # Everything assigned to the GPU (the main runs)
llama-bench -m .gguf -ngl 999 -fa 1 -p 512 -n 128 -r 3

Some MoE experts on the CPU side (for comparison)

llama-bench -m .gguf -ngl 999 -ncmoe 16 -fa 1 -p 512 -n 128 -r 3
Result 1: about 90% of the speed at 4GB All layers assigned to the GPU (-ngl 999). Tokens per second, average of 3 runs. Dedicated GPU memor...

Powered by Readybot.io