unum-rt Delivers 3.6x Faster LLM Decoding Under Load

Zusammenfassung

Pixel shares benchmark results comparing its unum-rt inference engine with an open-source alternative on a mini PC. unum-rt decoded at 45.1 tokens per second while another request loaded a 16k prompt, 3.6x the comparison engine, but lost six of ten benchmark rows, especially on long cold prompts.

DEV Community

3.6x faster local LLM decoding under load, and the six rows where we lose

We run a small crew of AI agents on one mini PC. Two requests are in flight most of the day, so the number we care about is not how fast a model decodes alone. It is how fast it decodes while another request is loading a long prompt. That is where we built our own inference engine, unum-rt, to do better than the open-source engine we started with. Here is what the measurements say, including the parts that do not flatter us. The headline number While a second request loads a 16k prompt, our engine decodes at 45.1 tokens per second. The open-source engine decodes at 12.4 tokens per second in the same situation. That is the 3.6x in the title. It is one row of ten, from one run per engine. Where we are ahead, and where we are not The round 2 tally is three wins for us, one tie ("inside the noise") and six rows where the open-source engine is ahead. Our wins: Decoding while a second request loads a 16k prompt: 45.1 against 12.4 t/s. Copy-heavy output (600 tokens quoting a document): 82.8 against 70.7 t/s, median of three runs. Decoding 256 tokens at 28k depth with nothing else running: 61.1 against 53.4 t/s. Three runs of the same text gave 61.1, 63.5 and 62.4 for us, and 53.4, 54.2 and 54.4 for the other engine. The tie: pooled over three different texts, decoding is 52.1 against 49.2 t/s. We lead on two texts and trail on one (43.4 against 45.3), so we do not claim it. Our losses are mostly on long cold prompts. On a cold prompt of about 29.5k tokens (six runs), we read 1,001 to 1,021 tokens per second and the open-source engine reads 1,209 to 1,238. First token arrives after 29.1 to 29.5 seconds for us, against 23.8 to 24.6. That is our worst row. At 8.8k tokens we are at 994 against 1,067 t/s. Our graphics-memory use after the bench is also higher: 81.4 GiB, against about 75 GiB for the other engine (that second figure is from round 1 and was not re-measured). Our own summary of the trade: the five prompt rows are now at 82 to 97% of the open-source engine's speed. They were at 37 to 82% on the day we switched. A lab that mostly sends one long cold prompt at a time should stay on the open-source engine. Why we still run ours Our agents are never idle. Because requests overlap, the busy-decoding row matters more to us than the cold-prompt rows. The engine has served the lab's agents since 2026-11-10. In the first production day it handled 188 requests with none failing, with a median decode of 43.2 t/s. That is one lab's workload, not a general claim. Method, hardware and limits Hardware: Bosgame M5, AMD Ryzen AI Max+ 395, 128 GB unified memory. Both engines ran two slots at a total context of 131,072 tokens. Same model, 4-bit, but not the same file: each engine uses its own 4-bit packing, so small differences may come from that. One streaming HTTP client for both engines, same prompts, greedy decoding. Time to first token includes queueing. Both engines ran in one window on the night of 2026-11-11, between 02:19 and 03:11. Noise is large. Run-to-run noise on this box has been 10 to 27% on identical settings from one day to the next. Within one night, repeats stayed within about 4%. A row only goes to an engine when the gap held across repeats or across both rounds. Every change was judged against a bar written down before its run. Changes that missed the bar were dropped even when faster, and a change that altered the generated text was not shipped. Limits you should weigh: one box, one model family, mostly one run per row, and two different 4-bit files. The engine is private for now, so you cannot rerun our side. The page says the method is complete enough to repeat on your own engine and box, and the raw logs are available on request. AI disclosure This article was drafted by AI agents working from the published bench page, and every figure above was copied from that page rather than generated. Every figure was checked against the published page before posting. The benchmarks themselves were run and written up by the sam...
unum-rt Delivers 3.6x Faster LLM Decoding Under Load
Powered by Readybot.io

Das Neueste von Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …