GLM-5.3 KV Cache Exposure: Causes and Fixes

Resumen

Pixel shares a deep dive into a cross-request KV cache exposure affecting GLM-5.3 on Workers AI. The article explains how an SGLang configuration mismatch and insufficient request isolation could leak cached state, then outlines remediation approaches—important lessons for secure, high-throughput AI inference.

DEV Community

Cross‑Request KV Cache Exposure in GLM‑5.3 Workers AI: A Deep Dive and SGLang Remediation

This article was originally published on FutureStack Introduction When a high‑throughput inference service leaks state between independent client calls, the breach is both a security risk and a performance nightmare. In late 2024 Cloudflare disclosed a cross‑request KV cache exposure in the GLM‑5.3 model running on Workers AI. The root cause was a subtle configuration mismatch that triggered a bug in SGLang, the lightweight runtime used to orchestrate token generation. Below we dissect the architecture, reproduce the failure mode, and walk through the remediation strategy we implemented at FutureStack. 1. Architectural Deep‑Dive 1.1 GLM‑5.3 on Workers AI – The Stack Layer Technology Role Model GLM‑5.3 (Meta) Large language model (≈ 5.3 B parameters) Runtime SGLang (C++/Rust hybrid) Token‑level scheduler, KV‑cache manager Edge Compute Cloudflare Workers (V8 isolates) Request isolation, network I/O State Store Workers KV (distributed LSM) Persistent key‑value cache for attention KV pages Transport HTTP/2 + binary protobuf Low‑latency inference payload The model is sharded across three V8 isolates. Each isolate holds a local KV cache for the attention mechanism (the so‑called KV pages). The cache lives in Workers KV, which offers eventual consistency and high read‑through performance but no built‑in request‑level isolation. 1.2 KV Cache Memory Model During generation, each transformer layer emits a key and a value matrix for the current token. These matrices are stored as pages (e.g., 64 KB each) in Workers KV under a composite key: kv:{model_id}:{request_id}:{layer_idx}:{token_idx}
model_id – static identifier (e.g., glm5.3). request_id – UUID generated per HTTP request. layer_idx – layer number (0‑N). token_idx – position of the token in the generated sequence. When the runtime needs to attend to past tokens, it fetches the appropriate pages via a batched GET. The latency budget per token is ~2 ms on the edge, so the KV fetch must be cache‑friendly and avoid cross‑request contention. 1.3 SGLang’s KV‑Cache Manager SGLang abstracts the KV store behind a CacheHandle object. Internally it: Pre‑allocates a buffer pool for each request. Maps KV pages to buffer slots via a hash table keyed by the composite string. Releases buffers back to the pool when the request completes. The critical invariant is that no buffer may be reused until the request’s lifecycle ends. The pool is global to the isolate to amortize allocation cost. 1.4 The Configuration Mismatch Workers AI ships with two mutually exclusive modes: Mode sglang.kv_isolation Description Strict true Each request gets a private pool; buffers are zero‑filled on reuse. Relaxed false Buffers are recycled across requests for throughput. A deployment script mistakenly set sglang.kv_isolation = false while the model’s KV‑page TTL was set to 0 (no expiration). The combination meant that after a request finished, its buffers were immediately recycled and could be served to a subsequent request before the KV store had flushed the stale pages. An attacker could therefore infer previous token embeddings by timing the fetch latency or, in the worst case, read the raw KV payload. 2. The SGLang Bug The bug manifested in the CacheHandle::acquire routine: CacheSlot CacheHandle::acquire(const std::string& key) {
auto it = pool.find(key);
if (it != pool.end()) {
// BUG: missing guard that ensures the slot belongs to the current request
return it->second;
}
// allocate new slot …
}
Because the pool key omitted the request_id component when kv_isolation was disabled, two distinct requests could resolve to the same slot. The isolation contract was silently broken, leading to the cross‑request exposure. 3. Remediation Blueprint FutureStack’s remediation strategy had three pillars: Enforce request‑scoped keys – always embed request_id in the KV lookup, regardless of the isolation flag. Introduce a
generation fence – a lightweight epoch counter that invalidates a...
GLM-5.3 KV Cache Exposure: Causes and Fixes
Powered by Readybot.io*

Lo último de Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …