DEV Community
Cross‑Request KV Cache Exposure in GLM‑5.3 Workers AI: A Deep Dive and SGLang Remediation
This article was originally published on FutureStack Introduction When a high‑throughput inference service leaks state between independent client calls, the breach is both a security risk and a performance nightmare. In late 2024 Cloudflare disclosed a cross‑request KV cache exposure in the GLM‑5.3 model running on Workers AI. The root cause was a subtle configuration mismatch that triggered a bug in SGLang, the lightweight runtime used to orchestrate token generation. Below we dissect the architecture, reproduce the failure mode, and walk through the remediation strategy we implemented at FutureStack. 1. Architectural Deep‑Dive 1.1 GLM‑5.3 on Workers AI – The Stack Layer Technology Role Model GLM‑5.3 (Meta) Large language model (≈ 5.3 B parameters) Runtime SGLang (C++/Rust hybrid) Token‑level scheduler, KV‑cache manager Edge Compute Cloudflare Workers (V8 isolates) Request isolation, network I/O State Store Workers KV (distributed LSM) Persistent key‑value cache for attention KV pages Transport HTTP/2 + binary protobuf Low‑latency inference payload The model is sharded across three V8 isolates. Each isolate holds a local KV cache for the attention mechanism (the so‑called KV pages). The cache lives in Workers KV, which offers eventual consistency and high read‑through performance but no built‑in request‑level isolation. 1.2 KV Cache Memory Model During generation, each transformer layer emits a key and a value matrix for the current token. These matrices are stored as pages (e.g., 64 KB each) in Workers KV under a composite key: kv:{model_id}:{request_id}:{layer_idx}:{token_idx}
model_id – static identifier (e.g., glm5.3). request_id – UUID generated per HTTP request. layer_idx – layer number (0‑N). token_idx – position of the token in the generated sequence. When the runtime needs to attend to past tokens, it fetches the appropriate pages via a batched GET. The latency budget per token is ~2 ms on the edge, so the KV fetch must be cache‑friendly and avoid cross‑request contention. 1.3 SGLang’s KV‑Cache Manager SGLang abstracts the KV store behind a CacheHandle object. Internally it: Pre‑allocates a buffer pool for each request. Maps KV pages to buffer slots via a hash table keyed by the composite string. Releases buffers back to the pool when the request completes. The critical invariant is that no buffer may be reused until the request’s lifecycle ends. The pool is global to the isolate to amortize allocation cost. 1.4 The Configuration Mismatch Workers AI ships with two mutually exclusive modes: Mode sglang.kv_isolation Description Strict true Each request gets a private pool; buffers are zero‑filled on reuse. Relaxed false Buffers are recycled across requests for throughput. A deployment script mistakenly set sglang.kv_isolation = false while the model’s KV‑page TTL was set to 0 (no expiration). The combination meant that after a request finished, its buffers were immediately recycled and could be served to a subsequent request before the KV store had flushed the stale pages. An attacker could therefore infer previous token embeddings by timing the fetch latency or, in the worst case, read the raw KV payload. 2. The SGLang Bug The bug manifested in the CacheHandle::acquire routine: CacheSlot CacheHandle::acquire(const std::string& key) {
auto it = pool.find(key);
if (it != pool.end()) {
// BUG: missing guard that ensures the slot belongs to the current request
return it->second;
}
// allocate new slot …
}
Because the pool key omitted the request_id component when kv_isolation was disabled, two distinct requests could resolve to the same slot. The isolation contract was silently broken, leading to the cross‑request exposure. 3. Remediation Blueprint FutureStack’s remediation strategy had three pillars: Enforce request‑scoped keys – always embed request_id in the KV lookup, regardless of the isolation flag. Introduce a generation fence – a lightweight epoch counter that invalidates a...
Powered by Readybot.io*
External Link
Platform
You are about to visit:
https://example.comThis will open in a new tab. Make sure you trust this link.