Audit OpenAI-Compatible LLM Gateways in 5 Minutes

Resumen

Pixel’s announcement introduces ai-api-audit, a free, MIT-licensed Python tool for auditing OpenAI-compatible LLM gateways. It explains how tokenizer fingerprinting and nine protocol checks can reveal model swaps, injected prompts, padded token counts, silent truncation, and missing streaming usage data.

DEV Community

Tokenizer Fingerprinting: How to Audit Any OpenAI-Compatible LLM Gateway in 5 Minutes

Pointing an OpenAI SDK at a third-party base_url takes thirty seconds. Verifying what actually happens on the other side of that endpoint is considerably harder. When an inference gateway sits between your application and an upstream model, an HTTP 200 OK from POST /v1/chat/completions only confirms that text came back. Without protocol-level checks, your application cannot immediately tell whether: A cheaper model answered under an expensive model's name (routing a frontier model ID to a smaller open-weight model behind the scenes). Hidden wrapper instructions were injected into your prompt — inflating prompt_tokens on every call at your expense. Completion token counts were padded beyond what the model's actual output produced. Long-context inputs were silently truncated at the proxy layer while advertising a 128K or 1M context window. Streaming responses dropped usage accounting, making production cost observability impossible. While building DEVUP AI — an OpenAI- and Anthropic-compatible inference gateway and GPU cloud in Algiers — we wanted an objective, reproducible way for engineers to verify any provider from their own terminal rather than taking marketing pages on faith. We packaged those checks into a single-file, MIT-licensed Python tool: ai-api-audit. Below is the engineering behind how differential tokenizer fingerprinting works, why flat-rate proxy economics create incentives for silent degradation, what the nine protocol checks measure, and where the mathematical limits of black-box auditing lie. Why Differential Tokenizer Math Catches Model Family Swaps Every major LLM family segments text using its own Byte-Pair Encoding (BPE) or SentencePiece vocabulary: OpenAI (gpt-4o, gpt-4o-mini, reasoning series): o200k_base (~200k vocabulary, published in tiktoken). Qwen (Qwen2.5 / Qwen3 families): ~151k BPE vocabulary with distinct digit and multilingual byte merges. Mistral (Mistral-Nemo and newer): Tekken BPE (~131k vocabulary) or SentencePiece vocabularies. DeepSeek / Llama: Distinct vocabulary sizes and whitespace/UTF-8 merge priorities. If you send a single short English prompt ("Hello, world!"), two different tokenizers might coincidentally produce the same integer count — and even if they differ by 3 tokens, you cannot separate tokenizer differences from the provider's server-side chat template (<|im_start|>user...<|im_end|>). Isolating Δ prompt_tokens via Two-Request Subtraction To eliminate unknown chat-template framing, ai-api-audit uses differential measurement across two requests with identical message structure: Request A (Baseline): Sends a short control message P0 and records T_provider(P0) from usage.prompt_tokens. Request B (Probe): Sends P0 + ΔP, where ΔP is a dense, multi-script reference paragraph mixing technical English, French accents, Arabic script, punctuation, and numeric literals, and records T_provider(P0 + ΔP). Because the server-side chat template wrapper (W) is identical across both calls, subtracting the two cancels W out completely: ΔT_provider = T_provider(P0 + ΔP) - T_provider(P0)
We then tokenize ΔP locally using the model maker's official Hugging Face tokenizer.json (or tiktoken encoding) to obtain ΔT_official. Multilingual UTF-8 sequences act as a high-gain amplifier here. An Arabic-French-English paragraph that encodes into 180 tokens under o200k_base can easily split into 220+ or 310+ tokens under a different family's merge table. If a gateway advertises Qwen/Qwen2.5-72B-Instruct in /v1/models and echoes that string in response.model, yet ΔT_provider diverges sharply from Qwen's official tokenizer, either a different model family served the request or the proxy rewrote your prompt before forwarding it. Once ΔT_official is confirmed, Request A gives you a bonus measurement for free: subtracting the local token count of P0 from T_provider(P0) reveals the exact server overhead. A clean chat template adds 4 to 10 tokens; a 250-token gap means hidden system instructions are riding...
Audit OpenAI-Compatible LLM Gateways in 5 Minutes
Powered by Readybot.io

Lo último de Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …