DEV Community
TokenBpe A editorial on how tokenizers can be sped up.
Introduction Every LLM pipeline starts with a tokenizer and almost no one thinks about it. The model gets the headlines, the tokenizer is quietly burning microseconds per token per request, forever. If you're serving millions of requests or just training on a corpus that doesn't fit in memory that "quietly" adds up. This post is a walk through TokenBpe, a BPE tokenizer that I've written from the ground up in C++20. It covers what the various components are, how each one works, what specific techniques give you a performance gain and why. All of this is based on the actual code (include/tokenizer/, src/, utils/) and the benchmark figures are from a properly audited 3 way comparison against OpenAI's tiktoken (Rust) and Hugging Face tokenizers. Headline numbers (Shakespeare corpus, 1.06 MB, GPT-2 vocabulary, 12 hardware threads, Intel i5-11260H): Operation TokenBpe tiktoken HF tokenizers Encode, 12T (zero-copy) 239.98 MB/s 45.83 MB/s 12.08 MB/s Encode, 1T 34.22 MB/s 11.55 MB/s 2.11 MB/s Decode, 1T (zero-copy) 341.01 MB/s 117.34 MB/s 24.93 MB/s ~5.2× tiktoken and ~19.9× HF on 12-thread encoding. And ~6.1× scaling from 1 to 12 threads natively. Let's open the hood. What a BPE tokenizer actually does BPE encoding has two phases: Pre-tokenisation: split the raw text into word-like pieces ("pre-tokens"). GPT-2's approach is to split on letter runs, digit runs, whitespace runs, contractions (i.e. 's, 're, 'll etc), punctuation runs and to fold the leading space into the next token (so " hello" is one pre-token rather than " " + "hello"). Merge based encoding: each pre-token is a sequence of single bytes. You repeatedly find the adjacent pair with the lowest merge rank and replace it with the merged token id, until no more merges can be applied. Decoding is trivial really: just map each token id to its bytes and concatenate. The naive implementation of this is fine. The real question is where the time goes and the answer, as ever, is everywhere, a little bit at a time. So TokenBpe attacks every layer: text ─▶ SWAR pre-tokenizer ─▶ dual-cursor split ─▶ per-thread cache ─▶ linked-list merge ─▶ token ids
with a persistent thread pool under the whole thing, a flat binary model format and a zero copy NumPy boundary on top. Component tour 1. The pre-tokenizer: SWAR instead of regex The problem is GPT-2's official tokenizer uses a regular expression to split the text. Regex engines are general purpose monsters backtracking, capture groups, Unicode classes all of which are completely unnecessary when you're just trying to determine if a run of bytes is letters or not. The technique. SWAR (SIMD Within A Register) does the 8 bytes per 64 bit word classification using pure integer arithmetic. No SSE/AVX intrinsics at all, it's all done in the 64 bit ALU. // swar.hpp (simplified)
// All 8 lanes are bytes; HIGH_BITS is OR'd in before subtracting
// min so no lane can borrow from its neighbour.
bool all_bytes_in_range(uint64_t v, uint8_t min, uint8_t max);
bool all_ascii_lower(uint64_t v); // [a-z]
bool all_ascii_upper(uint64_t v); // [A-Z]
bool all_ascii_letters(uint64_t v); // both, via v | 0x20
bool all_ascii_digits(uint64_t v); // [0-9]
bool all_bytes_equal(uint64_t v, uint8_t b);
The range check is written very carefully indeed: it rejects the high bit bytes first, then ORs in HIGH_BITS prior to subtracting min, so a lane can never borrow from its neighbour. load64 does an unaligned 64-bit load via memcpy which ends up as a single instruction. The splitter (pre_tokenizer.hpp) then scans: letter runs and digit runs 8 bytes at a time, scalar tail; space runs 8 bytes at a time with all_bytes_equal; contractions's, 't, 'm, 'd, 're, 've, 'll; punctuation runs (all other non-alphanumeric/non-space/non-apostrophe characters are grouped) And crucially next_token() returns a std::string_view into the original buffer. No pre-token is ever materialised as a string. The caller owns the text lifetime, the splitter just points into it. The speed pa...
Powered by Readybot.io
External Link
Platform
You are about to visit:
https://example.comThis will open in a new tab. Make sure you trust this link.