DEV Community
AI Embeddings Break Columnar Assumptions in Parquet and Arrow
TL;DR — Columnar engines like DuckDB and Arrow assume you scan a few columns and filter on values. An embedding column violates every one of those assumptions: it is wide, high-entropy, never filtered by value, and always read whole. Treat it as a separate physical tier, use fixed-size layouts, and late-materialize vectors after metadata filters. Columnar storage is a bet on a specific access pattern: read a few columns out of many, filter on their values, and aggregate. Parquet, Arrow, DuckDB, and GPU dataframe engines are all optimized around that bet. It's a good bet for analytics. Then we add an embedding column and quietly lose it. My thesis: an embedding column is the one column type where the columnar assumptions invert, and most "slow AI data prep" is the result of storing it like any other column. The fix isn't a faster engine. It's treating vectors as their own physical tier. The access pattern flips Look at what a normal analytical column gets for free: Value predicates. Min/max statistics, dictionary pages, and bloom filters let the engine skip data based on what's inside. Compression. Low-cardinality strings, sorted integers, and timestamps compress dramatically with dictionary, run-length, and delta encodings. Narrow reads. The query touches three columns out of fifty. Now look at an embedding column: No value predicates. Nobody writes a filter on the seventh dimension of a vector. Min/max stats on a float inside a vector are noise. You pay for the metadata and never use it. Poor compression. Learned floats are high-entropy. Dictionary and run-length encodings have nothing to grab. Byte-oriented transforms for floats help a bit, and general-purpose codecs buy you little for the CPU they burn. All-or-nothing reads. A consumer wants the whole vector or none of it. There is no useful "read half of this column." So the column that is usually the largest in your table, by bytes, gets none of the features that justify a columnar format. You're paying the format's costs (row-group overhead, encoding and decoding, nested-type bookkeeping) without collecting its benefits. Nested lists are the quiet tax The default way people land embeddings in a table is a variable-length list of floats. In Parquet, nested and repeated data is represented with repetition and definition levels, extra bookkeeping the reader must decode to reconstruct list boundaries. For a variable-length list, that work is legitimate. For a vector where every row has exactly the same dimension, it's pure waste: you're encoding "where does each list end" for data whose answer is a constant. Arrow has the right abstraction: a fixed-size list. Every row is the same width, so the whole column is one flat contiguous buffer of floats plus a known stride. That layout is what NumPy, a GPU kernel, or an ANN index wants. Row i lives at offset i * dim. No offsets array, no per-row decode. The practical rule: if the dimension is a schema invariant, encode it as one. Declare fixed-size at the Arrow layer, and verify what your Parquet writer and reader actually do with it end to end. Whether you get a zero-copy view or a silent conversion back through variable-length lists depends on the toolchain, and the failure mode is a slow pipeline, not an error. Check by measuring a round trip, not by trusting the type name. Split the vector from the metadata The bigger win is physical, not logical. In a typical embeddings table you have a handful of small, filterable columns (document id, source, timestamp, tenant, language) and one huge opaque blob of floats. Storing them in the same files means every metadata scan drags row groups sized for vectors. Row-group sizing is where this bites. A row group is the unit of skipping and the unit of reading. If the vector column dominates the bytes, a row group sized by byte target holds very few rows. Your metadata statistics then cover tiny spans, footers balloon, and a filter that should prune aggressively instead opens thousands o...
Powered by Readybot.io
External Link
Platform
You are about to visit:
https://example.comThis will open in a new tab. Make sure you trust this link.