How a full-text and vector index fits inside a valid Parquet file
Infino runs BM25 full-text search, vector search, and SQL against a single file on object storage that is a valid Apache Parquet file. The same bytes we search, pyarrow and DuckDB open as a plain table with no Infino code. We call it a superfile.
Two terms first. Parquet is a columnar file format: it stores data in blocks called row groups, with a footer that describes where everything lives. BM25 is the standard keyword-search ranking function, scoring a document by how often the query terms appear, adjusted for how common those terms are.
The layout, end to end
A superfile is a normal Parquet file with up to two extra blobs before the footer. Each blob is present only if that index was built. Front to back, the bytes are:
PAR1magic, the four bytes that begin every Parquet file.- The Parquet row groups, written by the normal Arrow writer and left untouched.
- The full-text blob, starting with the 8-byte magic
INFFTS01: an inverted index of postings, the list per term of the documents that contain it. - The vector blob, starting with the 8-byte magic
INFVEC01: IVF clusters plus 1-bit quantized codes. IVF, inverted file index, groups vectors so a search scans only the clusters nearest the query, not every vector. - A standard Parquet footer, rewritten with
inf.*offsets that point at the blobs. - The trailing
PAR1magic that ends every Parquet file.
The blobs go in the gap after the row groups, and the footer stays last where Parquet requires it.
How we build it
The normal Arrow ArrowWriter writes a complete Parquet file. We decode its footer, truncate the buffer to the end of the last row group, and append the full-text blob and then the vector blob. Then we re-emit a standard footer. It carries the original row groups and column and offset indexes forward unchanged, plus key/value metadata under an inf.* namespace.
A full-text index adds inf.fts.offset, inf.fts.length, and inf.fts.columns; a vector index adds inf.vec.offset, inf.vec.length, and inf.vec.columns. Five identifying keys are always written: inf.format (the constant infino-superfile), inf.format_version, inf.id_column, inf.n_docs, and inf.builder. The columns keys hold a small JSON blob per indexed column; everything else is a plain string.
Why afterward? The offsets exist only once the row groups are encoded, so you cannot set them in the writer properties up front. Parquet’s footer metadata is also immutable in the writer library we use, so we build a fresh footer around the original row groups rather than editing one in place.
Why it stays valid Parquet
Three things keep it valid:
- Row groups are addressed by absolute byte offset in the footer. Appending the blobs before the footer moves no row group, so every offset stays correct.
- The footer is an ordinary Thrift Parquet footer (Thrift is the binary serialization format Parquet uses for it), and the file still begins and ends with PAR1.
- Parquet readers ignore key/value metadata they do not recognize, so the
inf.*entries are invisible to everyone but us.
The blobs live in the gap between the last row group and the footer, which a standard reader never addresses.
The proof you can run
Two examples let you check it. A Rust example, examples/demo.rs, builds a one-document superfile with a text and a vector index, then reads the bytes back with vanilla DataFusion. A Python script, infino-python/examples/parquet_interop.py, writes a small corpus and reads it back with pyarrow and DuckDB, where it behaves as a table while the indexes ride along unseen.
A text column is stored twice: the raw text stays a normal Parquet column you can SELECT, and its terms also go into the full-text blob. A vector column differs: its floating-point values go into the vector blob and are never written as a Parquet column, so a standard reader sees the scalar columns and id, not the embeddings. The code lives in the Infino repository.
Compatibility is one-directional
Reading is free; rewriting is where it breaks. Load the file in an ordinary Parquet writer and write it back out, say through pyarrow to a new file, and you get valid Parquet with the indexes silently dropped. The generic writer knows only columns and a footer; it never saw the blobs, so it cannot carry them forward. The output is still valid Parquet, but no longer a superfile.
Reading through any tool is safe; avoid round-tripping the file through a writer that is not ours.
Inside the two blobs
The full-text blob opens with a fixed 48-byte header, then an FST term dictionary, a postings region, and a document-length table, in that order. The header records where each region begins. An FST, finite state transducer, is a compact automaton that maps each term to its location, holding a large vocabulary in little space. The postings region stores delta-encoded document ids and term frequencies in blocks; each block carries a block-max BM25 value so a query can skip whole blocks that cannot beat the current top results.
Each region ends with a 4-byte CRC-32C checksum (the Castagnoli polynomial, the same one Parquet uses for its page checksums), so a corrupt read fails loudly on the exact region. The default eager open path verifies the checksums before any query runs. The object-storage path skips the scan by default, since reading every byte to check a CRC would defeat fetching only the ranges a query touches; a deployment can turn the check back on.
The vector blob stores, per indexed column, a header, the cluster centroids, a cluster index, the 1-bit codes, and an optional rerank tier: full-precision floats, a compact quantized code with a residual correction (the default), or none, in which case the 1-bit codes are the final ranking. Those codes come from RaBitQ, which turns each rotated vector into one bit per dimension. Codes and their document ids are laid out cluster by cluster, so scanning a cluster is one contiguous read. That matters on object storage, where one sequential range request is far cheaper than scattered reads.
How a superfile ties to a table
One superfile is immutable. A table is a manifest snapshot pointing at many superfiles, and that manifest doubles as a data-skipping index, holding per file the min/max values, term Bloom filters, and centroids. A Bloom filter is a compact probabilistic set that cheaply answers "this term is definitely not in this file", which is what pruning needs.
A query prunes in two tiers. The manifest drops whole superfiles that cannot match, before any file bytes are fetched. The survivors' bytes then go to DataFusion’s Parquet reader, which does its own row-group and page pruning. Scalar filters and keyword terms prune through this manifest layer today, and the per-file centroids sit in the same structure for vector pruning. All three signals resolve against one layer, not three systems with three score scales to reconcile.
Prior art, and where we go further
The DataFusion project has written about embedding indexes in Parquet and names full-text search as a use case, but builds only a trivial example. Infino ships the full version: full-text plus vector, with the blob internals, checksums, and two-tier pruning above.
One risk
A stricter future Parquet reader could, in principle, object to bytes living between the last row group and the footer. Today’s readers walk past that region; a future one might not. The payoff for that risk: the copy of your data you already keep is the searchable one, with no second system to feed.