Why we added a search index to Parquet instead of designing a new format
The strongest argument against what we built cuts against us, so we will make it first. Columnar formats are poorly suited to search. Lance’s own paper argues that columnar layouts are bad at the random access and point lookups retrieval needs, and the DataFusion maintainers say native full-text search is out of scope for Parquet itself. If both are true, the obvious move is to stop fighting the format and design a search-first one. That is what Lance did.
We went the other way. Instead of replacing Parquet, we added an index layer inside it. The file stays a valid Parquet file, and the full-text and vector indexes ride along as embedded blobs plus inf.* footer metadata that standard readers ignore.
Three options, and the trade each one makes
There are three options. A separate search cluster or sidecar index next to your data. A search-first file format. Or an index layer embedded inside the Parquet file you already have.
Option A keeps a second, searchable copy in sync. Option B gives you one copy but asks everyone to adopt a format only you can read. We picked C, for three reasons.
Reason one: you inherit the whole ecosystem
Every Parquet reader reads the columns with no infino code and no export step: DuckDB, DataFusion, pyarrow, Spark, Polars, pandas. They open the file today because it is still a Parquet file, and the index bytes are just metadata they skip. You get a mature format and its entire toolchain for free.
A new format inverts that. Now you write and maintain readers for all of those tools and talk everyone else into adopting yours. That is a permanent tax, and your users pay it as much as we do.
The compatibility is one-way: any Parquet tool reads a superfile, but only infino writes one, since rewriting through an ordinary Parquet writer drops the index regions. That is the right side of the trade, because reads are what interop is usually for.
Reason two: one copy, one consistency model
Scalar columns, full-text, and vectors share one storage unit. No second system to sync, no duplicate data. That leaves one consistency model: immutable files plus atomic manifest commits. With a sidecar, the search copy can silently drift from the source of truth, and someone has to reconcile it.
Reason three: it is storage-agnostic
The same bytes work over local disk, S3, and Azure. Search depends on no proprietary cloud feature. Where the file lives is your decision, not the format’s.
The concession: point lookups are exactly Parquet’s weak spot
Plain Parquet is weak at the one thing search needs most: point lookups and random access. That is the whole reason we had to add indexes at all. If Parquet were already good at fetching individual rows, none of this index layer would exist.
A point lookup in Parquet decodes a whole page to reach one row. Most of our columns use 1 MiB pages, fine for scans but wasteful for a single row: one page touches about 65,000 rows to return one. So the id column, the one point lookups hit, uses 8 KiB pages instead, roughly 512 rows a page, with compression still on.
Shrinking every column’s pages rather than just the id column was the wrong move. On 320,000-row segments at k=10 it left full-row reads flat but made the id-plus-score path about 8x slower, because per-hit resolve cost scales with the number of pages the reader plans over, not how much each page decodes. Small pages pay off only on the column point lookups actually hit.
That is a workaround for a real limitation, not a feature. A non-Parquet, search-first format sat on our "decided against" list, rejected because a bespoke format’s storage savings did not outweigh losing the Parquet ecosystem. The bet was that we could buy back the point-lookup weakness with an index layer while keeping full read compatibility. We still think it was the better deal.
The risk we are taking, named plainly
Embedding index bytes next to the Parquet footer is a sharp edge. Today’s readers ignore the inf.* metadata and skip the index regions, which is what lets the file stay universally readable. But that behavior is convention, not a guarantee. A stricter future Parquet reader could object to bytes it did not put there. We are betting the compatibility win is worth that risk. The byte layout is documented in the infino repo.
Pick the format that matches the compatibility you need
Parquet is the right base for search over data that should stay usable by ordinary OLAP tools, because the file never stops being a normal Parquet file. It is the wrong choice for a transactional or graph workload, where the access patterns and the format do not line up. The question is not whether Parquet is good at search. It is what this data has to stay compatible with. For us, that was every reader our users already run, and that settled it.