The one idea behind our speed on Parquet
How can search be fast when the data lives as Apache Parquet files on object storage (S3 and its equivalents), with no always-on cluster in front of it? One idea does the work: make the single Parquet copy the searchable copy.
We embed the full-text index and the vector index inside a valid Parquet file. We call the result a superfile: it still opens as ordinary Parquet, and it carries the indexes search needs. Across many files, one thin manifest layer handles pruning. Search and point lookups run against the file already on S3, so there is no reindex step, no second copy, and no separate cluster to operate.
The usual build is three systems; ours is one copy
The classic way to get full-text search, vector search, and structured filters is three systems: a search cluster for fuzzy text, a vector store for embeddings, and a warehouse for structured columns. Each holds a second, searchable copy of your data, plus glue code to keep the copies in sync and merge their results.
That second copy is the expensive part. We make the copy you already have the searchable one. Retention stops being a pricing tier: it is however much S3 you keep, and nothing can drift out of sync.
One pruning layer across three query types
For every superfile the manifest carries a small summary: per-column min and max values, a term-presence Bloom filter (which answers "definitely absent" or "possibly present") tagged with the file’s lexicographic range of terms, and the vector cluster centroids. A query reads this summary to pick which files are worth touching. The summary stays resident, so this pass fetches no file bytes at all.
That one layer serves all three query shapes. A WHERE predicate prunes on min and max, a keyword prunes on the Bloom filter and its term range, and a vector query ranks each file’s centroids to probe only the closest. The Bloom occasionally lets an irrelevant file through: the default is a 64 KiB block Bloom per column, tuned to about a 7 percent false-positive rate at roughly 100 thousand distinct terms.
The scalar and keyword prunes are lossless: they only drop a file that provably cannot match, so a false keep costs a little time and never changes a result. The vector centroid step is different. It picks the closest clusters rather than proving the rest empty, so it inherits the usual approximate-nearest-neighbor recall trade, which is why our vector recall sits below 1.0.
An object-storage-native vector index
On object storage the dominant cost is request count and latency, not the RAM bandwidth that governs an in-memory index. That settles which vector index wins.
HNSW, the popular in-memory graph index, chases pointers: each hop depends on where the last one landed, so you cannot prefetch, and on object storage you pay the round-trip latency again on every hop. HNSW is genuinely faster when the whole index fits in memory, which is not the case here.
IVF (inverted file) groups vectors into clusters instead, so a probed cluster is one contiguous range you fetch in a single request. Our coarse scan uses 1-bit RaBitQ codes (one bit per dimension), so at 384 dimensions it reads about 32x fewer bytes than the full-precision vectors; those are fetched only for the small rerank set at the end. RaBitQ carries a theoretical error bound, so we size that rerank set from a calculation rather than a guess. Several object-storage engines (turbopuffer, Databricks, VectorChord) chose IVF-style indexes over HNSW for the same reason.
Open in at most two GETs, then read only what you touch
A surviving file is still not loaded whole. A lazy byte-source opens the superfile in at most two GET requests: one suffix-range read fetches the Parquet footer and the object size together, saving the usual HEAD round-trip, and a second follows only when the footer starts earlier than that read reached. After that we fetch only the bytes a query needs: the posting lists for its terms, the clusters the vector search probes. Since the constraint is request count, not bandwidth, a narrow question never pulls the full file into memory.
Indexes are physical access paths inside SQL, not a bolt-on API
The indexes are wired into query planning, not a side API you call separately. An equality or IN filter on an indexed text column resolves through the inverted index (a term-to-rows map) to a candidate row set before any column is decoded. A plain Parquet reader handles this worst: on an unsorted column the per-file min and max span most values, so column statistics prune almost nothing, while the term index still names the few matching rows. On our 10-million-row benchmark that made WHERE key = ? on an unsorted column 30x faster: 78.27 ms for a plain scan against 2.58 ms with index pushdown. Aggregates over that one-row candidate set (COUNT, SUM, MAX, AVG) ran about 26x to 29x faster.
Infino owns which files and rows are relevant; DataFusion executes over what survives, doing its own row-group and page pruning inside each Parquet file. The pushed filter is reported as inexact, so DataFusion always re-applies the full predicate above the scan: the index narrows the candidate set, it never decides the answer.
The numbers, including the trade-offs
On 10 million documents over real AWS S3, warm queries land at about 6.8 ms for bm25_search (BM25 is the standard full-text relevance ranking), about 10.4 ms for vector_search, and about 13.0 ms for hybrid_search.
The trade-offs come from being a file on object storage, not an always-on server. In a separate, smaller test swapping Solr out of a working job-search application (about 81 thousand postings, not the 10 million above), Solr won on warm latency: BM25 3 ms against our 9 ms, vector 3 ms against our 8 ms. Solr holds its whole index in RAM, exactly the resident-memory case we give up.
On that corpus we came out ahead on dense recall@10 (0.923 versus 0.884) and index size (509 MB versus 650 MB, about 22 percent smaller, columnar data included). Two more losses stand. The first query against an un-cached file is cold and pays the S3 round trips: tens to hundreds of milliseconds to open, and hundreds of milliseconds to low seconds to search a large table. And we store no token positions, so there is no phrase index; on an exact-phrase or position-sensitive query, a position-aware engine has a real advantage over us.
The trade is deliberate: a few milliseconds warm, in exchange for no cluster to run, no second copy of your data, and no idle-time bill.
The core idea
Embedding the indexes in the single Parquet copy, with object-storage-native pruning and a vector index, is what makes fast search directly on Parquet on S3 possible at all, without standing up a second system. The code is open source at github.com/infino-ai/infino.