How we built the engine: files, manifests, and stateless compute
Most search stacks are an always-on server: the index resident in RAM, a second copy of your data, and a bill that runs whether or not anyone queries. The Infino engine is the opposite. No resident index, no second copy: it reads files on object storage, fetches only the byte ranges a query touches, and scales to zero when idle.
Three ideas
- Data is files. Each chunk is one self-contained Parquet file with its search indexes embedded inside. No sidecar index to keep in sync.
- Files are never edited. We only add new files and publish manifests atomically. Nothing written is mutated in place.
- Storage and compute are separate. Object storage, such as Amazon S3, is the durable source of truth. Stateless compute fetches byte ranges on demand, caches the hot set locally, and scales to zero when idle.
If you have not met Parquet: it is a columnar file format, values from the same column stored together, ending with a footer that describes where each column lives. That footer matters later.
Three layers
Three stacked layers, each with a narrow job.
The connection is the catalog: it knows what exists but does not search. A supertable is where work happens: a manifest snapshot over a set of immutable files, plus the stateless compute and local cache that serve queries against it. Underneath, object storage is the durable source of truth that never moves; reads are byte-range GETs, specific offset ranges rather than the whole file.
The superfile
The unit of data is the superfile: one immutable file carrying everything a query needs about its rows. Inside: its columns as standard Parquet row groups, a full-text blob (an inverted index, the term-to-documents map keyword search walks), a vector blob (IVF clusters plus quantized codes, below), and a rewritten Parquet footer.
We write the columnar body with the ordinary Arrow writer, append the two blobs after the last row group, then re-emit a standard Parquet footer whose metadata records the blobs' absolute offsets and lengths under a reserved inf.* namespace. Row groups use absolute offsets, so appending bytes before the footer does not move them, and a Parquet reader ignores metadata keys it does not recognize. So pyarrow, DuckDB, or DataFusion open the file as a plain columnar table, while the engine uses the same footer to find its embedded indexes. No export step; compatibility is a property of the bytes.
The superfile opens lazily: the engine reads the footer and small headers, learns the layout, then fetches only the byte ranges a query touches. A query needing one column and a few postings never moves the rest.
IVF, an inverted file index, partitions vectors into clusters with k-means, so a search probes a few nearby clusters instead of every vector. Within a cluster, each vector is first reduced to a 1-bit code: a deterministic random rotation, then one sign bit per dimension (using RaBitQ, whose rotation makes those sign bits an unbiased estimator of the true inner product). The 1-bit codes drive a fast first-pass scan small enough to stay in cache.
A second, per-column rerank tier then orders the survivors precisely. Its default is Sq8Residual: an 8-bit quantized copy of the vector plus a signed 8-bit residual correction, two bytes per dimension. The tier is configurable, from full-precision floats down to the 1-bit codes alone, which shrinks the superfile by roughly 30x at a million vectors of dimension 384, at the cost of recall.
The supertable: manifest, reader, writer
The manifest is an immutable list of superfile entries. Each records the file location, its row range, per-column min and max values, a full-text Bloom filter and term range, and vector centroids (the representative points of the clusters). A Bloom filter is a compact structure that can say a term is definitely absent from a file, exactly what you want for skipping files cheaply. At scale the manifest is hierarchical and content-addressed, so it stays small to read and its parts are cached by content hash.
The reader pins a manifest version at the start of a query with an atomic pointer swap (ArcSwap). Pinning gives snapshot isolation without locks: the query sees a consistent set of files for its whole life, and concurrent writers cannot disturb it, since they publish new versions rather than edit the pinned one.
The writer is covered next.
Query dispatch, in three phases
Prune, search, merge.
Prune. The manifest drops every superfile that cannot match, before touching object storage. Per-column min and max rule out files outside a range; the term Bloom filter rules out files that cannot contain a keyword; centroids rule out files whose vectors are all too far away. Whatever survives is all we read.
Search. Each surviving superfile is searched in parallel across cores (using rayon, a Rust data-parallelism library). For full-text, we resolve query terms through the FST (a finite-state transducer: a compact ordered dictionary mapping each term to its postings-list location) and walk the postings. Postings sit in fixed-size blocks, each recording the largest BM25 contribution any of its documents can make, so the walk skips whole blocks that cannot enter the current top-k. The same postings answer unranked matching: intersect for all-terms, union for any-term, no scoring. For vectors, we score against the cluster centroids, probe the nearest clusters, scan their 1-bit codes into a shortlist, then rerank by true distance.
Merge. The per-file results combine into a single global top-k.
The design point underneath: the search operators are DataFusion table-valued functions. DataFusion is a Rust query engine; a table-valued function returns rows, so it can appear anywhere a table can in a query plan. A ranked candidate set is the first stage of a query, not the whole answer. You can JOIN it, GROUP BY it, or filter it further, and negation becomes set algebra: one candidate set EXCEPT another. Search is a relation you compose with, not a black box that emits a list.
The write and commit path
Writes never touch existing files. A writer stages new superfiles, then writes the new manifest list and its content-addressed parts to storage in parallel. Each part is named by the blake3 hash of its own bytes, so its location is known before any request goes out and all the writes issue at once; two writers producing byte-identical parts target the same object. None of that is visible yet.
Making it visible is a single atomic conditional write on one pointer file: a put-if-match against the prior version’s etag (the first commit uses create-if-absent). That one swap publishes the new data. If two writers race, one wins; the other loses the compare-and-swap, refreshes from the now-current manifest, rebuilds on top, and retries with backoff. The swap also makes commit crash-safe: anything written before it is simply unreferenced. A writer that dies mid-staging leaves orphaned bytes no reader ever saw, and the pointer still names the old, complete version.
Deletes follow the same rule; nothing is erased in place. A delete is a tombstone: a roaring bitmap (a compressed set of row identifiers) layered over the immutable files to mark rows gone. Reads stay snapshot-isolated against whatever manifest they opened, so a delete published after a query starts does not disturb it.
A deeper cut on vectors
Vector search hides a layout problem, and part of our answer is still being built. Filtered and unfiltered queries want opposite physical arrangements. A filtered query, where scalar, time, tenant, or text predicates narrow the rows first, wants table locality so those predicates prune before any vector math. A pure nearest-neighbor search across everything wants global vector locality, so similar vectors sit near each other regardless of file. Optimize for one and you pay on the other.
What ships today is the first half: user superfiles carry vectors organized for local search inside each file, which is what filtered and hybrid queries want, since a predicate prunes superfiles first and vector search runs only inside the survivors. Because superfiles are ordered roughly by write time, local recall is good, but there is no continuity of distance across files, so a pure unfiltered search across the corpus touches more files than it should.
The direction we are building toward keeps both layouts. Alongside the user table, a hidden vector-only supertable would organize the same vectors globally by distance for unfiltered approximate nearest-neighbor (ANN) search. A second copy is affordable only because the rerank encoding above already compresses the vectors hard, so the duplicate stays small. New writes would land as a bounded set of fresh cells, an SPFresh-style staging layer searchable before any global re-clustering, and a background compaction job would fold them into the main layout later, keeping writes fast without degrading the index as data accumulates. We call the approach Residual Shadow Indexing; the shadow table is the part still in progress. Today, filtered and hybrid vector search are the strong paths.
The stack, and what it costs
The engine is written in Rust (edition 2024) on Arrow and Parquet 58, DataFusion 53, object_store 0.13, and roaring bitmaps. It is Apache-2.0 licensed. Storage is pluggable across local disk, S3, and Azure behind a single trait, so the source-of-truth layer is a swap, not a rewrite. The code is on GitHub.
The numbers we can share come from a run at 10 million documents over real S3, on an 8-core machine, with the table spread across 256 superfiles in 16 commits. Full-text ingest ran at about 33,100 docs per second and vector ingest at about 62,200 docs per second at dimension 384. Against a warm cache, a keyword, vector, or hybrid search returned its top-k in single-digit to low-double-digit milliseconds.
Not every axis is a win. The first query against an un-cached superfile pays object-storage round trips and lands in the hundreds of milliseconds to low seconds, not the warm figures above; the design optimizes for the warm path. On bulk ingest throughput we currently trail the specialized incumbents. The strength is warm query speed-per-dollar.
None of this makes an always-on server obsolete for every workload. Reading a file on object storage is not free, and a resident in-RAM index will win a warm point lookup. What you get in exchange is the architecture: one copy of your data that is also the index, no cluster to keep alive between queries, and scale-to-zero when no one is asking. That is the trade: data is files, files are never edited, and storage and compute stay separate.