Querying a billion vectors from object storage, on one box
An earlier post walked one engine from a sub-millisecond query at a million vectors in RAM out to a few milliseconds off object storage for the vast cold set, and ended on a promise: a deeper billion-scale story off object storage. This is that story. We built a one-billion-vector index, kept it in object storage the whole way, and queried it from a single 16 vCPU / 64 GB machine. Here is what the latency and recall actually look like, and where the cost still sits.
The shape of the test
The dataset is deep1b: one billion 96-dimensional vectors, cosine distance, with a published query set and ground truth. We care about this scale for a plain reason. A billion vectors in a conventional in-memory graph is a fleet of large machines, most of them holding data that is almost never queried. The question is whether one engine can serve that corpus from object storage, on a single modest box, at a latency anyone would accept.
The index lives in object storage as Parquet superfiles. Nothing about it is resident by default. A query pulls the small set of byte ranges it needs into a local block cache, and once a neighborhood is warm, repeat queries into it serve from cache. The machine is one 16 vCPU / 64 GB box. The full index does not fit in that RAM and is not meant to. The working set does.
Latency and recall
Once a query's working set is cached, a single top-10 search over the full billion settles into tens of milliseconds. The first touch into a cold neighborhood is seconds, the object-storage fetch that fills the cache, after which it stays warm. Where the warm number lands depends on how much recall the engine is calibrated to reach, which it stamps per table against a recall target rather than leaving as a knob a caller turns. Two useful points on that curve, against the published ground truth:
| recall@10 | latency (median) |
|---|---|
| 0.90 | ~37 ms |
| 0.95 | ~74 ms |
deep1b, one billion vectors, 96-dim cosine, top-10, served from object storage on a single 16 vCPU / 64 GB box, against the dataset's published ground truth.
At the higher-recall point, most individual queries return all ten of their true neighbors, so the per-query median recall is a perfect 1.0. The average of 0.95 is held down by a small tail of hard queries whose nearest neighbors sit deep in the grid, and widening the probe past this point buys little recall for a lot more latency.
Put that next to the one benchmark constant everyone shares at this size: storage. Keeping a billion vectors and their graph resident in RAM is a large fleet and a large bill. Serving the same corpus from object storage trades a small amount of latency for storage that is roughly 100 times cheaper per gigabyte and effectively unbounded. Tens of milliseconds for a query is the latency side of that trade, and it stays usable.
How the engine gets there
Three things carry it. First, the object-storage index: the vectors are stored as compact codes in immutable Parquet superfiles, grouped into clusters, with a small routing structure that picks the handful of clusters worth reading. The cost of object storage is round trips, so the routing structure names the byte ranges a query needs before any data arrives, which lets them be issued together, coalesced, and cached rather than fetched one dependent request at a time.
Second, placement. The router ranks clusters by their centroids, and the biggest lever on a query at this scale is how many clusters it has to open to find its neighbors. Writing each vector into the cluster its own nearest routing centroid points at keeps that number small: a query's true neighbors land in the clusters it reads first, rather than scattered into clusters the router only reaches with a wider, slower probe. Aligning where data is written with where the router looks is what holds both the latency and the round-trip count down as the grid coarsens relative to the corpus.
Third, the cache, which has two tiers. The durable copy always lives in object storage. In front of it the box keeps the byte ranges queries have actually touched in a local block cache on disk, sized well past RAM, with the hottest of those pages resident in memory on top. A query fills this cache on first touch and serves from it after: from memory when the neighborhood is hot, from local disk when it is merely warm, and back to object storage only for ranges it has not seen. The capacity limit stops being RAM, and it stops being the index size. The cache is sized to the working set, which is what queries actually reach, not the whole corpus.
Built to keep going
A billion fits on one box because nothing about a query scales with the corpus. A search reads the router and the few clusters it probes, and that is all it pulls from object storage. The bytes a query moves track what the query needs, which grows far more slowly than the index itself, rather than with the total size. That is the property we built the engine around, and it is why the design targets corpora well past a billion, into the tens and hundreds of billions, where the index is far too large to hold in any memory but a query still only ever touches its own small slice of it. We have measured a billion here. The architecture is built for the orders of magnitude above it, where the open question is capacity and calibration rather than a different system.
Where the work still is
Serving a billion vectors is one half of the problem. Building and maintaining the index is the other, and it is where the real cost sits today. As the corpus grows, a background optimize periodically recompacts the index to keep it efficient, and at this scale that pass currently takes on the order of 6000 seconds. It runs in the background against a fully queryable index, so it is maintenance cost, not search downtime, and it is the number we are working to bring down.
Reproducing it
- Workload: deep1b, one billion vectors, 96-dim, cosine, top-10, against the published query set and ground truth.
- Storage: the index in object storage as Parquet superfiles, read through a local block cache.
- Hardware: a single 16 vCPU / 64 GB box.
- Latency: the per-query floor once the working set is cached. A single-query measurement, not throughput under concurrent load.
- Recall: recall@10 against the dataset's ground truth, averaged over the query sample.
Next in this series: the build path, and bringing the optimize cost down.
One engine, a million vectors in RAM to a billion off object storage, same API.
Free tier · No credit card to start
or run it yourself: pip install infino · Apache-2.0 · quickstart →
Deeper: the self-transforming engine · vector search on object storage · what a billion costs.