Lessons from building a search engine in Rust
Infino is a retrieval engine written in Rust (edition 2024, Apache-2.0). It runs inside your process, not as a server you reach over the network. That decision shapes most of what follows, starting with our most expensive mistake.
Lesson 1: tokio drives I/O, rayon drives CPU. Do not mix the pools.
Infino’s public API is synchronous: you call a function, it returns an answer. But storage I/O is async. The data lives on object storage, and we read it with tokio (Rust’s async runtime for I/O). So a sync function drives async work underneath, through a sync-to-async bridge.
The mistake was inside it. We ran query CPU work (decoding) on the same tokio runtime that drove the I/O. tokio’s scheduler only yields at await points, so a CPU-bound task holds its worker thread until it returns, stalling every other task on that thread. Decode sat on the threads meant to issue object-storage reads and starved them. Cold vector search, once about 1.1 s, blew out to 3.7 s to 11 s: a 3x to 10x regression from one wrong pool assignment.
The rule: tokio drives I/O, rayon (a data-parallel thread pool) drives CPU. When an async task needs CPU work, it hands the work to rayon and awaits a oneshot channel (a single-use async channel) for the result. The I/O threads stay free, and no thread is ever both an async-runtime driver and a rayon worker. Some teams instead run a second dedicated tokio runtime for CPU work. We did not need it: our compute bursts are short and we never cancel one mid-flight, the case that argument solves for. spawn_blocking is not the fix either: it hands out one OS thread per task, not the fixed-size, work-stealing pool that chunked parallel decode wants.
The bridge handles three cases, since we cannot assume anything about the caller’s runtime:
- Ambient multi-thread runtime: use block_in_place, which tells the scheduler "I am about to block" so it can move other tasks off this thread.
- No ambient runtime: build a small current-thread runtime on the fly.
- Ambient current-thread runtime: block_in_place is illegal here, so spawn a dedicated worker thread and run the work there.
A second, separate mistake lived in the no-ambient-runtime branch, about runtime flavor rather than the pool split. An earlier version routed every sync caller through one shared multi-thread runtime pinned to a single worker, on the theory that reuse beats building one per call. Reuse does win, but a multi-thread runtime’s block_on pays per-poll coordination with its worker thread; a current-thread runtime polls inline with no handoff. That cost measured at 6 to 17 percent on multi-term full-text search at 10M documents, worse the more a query fanned out. Single-term queries were flat. Building a current-thread runtime for that case fixed it.
Lesson 2: no unwrap in production
A stray unwrap is a panic waiting for the wrong input. We deny clippy’s unwrap_used, so the build fails instead of a query. For a fallible result, propagate with the question-mark operator. On genuinely infallible paths, use expect with an "invariant:" message that documents why. Test and benchmark code may use expect freely; production code may not lean on it as a shortcut.
Lesson 3: small stable public error, rich internal errors
The public API returns one coarse, non-exhaustive error enum of about seven variants: not found, already exists, schema, cardinality, io, query, and backend. Internally, each stage has its own fine-grained error type that converts inward into the public enum. The public contract stays stable while the internals churn. We can reshape an internal error without breaking a caller, and because the public enum is non-exhaustive, adding a variant later is not breaking either.
Lesson 4: allocator choice matters at scale
Indexing millions of documents means millions of small allocations, and the default allocator becomes the bottleneck. Our global allocator is mimalloc. One case must opt out: an embedding loaded into a host that already installs its own global allocator, notably our Python extension module. Two global allocators dlopened into one live process segfault, so mimalloc is default-on for the standalone library and off for that build.
Above the global allocator, hot paths that churn short-lived allocations use bumpalo arenas (bump allocation: hand out bytes by advancing a pointer, then free everything at once). Per-query scratch is one. The full-text index builder interns every term into a per-column bump arena, one pointer advance and a memcpy per term, so the bytes sit densely in one contiguous block that byte comparisons keep hitting in cache.
Lesson 5: portable SIMD, with a parity test as the safety net
SIMD (single instruction, multiple data: one instruction over a whole vector of numbers) is how vector search stays fast. We dispatch at runtime by what the CPU supports: AVX-512 first, then AVX2 with FMA (fused multiply-add), then a portable path on the wide crate’s f32x8 (eight f32 lanes) for everything else. The AVX-512 gate checks several feature bits, not just the base one, because the kernels use byte and doubleword variants a bare avx512f flag does not guarantee. An environment variable forces the lower tier, so we can A/B tiers on one host.
Every tier must produce byte-identical output, down to the last quantized byte, and that is not automatic. The scalar tail uses a fused multiply-add, so it rounds exactly once. A plain multiply-then-add would double-round and drift by up to one unit in the last place, enough to flip a quantization boundary and change the stored byte. A parity test checks every tier against a scalar reference across a sweep of dimensions chosen to hit each tier’s tail shape, so the fast path can never quietly disagree with the portable path.
The surprise: for the vector rotation step, a 16-lane AVX-512 butterfly measured about 8 percent slower than the 8-lane AVX2 one, so that step stays on AVX2. The butterfly is memory-bound, two loads and two stores per step with almost no arithmetic between. The wider lane buys nothing, and the clock AVX-512 gives up to stay in its power budget is a net loss. Measure; do not assume the widest instruction set wins.
Lesson 6: lock-free reader/writer coordination
Readers pin a snapshot with ArcSwap over the manifest (the small file listing which parts make up the current index). A reader loads the snapshot once when it opens and holds it, so a writer committing a new snapshot never disturbs a query in flight, and queries see a consistent view without a lock.
For caching manifest parts we use a DashMap (a concurrent hash map) from PartId to a OnceCell (a cell that initializes exactly once). The OnceCell is the coalescing point. When many queries touch the same cold part at once, they all await one initialization and share the single fetch, instead of each issuing its own object-storage GET. The hot path, a cache hit on a loaded part, touches only one DashMap shard, so it does not contend with lookups on other parts.
Lesson 7: testing unsafe code needs both miri and asan
Unsafe code is where you promise the compiler an invariant it cannot check. Our unsafe surface is small: one lifetime-extending transmute in the full-text index builder (the arena term interning above), the byte parsing in the Parquet footer splice, and the hand-written SIMD kernels. We test it with two tools that catch different things.
miri interprets the program against Rust’s abstract machine and catches language-level undefined behavior that real hardware happens to tolerate: aliasing violations under Stacked and Tree Borrows, pointer provenance bugs, reads of uninitialized memory, misaligned access (undefined on ARM even where x86 shrugs), and data races. asan (AddressSanitizer) instruments the real binary and catches hardware memory errors miri cannot simulate, including bugs across the FFI boundary into mimalloc’s C code: use-after-free, buffer overflow, use-after-return. miri runs about 100 to 1000 times slower than native, asan about 2 to 3 times. Neither is a superset of the other, so we run both.
The stack, briefly
Rust edition 2024, tokio and rayon for concurrency, Arrow and Parquet 58 for columnar data, DataFusion 53 for query, object_store 0.13 for storage, the fst crate (finite-state transducers, a compact way to store and search a sorted key set) for terms, roaring bitmaps for postings, and wide for portable SIMD. It is Apache-2.0, and the source is on GitHub. The tokio/rayon regression is one to take from our logs rather than reproduce in your own.