Making S3 the source of truth: the tricks that are not in the docs
Most systems treat object storage as a backup target: live data on local disks, a copy on S3 (Amazon’s object store, where you read and write whole objects over HTTP rather than seeking within a file) for safekeeping. We built it the other way around. In Infino the files on S3 are the source of truth: the engine reads and writes them directly, with no second always-on copy in RAM.
That changes what you optimize for. On a local SSD the scarce resource is bandwidth. On object storage it is requests: a single GET costs a round trip whether it returns 32 bytes or 32 megabytes. So the whole design bends toward one thing, fewer round trips, and every trick below comes out of that.
Trick 1: open a file in at most two GETs
Our on-disk unit is a superfile: an Apache Parquet file carrying columnar data alongside a full-text index and a vector index. Parquet keeps its metadata, the footer that says where every column lives, at the end of the object. The obvious open is a HEAD for the object’s size, then a range read of the last few kilobytes for the footer. Two round trips before you read a single row.
One trip is enough. S3 honors a suffix range, bytes=-N, meaning the last N bytes, and reports the total object size in its Content-Range header. So one GET returns the file’s tail and its size, the two facts HEAD-then-read takes two trips to learn. We guess a 64 KiB window, which covers the footer in the common case. A second GET fires only when the metadata exceeds 64 KiB and the footer starts before the window.
Index sections have their own fallbacks. When the manifest carries only an offset-and-length hint, the vector header is 32 bytes and the full-text header is 48 bytes, so we fetch just those, then range-read the body.
The manifest can cut the count further. When it records exact open ranges, the cold open issues the footer read and the vector and full-text ranges as one parallel batch, so several sections cost a single round trip. And since we fetch the manifest to plan the query anyway, a small superfile can ride its whole open window inside that read, at zero further GETs. Collapsing footer discovery to one suffix GET saves roughly 25 to 50 ms per cold open in-region; the batched and piggybacked cases save more.
Trick 2: size the connection pool for fan-out
A single query can fan out to hundreds of superfiles, each with its own range GETs, and Trick 1 keeps those cheap only if the connection under each is already open. A fresh TCP connection adds a TLS handshake, an extra round trip on top of the request itself. At the p99 tail, the slowest one percent of requests, that doubling is what shows up.
So we keep a deep warm pool of sockets, with three deliberate constants:
- Max idle connections per host, 1024. This lets wide fan-out reuse already-handshaked sockets instead of opening new ones.
- Pool idle timeout, 10 s. S3 closes an idle connection server-side at roughly 20 s, so we retire ours first. Reusing a server-closed socket does not fail cleanly; it surfaces later as a send error on some unrelated query that drew that socket. The timeout is per-backend for this reason: the Azure client sits at 90 s, below Azure’s close window rather than S3's.
- Connect timeout, 5 s. This bounds a slow SYN or TLS handshake so one stalled connection cannot hang the query. A separate retry layer handles genuine drops, so this timeout can be short.
The settings are ordinary but easy to get wrong: a pool sized for a normal request rate is starved by a query opening hundreds of ranges at once, and the failure is silent, a p99 that doubles under fan-out as connections churn.
Trick 3: the whole commit is one compare-and-swap
Superfiles are immutable and append-only: once written, a file’s bytes never change. The manifest, the small index of which files make up the table, is split into content-addressed parts, each named by a hash of its own bytes. Identical content lands at the same name, and a part can never be silently rewritten.
Immutable pieces make the commit simple. A writer stages everything, all the new superfiles and manifest parts, in parallel, since none depend on each other. These objects sit unreferenced in the bucket, invisible. Then the writer does one conditional write against a single pointer file, _supertable/current: put-if-match against the previous version’s ETag (S3's per-object version tag), or create-if-absent on the first commit. That swap is the only step that makes anything visible.
Three properties come free:
- A process that dies mid-commit leaves only unreferenced objects. Nothing points at them, so the table cannot be corrupted; the orphans are garbage to collect later.
- A stale writer, one whose ETag no longer matches, loses the compare-and-swap and retries against the new version.
- Two writers producing identical bytes resolve to the same content-addressed object rather than colliding. The second create fails its precondition, but the bytes already there are bit-identical to what it would have written, so we treat the failure as success.
S3 conditional writes make this work. A swap like this once needed an external lock service or consistent metadata store to arbitrate; the pointer object plus put-if-match now plays that role, with nothing to run alongside the bucket. The atomic step is one request, the same accounting as everything else.
Trick 4: a cache with two separate budgets
Once fetched, a file is memory-mapped (mmap): its bytes map into the process’s address space so the code reads them as ordinary memory, and the OS faults pages in from disk on demand. A mapped file occupies two resources, tracked apart: how much is resident in RAM, and how much sits on local disk.
Separate budgets let us reclaim RAM without discarding the local copy. To evict a cold file from memory we call madvise with MADV_DONTNEED, which drops the file’s pages from RAM but leaves the file on local disk. Touch it again and its pages re-fault from local disk, not from S3, so we skip the re-download, the expensive part. A background thread sweeps files untouched past a threshold, and a separate budget-driven pass drops the oldest pages first when resident memory crosses its ceiling.
MADV_DONTNEED is only safe here because of what these files are. On a writable mapping it is a foot-gun: the pages are truly discarded, and a later read sees zeros. Our cache files are immutable and the mapping is read-only, so a dropped page re-faults from the backing file bit-for-bit. The invariant that makes the system simple, that a superfile’s bytes never change, also makes releasing its memory risk-free.
Trick 5: a running query survives eviction, for free
Caches usually coordinate with readers so they do not delete a file out from under an in-flight query. POSIX handles that for us: an mmap stays valid after the file is unlinked, until the last reference goes away. So the cache can evict during a query without asking what is running; the worst case is a later re-fetch, never a correctness bug.
Trick 6: retry transient failures, bound the prefetch fan-out
Object storage occasionally returns transient errors. Those get exponential backoff, each retry waiting longer than the last, so a brief hiccup does not fail the query. The object-store client retries the usual transient statuses itself. On top, we re-issue a small class it will not retry, chiefly the dropped-socket send failure from Trick 2, on a separate budget whose backoff starts at 50 ms and caps at 2 s. A fresh dial also discards the dead pooled connection, fixing the cause, not just the symptom. When warming a cold table, cold fills and promotions prefetch with a bounded fan-out, default concurrency 8, so a many-segment table fills without a thundering herd hitting S3.
One storage trait, three backends
Every trick sits behind a single storage trait, so the same code runs over local disk, S3, and Azure Blob Storage. Two places differ. First, the atomic commit: local disk has no put-if-match, so we emulate the conditional write with an advisory file lock (via the fs4 crate), closing the read-then-overwrite window that would otherwise let two writers clobber each other. Second, the two-GET open: Azure’s client rejects a suffix range, so its cold open falls back to a HEAD plus a bounded read, two round trips rather than one. That penalty only lands on a standalone cold open, since a table read already carries the object size in the manifest and skips size discovery. The engine is open source; the storage layer lives in the Infino repository.
The tradeoffs
This design optimizes warm latency and pays for it at the cold edge, where the first query against an un-cached file pays real object-storage round trips. On a 10-million-document table over real AWS S3, us-east-1, opening a single superfile cold measures around 164 ms, and a fresh table that fans out across a couple hundred superfiles pushes the cold open toward 556 ms. Warm point queries land in the single-digit-millisecond range. Cold, they vary by shape: a point lookup returns in a few hundred milliseconds, while a cold analytic aggregate that touches every segment ran as high as eight seconds. If your workload is all first-touch, this is the wrong shape of system.
Some limits are structural, by design:
- The table is append-only. The atomic manifest commit is the durability boundary; updates and deletes are tombstones, roaring bitmaps (a compressed set of row ids) layered over the immutable files rather than in-place edits.
- Infino runs in your process today. There is no wire protocol and no SQL endpoint yet; you link it in as a library.
No trick here is clever alone. They work together because they share one premise: the constraint is round trips, not bytes, and the copy on S3 is the source of truth, not a backup of it. Accept that, and opening in two GETs, committing with one swap, and caching against two budgets are just how you build it.