The third of three ideas behind modern ANN: the winning index depends on where the data lives. RAM, SSD, and GPU reward very different layouts.
Part 4 of Approximate Nearest Neighbors
The first two articles turned search into a two-step sketch. Partition cuts the dataset into a handful of candidate regions. Compress replaces full vectors with tiny codes. Between them, the work you actually have to do per query drops by several orders of magnitude.
The remaining question is almost boring to say out loud: how is all of this laid out in storage. If your data fits in RAM, the answer is one thing. If it sits on an SSD, the answer is another. If you are answering thousands of queries at once and have a GPU to throw at them, the answer is a third thing entirely. The modern ANN story ends in a place that is less about algorithms and more about memory.
Every computer has a hierarchy of memory. A register read is free, an L1 cache hit takes about a nanosecond, DRAM takes a hundred, an SSD takes a hundred thousand, and reading across a network is in the millions. Each step down the ladder is roughly a hundred times slower than the one above it.
A nanosecond is invisible. A microsecond feels instant. A millisecond is noticeable. These feel like the same region of "fast" until you stack a million of them on top of each other, which is exactly what a search index does. An index that misses cache on every vector it touches can be a thousand times slower than one that streams through memory in order, even if the algorithm is otherwise identical.
The practical consequence is that the right ANN index depends on which rung of this ladder your data lives on. The same algorithm that is fastest in RAM can be the worst choice on an SSD, because on an SSD each "random pointer chase" costs a hundred thousand times more than it did in memory.
This is the classical regime, and for many applications it is still the right one. The entire database of vectors fits in RAM on one machine, usually after compression. The CPU can move through it at tens of gigabytes per second. Modern SIMD instructions let a single CPU core compute sixteen float-by-float products in parallel. Even touching every vector in a ten-million-row database is a matter of tens of milliseconds.
FAISS-flat (a linear scan with SIMD) and HNSW (a small-world graph walked in memory) both live here. The right layout is a single contiguous block of floats per index, with pointers between them that the CPU can prefetch. Graph walks are fine because each hop costs a few nanoseconds, not a few microseconds.
In-memory indexes dominate the "small enough, fast enough" niche: tens of millions of vectors at single-millisecond latency, where paying for the RAM to hold them is cheaper than the engineering effort to squeeze them onto disk.
At a billion vectors, the math changes. A billion 768-dimensional float32 vectors is three terabytes. RAM that size exists, but renting it is expensive. SSDs of the same size are cheap. The question becomes how to answer a query without reading most of the disk.
The brutal fact is that a random SSD read takes about a hundred microseconds. That is a thousand times slower than a random RAM read. A graph-based index that hops through a hundred neighbors during a query, which finishes in a millisecond in RAM, would take a tenth of a second on disk. That is too slow for interactive use.
This is the problem DiskANN (also called Vamana) solved. The index is a single graph on disk, with edges carefully chosen so the graph has a small diameter, and the vectors for each node stored in the same block as the edge list pointing out of it. Queries run a beam search: they keep a frontier of the most promising candidates so far and fetch all of their neighbors together, amortizing the per-read cost over many lookups. The compressed codes from the previous article also help, since you can rank candidates using the small codes in memory before fetching full vectors from disk.
A well-tuned DiskANN index can serve a billion vectors at tens-of-milliseconds latency from a single machine. Ten years ago that would have needed a cluster.
A GPU sees the world differently. A single CPU core and a single GPU thread run at comparable clock speeds, but a GPU has thousands of threads in flight at once. The cost per thread is higher (data has to be shipped across the PCIe bus, results shipped back) and the kernel launch overhead is non-trivial. But once a batch of queries is in GPU memory, the cost of the tenth thousand query is barely more than the cost of the first.
This makes GPUs a bad choice for one-query-at-a-time, low-latency serving, and a great choice for anything batch-shaped: embedding an entire corpus, building a kNN graph, training-time retrieval, bulk re-indexing. The crossover point depends on the hardware and the workload, but it usually sits around a batch size of a few dozen.
FAISS has a GPU implementation precisely for this regime. Training pipelines often use a GPU-resident brute-force kNN search because it can process tens of thousands of queries in a single kernel launch, which is faster than any CPU index even accounting for data transfer.
Between "all in RAM" and "graph engineered for SSD" there is a broad middle ground occupied by most of the production systems people actually reach for. Lancedb is a clean example. It stores an IVF-PQ index columnarly, in Arrow format, on disk. When a query arrives, it picks the two or three nearest partitions and reads them from disk sequentially. The operating system's page cache handles the rest: hot partitions stay resident, cold ones get evicted.
This is less clever than DiskANN, but it inherits two useful properties. Columnar layout means reads within a partition are sequential, which is the access pattern SSDs are fastest at. And the product quantization codes from the previous article are small enough that a medium-sized dataset often has its hot partitions in RAM after a few queries, with the full codes still on disk as a backstop.
There is no single best index. The choice is a function of how much data you have, how fast you need answers, how many queries arrive per second, and what you are willing to spend on hardware. The good news is that the space of reasonable choices is small and mostly non-overlapping. If you know your dataset size and your latency target, the right index is usually obvious.
A useful mental shortcut: in-memory indexes are bought with RAM, disk-based indexes are bought with engineering, and GPU indexes are bought with batch size. Each one converts a scarce resource into latency.
We now have the full modern stack. Partition reduces how many vectors you look at. Compress reduces how many bytes per vector. Hardware-aware layout chooses a storage pattern that matches the device you are reading from. Production systems stack all three. FAISS-IVF-PQ stacks the first two in RAM. Lancedb stacks all three on a laptop-sized SSD. DiskANN stacks all three to put a billion vectors on a single box.
The next article is the payoff. Two of the most widely used tools in applied machine learning, UMAP and semantic search, are both built on top of the same ANN primitive we have been developing. They use it in very different ways, and lining them up side by side makes both of them clearer.