LLM Inference From Flash: Running Models Larger Than Your DRAM

Here is a problem that sounds impossible at first. A language model with seven billion parameters needs roughly 14 GB just to hold its weights in 16-bit precision. Your board has 8 GB of RAM, and the rest of the system needs some of that. Conventional wisdom says the model will not run. It can, and the method behind it, LLM inference from flash, turns out to be more interesting to an embedded engineer than to a machine-learning researcher, because it is really a storage and memory-hierarchy problem in disguise.

The problem: a model that does not fit in DRAM

A modern language model can hold billions of parameters. At 16-bit precision, a 7-billion-parameter model needs about 14 GB for its weights alone. Many phones, single-board computers, and embedded gateways do not have that much DRAM to spare. The weights do fit comfortably in flash storage (eMMC, UFS, NVMe, or an SSD), but flash is far slower than DRAM. The naive fix, reloading the whole model from flash for every forward pass, can take seconds per token and burns energy doing it.

So the central question is this: if the model lives in flash and only part of it fits in DRAM at once, how do you generate each token without paying the full cost of reading the model from flash every single time?

The key observation: most neurons stay silent

The answer starts with a property of these networks. In the feed-forward (FFN) layers, which hold most of the weights, ReLU-style activations mean that for any given token the large majority of neurons output zero. Their weights contribute nothing to that token’s result, so they never needed to be read. In measured models this sparsity is dramatic: an OPT 6.7B model shows about 97% sparsity in its FFN layer, and Falcon 7B, after fine-tuning to a ReLU activation, reaches about 95% sparsity with essentially the same accuracy.

If you could know in advance which neurons a token will use, you would only need to read those weights from flash, a small fraction of the whole model. That single idea is what makes the whole approach possible.

Three moves that make flash fast enough

Knowing sparsity exists is not enough; you also have to fetch the right weights cheaply from a device that hates small random reads. Three techniques work together to do that:

• A low-rank predictor that, for each layer, predicts which FFN neurons will be active for the current token, so only the corresponding weight rows and columns are loaded rather than the entire layer.
• Windowing, which keeps the weights for recently active neurons resident in DRAM. Because consecutive tokens activate overlapping sets of neurons, a sliding window over the last few tokens means most of what the next token needs is already in memory, and only the small difference is read from flash.
• Row-column bundling, which stores the matching column of the up-projection matrix and row of the down-projection matrix for a neuron next to each other, so one larger contiguous read fetches both. This directly attacks the fact that flash pays a fixed latency before each read begins, so fewer and larger reads beat many small ones.

All three serve one goal that comes straight from the storage hardware: transfer less data from flash, and read what you do transfer in larger contiguous chunks.

What the numbers look like

Reported benchmarks for OPT 6.7B in 16-bit precision on an Apple M1 Max, with only about half the model resident in DRAM, show the effect of each move added in turn:

• Naive loading (reload everything): about 13.4 GB moved per token, roughly 2196 ms of I/O.
• Hybrid baseline (half the model kept resident, half streamed, no sparsity): about 6.7 GB, roughly 1090 ms.
• Add the activation predictor and windowing: only about 0.9 GB moved, roughly 738 ms. Throughput drops here, because the reads are now smaller and more scattered.
• Add row-column bundling: about 0.2 GB, roughly 164 ms, because coalescing restores larger reads.
• With the full method: roughly 87 ms of I/O latency per token.

Going from about 2196 ms to about 87 ms is roughly a 25x reduction in I/O latency, which is where the headline “20-25x on GPU” figure comes from: on a GPU, compute is fast, so I/O dominates the time and cutting it dominates the speedup. On a CPU, compute is slower and takes a larger share of each token, so the same I/O savings translate into a smaller overall gain of about 4-5x. Across the setups, the method enables running models up to twice the size of the available DRAM. The reported results cover OPT 6.7B, Falcon 7B, and Persimmon 8B, measured on an Apple M1 Max (1TB SSD), an M2 Ultra (2TB SSD), and a Linux machine with a 24 GB NVIDIA RTX 4090.

Reading LLM inference from flash as a systems problem

Strip away the model and this is a case study in matching an access pattern to a storage device. Two observations drive the whole design, and both are familiar to embedded Linux engineers.

First, flash rewards large sequential reads and punishes small random ones. On the M1 Max used in these measurements, the 1 TB flash exceeds 6 GiB/s on a 1 GiB linear read of an uncached file, but that bandwidth collapses for small random reads because each read pays a fixed “latency to first byte” through the OS, driver, interrupt path, and flash controller. The practical rule that emerged is that useful throughput for this workload needs reads of 32 KiB or larger, issued across many threads (32 in these measurements). That is exactly why bundling exists: pay the per-read latency once for a larger payload.

You can watch the same behaviour on your own board with fio. The point is not the absolute numbers, which depend on the device, but the shape of the curve as block size grows:

# Sequential 1 MiB reads: throughput is high
raghu@techveda.org:~$ fio --name=seq --rw=read --bs=1M --size=1G --filename=./testfile --direct=1

# 4 KiB random reads: throughput collapses, latency to first byte dominates
raghu@techveda.org:~$ fio --name=rand4k --rw=randread --bs=4k --size=1G --filename=./testfile --direct=1

# 32 KiB random reads at queue depth 32: most of the throughput returns
raghu@techveda.org:~$ fio --name=rand32k --rw=randread --bs=32k --iodepth=32 --size=1G --filename=./testfile --direct=1

Second, to measure flash honestly you must defeat the page cache, exactly as these benchmarks did by reading without OS caching. On Linux you drop caches before an uncached read:

raghu@techveda.org:~$ sync
raghu@techveda.org:~$ echo 3 | sudo tee /proc/sys/vm/drop_caches

The predictor plus windowing scheme is also a familiar idea in new clothing: it is caching and prefetching driven by predicted access, the same reasoning behind readahead heuristics and working-set management in the kernel. The novelty is applying that reasoning to model weights instead of file pages.

Where this approach breaks down

The method depends on high activation sparsity, which comes from ReLU-style activations. Models built with other activations must be fine-tuned to a ReLU variant first, which is extra work and is not always possible. The reported I/O latencies are best-case theoretical figures used for a fair comparison, so real deployments may see higher numbers. The approach processes one sequence at a time, which fits on-device assistants but not batched serving. The measurements were taken on high-end laptop and desktop silicon rather than a low-cost embedded board, so the exact gains on eMMC or slower UFS will differ, even though the underlying design principle still holds. Finally, the technique is orthogonal to quantization; combining the two is left largely as future work.

What this means for embedded and kernel engineers

The lasting value here is the method, not the specific model. When a working set does not fit in RAM, the winning move is to stream it from flash while shaping every access to the device: read less by exploiting structure in the workload, and read in larger contiguous chunks to amortise per-read latency. That reasoning applies to on-device inference, but it applies equally to large memory-mapped datasets, over-the-air update images, and any data-heavy service on a constrained device. For teams that reason carefully about the storage stack, the page cache, and I/O access patterns, running a model twice the size of DRAM stops being surprising and becomes an engineering exercise.

Key takeaways

• A model can run even when its weights exceed DRAM, by keeping the weights in flash and loading only what each token needs.
• Activation sparsity (about 97% in OPT 6.7B’s FFN) means most weights can be skipped per token; a small predictor picks the active ones.
• Windowing reuses recently loaded neurons; row-column bundling turns many small reads into fewer large ones.
• On OPT 6.7B the per-token I/O latency drops from about 2196 ms to about 87 ms, giving roughly 4-5x faster inference on CPU and 20-25x on GPU.
• The transferable lesson is storage-aware design: transfer less data and read it in larger contiguous chunks, which is standard embedded I/O reasoning.

Frequently asked questions

How can a model larger than the available RAM run at all?
By keeping the full set of weights in flash storage and loading, for each generated token, only the small subset of weights that token actually needs. With this approach a device can run models up to twice the size of its available DRAM.

Why is reading from flash the hard part?
Flash has far lower bandwidth than DRAM and pays a fixed latency before each read starts. Small random reads are dominated by that startup latency, so the techniques exist to read less data and to read it in larger, more contiguous chunks of 32 KiB or more across many threads.

How large are the speedups?
Compared with naively loading the whole model on every pass, reported benchmarks show roughly 4-5x faster inference on CPU and 20-25x on GPU. For OPT 6.7B, per-token I/O latency fell from about 2196 ms to about 87 ms on an Apple M1 Max.

Why should an embedded engineer care about an AI workload?
The core method is standard storage engineering: exploit structure in the workload to move less data, and shape access patterns to the device. That reasoning transfers to any case where a working set does not fit in RAM, not only to language models.

Further reading

• The technique described here was introduced in: Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar, “LLM in a flash: Efficient Large Language Model Inference with Limited Memory,” ACL 2024 (arXiv:2312.11514).
• Apple Machine Learning Research: project page with additional detail.
• Mirzadeh et al., “ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models,” 2023 — the sparsity result this method builds on.
• Linux kernel documentation: memory management concepts (page cache and reclaim).