On-Device LLM Memory: What the Kernel Evicts, and What Restoring It Costs

If you have measured an on-device LLM on an idle board, you have measured the easy case. The hard case is the device doing its real job, with a camera pipeline running and a browser in the background. Memory runs short, the kernel reclaims whatever looks coldest, and much of that belongs to the model you loaded twenty minutes ago and have not touched since. The next time the user types, it must be reassembled before one token appears.

What an on-device LLM holds in memory, and which part grows

Memory budgets are usually written around the weights, the one number the model card gives you. That number is fixed; the key-value cache is not. It grows linearly with the tokens in the conversation and on long-context configurations overtakes the weights entirely. For a decoder with grouped-query attention:

bytes_per_token = 2 * num_hidden_layers * num_key_value_heads * head_dim * bytes_per_element

The leading 2 is the key tensor and the value tensor. Take a model teams actually ship, Qwen3-0.6B, and read its config.json: 28 layers, 16 attention heads but only 8 key-value heads, head dimension 128, bfloat16, position limit 40,960 tokens. So 2 x 28 x 8 x 128 x 2 bytes, which is 114,688 bytes, or 112 KiB of key-value cache per token.

The right-hand column uses roughly 1.2 GB for the weights, what 0.6 billion parameters costs at bfloat16. Treat the weight figures as approximate: they depend on the quantisation scheme and on whether the tied embedding is quantised with the rest. Ship 4-bit weights, the normal choice, and they fall to roughly 0.4 GB, so at a 32,768-token context the cache is around nine times the model it belongs to. Budget an on-device LLM accordingly: a board with 8 GB of RAM, a 4-bit 1.2B model and a 32k window is running closer to 5 GB than to 0.5 GB.

What the kernel does when memory runs short

Those gigabytes are not one kind of memory. Weights are usually mapped from a file, and clean file-backed pages are the cheapest thing to reclaim: the kernel writes nothing, drops them, updates the page tables, and the memory is free. From the application’s side, a layer of weights has silently become a future storage read.

The key-value cache is the opposite: anonymous memory that exists nowhere else, so the kernel cannot drop it. It can only compress it into zram or write it to swap.

Which leads to the result that surprises people. Compression does almost nothing here: measured with both lz4 and zstd, the saving on key-value data is 0.2 to 9.3 percent, against the two to three times ordinary application heap achieves. That is not a defect in the compressor. A key-value cache is dense numeric tensor data with high entropy and almost no repeated byte patterns, the input a general-purpose compressor cannot exploit. So letting zram absorb the pressure of an on-device LLM absorbs nothing, and the next actor to respond kills processes.

On Android that is lmkd, a userspace daemon; the in-kernel lowmemorykiller driver was removed in Linux 4.12. It watches pressure stall information, treating a partial stall beyond ro.lmk.psi_partial_stall_ms as a low-memory notification and a complete stall beyond ro.lmk.psi_complete_stall_ms as critical; defaults are 200 ms partial on low-RAM devices, 70 ms on high-end devices and 700 ms complete. Cross the critical threshold while reclaim lags and a process dies. If it held your on-device LLM, the next prompt pays a cold start rather than a restoration.

Why restoring the memory is slower than it looks

On one test device, time to first token rises from 0.8 seconds with everything resident to 16 seconds when evicted memory must be restored first, caused by nothing the application did. On a Galaxy S25+ running Qwen3-0.6B, full eviction multiplies it by 18 to 20 times when the swap path is flash and by 5 to 6 times when it is zram. Attributed on a OnePlus 12 with the same model, storage reads are 59 to 61 percent of it and allocation a further 10 to 13 percent. Two thirds of the delay is not computation.

The cause is a bandwidth gap that does not close. On these devices UFS 4.0 sequential reads measured 4.2 GB/s against 84.8 GB/s for LPDDR5X, and read throughput saturates around 512 KB per operation, so smaller requests waste the little bandwidth available while larger ones buy nothing.

A structural problem compounds it. The conventional way to handle a model that does not fit is to offload whole layers and reload on demand, which serialises: the runtime finds it needs layer 11, reloads it, computes it, then finds it needs layer 12. Each reload is a full stall, because the compute engine has nothing it is allowed to do meanwhile. A twenty-fold bandwidth deficit multiplied by a serialised access pattern is the 16 seconds.

The key idea: make the model restorable in pieces

Eviction cannot be prevented, so the target is not residency. It is overlap: if computation on the resident part of an on-device LLM proceeds while the evicted part reloads, storage time hides behind useful work instead of adding to it. Three decisions make that possible.

Allocate in fine-grained shared buffers. Weights are split per layer and the cache into 256-token chunks, each allocated as a buffer both CPU and GPU address directly, which a phone SoC permits because the memory is physically unified; the mechanism here is OpenCL shared virtual memory. Eviction then becomes partial rather than all-or-nothing, so the system can surrender a quarter of the model instead of all of it. Less obviously, the CPU writes a restored chunk straight into the buffer the GPU is reading, with no copy. Without shared addressing every restored byte is copied across, putting restoration back on the critical path.

Evict backward, restore forward. Inference runs through layers in order, so eviction takes the last layers first and leaves the early ones resident. When a prompt arrives the GPU starts on layer 0, because layer 0 was never evicted, while the CPU restores from the earliest evicted layer forward, moving in the same direction as the computation and staying ahead of it. This looks arbitrary until the ordering argument is clear: evict the other way round and the GPU has nothing to do until restoration finishes, which is the serialised behaviour again. The direction of eviction converts a stall into a pipeline.

Split the swap path by cost, not by type. Cache chunks go to two destinations at once, some into an in-memory compressed region and some to storage, divided in inverse proportion to each path’s time per unit of data. The paths are independent resources: all to flash and the flash queue bottlenecks while the compressor idles; all to the compressed region and pressure is barely relieved. Eviction then runs in stages, cache chunks first, then chunks with weights, then weights alone, and finally compressed cache data out to storage, so the cheapest thing to restore is always surrendered first.

What the numbers look like

Against a layer-level partial-offload baseline on two Qualcomm phones, time to first token improves by 2.1 to 5.5 times at residency levels of 0, 25, 50 and 75 percent. Against an OS-paging baseline, which maps weights with mmap() and leaves the cache to zstd-compressed zram, the gain at full eviction is 2.5 to 3.0 times on the Galaxy S25+ and 9.2 to 25.9 times on the OnePlus 12 — a spread wide enough to warn against treating any single figure as general.

The component breakdown on the Galaxy S25+ at 50 percent residency shows where the gain comes from: allocation 1.37 times, reload 2.15 times, prefill 2.33 times, and overlapping all three a further 1.61 times. About a third of the total improvement comes not from any phase being faster but from the phases no longer waiting for each other.

The overlap has a cost. Peak power rises to 19.2 W from 14.6 W, because CPU, GPU and storage controller are active at once instead of taking turns, although total energy per request falls because the work finishes sooner. On a product with a peak-power or thermal ceiling, that trade may be wrong.

More useful still is mixed real use. Over ten rounds across 45 minutes of application switching on the S25+, the OS-paging configuration was killed by the low-memory killer every round. Surrendering 25 percentage points of memory in advance survived all ten; 15 points was killed in four of ten.

Where this does not apply

• One context, dense models only. Two concurrent contexts are left open. Backward eviction assumes every layer is needed in order, so mixture-of-experts models and sparse attention, which touch only part of the weights or cache, would need restoration order predicted from the input, and that prediction does not exist here. The split between the compressed and storage paths is also profiled offline, so expect drift on a hot, fanless enclosure.
• Qualcomm GPUs only, through OpenCL. Measured on Adreno 830 and Adreno 750 with device-specific GPU kernels. There is no NPU result, and an NPU with its own driver and memory import rules needs the unified-buffer assumption re-checked. A vendor runtime that hides allocation granularity removes the first design decision entirely.
• Two models under 3B parameters. Qwen3-0.6B and a 1.2B model at 8k, 16k and 32k contexts. Nothing establishes behaviour for a 7B-class on-device LLM, where the weights rather than the cache dominate. Output quality was checked only as F1 on 428 question-answer pairs from one dataset.

How to measure on-device LLM memory pressure on your own board

Every quantity above is observable on a normal Linux system. First, compute the cache size from the model’s own configuration file:

raghu@techveda.org:~$ python3 -c '
import json
c = json.load(open("config.json"))
b = 2 * c["num_hidden_layers"] * c["num_key_value_heads"] * c["head_dim"] * 2
print("KV bytes per token:", b, "=", b/1024, "KiB")
for n in (4096, 8192, 16384, 32768):
    print(n, "tokens ->", round(b*n/1024/1024, 1), "MiB")
'
KV bytes per token: 114688 = 112.0 KiB
4096 tokens -> 448.0 MiB
8192 tokens -> 896.0 MiB
16384 tokens -> 1792.0 MiB
32768 tokens -> 3584.0 MiB

Second, check whether the system is already stalling. Pressure stall information is the same signal lmkd acts on: some is time with at least one task delayed, full time with every runnable task delayed. A full avg10 persistently above zero while the model is loaded means eviction of your on-device LLM is a question of when, not whether. Without CONFIG_PSI the file does not exist, which is worth fixing on a product kernel first.

raghu@techveda.org:~$ cat /proc/pressure/memory
some avg10=4.22 avg60=2.89 avg300=1.14 total=91833472
full avg10=1.07 avg60=0.71 avg300=0.26 total=24117043

Third, confirm reclaim is touching the model and not something else. Put the inference process in its own cgroup and read the refault counters, which count pages evicted and later needed again. A rising workingset_refault_file is your weights being dropped and read back; workingset_refault_anon is the cache coming back out of swap. A non-zero high in memory.events means direct-reclaim throttling.

raghu@techveda.org:~$ grep -E 'workingset_refault|^pgsteal ' /sys/fs/cgroup/inference/memory.stat
pgsteal 1203884
workingset_refault_anon 402119
workingset_refault_file 1588003
raghu@techveda.org:~$ cat /sys/fs/cgroup/inference/memory.events
low 0
high 214
max 0
oom 0

Fourth, measure what compression buys on your own data. Force the cache out to zram and read the first two fields of mm_stat. A ratio near 1.04x is the expected result, not a misconfiguration.

raghu@techveda.org:~$ read orig comp rest < /sys/block/zram0/mm_stat
raghu@techveda.org:~$ echo "uncompressed $((orig/1048576)) MiB, compressed $((comp/1048576)) MiB"
uncompressed 2048 MiB, compressed 1961 MiB

Fifth, apply pressure deliberately rather than waiting for the field to do it. On cgroup v2, writing a byte count to memory.reclaim reclaims that much from the cgroup without signalling pressure to the rest of the system:

raghu@techveda.org:~$ echo 2G | sudo tee /sys/fs/cgroup/inference/memory.reclaim
2G
raghu@techveda.org:~$ time ./run_prompt --tokens 8
real    0m14.880s

If your kernel has that file, this is the most useful test here: it turns an intermittent field complaint into a reproducible measurement. The kernel may reclaim more or less than asked and the write returns EAGAIN if it reclaimed less, so check the return value. Time the same prompt on a freshly loaded on-device LLM for your own version of the 0.8-versus-16-second figure.

What this changes for an embedded product

For an on-device LLM, context length is a memory decision rather than a marketing one. A 32k window on a small model is a multi-gigabyte commitment that grows during use, and belongs in the memory budget beside the weights. A team that advertises a long context and budgets only for the weights has shipped a device its own low-memory killer will terminate in the field, never on the bench.

What to do next depends on the device, and the two cases pull in opposite directions. On a fixed-function appliance, where inference is the product and nothing else competes, build none of this machinery: give the cgroup a memory.min covering the model and keep the rest of the system small. Protection is cheaper and more predictable than restoration. On a general-purpose device that multiplexes, residency is unachievable, so an on-device LLM must assume partial residency from the first allocation, controlling allocation granularity and eviction order itself. That is a decision about which runtime you adopt, and it must be made before integration: a runtime that allocates the cache as one contiguous block cannot be retrofitted into chunked eviction.

Proactive eviction is also a tuning parameter you now own: surrendering a quarter of the model in advance survived ten rounds where 15 percent did not. This is why on-device LLM work keeps turning into kernel work. Nothing here is a machine-learning problem. It is reclaim policy, swap behaviour, allocation granularity and the cost of a storage read, the material our training tracks cover for engineers who make these calls on real hardware.

Key takeaways

• The key-value cache grows linearly with context and overtakes the weights: for Qwen3-0.6B, 112 KiB per token, or 3.5 GiB at a 32,768-token context against roughly 1.2 GB of bfloat16 weights.
• Weights are file-backed and dropped for free. The cache is anonymous, can only be compressed or swapped, and compresses by just 0.2 to 9.3 percent.
• Restoration, not computation, is the cost: 0.8 seconds resident against 16 seconds evicted, storage reads 59 to 61 percent of it.
• The low-memory killer decides, not the application. Protect the memory with memory.min on a fixed-function device; assume partial residency on a multiplexing one.

Frequently asked questions

Why does the key-value cache compress so badly when ordinary application memory compresses well?
It is dense numeric tensor data with high entropy and almost no repeated byte patterns, which a general-purpose compressor cannot exploit. Measured with lz4 and zstd, the saving is only 0.2 to 9.3 percent, so zram does not relieve the pressure as it does for a normal heap.

Why evict the last layers first instead of the least recently used ones?
Inference runs through layers in order. Leaving the early layers resident lets the compute engine start immediately while restoration proceeds forward from the earliest evicted layer, so reloading overlaps with computation instead of stalling it.

Can I stop the kernel from evicting my on-device LLM?
On a fixed-function device, largely yes: set a memory.min on the inference cgroup covering the model and keep the rest of the system small. On a device running other applications you cannot: the low-memory killer decides, and no setting guarantees survival.

Further reading

• Hongseung Yu, Minsung Kim, Jongseok Park, Kyunghan Lee. “mzCache: On-Device LLM Memory Management under Multitasking.” arXiv:2609.01338, September 2026 (ACM DOI 10.1145/3795866.3844495) — arxiv.org/abs/2609.01338
• Linux kernel documentation, zram: docs.kernel.org/admin-guide/blockdev/zram.html
• Linux kernel documentation, Control Group v2 memory controller: docs.kernel.org/admin-guide/cgroup-v2.html
• Linux kernel documentation, Pressure Stall Information: docs.kernel.org/accounting/psi.html
• Android Low Memory Killer Daemon (lmkd): android.googlesource.com/platform/system/memory/lmkd
• Qwen3-0.6B model configuration: huggingface.co/Qwen/Qwen3-0.6B