File-Backed Weights: Why Your Edge Device Holds the Model Twice

Most inference stacks on embedded Linux load a model in two steps that look like one. The runtime opens the model file, and the kernel places the bytes it reads into the page cache. The runtime then copies those bytes into an allocation it owns, because that is what its tensors and kernels expect to operate on. On a device with unified memory, where the CPU and the accelerator address the same DRAM, that second copy moves data from one part of RAM to another part of the same RAM and changes nothing about where the bytes physically are. File-backed weights are the alternative: the mapped pages of the model file become the tensor storage directly, with no private duplicate. This article shows what the duplicate costs, why runtimes create it anyway, how file-backed weights remove it, and where they should not be used.

The same model, resident twice

The cost is easy to see rather than argue about. The program below opens a file and loads it two ways. In one mode it reads the file into a buffer from malloc(). In the other it maps the file with mmap() and MAP_SHARED. Both modes then touch one byte per page, so the pages are genuinely resident in both cases. Between the two it prints the resident-set breakdown that the kernel exposes in /proc/self/status.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>

static void show(const char *tag)
{
    FILE *f = fopen("/proc/self/status", "r");
    char line[256];
    printf("--- %s ---\n", tag);
    while (fgets(line, sizeof(line), f))
        if (!strncmp(line, "VmRSS", 5) || !strncmp(line, "RssAnon", 7) ||
            !strncmp(line, "RssFile", 7) || !strncmp(line, "RssShmem", 8))
            fputs(line, stdout);
    fclose(f);
}

int main(int argc, char **argv)
{
    const char *path = argv[1];
    int mode = atoi(argv[2]);        /* 0 = read into malloc, 1 = mmap MAP_SHARED */
    struct stat st;
    int fd = open(path, O_RDONLY);
    fstat(fd, &st);
    size_t len = st.st_size;
    volatile unsigned long long sum = 0;
    unsigned char *p;

    if (mode == 0) {
        p = malloc(len);
        ssize_t got = 0, n;
        while (got < (ssize_t)len) {
            n = pread(fd, p + got, len - got, got);
            if (n <= 0) break;
            got += n;
        }
    } else {
        p = mmap(NULL, len, PROT_READ, MAP_SHARED, fd, 0);
    }

    for (size_t i = 0; i < len; i += 4096)
        sum += p[i];

    show(mode == 0 ? "read() into malloc buffer" : "mmap(MAP_SHARED) + touch");
    return 0;
}

Built and run against a 512 MiB file whose pages are already in the page cache:

raghu@techveda.org:~$ gcc -O2 -o wdemo wdemo.c
raghu@techveda.org:~$ cat weights.bin > /dev/null
raghu@techveda.org:~$ ./wdemo weights.bin 0
--- read() into malloc buffer ---
VmRSS:	  525448 kB
RssAnon:	  524384 kB
RssFile:	    1064 kB
RssShmem:	       0 kB

raghu@techveda.org:~$ ./wdemo weights.bin 1
--- mmap(MAP_SHARED) + touch ---
VmRSS:	  525428 kB
RssAnon:	      92 kB
RssFile:	  525336 kB
RssShmem:	       0 kB

The two totals are nearly identical, which is why this is so easy to miss when a team watches only resident set size. The breakdown is the interesting part. In the read path, 524,384 kB of the resident set is anonymous memory: a private copy that belongs to the process alone and is backed by nothing on disk. In the mapped path, anonymous memory stays at 92 kB and the resident pages are counted as file pages, because they are the page-cache pages of the model file. The process did not get its own copy. It got the kernel’s.

The system-wide view shows what that costs the device. Measured against a warm page cache holding the same file, one reader on each path:

raghu@techveda.org:~$ free -m
   baseline : used=139MB  buff/cache=636MB
   read()   : used=655MB  buff/cache=636MB    <- anonymous copy plus the still-cached file
   mmap     : used=156MB  buff/cache=636MB    <- the cached file only

The read path added roughly 516 MB of used memory while the page cache stayed exactly where it was. Those 512 MiB of weights are now in DRAM twice. On a workstation that is wasteful. On a board with 2 GB or 4 GB of DRAM, running a model sized to fit that board, it decides whether the product runs at all.

Why runtimes make the second copy

The copy is not an oversight. A framework owns its memory: a tensor is a handle into a buffer that the framework’s allocator produced, and every kernel, stream and lifetime rule assumes that buffer came from that allocator. When the loader is handed a file, the shortest path to a valid tensor is to ask the allocator for storage and copy the file bytes into it. The copy is what converts foreign memory into memory the framework can reason about.

That design was correct for discrete accelerators. On a card behind PCIe the device cannot read host DRAM at useful speed, so the bytes must physically move into device memory before any kernel touches them. The copy is not overhead there; it is the transfer that makes execution possible. Loading paths were written under that assumption and then carried unchanged onto integrated and coherent-memory parts, where the assumption stopped being true. On an SoC whose accelerator addresses the same DRAM as the CPU, the file pages are already in memory the accelerator can read, and the copy has become pure duplication.

Two further costs usually ride along with it. Data-type expansion often happens during the copy, so a checkpoint packed as int8 is widened to a float format on the way in, multiplying the footprint again. And when the framework-owned copy is too large to hold, the loader switches to refilling it as layers execute, so the same weights are re-read and re-copied on every token. That is the regime in which duplication stops being a memory problem and becomes a throughput problem, and it is exactly the regime file-backed weights were designed for.

What file-backed weights change

The mechanism behind file-backed weights is a shared file mapping. mmap() with MAP_SHARED does not create a copy of the file’s pages; it installs page-table entries that point at the page-cache pages themselves. Two processes mapping the same file map the same physical pages. A page that is read and never written stays clean, and a clean file-backed page can be dropped by the kernel at any time and read back from the file when it is touched again.

That last property is where most of the benefit comes from, and smaps_rollup shows it directly. Comparing the two readers again:

raghu@techveda.org:~$ grep -E '^(Rss|Pss|Private_Clean|Private_Dirty)' /proc/$PID/smaps_rollup

  mmap    Rss:            525248 kB     read()  Rss:            525256 kB
  mmap    Pss:            524451 kB     read()  Pss:            524458 kB
  mmap    Private_Clean:  524292 kB     read()  Private_Clean:       4 kB
  mmap    Private_Dirty:      84 kB     read()  Private_Dirty:  524380 kB

Half a gigabyte of clean pages against half a gigabyte of dirty pages. Clean file pages are reclaimable for free: the kernel drops them and re-reads from the file when they are next touched. Dirty anonymous pages are not. With swap they can only be freed by writing them out first, and without swap they cannot be paged out at all. That does not make them permanent, and it is worth being precise about why: under enough pressure the kernel still reclaims that memory, but the only mechanism left to it is killing the process that owns it. Most embedded products run with no swap, so on those devices the anonymous copy is memory the kernel cannot reclaim without terminating your inference daemon.

Sharing follows from the same property, though the accounting only makes it visible once sharing actually happens. The single-reader figures above show the mapped pages under Private_Clean, because one process is mapping them; Shared_Clean and a divided Pss appear only when another process maps the same pages. Run two readers and that is exactly what happens: they report proportional set sizes of 262,297 kB and 262,287 kB against resident sets of about 525 MB each, because the kernel is dividing one set of physical pages between two mappers. Two processes on the read path would report two full private copies instead.

Making the accelerator read the same pages

Mapping the file solves the CPU half. The accelerator half needs an import path, because a runtime cannot hand a raw pointer to a GPU or NPU and expect it to work. The memory has to be brought into the device’s address space through whatever interface the driver offers.

In Vulkan, the relevant interface is the VK_EXT_external_memory_host extension. An application queries vkGetMemoryHostPointerPropertiesEXT to learn which memory types accept a given host pointer, then chains a VkImportMemoryHostPointerInfoEXT structure into vkAllocateMemory to import that host allocation as a Vulkan memory object instead of allocating fresh device memory. Address and size must both be multiples of minImportedHostPointerAlignment, which the implementation reports. Importing shares ownership: the application may still reach the memory through the host pointer, and is responsible for synchronising host and device access. On unified-memory parts this is the natural fit, because the pages being imported are already in memory the device can read.

Two limits on this section are worth stating plainly. The published work behind file-backed weights implemented the import on Apple’s Metal and, separately, on Vulkan on one AMD integrated part running Windows drivers; it did not report a Linux NPU integration. And I could not verify that any current embedded NPU runtime from Rockchip, NXP, TI or NVIDIA exposes a host-pointer import of this kind. Several embedded stacks instead expect buffers as a dma-buf file descriptor, which would make dma-buf the natural Linux carrier here, but I found no published implementation of that path and am not claiming one exists. Treat the Vulkan route as verified and the rest as a direction.

Zero copy alone is not enough

This is the part that a summary of the idea always leaves out, and it is the part that decides whether an implementation helps or hurts.

Removing the copy is one of three conditions that must hold together. The first is that the kernel actually reads the mapped file pages with no per-use copy reintroduced behind the scenes. The second is that activations stay resident on the accelerator across layer calls, rather than being copied out to host arrays and back between layers. The third is that dependencies between operations are established on the accelerator itself, inside the framework’s own stream or with an on-device event, so the host never has to wait for outstanding work to drain.

An implementation that satisfies only the first condition is measurably worse than the copy it replaced. In the reported experiments, mapping the weights but dispatching from a private queue outside the framework, and round-tripping activations through the host, ran a dense decode stage at 5.96 seconds against 2.62 seconds for the ordinary copying path. That is 2.3 times slower after successfully eliminating the copy. The synchronisation that the arrangement introduced cost more than the copy ever did.

The costs were then separated one at a time, measured as extra milliseconds per pass over a 1.06 GB working set. Removing the first condition added 12.38 ms, removing the second added 5.62 ms, and removing the third added 3.46 ms. Removing all three added 17.90 ms. The ordering cost was measured separately and forms a clear hierarchy for the same pass: 38 ms when dependencies stay in the framework’s stream, 49 ms with a plain event, 55 ms with a shared event, and 124 ms when the host drains the queue. The cost appears where a signal becomes visible to the host, not where the queue changes owner. For anyone building this on an embedded stack, that is the single most useful thing to know before starting.

What the numbers look like

On a microbenchmark isolating how weights reach the kernel, default framework constructors delivered 53 to 82 GB/s, the strongest pipelined and double-buffered copy reached 134 GB/s, and the adopted mapping reached 516 GB/s, matching the same kernel over resident framework storage to within a fraction of a percent. End to end on a matched Qwen2.5-72B in int8, the adopted path reached 7.14 tokens per second against 7.23 for a control that copied once into framework-owned storage, and 0.94 for the stock path that re-ingests weights on every use: within 1.3 percent of the resident control, and about 7.6 times faster than per-use ingestion, while holding one set of reclaimable file pages instead of a second owned copy.

Concurrency is where the difference becomes severe rather than incremental. With several decoders of one checkpoint running at once, mapped processes shared a single set of file pages while resident loaders added a full private copy each, until the machine ran out and began evicting the shared cache; at capacity the mapped arm sustained 5.5 tokens per second per process and the resident arm 0.08. Time to first token on a roughly 65 GB checkpoint improved 6.4 times against stock loading, of which 1.6 times came from the storage change alone. On a trillion-parameter mixture-of-experts model the dense stage of each token fell from 2.62 to 0.35 seconds, a factor of 7.5, of which 3.8 came from storage and the rest from a packed kernel. Applied to llama.cpp on an AMD integrated part, decode rose from 2.82 to 3.42 tokens per second while peak working set fell from 8.13 to 4.20 GiB.

Those figures were measured on Apple silicon and on one AMD integrated part. To see how much of the memory argument for file-backed weights survives on arm64 Linux, I reproduced the residency half on a 4 GB aarch64 system running Linux 6.8 with no swap, using a 512 MiB file and a warm page cache. Each row runs N readers of the same file at once:

At six readers the copying path had already driven the page cache down from 1405 MB to 643 MB, because the kernel was evicting cached file pages to make room for anonymous copies. At ten it exhausted the machine and the OOM killer terminated three processes, while the mapped path used 184 MB, left 3585 MB available, and kept the model file fully resident. Load time moved the same way: reading 512 MiB out of a warm page cache into a private buffer took 73 to 78 ms across three runs, while mapping took under a tenth of a millisecond and faulting the whole range in on first touch took 7.5 to 9.5 ms.

Where file-backed weights break down

The technique has a narrow and well-defined domain, and the failure modes outside it are not subtle.

• Discrete accelerators. If the device reads host memory across PCIe, every mapped access crosses the bus. The same path measured 39 times slower on a discrete card. The rule is to adopt file pages only where the accelerator can already read them at full speed.
• The residency precondition. The active mapped set must stay in the page cache. Once it does not, every access becomes storage traffic. This failure can be self-inflicted: on one smaller test system, creating the framework copy was itself enough to evict the mapping, after which none of it remained cached.
• Alignment. Tensor ranges must be page-aligned, either because the container was written that way or through a one-time relayout. A partial page cannot be wired to one tensor without also exposing its neighbour.
• Driver ceilings. On the tested integrated part the driver capped a single imported allocation at 2 GiB, with a separate cumulative limit of 12.5 GiB on importable host memory. Sharding satisfies the first and does nothing about the second. Forcing the whole model through one oversized buffer made the adopted path 14 percent slower than stock.
• Workload shape. The reported gains are for low-batch decode, where the same weights are re-read for every token. They do not extend to batched matrix multiplication, prefill, or concurrent serving.
• Immutability is not enforced. The read-only flag on the interchange format records intent, but consumers still accept in-place writes, and behaviour then depends on the substrate: a store to the read-only mapping was silently discarded on one platform, faulted on another, and rejected at import time by a third.
• Data-type expansion survives. File-backed weights remove the copy. A framework with no kernel for the checkpoint’s packed format will still widen the data, and the footprint returns.

Checking file-backed weights on your own board

Whether your device is already paying for a second copy takes a few minutes to determine. Start with the resident-set breakdown of the running inference process; these fields were split out of total RSS long ago and are present on any kernel you are likely to be running.

raghu@techveda.org:~$ pidof my-inference-daemon
2417
raghu@techveda.org:~$ grep -E '^(VmRSS|RssAnon|RssFile|RssShmem)' /proc/2417/status
VmRSS:	  742100 kB
RssAnon:	  698340 kB
RssFile:	   43664 kB

An RssAnon close to the size of your model file is the signature of the copy; a large RssFile instead means the weights are already mapped. Confirm which pages are shared and whether they are clean:

raghu@techveda.org:~$ grep -E '^(Pss|Private_Clean|Private_Dirty|Shared_Clean)' /proc/2417/smaps_rollup

Then check that the model file is actually resident in the page cache, the precondition file-backed weights depend on. The fincore tool from util-linux reports this per file:

raghu@techveda.org:~$ fincore /opt/models/model.gguf
  RES  PAGES  SIZE FILE
 512M 131072  512M /opt/models/model.gguf

If RES is well below SIZE while the workload runs, the mapping is being evicted and file-backed weights will not help until that is fixed. Where the service runs under a memory limit, the same split is visible per cgroup, with anon counting private copies and file counting page cache:

raghu@techveda.org:~$ grep -E '^(anon|file) ' /sys/fs/cgroup/system.slice/inference.service/memory.stat
anon 715100160
file 44711936

If you run llama.cpp, the loader mode is a flag rather than a rebuild. The current option is -lm or --load-mode, taking auto, none, mmap, mlock, mmap+mlock or dio. The default is auto, which maps the model unless a device does not support it, so the CPU-side half of file-backed weights is usually already in place. Note that mlock and mmap+mlock pin the pages, which prevents the eviction mapping otherwise permits.

raghu@techveda.org:~$ llama-cli -m /opt/models/model.gguf --load-mode mmap -n 64
raghu@techveda.org:~$ llama-cli -m /opt/models/model.gguf --load-mode none -n 64

Run both and compare RssAnon. The difference is the size of the copy your device is carrying. The same two commands answer the question for any runtime, which matters because most vendor NPU stacks do not document their loading path: whatever ONNX Runtime, TensorFlow Lite, or your SoC vendor’s SDK does internally, a large RssAnon next to a fully cached model file tells you it is copying, and no documentation is required to establish that. Finally, if your accelerator is programmed through Vulkan, check whether the host-pointer import path exists at all:

raghu@techveda.org:~$ vulkaninfo | grep -i external_memory_host
        VK_EXT_external_memory_host            : extension revision 1

Reasoning about page cache behaviour, mapping flags and reclaim is ordinary Linux memory management applied to a new workload, and it is the part of the stack that decides whether an edge AI product fits its board. Our Linux systems engineering programme covers this material in depth.

Key takeaways

• On unified-memory hardware an ordinary model load leaves the weights in DRAM twice: once as page cache, once as a private copy inside the runtime. Total RSS hides this; only the anonymous versus file split shows it.
• File-backed weights make the mapped file pages the tensor storage, so the pages stay clean, shared between processes, and reclaimable by the kernel.
• Removing the copy is necessary but not sufficient. An implementation that keeps activations on the host or synchronises through it ran 2.3 times slower than the copying path it replaced.
• File-backed weights apply to low-batch decode where the accelerator already reads host DRAM. On a discrete card across PCIe the same path was 39 times slower and should be rejected.
• Everything depends on the mapped set staying resident in the page cache. Check that first with fincore, before changing anything else.

Frequently asked questions

How do I tell whether my device is already paying for a second copy of the model?
Read /proc/<pid>/status for the running inference process and compare RssAnon with RssFile. An RssAnon value close to the size of your model file means the runtime holds a private copy. A large RssFile instead means the weights are already mapped from the file.

Do file-backed weights help on a discrete GPU?
No. If the accelerator reads host memory across PCIe, every mapped access crosses the bus, and the same path was measured 39 times slower than copying into device memory. File-backed weights are for hardware where the accelerator already reads host DRAM at full speed.

Is it enough to just map the model file instead of reading it?
Not on its own. Mapping removes the copy, but an implementation that then round-trips activations through the host, or waits on the host between layers, ran a decode stage 2.3 times slower than the ordinary copying path. Activations must stay on the accelerator and ordering must be established there too.

What breaks if the model file is evicted from the page cache?
Every access to an evicted page becomes storage traffic instead of a memory read. This is why the mapped set must stay resident, and why fincore is the first thing to check.

Does mlock make file-backed weights better?
It changes the trade rather than improving it. Pinning guarantees the pages are never evicted, which suits a latency-sensitive product, but gives up the elasticity that lets the kernel reclaim clean file pages under pressure. That elasticity is what allowed ten concurrent readers to run where the copying path lost three processes to the OOM killer.

Further reading

• Yuan Si, Yufeng Lin, Daming Li and Jialu Zhang, The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks, arXiv:2608.12114 [cs.OS], 2026.
• The /proc Filesystem — Linux kernel documentation for the RssAnon, RssFile and RssShmem fields and for smaps_rollup.
• Control Group v2 — Linux kernel documentation for the anon and file entries in memory.stat.
• mmap(2) — the manual page describing MAP_SHARED and shared file mappings.
• VK_EXT_external_memory_host — Khronos reference for importing host allocations into Vulkan memory objects.
• llama.cpp CLI options — current documentation for the --load-mode loader flag.