Concurrent Inference Stops Scaling at Half Your CPU Cores

Somebody hands you a board that is already running one model and asks whether it can run a second, and then a third. You do the obvious check. You watch the GPU utilisation figure while the existing model runs, see it sitting near 100 per cent, and conclude that the accelerator is saturated and there is no room. That conclusion is usually wrong, and the way concurrent inference actually degrades on an edge SoC is worth understanding, because the thing that breaks is on your side of the hardware boundary. It is the CPU, the run queue and the scheduler, and those are all objects you own.

What the GPU utilisation number actually measures

The utilisation figure reported for an integrated GPU is a duty cycle: the fraction of the observation window in which the GPU had at least something to do. This is not an interpretation. NVIDIA’s own tegrastats documentation defines the GR3D_FREQ field as the proportion of GPU activation time in a period, measured against the current running frequency. It is a time-based measure of whether work was assigned, not a measure of how much work the GPU did, and not a measure of how much more it could have taken.

The gap between those two things is large. Profiling a ResNet50 model in fp16 on a Jetson Orin Nano, with the model already converted to a TensorRT engine, reports over 98 per cent GPU utilisation while GPU memory usage stays below 3 per cent. Going one level down changes the picture completely. Streaming multiprocessor active cycles, meaning the fraction of cycles in which a multiprocessor had at least one warp resident, do run high, typically 75 to 100 per cent. But the issue-slot figure, meaning the fraction of cycles in which an instruction was actually issued, sits around 25 per cent on average and never rises above 80 per cent for any model at any precision. Tensor core utilisation is lower still, around 25 to 30 per cent for most configurations.

So the hardware is occupied and simultaneously idle. Warps are resident on the multiprocessors, which is what makes the duty cycle read high, but three cycles out of four issue nothing because the warps are stalled. A duty-cycle metric cannot distinguish “doing work” from “waiting with work assigned”, and on an edge board those two states look identical from the outside.

This is the same class of error as reading a load average and concluding a machine is busy. The number is real, the measurement is correct, and the conclusion drawn from it is not supported. It also means the capacity question you were trying to answer, whether the board has room for concurrent inference, has not been answered at all.

What happens when you add the third and the fourth process

The useful way to test concurrent inference is to hold the model fixed and vary only how the work is packaged: more images per batch in one process, against the same images spread across more processes. The two are not interchangeable, and the difference is large.

Take a YOLOv8n detection model on Jetson Orin Nano. With one process, throughput per process is around 210 images per second at batch size 1, rising to around 320 at batch size 16. Larger batches help, with diminishing returns, which is what you would expect. Now hold the batch and increase the process count from one to eight: throughput per process falls to nearly 10. That is not a gentle degradation from sharing. That is a collapse.

The mechanism becomes visible when you time a single unit of inference work, the execution context that handles one batch from submission to completion. At one or two concurrent processes, that duration is stable at 1 to 2 milliseconds. At four processes it grows by approximately 30 times. At eight, by approximately 70 times.

Note what is missing from that table. Three processes were not measured. The tested points are one, two, four and eight, so the exact position of the knee between two and four is not established. What is established is that it exists, and that it is below four.

Why concurrent inference breaks on the CPU side

Decompose the time to process one batch and it separates into four parts: the time to launch each GPU kernel, the time for a CPU thread to start running again after it was preempted, the actual computation, and the time the process spends blocked. Only the third of those is accelerator work. The other three are scheduling.

Each of the three CPU-side terms grows with process count, and they grow for different reasons.

Blocking time. At one or two processes it is negligible. At four or eight it becomes the dominant term, with individual blocking intervals typically 1 to 2 milliseconds each. Compare that against an execution context that used to take 1 to 2 milliseconds in total: the process now spends more time off the CPU than it previously spent completing the whole batch.

Thread rescheduling and kernel launch. An individual GPU kernel launch is short, roughly 20 to 100 microseconds. There are many of them per batch, and each one is issued by a CPU thread that may have been preempted since the last one. The cumulative overhead becomes substantial specifically because of frequent context switching, not because any single launch got slower.

Migration and cache. Once processes are being time-sliced, the load balancer moves them between cores. Each migration costs the L1 and L2 working set the task had built up, which raises the computation term itself. This one comes with a caveat worth stating: the rise in L1 and L2 miss rates is offered as the cause, but no miss-rate measurement is published alongside it. Treat it as the mechanism that fits, not as a number you can quote.

None of this is specific to machine learning. It is ordinary behaviour of the kernel’s fair-class scheduler on an oversubscribed run queue, and it would look the same for four database processes or four video encoders. Which fair-class scheduler is worth knowing before you go looking for tuning advice: the Jetson Linux 36.x releases that JetPack 6 ships are built on Linux 5.15, so they run CFS, while mainline replaced the CFS implementation with EEVDF in 6.6. The class of problem is identical either way, a runnable task waiting behind other runnable tasks, but the pick rule is not, and guidance written against one does not automatically carry to the other. What makes the problem unexpected in an inference context is that the monitoring tool people use first reports on the GPU, so the GPU is where they look, and the GPU reports nothing about run-queue waiting.

This is the CPU half of a pattern that appears repeatedly on edge boards, where the accelerator has capacity to spare and something else sets the ceiling. The memory half of it is the part of a model the kernel cannot reclaim, where the binding constraint is an anonymous mapping that no amount of page-cache pressure will free. The structure is the same in both cases: the number on the accelerator is healthy, and the board still cannot take the workload.

The explanation that does not hold up

The rule that emerges from the measurements is stated plainly: if the number of processes is at or below half the available CPU cores, batch duration stays stable and execution contexts are not preempted; above that, batch duration and kernel launch time both grow, and the growth looks exponential in the process count. Larger batches, by contrast, stabilise batch duration and reduce kernel launch time.

That rule fits the data on both boards. Orin Nano has six cores and breaks at four. Jetson Nano has four cores and shows roughly a doubling of batch duration at four processes. Both collapse points sit above half the core count.

The explanation offered for the rule is that these Arm CPUs use a big core and little core arrangement, with three cores on Orin Nano dedicated to heavy loads and two on Jetson Nano. That description is wrong in one way and closer to right in another, and the difference decides whether the rule ports to your board.

It is wrong about core types. All six Orin Nano cores are Cortex-A78AE v8.2. There is no smaller companion core, so this is not a big.LITTLE arrangement and no cores are set aside for light work. Jetson Nano is four Cortex-A57 cores, also a single type.

It is closer to right about topology. NVIDIA’s Jetson Orin Nano series datasheet describes the CPU as a heterogeneous multi-processing architecture built from two clusters, one of four cores and one of two, with 64 KB of L1 instruction cache, 64 KB of L1 data cache and 256 KB of private L2 per core, and a separate 2 MB of cluster-level L3 for each cluster. NVIDIA’s word “heterogeneous” here refers to that cluster topology, not to differing core designs. The shorter marketing specification, six cores with 1.5 MB L2 and 4 MB L3, is the same numbers with the clusters summed and the boundary hidden.

Two clusters strengthen the migration argument rather than weakening it. A task moved within a cluster loses its L1 and L2 working set but keeps the cluster’s L3. A task moved across the boundary loses the L3 as well and pays for coherency traffic between clusters. The cost of a migration on this part is therefore not uniform, and a four-plus-two split is a credible reason for a knee somewhere in the middle of the core count. It is not evidence for a knee at three, which is what the source’s own explanation implies and which matches neither cluster.

So take the empirical rule, discard the big.LITTLE reasoning, and keep the cluster boundary in mind as a real structure the scheduler moves tasks across. The distinction matters when you port the rule. If you believe the cause is a three-core performance cluster, you will predict a knee at three on any six-core board. If the cause is ordinary oversubscription combined with a non-uniform migration cost, the knee depends on how much CPU each process demands and on where the cluster boundary falls, and both vary by part, by model, by preprocessing and by runtime. On these two boards the competing explanations happen to predict a similar number. On your board they will not. Measure it rather than inheriting it.

One more constraint sits underneath all of this and is easy to hit first. Each process allocates its own copy of the engine and its own working buffers out of the same unified memory the CPU uses. Attempting four processes of the segmentation model on Jetson Nano did not merely run slowly; it exhausted memory and the system rebooted. On a board with unified memory, process count is a memory decision before it is a scheduling decision.

Why more processes is the tempting answer anyway

It is worth asking why anyone fans the work out across processes instead of batching it inside one. Three reasons, and they are all reasonable. Batching only helps within a single engine, so a detector and a classifier cannot share a batch, and the simplest way to get two execution contexts is two processes. One process per model also gives isolation: a crash does not take the others down, and each can be restarted, upgraded and resource-limited on its own. That is good practice everywhere else in an embedded system.

The third reason is that the spatial alternative was, for years, unavailable. On discrete NVIDIA GPUs a multi-process service lets several processes share the device by partitioning it spatially rather than taking turns. That was absent on Tegra, and the measurements above were taken on a system without it, so what they describe is time multiplexing in its pure form, which is precisely the mode in which preemption and deferred scheduling appear.

Check this one against your own stack rather than inheriting it, because it has changed. NVIDIA’s CUDA for Tegra documentation states that multi-process service support came to Tegra platforms with CUDA 12.5, which reaches Orin-class modules through JetPack 6.1 and later. If you are on that combination, spatial sharing is an option the measurements did not have, and it is worth testing against a process-count budget rather than assuming time multiplexing is the only mode available. If you are on an older JetPack, or on a pre-Orin module, it is not available and the budget is the whole answer. Either way the response is not to avoid multiple processes but to bound their number deliberately, on evidence from your own board.

How to check concurrent inference on your own board

You do not need a GPU profiler for this, and you should not start with one. The claim under test is that inference processes are waiting for CPU, and the kernel already measures that. Start with your inference process running alone, then add instances and watch these four things move.

First, run-queue waiting time per process. Scheduler statistics are off in many production kernel configurations, so enable them first:

raghu@techveda.org:~$ cat /proc/sys/kernel/sched_schedstats
0
raghu@techveda.org:~$ sudo sysctl -w kernel.sched_schedstats=1
kernel.sched_schedstats = 1
raghu@techveda.org:~$ cat /proc/1847/schedstat
41883294617 9127740233 58211

The three fields are time spent on the CPU in nanoseconds, time spent waiting on a run queue in nanoseconds, and the number of timeslices run. In the reading above the process has waited about 9.1 seconds against about 41.9 seconds of CPU time, which is roughly 18 per cent of its lifetime spent runnable and not running. Sample this before and after adding a process and take the difference. If the second field climbs faster than the first as you add instances, your inference processes are queuing for CPU and the accelerator is not the problem.

If the file is absent or reads as zeros after enabling the sysctl, the kernel was built without CONFIG_SCHEDSTATS. On a vendor BSP that is common, because the symbol lives under Kernel hacking, in the Scheduler Debugging submenu, and depends on CONFIG_DEBUG_KERNEL. Turning it on costs very little at runtime and it is worth having in a development image.

Second, involuntary context switches. These distinguish “my process gave up the CPU to wait for the accelerator” from “my process was thrown off the CPU”:

raghu@techveda.org:~$ grep ctxt /proc/1847/status
voluntary_ctxt_switches:        184203
nonvoluntary_ctxt_switches:     2941

A process feeding an accelerator should show mostly voluntary switches, because it submits work and sleeps. When the non-voluntary count starts rising steeply as you add instances, the scheduler has begun time-slicing your inference processes against each other, and that is the transition the batch-duration collapse is made of.

Third, CPU pressure for the whole board:

raghu@techveda.org:~$ cat /proc/pressure/cpu
some avg10=31.42 avg60=27.08 avg300=19.55 total=884213975
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The some line is the share of time in which at least one task was stalled waiting for CPU. Watch avg10 as you add instances. For CPU, the full line is undefined at the system level and is reported as zero, so ignore it here; if you want per-workload figures, cgroup v2 exposes the same format in each group’s cpu.pressure file. If /proc/pressure does not exist at all, this kernel was built without pressure stall information.

Fourth, migrations, which is the term that raises computation time without appearing in any accelerator metric:

raghu@techveda.org:~$ sudo perf stat -e context-switches,cpu-migrations,task-clock -p 1847 -- sleep 30

Then the two experiments worth running. Pin each inference process to its own core and see how much of the collapse disappears:

raghu@techveda.org:~$ taskset -c 2 ./infer --model detector.engine
raghu@techveda.org:~$ grep Cpus_allowed_list /proc/1847/status
Cpus_allowed_list:      2

And re-run the throughput comparison with the work repackaged: same total images per second target, but expressed as a larger batch in fewer processes. If throughput per process rises and run-queue waiting falls, you have confirmed the effect on your own hardware, with your own model and your own runtime, and you now have a number you can design against.

What this changes for a product

The first consequence is a sizing rule. Do not size a board for concurrent inference by its accelerator figure alone. A module advertised with a large TOPS number and a modest core count can be CPU-bound for your workload long before the accelerator is, and no amount of model optimisation will recover it. When you compare candidate modules, compare cores per concurrent model, not only TOPS.

The second is an architecture choice to make deliberately. If several streams run the same model, one process with a larger batch beats several processes with small ones, and the measurements say so twice: larger batches raise throughput per process and stabilise batch duration, while more processes do the opposite. If the models genuinely differ, then process count is a budget, and cores set that budget, not accelerator capacity.

The third is that once you have chosen a number, enforce it. Pin the inference processes and use a cgroup v2 cpuset to keep everything else off those cores, so that a logging daemon or an update agent waking up does not push you across the knee at the worst moment. This is the reasoning behind CPU isolation for a real-time task, applied to a throughput-sensitive one.

The fourth is about what you monitor in the field. A dashboard showing accelerator utilisation will report a healthy board while inference latency doubles. Export the run-queue wait from /proc/<pid>/schedstat and the non-voluntary context-switch count instead, or the CPU pressure figure if you want a single number. Those move when concurrent inference starts to degrade. The utilisation figure does not.

What is not settled here

Several things are open, and they bound how far you should carry these numbers.

The software stack is not pinned. The measurements record no JetPack or L4T version, no CUDA or TensorRT version, no nvpmodel power mode and no statement of whether the clocks were locked. That last one matters a great deal on Orin Nano, which has power modes between 7 and 25 watts, and dynamic voltage and frequency scaling was observed capping board power near 7 watts and reducing throughput to stay under it. A different power mode would move the absolute numbers.

The measurement itself was intrusive at the detailed level. Collecting the low-level counters reduced throughput by 50 per cent, so the fine-grained figures describe a system that was itself running slower than production.

The throughput figures are an upper bound by construction. The harness pre-enqueues a batch, which removes CPU-side image preprocessing from the critical path. A real pipeline that decodes, resizes and normalises frames on the CPU adds exactly the kind of CPU work that makes this problem worse, so a shipping product running concurrent inference should expect the knee earlier, not later.

Pinning was not tested. The measurements establish that the collapse happens and what it is made of; they do not establish how much of it CPU affinity or a cgroup cpuset would recover. That is offered here as the experiment to run, not as a result already in hand.

The concurrency options have moved since the measurements were taken. They describe a system with no spatial multi-process sharing on the GPU, which was the state of Tegra for years. Multi-process service support has since reached Tegra with CUDA 12.5, so on Orin with JetPack 6.1 or later a reader has a mechanism the measured system did not. The scheduling arithmetic still applies to the CPU side of the work, but do not assume the accelerator-sharing conclusion is final on a current stack.

The results are GPU results on two Jetson modules. The scheduling behaviour is a Linux property and will appear on any oversubscribed board. The specific numbers will not transfer to an i.MX with an eIQ Neutron NPU or to an RK3588, because how much CPU each accelerator demands per unit of work differs, and that demand is what sets the knee. The method transfers. The constants do not.

Key takeaways

• Accelerator utilisation on an edge board is a duty cycle, not a work measure. A reading above 98 per cent coexists with tensor cores near 25 per cent and issue slots near 25 per cent.
• Concurrent inference degrades sharply once processes exceed roughly half the CPU core count: batch duration grew about 30 times at four processes and about 70 times at eight on a six-core Orin Nano.
• The knee is a CPU budget, not an accelerator budget. Size a board for concurrent inference by cores per model as well as by TOPS.
• The causes are all CPU-side — blocking on the run queue, thread wake-up after preemption, per-launch overhead multiplied by context switching, and cache loss from migration.
• The published explanation for the rule, a big core and little core split, does not match the hardware: all six Orin Nano cores are Cortex-A78AE. But the part does have two clusters, four cores and two, each with its own L3, so migration cost is not uniform. Take the rule, correct the reason, and measure the knee on your own board.
• Batching and process count are not interchangeable. Larger batches raised throughput per process from about 210 to about 320; going from one process to eight dropped it to about 10.
• Measure it from the kernel: run-queue wait in /proc/<pid>/schedstat, nonvoluntary_ctxt_switches in /proc/<pid>/status, and /proc/pressure/cpu. Then pin the processes and hold the cores with a cgroup cpuset.

Frequently asked questions

If GPU utilisation reads 99 per cent, is the accelerator really saturated?
No. That figure is GPU compute time divided by wall-clock time, so it reports that the GPU had something assigned, not that it was doing useful work. On Jetson Orin Nano a reading above 98 per cent was measured alongside tensor core utilisation near 25 per cent and instruction issue slots near 25 per cent.

How many concurrent inference processes can one edge board run?
The measured guidance is that batch duration stays stable while the process count is at or below roughly half the available CPU cores, and degrades sharply above that. On a six-core Jetson Orin Nano the collapse appeared at four processes. Three processes was not tested, so treat the boundary as something to measure on your own board rather than a fixed number.

Is it better to increase the batch size or to run more processes?
Increase the batch size where the model allows it. Larger batches raised throughput per process from about 210 to about 320 images per second for a YOLOv8n model and stabilised batch duration, while increasing the process count from one to eight cut throughput per process to nearly 10.

Which Linux interfaces show this problem?
Field 2 of the per-process schedstat file gives time spent waiting on a run queue, nonvoluntary_ctxt_switches in the per-process status file shows when the scheduler is throwing your process off the CPU, and /proc/pressure/cpu gives a whole-board figure. Scheduler statistics may need enabling first through the kernel.sched_schedstats sysctl.

Does this apply to NPUs as well as GPUs, and to other vendors?
The scheduling behaviour is a Linux property and will appear on any board where concurrent inference processes are oversubscribed to cores. The specific figures quoted here were measured on Jetson GPU modules, and how much CPU time each accelerator demands per unit of work differs by design, so the position of the knee has to be measured per platform.

Further reading

• Abhinaba Chakraborty, Wouter Tavernier, Akis Kourtis, Mario Pickavet, Andreas Oikonomakis, Didier Colle — “Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson — Extended”, arXiv:2508.08430 [cs.DC], August 2025
• Scheduler Statistics, including the /proc/<pid>/schedstat fields — The Linux Kernel documentation
• PSI, Pressure Stall Information — The Linux Kernel documentation
• The /proc Filesystem, Table 1-2 status fields — The Linux Kernel documentation
• Documentation for /proc/sys/kernel/, including sched_schedstats — The Linux Kernel documentation
• Jetson Orin module specifications — NVIDIA
• tegrastats utility, including the GR3D_FREQ field — NVIDIA Jetson Linux Developer Guide
• CUDA for Tegra application note, including multi-process service support — NVIDIA