GPU Isolation on an Edge Board: Why Partitioning Alone Misses Deadlines

A very common edge design puts two inference processes on one board. One must produce a result on every frame and has a deadline in the product specification; the other can wait. Both want the same GPU. The standard answer is GPU isolation: partition the accelerator so the deadline-bound process gets a fixed share its neighbour cannot take away. On a rack server that is sound. On a power-constrained embedded module it is only half right, and the wrong half decides whether the deadline is met.

One GPU, two processes, one deadline

Take two image classifiers, both converted to TensorRT engines, both at batch size 1 on a Jetson Orin Nano. Run the first alone and find its highest sustainable rate: the largest number of inferences per second at which every inference still finishes inside its slot — the number you would quote in a design review. Now start the second process and raise its request rate step by step. Nothing about the first has changed, yet its inferences begin finishing late, and the share that miss their slot climbs as the neighbour asks for more.

That share is the number to track, and it is not the one most teams measure. A mean latency figure hides this failure: the mean can sit well inside budget while two or three percent of frames arrive after the moment they were needed, which on a machine-vision line or a driver-monitoring camera is the only statistic that matters. So the question is concrete: which GPU isolation mechanism removes the interference?

What the GPU isolation mechanisms actually partition

Three mechanisms exist on current NVIDIA hardware, and they are not three strengths of the same thing: they partition different resources, for different reasons.

Multi-Process Service (MPS) was not built for isolation at all. It exists to remove context-switch cost. Without it, two processes sharing a GPU each hold their own context and the hardware alternates between them, paying for every switch; MPS runs a server giving clients a shared context so their kernels execute concurrently instead of taking turns. A cap is available: NVIDIA documents CUDA_MPS_ACTIVE_THREAD_PERCENTAGE as setting “the portion of the available threads that can be used by the client contexts”. But read what that cap is — a portion of threads. It is not described as a timing guarantee, it reserves no memory bandwidth, and it promises nothing about when a kernel is scheduled. MPS raises throughput, most visibly for large models losing the most to context switching, but it does not hold a deadline.

Multi-Instance GPU (MIG) is the opposite: it splits the GPU physically. NVIDIA’s MIG documentation states that the on-chip crossbar ports, L2 cache banks, memory controllers and DRAM address busses are assigned uniquely to each instance, and lists SM performance isolation, memory protection, memory-bandwidth quality of service and error isolation as properties of the result. The smallest unit it hands out is a GPU Processing Cluster — on the A100, one of seven. This is genuine GPU isolation, paid for in granularity and rigidity: the documentation lists MIG reconfiguration as an idle-time operation, so a partition cannot be resized underneath a running workload.

MIG’s supported-product table is the part embedded teams must read. MIG needs compute capability 8.0 or higher, and the table lists data-centre parts — A100, A30, H100, H200, B200, GB200 and the RTX PRO Blackwell line — plus one embedded entry, the Thor integrated GPU, limited to two concurrent instances. Orin does not appear in that table. If your product ships on an Orin module, MIG is not the mechanism you have.

Green Contexts are the mechanism you do have. Present in the CUDA driver API from CUDA 12.4, a Green Context is a CUDA context restricted to a stated number of streaming multiprocessors; a process inside it runs on those SMs and no others. It is software only, needs no special GPU mode, can be created and destroyed at run time, and its throughput cost was not measurable — unlike MIG, which charges a real overhead. It isolates nothing other than SMs: no memory isolation, no partitioning of the memory subsystem or the interconnect.

Why Green Contexts hand out SMs in fixed steps

A Green Context cannot be given an arbitrary SM count, and the constraint comes from the chip rather than the driver. SMs are grouped for co-scheduling, so a partition must align to that grouping. The driver API exposes this instead of fixing a number: the SM resource carries an sm.smCoscheduledAlignment value and a minimum partition size, and portable code is expected to make its requested count a multiple of the alignment the device reports. On Turing-class and Ampere-class parts, and on Tegra SoCs, that alignment is 2; from compute capability 9.0 it is 8.

The consequence on a small board is blunt. The Orin Nano GPU has eight SMs in total, so with a minimum useful partition of four it supports exactly two non-overlapping Green Contexts. Any design assuming it can carve a small edge GPU into several isolated slices must be checked against the real module’s SM count first.

So query, never hard-code. Read the device’s SM resource with cuDeviceGetDevResource, split it with cuDevSmResourceSplitByCount, build a descriptor with cuDevResourceGenerateDesc, create the context with cuGreenCtxCreate, and take a stream with cuGreenCtxStreamCreate. An SM count of zero puts the split into a discovery mode reporting what the device will really give you, and the split may return fewer groups than requested. Code that ignores the returned count is running a GPU isolation layout it never verified.

The measurement that changes the design

With two Green Contexts of four SMs each on the Orin Nano, the two-process experiment did not give the expected result. All three configurations — the bare shared GPU, MPS, and Green Contexts — produced very similar deadline-miss behaviour, for all three models regardless of size. That last detail is the clue: model size should matter a great deal to contention, so when it stops mattering the bottleneck has moved somewhere unrelated to how the SMs were divided.

Logging frequency, power and temperature through the run located it. The GPU frequency dropped suddenly to close to half its maximum, and at those same moments the module’s power draw approached about 20 W, the ceiling for that board in its highest-power mode during these runs. The board was saturating its power budget.

The mechanism is then plain, and it is the central point here. The partition was correct: four SMs were reserved and the neighbour genuinely could not use them. The deadline was still missed, because a process running on four isolated SMs at half the clock it was characterised at is not the process you characterised. GPU isolation divided the streaming multiprocessors. Nothing divided the watts.

This is not an exotic interaction; it is written into the platform’s own configuration. Each Jetson power mode in /etc/nvpmodel.conf carries lines such as GPU MAX_FREQ and EMC MAX_FREQ — the mode stating in plain text that it caps the GPU and memory clocks. A power mode is a frequency ceiling with a wattage label on it.

The same partition on a board with power to spare

The clean test changes only the power headroom. The platform moved from an Orin Nano to an Orin AGX: same chip architecture, roughly double the maximum power at 50 W in peak mode, a maximum GPU frequency of 1.3 GHz, more memory bandwidth, and sixteen SMs instead of eight. The per-process allocation stayed at four SMs and the frequency was fixed at 1.02 GHz, the Orin Nano’s maximum. Frequency, power and temperature were logged first: the frequency stayed stable and power never came near the ceiling. Only then were the deadline measurements repeated, with the adjustable process pushed past the rate that guarantees its own timing. The share of late inferences fell drastically in every case, reaching close to the GPU isolation physical MIG partitions deliver on the A100.

Same partition size, same model, same software, same clock; the only change was whether the clock could be held. A further run filled the AGX with four partitions of four SMs and four processes, and Green Contexts held up there too.

What the model itself decides

One more factor belongs to the network rather than the platform. Compute-bound models tolerate having resources taken away, because the SM count was never their limit; small memory-bound models are the ones that suffer where there is no memory isolation, and Green Contexts provide none.

MIG’s own overhead appears in the same comparison. Measured as the maximum sustainable inference rate on the A100, the bare GPU and MPS matched for every model, while a three-GPC MIG partition fell from 147 to 123 for a lightweight mobile network and from 95 to 76 for a vision transformer. The strongest GPU isolation available costs the most throughput, and costs it most on the small models that needed isolating least.

Where GPU isolation does not help, and what it costs

• MIG is not complete isolation either. At high request rates, late inferences appeared on the A100 even with MIG partitions in place.
• MIG cannot be resized under load. Reconfiguration is an idle-time operation, so it does not suit a system whose requirements change while it runs.
• MPS is not an isolation mechanism, and should not carry a high-criticality process.
• Green Contexts isolate SMs only; a memory-bound neighbour can still cost you.
• These numbers are from Jetson Orin and A100 hardware, TensorRT engines, batch size 1. Treat the mechanism as general and the magnitudes as specific.

Two things were not established. The 20 W figure is what the board drew as it saturated, not a datasheet rating, so read it as the observed saturation point for that board. And the share of late inferences is published as graphs rather than numbers, which is why the table above gives direction and magnitude instead of percentages that were never stated.

How to check this on your own board

Four checks will tell you whether GPU isolation can work on your hardware. First, find the power mode, which is the frequency ceiling you are working under:

raghu@techveda.org:~$ sudo /usr/sbin/nvpmodel -q
raghu@techveda.org:~$ grep -E 'POWER_MODEL|GPU MAX_FREQ|EMC MAX_FREQ' /etc/nvpmodel.conf

Second, watch the GPU clock while your real workload runs, not while a benchmark runs for thirty seconds. On Jetson, tegrastats reports it. The GR3D_FREQ field has the form X%@[Y1,Y2]: X is the proportion of time the GPU was active, and Y1 and Y2 are the megahertz frequencies of the GPU’s two GPCs, each with its own frequency controller on AGX Orin parts:

raghu@techveda.org:~$ sudo tegrastats --interval 500 --logfile /tmp/clocks.txt &
raghu@techveda.org:~$ grep -o 'GR3D_FREQ [^ ]*' /tmp/clocks.txt | sort | uniq -c | sort -rn | head

More than one frequency value under load means your characterisation and production runs are not measuring the same machine.

Third, on any embedded Linux board and not only a Jetson, read the GPU’s frequency domain through the standard devfreq interface. The node name is SoC-specific, so list the nodes and identify the GPU one:

raghu@techveda.org:~$ ls /sys/class/devfreq/
raghu@techveda.org:~$ cd /sys/class/devfreq/<gpu-node>
raghu@techveda.org:~$ cat name governor cur_freq target_freq min_freq max_freq
raghu@techveda.org:~$ cat available_frequencies

The kernel ABI documents trans_stat as the time spent in each state plus the number of transitions between states. That table is the honest record of whether the clock stayed where you expected. Reset it, run the workload, read it back:

raghu@techveda.org:~$ echo 0 | sudo tee trans_stat
raghu@techveda.org:~$ ./run_my_real_workload.sh
raghu@techveda.org:~$ cat trans_stat

Fourth, remove the clock as a variable before comparing mechanisms, so that you measure GPU isolation and not power behaviour. max_freq overrides both the governor and min_freq, so writing a fixed value there pins the domain. On Jetson, save the current settings first so you can restore them:

raghu@techveda.org:~$ sudo /usr/bin/jetson_clocks --store
raghu@techveda.org:~$ sudo /usr/bin/jetson_clocks --show

Then repeat the two-process experiment with the clock pinned. If the late inferences disappear, GPU isolation was never the problem.

What this changes for a product

Most teams use the design order the wrong way round: GPU isolation is chosen during architecture, and power behaviour examined during integration. On a power-constrained module the clock ceiling decides the outcome before the partition gets a vote.

A better order is short. Measure whether the clock holds under the real combined workload. If it does not, raise the power budget, move to a module with headroom, or reduce the load until the clock is stable, and only then choose a mechanism. If it does hold, Green Contexts are a reasonable choice on Orin-class hardware. Size the partitions from the SM count the device reports, and keep memory-bound neighbours away from the deadline-bound process. Reasoning across the accelerator, the clock domains, the power mode and the kernel’s power-management interfaces is one skill, and our training tracks are built around that cross-layer work on real hardware.

Key takeaways

• GPU isolation mechanisms partition compute units. None of them partitions the board’s power budget, and the power mode caps the GPU clock.
• Track the share of inferences that miss their deadline, not mean latency.
• MPS removes context-switch cost and is not an isolation mechanism. MIG isolates compute and memory but is rigid, and Orin is absent from its supported-product table. Green Contexts isolate SMs only, at no measurable cost, and must align to the SM grouping the device reports.
• The same four-SM partition failed to hold the deadline on a power-constrained module and came close to MIG-level isolation on a larger one with the clock pinned, so pin the clock before comparing mechanisms and verify it with trans_stat.

Frequently asked questions

Why did partitioning the GPU not stop the deadline misses?
The partition divided the streaming multiprocessors while the board was running out of power. With both processes running, the module approached its power ceiling and the GPU clock dropped to roughly half its maximum, so the process on its four reserved SMs was no longer running at the clock it was characterised at.

Can I use MIG on a Jetson Orin module?
MIG requires compute capability 8.0 or higher, and its supported-product table lists data-centre GPUs plus the Thor integrated GPU, limited to two concurrent instances. Orin does not appear in that table, so on an Orin module Green Contexts are the mechanism to plan around.

How many Green Contexts can a small edge GPU support?
That depends on its SM count and the alignment the device reports. An Orin Nano GPU has eight SMs, so with a four-SM minimum partition it supports exactly two. Query it with cuDeviceGetDevResource and check how many groups the split call returns.

What is the fastest check to run on my own board?
Log tegrastats through your real combined workload and see whether the GR3D_FREQ values stay constant. If they move, pin the clock first, because comparing mechanisms on an unstable clock measures power behaviour rather than GPU isolation.

Further reading

• Juan José Martín, José Flich and Carles Hernández, “Performance Isolation for Inference Processes in Edge GPU Systems”, IEEE Access, vol. 14, pp. 156028-156042, 2026 (arXiv:2601.07600) — the source of the measurements above.
• CUDA Driver API: Green Contexts
• NVIDIA Multi-Instance GPU User Guide (PDF)
• NVIDIA Multi-Process Service: environment variables
• Linux kernel ABI: sysfs-class-devfreq
• Jetson Linux Developer Guide: Tegrastats Utility
• Jetson Linux Developer Guide: Platform Power and Performance, Orin series
• Concurrent Inference Stops Scaling at Half Your CPU Cores — the CPU side of the same board.