An IRQ Storm on an Embedded Gateway

By the time it reached us the failure was already a finished fact, so the work was to reconstruct it backward from the surviving artifacts — a persisted kernel log, a /proc/interrupts snapshot, and the source history — to the latent defect and the trigger that combined to cause it.

The failure as it arrived

A batch of industrial gateways running the latest firmware would lose Wi-Fi connectivity and never recover until the unit was power-cycled. Ethernet stayed up. The failure only appeared on units that were actively streaming from the on-board video capture bridge, and only after some minutes of load; units on the previous firmware never showed it.

We could not reproduce it on demand at first, but the gateways persist the kernel log to eMMC through systemd-journald, so the messages survived the reboot.

What the surviving artifacts showed

The persisted log held one clear splat. Reading it back:

raghu@techveda.org:~$ journalctl -k -b -1 | grep -A18 'nobody cared'
irq 61: nobody cared (try booting with the "irqpoll" option)
CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.12.0-tvgw #1
Hardware name: TechVeda IMX8MM Gateway (DT)
Call trace:
 dump_backtrace+0x9c/0x100
 show_stack+0x20/0x38
 dump_stack_lvl+0x74/0x94
 dump_stack+0x18/0x24
 __report_bad_irq+0x54/0x110
 note_interrupt+0x2fc/0x360
 handle_irq_event_percpu+0x54/0x100
 handle_irq_event+0x50/0xb8
 handle_fasteoi_irq+0xb8/0x220
 generic_handle_domain_irq+0x34/0x50
 gic_handle_irq+0x4c/0xd8
handlers:
[<0>] tvcap_irq [tvcap]
[<0>] ath_isr [ath9k]
Disabling IRQ #61

Two facts are stated directly. The kernel decided interrupt 61 was stuck (nobody cared) and then disabled the line (Disabling IRQ #61). The handlers: list shows two drivers registered on that one line: our new tvcap capture driver and the ath9k Wi-Fi driver. That sharing is expected here: by default ath9k does not enable MSI, so it registers its PCIe interrupt as a shared legacy INTx line with IRQF_SHARED, and on this board the capture bridge was routed to the same INTx line — so both ended up on IRQ 61.

A /proc/interrupts snapshot from the same unit, taken shortly before the hang, made IRQ 61 stand out:

raghu@techveda.org:~$ cat proc-interrupts.txt
           CPU0       CPU1       CPU2       CPU3
 61:    4001991          0    3998540          0     GICv3  59 Level     tvcap, ath9k

The count was in the millions and climbing on the CPUs that took the line, and the type was Level. A level-triggered line re-firing at that rate is a device holding its interrupt asserted while nobody clears it.

How the kernel decides an interrupt line is stuck

The mechanism behind the splat lives in kernel/irq/spurious.c. A handler returns one of three values, defined in include/linux/irqreturn.h: IRQ_NONE when the interrupt was not from this device or was not handled, IRQ_HANDLED when this device raised it and the handler serviced it, and IRQ_WAKE_THREAD to wake a threaded handler.

On a shared line, returning IRQ_NONE is normal. When an interrupt arrives, the core calls every handler on the line in turn; each checks whether its own device is the source and returns IRQ_NONE if not. The interrupt counts as handled as long as one handler claims it with IRQ_HANDLED.

The problem begins when no handler claims a real, persistent interrupt. Each time the combined result for a line is IRQ_NONE, note_interrupt() increments an unhandled counter. The rule, quoted from the source comment, is: “If 99,900 of the previous 100,000 interrupts have not been handled then assume that the IRQ is stuck in some manner.” When that threshold is crossed, the core prints the nobody cared message, lists the handlers, prints Disabling IRQ #N, and disables the line.

Two details matter. First, the storm must be dense: if more than a tenth of a second passes between unhandled interrupts, the counter resets, so only a device that pins the line and re-fires continuously reaches 100,000 in time. Second, on an ARM GIC these shared SPIs use the handle_fasteoi_irq flow, which signals end-of-interrupt at the controller after the handlers run but does nothing at the device. If the device still asserts the line, the GIC sees it pending again and re-enters — the loop the counter measured.

Reconstructing the shared IRQ storm

The log named tvcap, and tvcap was new. The source history confirmed which change introduced it:

raghu@techveda.org:~$ git log --oneline -1 -- drivers/media/tvcap/tvcap_core.c
a1f4c92 media: tvcap: add capture bridge driver for gateway rev C

That commit was part of the firmware bump the failing units were running. It was the only interrupt-handling code that had changed. The handler read like this:

/* drivers/media/tvcap/tvcap_regs.h */
#define TVCAP_ISR         0x0C        /* interrupt status, W1C */
#define TVCAP_INT_FRAME   BIT(0)

/* drivers/media/tvcap/tvcap_core.c */
static irqreturn_t tvcap_irq(int irq, void *dev_id)
{
        struct tvcap *tv = dev_id;
        u32 status = readl(tv->base + TVCAP_ISR);

        if (!(status & TVCAP_INT_FRAME))
                return IRQ_NONE;

        tvcap_handle_frame(tv);
        writel(TVCAP_INT_FRAME, tv->base + TVCAP_ISR);   /* ack */
        return IRQ_HANDLED;
}

The structure is correct for a shared handler: read the status register, return IRQ_NONE if this device did not raise the interrupt, otherwise service the frame and write the bit back to acknowledge it. The defect was the offset. The datasheet places the interrupt status register at 0x1C; offset 0x0C is a read-only device-identity register that reads back 0x00000000 on this silicon stepping. So status was always zero, the frame bit was never seen, and the handler returned IRQ_NONE every time — never reaching the writel() that clears the source.

That explains the trigger. The wrong offset was harmless while the bridge never raised an interrupt, which is why idle units and earlier firmware never failed. Once streaming began, the bridge asserted its frame-done interrupt and held the shared, active-low line. tvcap_irq read zero and declined it; ath_isr checked its own registers, saw nothing, and also declined it. No handler cleared the bridge, so the line stayed asserted and re-fired continuously until the counter reached the threshold and the core disabled IRQ 61. Because ath9k shared that line, it stopped receiving its own interrupts from that moment, which is why Wi-Fi died and stayed dead until reboot.

Root cause

A single wrong register offset made the new driver’s handler always return IRQ_NONE and never acknowledge its device. On a shared, level-triggered line, one unacknowledged persistent interrupt is enough to disable the line for every device on it. The Wi-Fi failure was a consequence, not the cause.

The fix and what would have caught it earlier

The change was one line — the correct offset from the datasheet:

-#define TVCAP_ISR         0x0C        /* interrupt status, W1C */
+#define TVCAP_ISR         0x1C        /* interrupt status, W1C */

With the correct offset, the handler reads a non-zero status when its device fires, services the frame, and writes the bit back to clear the source, so the device deasserts the line and returns IRQ_HANDLED. After the fix, /proc/interrupts for IRQ 61 advanced at the real frame rate, the splat did not return, and Wi-Fi stayed up through extended video load.

Several inexpensive checks would have surfaced this during bring-up rather than in the field:

• Request the interrupt without IRQF_SHARED during initial bring-up, so a stuck line trips nobody cared against that one driver in isolation instead of hiding behind an unrelated device.
• Watch /proc/interrupts while exercising the new device. A count jumping by roughly a hundred thousand per second signals an unacknowledged interrupt long before the line is disabled.
• Check every register offset against the datasheet; a value copied from a sibling register block is easy to miss in review.
• Read the status register back after the acknowledge write and confirm the source bit cleared, catching a device that fails to deassert.

Interrupt handling on shared lines, and the register discipline that keeps a handler honest, are exactly the kind of thing we cover in depth in our Linux Device Drivers training.

Key takeaways

• irq N: nobody cared then Disabling IRQ #N means about 99,900 of the last 100,000 interrupts on that line went unclaimed, so the core disabled it.
• On a shared line, returning IRQ_NONE is correct when the interrupt is not yours; the failure is when no handler claims a real, persistent interrupt.
• A handler that never acknowledges its device on a level-triggered line holds the line asserted and re-fires until the core gives up — and every device on that shared line loses interrupts, so the visible victim is often not the culprit.
• /proc/interrupts is the fastest first look: a runaway count on a Level line points to an unacknowledged source.

Frequently asked questions

What does the kernel message irq N: nobody cared mean?
The kernel’s spurious-interrupt detector in kernel/irq/spurious.c saw that about 99,900 of the last 100,000 interrupts on that line were not claimed by any handler. It treats the line as stuck, prints the registered handlers, and disables the line.

Why did disabling one interrupt line take down an unrelated device?
The two devices shared a single line. When the core disables a shared line, it stops delivering interrupts to every driver on it. The Wi-Fi driver did nothing wrong; it lost its interrupts because it shared the line with the driver that caused the storm.

Is returning IRQ_NONE a bug?
No. On a shared line each handler must return IRQ_NONE when its own device did not raise the interrupt. The bug was that the capture handler returned IRQ_NONE even when its device had raised the interrupt, because it read the wrong register and never cleared the source.

Why did the failure only appear under video load?
The wrong offset was dormant until the capture bridge actually raised an interrupt. Idle units never asserted the line, so the storm only started once video streaming began.

Further reading

• Linux generic IRQ handling — kernel.org, Documentation/core-api/genericirq.rst (flow handlers, including handle_fasteoi_irq and handle_level_irq).
• kernel/irq/spurious.c — the source of note_interrupt(), the 99,900-of-100,000 threshold, and the “nobody cared” / “Disabling IRQ” messages.
• include/linux/irqreturn.h — the definitions of IRQ_NONE, IRQ_HANDLED and IRQ_WAKE_THREAD.
• request_threaded_irq() — the API for registering handlers and the meaning of IRQF_SHARED.