Embedded Linux Device Heartbeat Design: How Often, What to Report, and What Failure Means

Most embedded products acquire a device heartbeat early and never revisit it. A timer fires every thirty seconds, a small message goes to the backend, and the backend marks a device offline after three of them fail to arrive. It works well enough to ship. Then the fleet reaches a few thousand units on cellular links, and the operations team starts each morning with a list of devices marked down, nearly all of which were fine the whole time. At the same time, devices that have genuinely stopped doing their job sit on the dashboard in green, because the one process still sending the heartbeat is the one process that never broke.

This article is about the design decisions behind that message: how often to send it, what to put in it, and what a failure to receive it actually means. It applies to embedded Linux devices reporting to a remote backend over a link the device does not control — cellular, customer Wi-Fi, anything behind NAT — running systemd as the service manager. It does not apply to a device on a wired network with an operator beside it, where a missing device heartbeat and a broken device are close enough to the same thing.

What a missed device heartbeat actually proves

Take the common design literally. The device runs a reporting process. Every thirty seconds it composes a message and sends it. The backend records arrivals, and if three consecutive expected arrivals do not happen, it raises an alert saying the device is offline.

Work backwards from that alert. For the message to arrive, all of the following must hold: the process is scheduled and reaches its send call; the network stack accepts the packet; the radio has a usable connection; the carrier delivers it; the backend’s ingress accepts it; and the backend records it against the right device. The alert fires when any one of those fails. A device heartbeat is therefore a single indicator standing in for six independent mechanisms, and it cannot say which one moved.

In practice the first of the six — the process running — is the least likely to fail, and the middle ones are the most likely. A device on a moving vehicle loses its connection routinely. A device on a customer’s site sits behind a firewall that someone reconfigures on a Tuesday. The alert says “device offline” and the engineer reads it as “device broken”, when the common cause is that the path was unavailable for two minutes.

The reverse failure gets less attention and is worse. The reporting process is usually small, has few dependencies, and does not touch the hardware the product exists to drive. It will keep sending while the sensor pipeline has stalled, the storage partition has filled, or the control loop has been stuck in an error path for an hour. A device heartbeat that only proves its own sender is alive is a test of the least interesting process on the device.

Three questions, three timescales, three owners

The way out is to stop treating this as one signal. There are three distinct questions, and conflating them is what produces both failure modes above.

Liveness asks whether a process is scheduled and making progress. It is a local property, and answering it requires no network — which is the point, because a device that has lost its connection is exactly when local supervision must keep working. The timescale is seconds and the owner is the service manager.

Reachability asks whether the backend can exchange messages with the device right now. It is a property of the path, of which the device is one end. Nothing the device does can make it true alone. The timescale is minutes, because transient loss on a wireless link is normal and not worth reporting.

Health asks whether the device is doing the job it exists to do. It is a property of the application, has to be defined per product, and cannot be inferred from either of the other two. The timescale is the product’s own cycle, often minutes to hours.

The three differ in timescale because they rest on different mechanisms, and one interval cannot serve all three. Thirty seconds is far too slow for liveness and far too noisy for reachability. The device heartbeat design decision is therefore not “what interval” but “which of these three am I measuring, and with what”.

What the transport already answers, and what it leaves to you

Before designing a device heartbeat message, check what the connection is already telling you, because part of the reachability question may be answered below the application. MQTT is the common case on embedded fleets. Its Keep Alive sets the maximum interval the client may go without sending a control packet, the client sends PINGREQ when it has nothing else to send, and the broker disconnects a client it has not heard from for one and a half times that interval. A Last Will and Testament, registered at connect time, is then published on the client’s behalf.

That is a genuine reachability mechanism and it is cheaper than anything you would build. What it does not give you is everything else in this article. The broker knows that MQTT control packets are arriving from a client. It does not know whether the application’s main loop is healthy, whether the sensor pipeline or storage is working, which firmware the device is running, how long it has been up, whether it restarted since the last message, or how many application messages were lost rather than merely delayed. A Last Will is published when the broker notices the connection has gone, which is a statement about the connection, not an assertion about the device.

So the division is: let the transport own reachability where it has a mechanism for it, and do not duplicate that with a device heartbeat whose only content is “still here”. Design the application message around the things the broker cannot see — the sequence number, the boot identifier, the health verdict. On a transport with no such mechanism, or where the broker’s disconnect events do not reach the system that raises alerts, the application message carries reachability as well, and the arithmetic in the next section applies to it directly.

Liveness belongs to the service manager, not to the network

systemd already implements process liveness, and a large number of embedded products do not use it. The mechanism is WatchdogSec= in the service unit. When it is set, the service manager passes the timeout to the process in the WATCHDOG_USEC= environment variable, and expects the process to call sd_notify() with the string WATCHDOG=1 at least that often. If the interval between two such calls exceeds the configured time, the service is placed in a failed state and terminated with SIGABRT, or with the signal named by WatchdogSignal=. Setting Restart= to on-watchdog, on-failure, on-abnormal or always then restarts it. If WatchdogSec= is used and NotifyAccess= has not been configured, it is implicitly set to main.

Two details in that mechanism are worth understanding rather than just copying. The first is that the timeout is handed to the process rather than kept private to the manager, which lets a daemon discover at runtime whether it is supervised at all and at what deadline, instead of hard-coding a number that has to be kept in step with the unit file by hand. The documented way to read it is sd_watchdog_enabled() rather than reading $WATCHDOG_USEC directly, because that call also checks $WATCHDOG_PID and so a forked child does not mistake its parent’s supervision for its own. The documented recommendation is to send the keep-alive every half of the returned value. That half is not caution for its own sake: the deadline is measured between two received notifications, so a process pinging exactly at the deadline fails on any scheduling delay at all.

The second is WATCHDOG=trigger, available from systemd 243 onward, which is worth checking before you rely on it because vendor BSPs often ship older. Sending that string tells the service manager that the process has detected an internal error and wants the configured watchdog action applied immediately, with the same effect as having missed the deadline — and it works even when WatchdogSec= is not enabled. A daemon that finds its own state unrecoverable therefore does not have to choose between exiting with a code the manager may read as clean and hanging until a timeout notices it. It can declare the failure.

The unit fragment is short:

[Service]
Type=notify
ExecStart=/usr/bin/telemetry-agent
WatchdogSec=30s
Restart=on-failure
RestartSec=5s

Two things about that fragment are easy to get wrong. The first is the restart policy. Restart=on-watchdog looks like the obvious choice and it is the wrong one: the documentation states that it restarts the service only if the watchdog timeout expires, so a segmentation fault, a non-zero exit or an OOM kill would leave the agent dead on a device nobody can reach. on-failure covers the watchdog case and those as well, and is what the documentation recommends for long-running services.

The second is that Type=notify places an obligation on the program beyond the keep-alive. The service manager waits for a READY=1 notification before it considers start-up complete, and the watchdog is only activated once start-up has completed. A binary that sends WATCHDOG=1 but never sends READY=1 will hang until the start timeout expires and the unit will fail, with the watchdog never armed. This is also the point at which the mechanism stops being free: it requires a process that cooperates. If the reporting binary came from a vendor BSP and you cannot change its source, none of this applies to it, and the honest options are a wrapper process that supervises it from outside or an external prober that decides it is stuck.

There is one more way to get this wrong, and it is the one that survives review because the code looks correct. If the keep-alive is sent from a dedicated timer thread, that thread will keep sending it while the main work loop is deadlocked on a mutex, blocked on I/O that never returns, or stuck waiting on an empty queue. The service manager sees notifications arriving on schedule and concludes the service is healthy, and the device heartbeat downstream reports the same thing, so the watchdog never fires and the supervision is worth nothing. The protocol measures whether notifications are timely, not whether any work is being done. The fix is to make the notification depend on progress: have the work loop update a counter or a completion timestamp, and have whatever sends WATCHDOG=1 refuse to send it unless that value has moved since the last one.

Where the process does cooperate, the result is that the liveness question is answered locally, in seconds, with no network involved, and that the answer is a restart rather than an alert. By the time anything reaches the backend through the device heartbeat, the device has already tried to fix itself.

How often to send: the arithmetic behind the interval

With liveness handled locally, the uplink device heartbeat is left doing one job — reachability accounting — and its interval can be chosen against that job. Two quantities matter, and they move in opposite directions.

Detection latency is the interval multiplied by the missed-beat threshold, minus however long the device had already been failing before the next send fell due. A failure that begins immediately after a successful send costs the full three intervals; one that begins just before the next send is due costs two. So the latency lies between (threshold − 1) and threshold intervals: with a thirty-second interval and a threshold of three, between 60 and 90 seconds. Backends differ in whether they start counting from the last message received or from the first one missed, which shifts the range by an interval, so confirm which convention yours uses rather than assuming.

The false-alarm rate depends on the chance of losing that many messages in a row. The following is an illustrative model, not a measurement from any fleet: assume each message has an independent 2% chance of not arriving, which is not unusual on a cellular link. Then three consecutive losses have probability 0.02³ = 8 × 10⁻⁶. At a thirty-second interval there are 2,880 messages per device per day, so a device produces about 0.023 false alerts per day. Across 20,000 devices that is roughly 460 false alerts a day, which is how an offline list grows large enough that operators stop acting on it.

Raising the threshold from three to six changes the same arithmetic dramatically. Six consecutive losses have probability 0.02⁶ = 6.4 × 10⁻¹¹, giving about 0.004 false alerts per day across the whole 20,000-device fleet, or roughly one per year. Detection latency goes from 60–90 seconds to 150–180 seconds. Stated plainly: under this model, ninety extra seconds of detection latency removes five orders of magnitude of alert noise.

There is a third cost that is easy to forget on a metered connection. At a 64-byte payload and a thirty-second interval, a device sends about 180 KiB per day, or 5.3 MiB per month, before any transport framing. At 256 bytes it is 21 MiB per month per device. On a tariff sold in tens of megabytes, the heartbeat alone can be a visible fraction of the plan.

There is a fourth constraint that only applies on some networks and is easy to discover too late. A device on a mobile network usually sits behind carrier-grade NAT, and the mapping that lets the backend reach it is held open by outbound traffic. If the interval is longer than the carrier expires that mapping in, the device can still send, and the backend can no longer reach it to deliver a command or a configuration change. There is no standard value to design against — it varies by carrier, by APN and by implementation — so the interval cannot be chosen from a published number here. Measure it on the SIM the product will actually ship with, or get the figure from the operator, and treat the result as a ceiling on the interval rather than a detail of the alerting rule.

Why raising the threshold helps less than the arithmetic says

The model above assumes each message is lost independently of the others. On a real wireless link that assumption is wrong, and it is wrong in the direction that matters. Loss on a cellular link is correlated in time: a device in a tunnel, at a cell edge, or in the middle of a handover loses a run of messages together, not scattered ones. The independent model treats six consecutive losses as astronomically unlikely; in a correlated regime, one two-minute outage produces exactly that run as a single event.

The practical consequence is that threshold tuning has a ceiling. Raising the number of missed beats reduces alerts caused by isolated loss, which the arithmetic handles well, and does far less against the clustered loss that dominates in the field. A team that keeps raising the threshold will find the improvement flattens while detection latency keeps growing, and will not know why the device heartbeat is still producing alerts nobody acts on.

This is the point at which the design should change rather than the numbers. If tuning cannot reduce the ambiguity of a gap, add information to the next successful message. That is what the payload is for.

Choosing the interval and the threshold for your own product

The arithmetic above is not a recommendation, because the right device heartbeat interval depends on what the product can tolerate. It does give a procedure, and it runs backwards from the consequence rather than forwards from a habit.

1. Start from the detection latency the product actually needs, not the one that sounds responsive. Ask what happens in the gap between a device failing and someone knowing. If the answer is that a technician is dispatched the next working day, a three-minute detection and a fifteen-minute detection are the same thing, and the faster one costs alert noise and data for nothing. If the answer is that a process runs unsupervised and product is scrapped, the latency has a real value attached and you can defend a shorter one. Call this number L.
2. Set a false-alert budget. Decide how many false “device down” alerts per day the people on the receiving end will tolerate before they stop reading the list, and divide by the fleet size to get a per-device figure. A fleet of 20,000 and a tolerance of five alerts a day is 2.5 × 10⁻⁴ per device per day.
3. Pick the threshold first, then the interval. The threshold is what buys down false alerts; the interval is what buys detection latency. Raise the threshold until the modelled rate fits the budget from step 2 for your device heartbeat, then set the interval to L divided by the threshold, since detection takes between (threshold − 1) and threshold intervals.
4. Check the result against the other two ceilings. Multiply the interval by the payload size and the fleet size and compare against the data plan. Then compare the interval against the NAT mapping lifetime measured on the target network. Either can veto the number, and the NAT ceiling in particular is not negotiable — exceed it and you lose downlink reachability entirely.
5. Stop tuning when the improvement stops arriving. Once the threshold is past the point where clustered loss dominates, further increases buy detection latency and very little else. At that point the remaining false alerts are telling you about the path, and the next improvement comes from the payload — the sequence number that says how many messages were actually sent — rather than from another increment of the threshold.

Two products with the same hardware and the same carrier can land on very different device heartbeat intervals from this procedure, which is the point. A vending machine where a fault means a day of lost sales and a laboratory freezer where a fault means lost samples do not deserve the same interval, and neither of them deserves thirty seconds because thirty seconds is what the first prototype used.

What to report in a device heartbeat, and why each field changes a decision

The test for including a field is not whether it is interesting. It is whether someone or something makes a different decision because the field is present. On that test, a small set earns its place in a device heartbeat.

A sequence number. This is the highest-value field in the message and it costs four bytes. Without it, the backend knows only that messages did not arrive. With it, the next message that does arrive states how many were sent in the meantime. A gap stops being ambiguous after the fact: a gap of five means the sender was running and allocated five messages in the meantime, so the process did not die. Where those five were lost — the radio, the carrier, the backend’s own ingress — the gap does not say, but it removes the device-stopped-running explanation, which is the one the alert had assumed. One field converts a question the interval can never answer into one that answers itself on recovery. Keep the counter in RAM so that it restarts at zero; persisting it across reboots is a defensible choice for other reasons and it costs you this distinction.

The boot identifier. Linux exposes /proc/sys/kernel/random/boot_id, a read-only random string generated once per boot. Reporting it makes a restart detectable as a change between two messages. Together these two fields separate messages lost in flight from a device that restarted. They do not go further than that: a device that crash-rebooted and one that was deliberately powered off and came back present identically — new boot identifier, counter from zero, low uptime — which is what the reset-cause field below is for. A device that is still off reports nothing at all, and is inferred only from continued silence.

Uptime, from the right clock. Report seconds since boot, and take them from CLOCK_BOOTTIME, not CLOCK_MONOTONIC. Both are non-settable and neither jumps when the wall clock is corrected, but they differ in one respect that matters on battery-powered hardware: CLOCK_MONOTONIC does not count time during which the system is suspended, while CLOCK_BOOTTIME is identical except that it does include suspended time. On a device that suspends between duty cycles, monotonic uptime under-reports by the whole suspend budget, and a backend using falling uptime as its reboot signal will report reboots that never happened.

Reset cause. Whatever the platform can report about why it last restarted belongs here, because it is the difference between counting reboots and understanding them. A device restarting daily from a watchdog reset and one restarting because an operator power-cycles it are the same number on a dashboard and different engineering problems.

A health verdict, plus the few numbers behind it. The device is better placed than the backend to judge whether it is working, because it can see its own sensor pipeline and error counters. Send a verdict the product defines, and the small number of values that justify it, so a disputed verdict can be examined without a field visit. This is the field that catches the up-but-wrong case, and the one that has to be designed per product rather than copied.

Firmware and configuration version. A change in fleet failure rate that correlates with a release is a finding; the same change without version information is noise.

Two things are deliberately absent. Wall-clock timestamps generated by the device are unreliable before time synchronisation completes after boot, and the backend can timestamp arrivals itself. General application metrics are discussed below.

Three designs, and what each one costs

Design 1 — One message, fixed interval, alert on missed beats

The common starting point. One process, one timer, one alert rule.

• For: trivial to implement and explain, and it does detect a device that has been powered off for a long period.
• Against: it conflates the three questions, so alerts are dominated by path failures while application failures pass unnoticed; and its only knob, the interval, trades detection latency against alert noise with no way to improve both.

Design 2 — Separate the three questions

Local liveness through WatchdogSec=, a sequence-numbered reachability message on the uplink, and a health report on the product’s own cadence carrying the verdict and its supporting values.

• For: each signal means one thing. Process hangs are caught in seconds and fixed by a restart without anyone being told. Reachability gaps are counted rather than alerted on. Application failures become visible as themselves.
• Against: three mechanisms to implement, and the health verdict is design work that cannot be delegated to a library. The backend must hold more per-device state than a last-seen timestamp.

Design 3 — Report on change, with a slow keepalive floor

The device sends when something meaningful changes, plus one message per long interval so that its continued existence is confirmed.

• For: by far the lowest data cost, and every message carries information rather than filling a schedule. Suits battery devices that spend most of their life suspended.
• Against: detection latency for a dead device is the keepalive floor, which may be hours, and silence is harder to interpret because it is the normal case. This device heartbeat design is only workable with sequence numbers, and it needs a well-designed health verdict, because there is no steady stream in which to notice anomalies.

Common wrong turns in device heartbeat design

Shortening the interval to detect failures faster. Attractive because it is the only knob visible from the dashboard, and because it does reduce detection latency. It also multiplies the false-alarm count and the data bill in direct proportion, and does nothing for the device that is up and not working. The incomplete part of the model is the assumption that a message not arriving is evidence about the device.

Carrying application metrics in the device heartbeat. Attractive because the message already flows on a timer, so adding fields looks free. It couples the liveness signal to the payload size, and the message grows over the product’s life until it is large enough to be the first thing that fails under poor radio conditions — removing the signal exactly when it is most needed. Keep the reachability message small and fixed.

Rebooting a device remotely when its device heartbeat stops. Attractive because it looks like automated recovery, and it does clear a class of real hangs. The command has to reach the device to be obeyed, so it can only act on devices that are reachable — which are not the ones that triggered it. Where it does work, through a command queued and delivered on reconnection, it reboots a device whose only problem was a firewall change. The incomplete model is again treating a missed message as a device fault.

Using CLOCK_MONOTONIC for uptime. Attractive because it is the standard advice for measuring intervals, and correct advice for that purpose. It is wrong for reporting uptime from a device that suspends, for the documented reason that it does not count suspended time.

Treating a device that has never reported as one that has stopped reporting. Attractive because both look like absence. A unit just commissioned and never seen has a provisioning or credential problem, not a reliability problem, and merging the two into one offline list buries genuinely new failures among units that never worked.

Check your own device heartbeat today

Start by finding out whether your reporting service is supervised at all. On a running board:

raghu@techveda.org:~$ systemctl show -p WatchdogUSec -p Restart -p NotifyAccess telemetry-agent.service
Restart=on-failure
NotifyAccess=main
WatchdogUSec=30s

A WatchdogUSec=0 here means the service has no liveness supervision and the only thing watching it is your backend, several minutes away across a network. Confirm the boot identifier is readable, since it costs nothing and is the field that makes reboots visible:

raghu@techveda.org:~$ cat /proc/sys/kernel/random/boot_id
c75123ed-252b-4c98-b491-dd3fe4bf2c24

Then check which clock your uptime field is using. This program prints both, and the difference between them is the accumulated suspend time:

#define _GNU_SOURCE
#include <inttypes.h>
#include <stdint.h>
#include <stdio.h>
#include <time.h>

static uint64_t clock_seconds(clockid_t id)
{
        struct timespec ts;

        if (clock_gettime(id, &ts) < 0)
                return 0;

        return (uint64_t) ts.tv_sec;
}

int main(void)
{
        printf("monotonic_s %" PRIu64 "\n", clock_seconds(CLOCK_MONOTONIC));
        printf("boottime_s  %" PRIu64 "\n", clock_seconds(CLOCK_BOOTTIME));
        return 0;
}

raghu@techveda.org:~$ gcc -Wall -Wextra -Wconversion -o hb hb.c
raghu@techveda.org:~$ ./hb
monotonic_s 214
boottime_s  214

On a system that has never suspended the two values are equal, as above. On a board that suspends between duty cycles, boottime_s will be the larger of the two by the total time spent suspended, and that gap is the error your backend is currently reading as a reboot.

Finally, produce the failure deliberately rather than waiting for it. Block the uplink at the device with a firewall rule for longer than your missed-beat threshold, then restore it, and read what the backend concluded. If it recorded a device failure rather than a path failure, the design has the defect this article is about. Decisions like this one sit across systems programming, kernel internals and driver work, which is the span TECH VEDA’s Embedded Linux Mastery Track covers as a single mentor-led sequence.

Key takeaways

• A missed device heartbeat is evidence about the path between the device and the backend. On its own it is not evidence about the device.
• Liveness, reachability and health are three questions with three timescales. One message on one interval cannot answer all three, and trying makes it answer none of them well.
• Put liveness where it belongs: WatchdogSec= answers it locally in seconds, with no network, and fixes it with a restart — but pair it with Restart=on-failure, not Restart=on-watchdog, and only on a process whose source you can change.
• Choose the device heartbeat interval and the missed-beat threshold by arithmetic. Under an independent-loss model, moving the threshold from three to six costs ninety seconds of detection and removes five orders of magnitude of false alerts — but real link loss is clustered, so that tuning stops helping beyond a point.
• A sequence number and a boot identifier separate messages lost in flight from a device that restarted, for a handful of bytes. Telling a crash-restart from a deliberate power-off needs the reset cause as well.
• Report uptime from CLOCK_BOOTTIME. CLOCK_MONOTONIC does not count suspended time and will make a suspending device appear to reboot.

Verify the design is working

1. systemctl show -p WatchdogUSec telemetry-agent.service returns a non-zero value, and Restart= is on-failure or always — not on-watchdog, which would leave a crashed agent dead.
2. Stop the keep-alive in the reporting process deliberately. Within the configured timeout the journal records the service failing, and the unit restarts. No alert reaches the backend.
3. Block the uplink for longer than the missed-beat threshold, then restore it. The first device heartbeat after restoration shows a sequence gap, and the backend records a path failure with a counted number of lost messages, not a device failure.
4. Reboot the device. The boot identifier in the first message after reboot differs from the one before it, and the backend records a restart rather than a gap.
5. Suspend and resume a battery-powered unit. The reported uptime continues to increase across the suspend; it does not fall.
6. Break the application without touching the reporting process — fill the data partition, or disconnect a sensor. The health verdict changes within its own cadence while the device heartbeat keeps arriving unchanged.
7. Classify a full day of fleet alerts as path, device or application. If path failures are more than a small minority, the threshold or the design is wrong.

Related reading on TECH VEDA

• Why a Watchdog Is Not a Reliability Strategy — the on-device side of the same problem: ordered detection timeouts, and keeping evidence across a reset that the kernel did not perform.
• Embedded Linux Observability: What to Instrument on a Device You Cannot Reach — what else is worth measuring on a device you cannot log into, and how to collect it without a live connection.
• Boot Confirmation: Who Decides an Update Worked? — the closely related question of which component is entitled to declare a device healthy, in the context of update rollback.

Frequently asked questions

How often should a device heartbeat be sent?
Choose the interval from the detection latency you need and the false-alarm rate you can tolerate, not from habit. Detection latency is the interval multiplied by the missed-beat threshold, minus the part-interval before the next send fell due, so a thirty-second interval with a threshold of three detects in 60 to 90 seconds.

Does a missed device heartbeat mean the device has failed?
No. It means a message did not arrive, and the message crosses a network path the device does not control. On wireless links the path is the most common cause. A sequence number in the payload lets the backend establish afterwards how many messages were sent during the gap.

Why report CLOCK_BOOTTIME instead of CLOCK_MONOTONIC?
CLOCK_MONOTONIC does not count time during which the system is suspended, while CLOCK_BOOTTIME is identical except that it includes suspended time. On a device that suspends, monotonic uptime under-reports, and a backend using falling uptime as its reboot signal will report reboots that did not happen.

Does MQTT keepalive make an application heartbeat unnecessary?
No. MQTT Keep Alive and a Last Will tell you the broker stopped receiving control packets from the client, which is a statement about the connection. They do not report the application’s own health, the firmware version, whether the device restarted since the last message, or how many application messages were lost. Let the transport own reachability where it has a mechanism, and carry the rest in the application message.

What does WatchdogSec= do that a network heartbeat cannot?
It answers process liveness locally, in seconds, without any network. If the interval between two WATCHDOG=1 notifications exceeds the configured time, systemd places the service in a failed state and terminates it with SIGABRT, and Restart=on-failure restarts it. It requires a process that cooperates, so it does not apply to a vendor binary whose source you cannot change.

Should application metrics travel in the heartbeat message?
Keep them separate. Adding fields makes the message grow over the product’s life until its size is what fails first under poor radio conditions, which removes the signal exactly when it is most needed. Send health reporting as its own message on its own cadence.

Further reading

• systemd.service(5) — WatchdogSec=, Type=notify, the implicit NotifyAccess=main, and the table showing which exit causes each Restart= setting covers.
• systemd.kill(5) — WatchdogSignal=, which is defined here rather than in systemd.service(5), and defaults to SIGABRT.
• sd_notify(3) — the WATCHDOG=1 keep-alive and the WATCHDOG=trigger assignment.
• sd_watchdog_enabled(3) — reading $WATCHDOG_USEC at runtime and the recommendation to ping at half the returned interval.
• clock_gettime(2) — the documented difference between CLOCK_MONOTONIC and CLOCK_BOOTTIME across suspend.
• random(4) — /proc/sys/kernel/random/boot_id and its once-per-boot semantics.
• The Linux Watchdog driver API — the separate hardware timer, which per-service WatchdogSec= never touches; systemd drives it through RuntimeWatchdogSec= in systemd-system.conf(5), which is off by default. Its boot-status flags can report a watchdog-caused reset.