Debugging a Proxmox RCU Stall: The Cause Was Not the NIC

The server went unreachable intermittently and needed a physical reboot to come back. I had just replaced the network card, so obviously the NIC was the culprit.

It wasn't. That assumption cost me a day.

The false correlation

The card swap and the crashes were independent events that happened to land close together in time. The logs were unambiguous once I actually read them: the interface was up, with no errors, throughout the entire stall window.

This is the trap worth naming. When two things change around the same time, the recent deliberate change absorbs all suspicion. It's a reasonable prior — and it's exactly why you check the logs before acting on it.

What was actually happening

pvestatd, the Proxmox statistics daemon, was segfaulting in Perl with a general protection fault. When it died, it left a spinlock held.

The chain from there:

pvestatd[1193] segfault in Perl
  → exits with preempt_count=1 (spinlock never released)
  → BUG: scheduling while atomic: pvestatd/1193
  → "Fixing recursive fault but reboot is needed!"
  → RCU stalls every 3 minutes for ~42 hours
  → UI completely inaccessible

The RCU stall was the symptom, not the cause. When a kernel-context process dies holding a spinlock, Read-Copy-Update can't make progress. The kernel detects this and reports it loudly — but it does not recover on its own.

I spent hours reading about RCU stalls. RCU was the thing complaining, not the thing broken.

The aggravating factor

pvestatd.service ships with Restart=no.

That single line turned a recoverable crash into 42 hours of downtime. With Restart=on-failure, systemd would have restarted the daemon in ten seconds and the stall would never have persisted long enough for me to notice.

The daemon crashing sporadically is a known, unresolved bug — there are multiple reports against this Proxmox version. I can't fix that. I can fix how the system responds to it.

How to diagnose this pattern

The key move is reading the previous boot, since the evidence is destroyed by the reboot that restores service:

# 1. List previous boots
journalctl --list-boots

# 2. Inspect the boot before the reboot
journalctl -b -1 -k | grep -E "BUG|segfault|general protection|panic" | head -20

# 3. Confirm ordering: did the crash precede the first RCU stall?
journalctl -b -1 -k | grep -E "rcu.*stall" | head -3

# 4. Identify the process
journalctl -b -1 -k | grep "segfault" | head -5

Step three is the whole technique. If the segfault appears before the first stall, the process that crashed is your root cause and RCU is noise. Cause precedes symptom in the log; you just have to look earlier than the alarming message that got your attention.

What I changed

The real fix was a systemd drop-in overriding the upstream default:

# /etc/systemd/system/pvestatd.service.d/restart.conf
[Service]
Restart=on-failure
RestartSec=10

I also installed kdump-tools with crashkernel=256M reserved, so the next serious crash captures a vmcore instead of leaving me to infer everything from the journal.

The lesson

The most expensive part of this incident was not the downtime. It was the hours spent investigating a component that was working correctly, because I trusted a coincidence over the logs.

Read the previous boot. Find the first thing that broke, not the loudest thing that's complaining.