netdev - Kernel lockup on bpf_probe_read_kernel in BPF_PROG_TYPE_RAW_TRACEPOINT sched

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <CAH6OuBRzMZfa2kR6KYw2-F3mo3LfqwdVDqLSAD6e-xU9C56Fdw@mail.gmail.com>
Date: Fri, 5 Sep 2025 16:54:18 +0200
From: Ritesh Oedayrajsingh Varma <ritesh@...erluminal.eu>
To: bpf <bpf@...r.kernel.org>, netdev@...r.kernel.org
Subject: Kernel lockup on bpf_probe_read_kernel in BPF_PROG_TYPE_RAW_TRACEPOINT
 sched_switch program

Hi,

We're running into an issue with our eBPF-based CPU profiler that
we've been attempting to debug for a week now. The issue is that as
soon as our eBPF program(s) are attached, the entire system (kernel)
hangs, but only on specific distributions. We've managed to reduce the
problem down to the following code:

SEC("raw_tracepoint/sched_switch")
int raw_tracepoint__sched__sched_switch(struct bpf_raw_tracepoint_args *ctx)
{
    struct task_struct *prev_task = (struct task_struct
*)BPF_CORE_READ(ctx, args[1]);
    struct task_struct *next_task = (struct task_struct
*)BPF_CORE_READ(ctx, args[2]);
    return 0;
}

i.e. this is just a raw tracepoint attached to sched_switch. Loading &
attaching this program, which just reads the tracepoint args, will
result in the system immediately locking up on some distributions. It
works fine on Ubuntu, Debian (on various distros & kernel versions),
but will hang on Arch and Fedora. As far as we can tell there is no
relation to kernel version; we've tested both old (5.11 on Fedora 34)
and new (6.16.3-arch1-1 on Arch) kernels and both hang immediately
when this program executes.

The code above calls BPF_CORE_READ, which compiles down to
bpf_probe_read_kernel, which we've verified by dumping the xlated
version of the program after loading it (without attaching it. And
indeed, if we replace the BPF_CORE_READ with direct calls to
bpf_probe_read_kernel, it hangs in the same way.

To debug, we've attempted the following:

- Enabled nmi_watchdog, hardlockup_panic, softlockup_panic and hard &
softlockup backtraces. This doesn't give any further information.
- There is no relevant output from dmesg
- Regarding Arch, of note is that a Manjaro distro with its own kernel
works fine, but an Arch kernel doesn't. We suspect there is some kind
of kernel config option this code is interacting badly with, but we
have been unable to find such an option by diffing the kernel configs
of both the working & non-working machines.
- Attaching a kernel debugger via network. Unfortunately, the lockup
also locks up the networking layer, so gdb is unable to remotely
inspect the hanging state. We're currently in the process of setting
up a serial debugger.

We'll continue debugging this on our end once we get the serial
debugger components, but we were wondering if anybody has seen this
before or has any tips with regards to tracking down the cause of the
hang? Any input would be greatly appreciated.

Thanks,
Ritesh