Message-ID: <d23d133b52ef574d669f1656789b78d07c91c9f5.camel@redhat.com>
Date: Mon, 27 Oct 2025 13:20:48 +0100
From: Gabriele Monaco <gmonaco@...hat.com>
To: Yunseong Kim <ysk@...lloc.com>, Nam Cao <nam.cao@...aro.org>
Cc: Sebastian Andrzej Siewior <bigeasy@...utronix.de>, Tomas Glozar	
 <tglozar@...hat.com>, Shung-Hsi Yu <shung-hsi.yu@...e.com>, Byungchul Park	
 <byungchul@...com>, syzkaller@...glegroups.com,
 linux-rt-devel@...ts.linux.dev,  LKML <linux-kernel@...r.kernel.org>
Subject: Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via
 RV (Runtime Verification) monitor rtapp:sleep

On Mon, 2025-10-27 at 15:54 +0900, Yunseong Kim wrote:
> Hi Nam,
> 
> I've been very interested in RV (Runtime Verification) to proactively detect
> "sleep in atomic" scenarios on PREEMPT_RT kernels. Specifically, I'm looking
> for ways to find cases where sleeping spinlocks or memory allocations are used
> within preemption-disabled or irq-disabled contexts. While searching for
> solutions, I discovered the RV subsystem.
> 

Hi Yunseong,

I'm sure Nam can be more specific on this, but let me add my 2 cents here.

The sleep monitor doesn't really do what you want, its violations are real time
tasks (typically userspace tasks with RR/FIFO policies) sleeping in a way that
might incur latencies. For instance using non PI locks or imprecise sleep.

What you need here is to validate kernel code, RV was actually designed for
that, but there's currently no monitor that does what you want.

The closest thing I can think of is monitors like scpd and snep in the sched
collection [1]. Those however won't catch what you need because they focus on
the preemption tracepoints and schedule, which works fine also in your scenario.

We could add similar monitors to catch what you want though:

                     |
                     |
                     v
                   +-----------------+
                   |   cant_sleep    | <+
                   +-----------------+  |
                     |                  |
                     | preempt_enable   | preempt_disable
                     v                  |
    kmalloc                             |
    lock_acquire                        |
  +---------------      can_sleep       |
  |                                     |
  +-------------->                     -+

which would become slightly more complicated if considering irq enable/disable
too. This is a deterministic automaton representation (see [1] for examples),
you could use an LTL like sleep as well, I assume (needs a per-CPU monitor which
is not merged yet for LTL).

This is simplified but you can of course put conditions on what kind of
allocations and locks you're interested in.

Now this specific case would require lockdep for the definition of lock_acquire
tracepoints. So I'm not sure how useful this monitor would be since lockdep is
going to complain too. You could use contention tracepoints to catch exactly
when sleep is going to occur and not /potential/ failures.

I only gave a quick thought on this, there may be better models/event fitting
your usecase, but I hope you get the idea.

[1] - https://docs.kernel.org/trace/rv/monitor_sched.html#monitor-scpd

> Here are my questions:
> 
> 1. Does the rtapp:sleep monitor proactively detect scenarios that
>    could lead to sleeping in atomic context, perhaps before
>    CONFIG_DEBUG_ATOMIC_SLEEP (enabled) would trigger at the actual point of
>    sleeping?

I guess I answered this already, but TL;DR no, you'd need a dedicated monitor.

> 2. Is there a way to enable this monitor (e.g., rtapp:sleep)
>    immediately as soon as the RV subsystem is loaded during boot time?
>    (How to make this "default turn on"?)

Currently not, but you could probably use any sort of startup script to turn it
on soon enough.

> 3. When a "violation detected" message occurs at runtime, is it
>    possible to get a call stack of the location that triggered the
>    violation? The panic reactor provides a full stack, but I'm
>    wondering if this is also possible with the printk reactor.

You can use ftrace and rely on error tracepoints instead of reactors. Each RV
violation triggers a tracepoint (e.g. error_sleep) and you can print a call
stack there. E.g.:

  echo stacktrace > /sys/kernel/tracing/events/rv/error_sleep/trigger

Here I use sleep as an example, but all monitors have their own error events
(e.g. error_wwnr, error_snep, etc.).

Does this all look useful in your scenario?

Gabriele

> 
> Here is some background on why I'm so interested in this topic:
> 
> Recently, I was fuzzing the PREEMPT_RT kernel with syzkaller but ran into
> issues where fuzzing wouldn't proceed smoothly. It turned out to be a problem
> in the kcov USB API. This issue was fixed after I reported it, together
> with Sebastian’s patch.
> 
> [PATCH] kcov, usb: Don't disable interrupts in kcov_remote_start_usb_softirq()
>  - https://lore.kernel.org/all/20250811082745.ycJqBXMs@linutronix.de/
> 
> After this fix, syzkaller fuzzing ran well and was able to detect several
> runtime "sleep in atomic context" bugs:
> 
> [PATCH] USB: gadget: dummy-hcd: Fix locking bug in RT-enabled kernels
>  -
> https://lore.kernel.org/all/bb192ae2-4eee-48ee-981f-3efdbbd0d8f0@rowland.harvard.edu/
> 
> [BUG] usbip: vhci: Sleeping function called from invalid context in
> vhci_urb_enqueue on PREEMPT_RT
>  -
> https://lore.kernel.org/all/c6c17f0d-b71d-4a44-bcef-2b65e4d634f7@kzalloc.com/
> 
> This led me to research ways to find these issues proactively at a
> static analysis level, and I created some regex and coccinelle scripts
> to detect them.
> 
> [BUG] gfs2: sleeping lock in gfs2_quota_init() with preempt disabled
> on PREEMPT_RT
>  - https://lore.kernel.org/all/20250812103808.3mIVpgs9@linutronix.de/t/#u
> 
> [PATCH] md/raid5-ppl: Fix invalid context sleep in
> ppl_io_unit_finished() on PREEMPT_RT
>  -
> https://lore.kernel.org/all/f2dbf110-e2a7-4101-b24c-0444f708fd4e@kernel.org/t/#u
> 
> Tomas, the author of the rtlockscope project, also gave me some deep
> insights into this static analysis approach.
> 
> Re: [WIP] coccinelle: rt: Add coccicheck on sleep in atomic context on
> PREEMPT_RT
>  -
> https://lore.kernel.org/all/CAP4=nvTOE9W+6UtVZ5-5gAoYeEQE8g4cgG602FJDPesNko-Bgw@mail.gmail.com/
> 
> 
> Thank you!
> 
> Best regards,
> Yunseong Kim

Powered by blists - more mailing lists