linux-kernel - Re: 'rcu_preempt detected stalls on CPUs/tasks...' issue of cyclictest on rt-linux

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAFwiDX_p6Uvcy0uQb0T7pRc2hy-iNZTHyyggzT4qBojLftbKzg@mail.gmail.com>
Date: Fri, 26 Jul 2024 13:48:48 +0530
From: Neeraj upadhyay <neeraj.iitr10@...il.com>
To: richard clark <richard.xnu.clark@...il.com>
Cc: paulmck@...nel.org, josh@...htriplett.org, 
	Lai Jiangshan <jiangshanlai@...il.com>, mathieu.desnoyers@...icios.com, 
	Steven Rostedt <rostedt@...dmis.org>, Mark Rutland <mark.rutland@....com>, 
	Linus Torvalds <torvalds@...ux-foundation.org>, rcu@...r.kernel.org, 
	linux-kernel@...r.kernel.org, linux-arm-kernel@...ts.infradead.org, 
	linux-rt-users@...r.kernel.org
Subject: Re: 'rcu_preempt detected stalls on CPUs/tasks...' issue of
 cyclictest on rt-linux

Hi,

On Fri, Jul 26, 2024 at 12:46 PM richard clark
<richard.xnu.clark@...il.com> wrote:
>
> Hi，
>
> On Wed, Jul 10, 2024 at 7:22 PM Neeraj upadhyay <neeraj.iitr10@...il.com> wrote:
> >
> > Hello Richard,
> >
> > On Wed, Jul 10, 2024 at 1:56 PM richard clark
> > <richard.xnu.clark@...il.com> wrote:
> > >
> > > Hi,
> > > I am running a Ubuntu 20.04.5 LTS on Nvidia Jetson AGX Orin platform
> > > with 12-cores as a guestOS, the kernel version is - 6.1.83-rt28.
> > > Kernel cmdline is:
> > > 'root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4
> > > console=ttyTCU0,115200 console=tty0 firmware_class.path=/etc/firmware
> > > fbcon=map:0 net.ifnames=0'
> > >
> > > The cyclictest command 'cyclictest -Smp99 -H 3000
> > > --histfile=orin_idle_hyp_4h.hist -D 4h' will hang randomly during the
> > > test, then the minicom console will show below messages:
> > > ...
> > >
> > > [97619.450889] [CPU11-E] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> > > [97619.450894] [CPU11-E] rcu:   1-...!: (0 ticks this GP)
> > > idle=dc88/0/0x0 softirq=0/0 fqs=2 (false positive?)
> > > [97619.450914] [ CPU1-E] NMI backtrace for cpu 1
> > > [97619.451912] [CPU11-E] rcu: rcu_preempt kthread timer wakeup didn't
> > > happen for 5251 jiffies! g6029253 f0x0 RCU_GP_WAIT_FQS(5)
> > > ->state=0x402
> > > [97619.451916] [CPU11-E] rcu:   Possible timer handling issue on cpu=1
> > > timer-softirq=342864
> >
> > This log indicates that jiffies timers are not getting handled on CPU1, due to
> > which GP kthread was not woken up. Can you check irq, softirq and timer traces
> > on CPU1, to see if the softirqs/timers are getting served on this CPU?
> >
> Please ignore this issue. As narrow down this is an issue from
> homemade hypervisor which unmasks the vtimer ctrl.IMASK bit by
> accident, so the vcpu can't get further vtimer interrupt, thus the
> issue happens...
> Thanks
>

Great! Thanks for providing details on the root cause!


- Neeraj

> >
> > - Neeraj
> >