linux-kernel - Re: [clocksource] 8e614d5b58: WARNING:at_kernel/time/clocksource-wdtest.c:#wdtest

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20210525073655.GE7744@xsang-OptiPlex-9020>
Date:   Tue, 25 May 2021 15:36:55 +0800
From:   Oliver Sang <oliver.sang@...el.com>
To:     "Paul E. McKenney" <paulmck@...nel.org>
Cc:     Thomas Gleixner <tglx@...utronix.de>,
        John Stultz <john.stultz@...aro.org>,
        Stephen Boyd <sboyd@...nel.org>,
        Jonathan Corbet <corbet@....net>,
        Mark Rutland <Mark.Rutland@....com>,
        Marc Zyngier <maz@...nel.org>, Andi Kleen <ak@...ux.intel.com>,
        Feng Tang <feng.tang@...el.com>,
        Xing Zhengjun <zhengjun.xing@...ux.intel.com>,
        Chris Mason <clm@...com>, LKML <linux-kernel@...r.kernel.org>,
        Linux Memory Management List <linux-mm@...ck.org>,
        lkp@...ts.01.org, lkp@...el.com
Subject: Re: [clocksource]  8e614d5b58:
 WARNING:at_kernel/time/clocksource-wdtest.c:#wdtest_func.cold

hi Paul,

On Fri, May 07, 2021 at 10:12:59AM -0700, Paul E. McKenney wrote:
> On Wed, May 05, 2021 at 11:03:12AM -0700, Paul E. McKenney wrote:
> > On Wed, May 05, 2021 at 10:36:16PM +0800, kernel test robot wrote:
> > > 
> > > 
> > > Greeting,
> > > 
> > > FYI, we noticed the following commit (built with gcc-9):
> > > 
> > > commit: 8e614d5b58992e722f07de7c2426f2c44668092b ("clocksource: Provide kernel module to test clocksource watchdog")
> > > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
> > > 
> > > 
> > > in testcase: boot
> > > 
> > > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
> > > 
> > > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
> > > 
> > > 
> > > +-------------------------------------------------------------------------+------------+------------+
> > > |                                                                         | bdbd9c673e | 8e614d5b58 |
> > > +-------------------------------------------------------------------------+------------+------------+
> > > | WARNING:at_kernel/time/clocksource-wdtest.c:#wdtest_func.cold           | 0          | 11         |
> > > | RIP:wdtest_func.cold                                                    | 0          | 11         |
> > > +-------------------------------------------------------------------------+------------+------------+
> > 
> > Might it be useful to address the lockdep issues that preceded this splat?
> > 
> > Leaving that aside, the system appears to still be booting.  There are
> > RCU CPU stall warning messages later on, and then the system hangs more
> > than six minutes while still booting, presumably due to the large number
> > of self-tests and debug options enabled.
> > 
> > The intent is that the clocksource-wdtest tests run after boot has
> > completed.  One approach would be to test it using modprobe after boot
> > has completed.  In addition, the clocksource-wdtest module is not designed
> > to handle CPU overload conditions, and making it do so would reduce the
> > effectiveness of the test.
> > 
> > I suggest setting clocksource-wdtest.holdoff=N, where "N" is in seconds
> > and is large enough that boot has completed.  Alternatively, use modprobe
> > to activate this module from userspace after boot has completed.
> > 
> > What I do is just set CONFIG_TEST_CLOCKSOURCE_WATCHDOG=y in an ordinary
> > rcutorture run, if that helps.
> 
> All that aside, does the patch below help in your environment?

sorry for late.
I applied below patch to f4c6b34ee1 ("clocksource: Provide kernel module to test clocksource watchdog")
still found the issue on both f4c6b34ee1 and patched kernel.

attached dmesg.xz FYI

[  157.028293] ------------[ cut here ]------------
[  157.031055] WARNING: CPU: 0 PID: 207 at kernel/time/clocksource-wdtest.c:154 wdtest_func.cold+0x6a/0x117
[  157.034366] Modules linked in:
[  157.036812] CPU: 0 PID: 207 Comm: wdtest Tainted: G S      W         5.13.0-rc1-00006-g2aaf396b0698 #1
[  157.040112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[  157.043283] EIP: wdtest_func.cold+0x6a/0x117
[  157.045833] Code: 00 69 55 00 a1 24 eb ea da 83 c4 10 85 c0 74 02 0f 0b a1 04 32 9a d9 39 1d b0 0f 9a d9 0f 93 c1 83 e0 40 0f 94 c2 38 d1 74 02 <0f> 0
b 85 c0 74 32 b8 c0 31 9a d9 e8 ca d7 3a fb b8 19 00 00 00 c7
[  157.052700] EAX: 00000040 EBX: 00000000 ECX: 00000001 EDX: 00000000
[  157.055738] ESI: d8abf847 EDI: d8abf836 EBP: c680ff84 ESP: c680ff78
[  157.058770] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010202
[  157.061854] CR0: 80050033 CR2: 00000000 CR3: 1a780000 CR4: 000406d0
[  157.064864] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  157.067797] DR6: fffe0ff0 DR7: 00000400
[  157.070414] Call Trace:
[  157.072676]  kthread+0xf6/0x140
[  157.074915]  ? wdtest_ktime_read+0x80/0x80
[  157.077140]  ? kthread_insert_work_sanity_check+0x60/0x60
[  157.079472]  ret_from_fork+0x1c/0x28
[  157.081558] irq event stamp: 533
[  157.083404] hardirqs last  enabled at (541): [<d19c60e5>] console_unlock+0x4c5/0x600
[  157.085717] hardirqs last disabled at (550): [<d19c6055>] console_unlock+0x435/0x600
[  157.087921] softirqs last  enabled at (368): [<d6bb1ff0>] __do_softirq+0x2f0/0x44b
[  157.090071] softirqs last disabled at (567): [<d18ce2a5>] call_on_stack+0x45/0x60
[  157.092111] ---[ end trace 0b325ef1942d2bda ]---


> 
> If so, I can adjust so that my testing gets done quickly and yours
> avoids false-positive failures.
> 
> 							Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> diff --git a/kernel/time/clocksource-wdtest.c b/kernel/time/clocksource-wdtest.c
> index 01df12395c0e..0d8542f8b1d2 100644
> --- a/kernel/time/clocksource-wdtest.c
> +++ b/kernel/time/clocksource-wdtest.c
> @@ -149,7 +149,7 @@ static int wdtest_func(void *arg)
>  			s = ", expect clock skew";
>  		pr_info("--- Watchdog with %dx error injection, %lu retries%s.\n", i, max_cswd_read_retries, s);
>  		WRITE_ONCE(wdtest_ktime_read_ndelays, i);
> -		schedule_timeout_uninterruptible(2 * HZ);
> +		schedule_timeout_uninterruptible(60 * HZ);
>  		WARN_ON_ONCE(READ_ONCE(wdtest_ktime_read_ndelays));
>  		WARN_ON_ONCE((i <= max_cswd_read_retries) !=
>  			     !(clocksource_wdtest_ktime.flags & CLOCK_SOURCE_UNSTABLE));

Download attachment "dmesg.xz" of type "application/x-xz" (40020 bytes)