linux-kernel - Re: [rcutorture] 613b00fbe6: WARNING:at_kernel/rcu/rcutorture.c:#rcu_torture_fwd_prog

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <YxHCPMftruuGYrnm@xsang-OptiPlex-9020>
Date:   Fri, 2 Sep 2022 16:43:40 +0800
From:   Oliver Sang <oliver.sang@...el.com>
To:     "Paul E. McKenney" <paulmck@...nel.org>
CC:     Neeraj Upadhyay <neeraj.iitr10@...il.com>,
        LKML <linux-kernel@...r.kernel.org>, <lkp@...ts.01.org>,
        <lkp@...el.com>, <qiang1.zhang@...el.com>
Subject: Re: [rcutorture]  613b00fbe6:
 WARNING:at_kernel/rcu/rcutorture.c:#rcu_torture_fwd_prog_cr[rcutorture]

hi Paul,

On Wed, Aug 31, 2022 at 03:57:07AM -0700, Paul E. McKenney wrote:
> On Wed, Aug 03, 2022 at 11:05:12PM +0800, kernel test robot wrote:
> > 
> > (please be noted we reported this since clean on parent, ~10% on this commit,
> > and found similar issue on latest maineline, e.g. v5.19)
> > 
> > 
> > Greeting,
> > 
> > FYI, we noticed the following commit (built with gcc-11):
> > 
> > commit: 613b00fbe64461f1c73e035e07c22cf3de65740b ("rcutorture: Add ability to limit callback-flood intensity")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > 
> > in testcase: rcutorture
> > version: 
> > with following parameters:
> > 
> > 	runtime: 300s
> > 	test: default
> > 	torture_type: tasks-tracing
> > 
> > test-description: rcutorture is rcutorture kernel module load/unload test.
> > test-url: https://www.kernel.org/doc/Documentation/RCU/torture.txt
> > 
> > 
> > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
> 
> I believe that this commit in -rcu fixes this problem.  Could you please
> check it?

I found dba268ff32101559cd3e3800b24993ae5c59844c includes 613b00fbe6:

linux$ git rev-list dba268ff32101559cd3e3800b24993ae5c59844c | grep 613b00fbe6
613b00fbe64461f1c73e035e07c22cf3de65740b

so I directly tested on dba268ff32101559cd3e3800b24993ae5c59844c and its
parent: 18d49f5bdb878c0567d8256f02b3dcadc6878edf

but still found similar issus on both commits, with similar rate:

18d49f5bdb878c05 dba268ff32101559cd3e3800b24
---------------- ---------------------------
       fail:runs  %reproduction    fail:runs
           |             |             |
         26:500          1%          33:500   dmesg.RIP:rcu_torture_fwd_prog_cr[rcutorture]
         26:500          1%          33:500   dmesg.WARNING:at_kernel/rcu/rcutorture.c:#rcu_torture_fwd_prog_cr[rcutorture]


attached config and dmesg FYI

[  316.326241][  T606] ------------[ cut here ]------------
[  316.328146][  T606] WARNING: CPU: 1 PID: 606 at kernel/rcu/rcutorture.c:2559 rcu_torture_fwd_prog_cr+0x300/0x3b1 [rcutorture]
[  316.331781][  T606] Modules linked in: rcutorture torture
[  316.333554][  T606] CPU: 1 PID: 606 Comm: rcu_torture_fwd Not tainted 6.0.0-rc1-00068-gdba268ff3210 #1
[  316.336520][  T606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-4 04/01/2014
[  316.339699][  T606] RIP: 0010:rcu_torture_fwd_prog_cr+0x300/0x3b1 [rcutorture]
[  316.342248][  T606] Code: 00 00 00 31 f6 49 83 fc 63 48 c7 c7 04 d5 01 a0 40 0f 9e c6 31 c9 31 d2 89 34 24 e8 df c1 22 e1 49 83 fc 63 8b 34 24 7f 04 90 <0f> 0b 90 31 c9 31 d2 48 c7 c7 d4 d4 01 a0 e8 c2 c1 22 e1 48 8b 0d
[  316.348610][  T606] RSP: 0018:ffffc90000f63e90 EFLAGS: 00010293
[  316.350834][  T606] RAX: 0000000000000000 RBX: 000000000000c350 RCX: 0000000000000000
[  316.353793][  T606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
[  316.356455][  T606] RBP: ffff888129999000 R08: 0000000000000000 R09: 0000000000000000
[  316.359211][  T606] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  316.363628][  T606] R13: 000000000000a922 R14: 0000000000000000 R15: 0000000000000000
[  316.372010][  T606] FS:  0000000000000000(0000) GS:ffff88842fd00000(0000) knlGS:0000000000000000
[  316.374987][  T606] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  316.379048][  T606] CR2: 00007fbfd2c6a000 CR3: 0000000127e76000 CR4: 00000000000406e0
[  316.387853][  T606] Call Trace:
[  316.389030][  T606]  <TASK>
[  316.390215][  T606]  rcu_torture_fwd_prog.cold+0x3e/0x147 [rcutorture]
[  316.395209][  T606]  ? rcu_torture_fwd_prog_cbfree+0x90/0x90 [rcutorture]
[  316.397657][  T606]  kthread+0xf1/0x110
[  316.401308][  T606]  ? kthread_complete_and_exit+0x20/0x20
[  316.404960][  T606]  ret_from_fork+0x1f/0x30
[  316.407609][  T606]  </TASK>
[  316.409694][  T606] irq event stamp: 2944523
[  316.412682][  T606] hardirqs last  enabled at (2944537): [<ffffffff8117b893>] __up_console_sem+0x93/0xa0
[  316.432771][  T606] hardirqs last disabled at (2944560): [<ffffffff8117b878>] __up_console_sem+0x78/0xa0
[  316.435881][  T606] softirqs last  enabled at (2944554): [<ffffffff82e0041a>] __do_softirq+0x41a/0x6d4
[  316.440503][  T606] softirqs last disabled at (2944545): [<ffffffff810ececf>] irq_exit_rcu+0x7f/0xa0
[  316.444348][  T606] ---[ end trace 0000000000000000 ]---


> 
> 							Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> commit dba268ff32101559cd3e3800b24993ae5c59844c
> Author: Zqiang <qiang1.zhang@...el.com>
> Date:   Sun Jul 31 18:53:56 2022 +0800
> 
>     rcutorture: Use the barrier operation specified by cur_ops
>     
>     The rcutorture_oom_notify() function unconditionally invokes
>     rcu_barrier(), which is OK when the rcutorture.torture_type value is
>     "rcu", but unhelpful otherwise.  The purpose of these barrier calls is to
>     wait for all outstanding callback-flooding callbacks to be invoked before
>     cleaning up their data.  Using the wrong barrier function therefore
>     risks arbitrary memory corruption.  Thus, this commit changes these
>     rcu_barrier() calls into cur_ops->cb_barrier() to make things work when
>     torturing non-vanilla flavors of RCU.
>     
>     Signed-off-by: Zqiang <qiang1.zhang@...el.com>
>     Signed-off-by: Paul E. McKenney <paulmck@...nel.org>
> 
> diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> index 7168dc8d61e98..bdd630f7a970c 100644
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -2603,12 +2603,12 @@ static int rcutorture_oom_notify(struct notifier_block *self,
>  	for (i = 0; i < fwd_progress; i++)
>  		ncbs += rcu_torture_fwd_prog_cbfree(&rfp[i]);
>  	pr_info("%s: Freed %lu RCU callbacks.\n", __func__, ncbs);
> -	rcu_barrier();
> +	cur_ops->cb_barrier();
>  	ncbs = 0;
>  	for (i = 0; i < fwd_progress; i++)
>  		ncbs += rcu_torture_fwd_prog_cbfree(&rfp[i]);
>  	pr_info("%s: Freed %lu RCU callbacks.\n", __func__, ncbs);
> -	rcu_barrier();
> +	cur_ops->cb_barrier();
>  	ncbs = 0;
>  	for (i = 0; i < fwd_progress; i++)
>  		ncbs += rcu_torture_fwd_prog_cbfree(&rfp[i]);

View attachment "config-6.0.0-rc1-00068-gdba268ff3210" of type "text/plain" (153902 bytes)

Download attachment "dmesg.xz" of type "application/x-xz" (21936 bytes)