linux-kernel - Re: [syzbot] [mm?] [input?] [usb?] INFO: rcu detected stall in brk (2)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANp29Y58o0O8jTSP_SpBLVp7VK0bM4Svo7jBNZgDpv=s1DWJqQ@mail.gmail.com>
Date: Fri, 1 Nov 2024 13:09:24 +0100
From: Aleksandr Nogikh <nogikh@...gle.com>
To: Lorenzo Stoakes <lorenzo.stoakes@...cle.com>
Cc: syzbot <syzbot+7402e6c8042635c93ead@...kaller.appspotmail.com>, 
	akpm@...ux-foundation.org, jannh@...gle.com, liam.howlett@...cle.com, 
	linux-input@...r.kernel.org, linux-kernel@...r.kernel.org, linux-mm@...ck.org, 
	linux-usb@...r.kernel.org, syzkaller-bugs@...glegroups.com, vbabka@...e.cz, 
	Alan Stern <stern@...land.harvard.edu>, Dmitry Vyukov <dvyukov@...gle.com>
Subject: Re: [syzbot] [mm?] [input?] [usb?] INFO: rcu detected stall in brk (2)

On Fri, Nov 1, 2024 at 12:11 PM Lorenzo Stoakes
<lorenzo.stoakes@...cle.com> wrote:
>
> On Fri, Nov 01, 2024 at 12:02:25PM +0100, Aleksandr Nogikh wrote:
> > Hi Lorenzo,
> >
> > On Thu, Oct 31, 2024 at 5:59 PM 'Lorenzo Stoakes' via syzkaller-bugs
> > <syzkaller-bugs@...glegroups.com> wrote:
> > >
> > > +Alan re: USB stalls
> > >
> > > On Thu, Oct 31, 2024 at 09:41:02AM -0700, syzbot wrote:
> > > > Hello,
> > > >
> > > > syzbot has tested the proposed patch and the reproducer did not trigger any issue:
> > > >
> > > > Reported-by: syzbot+7402e6c8042635c93ead@...kaller.appspotmail.com
> > > > Tested-by: syzbot+7402e6c8042635c93ead@...kaller.appspotmail.com
> > > >
> > > > Tested on:
> > > >
> > > > commit:         cffcc47b mm/mlock: set the correct prev on failure
> > > > git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/ mm-hotfixes-unstable
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=1304a630580000
> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=6648774f7c39d413
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=7402e6c8042635c93ead
> > > > compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
> > > >
> > > > Note: no patches were applied.
> > > > Note: testing is done by a robot and is best-effort only.
> > >
> > > OK this seems likely to be intermittant (and unrelated to what's in
> > > mm-unstable-fixes honestly) and does make me wonder if the fix referenced
> > > in [0] really has sorted things out? Or whether it has perhaps help
> > > mitigate the issue but not sufficiently in conjunction with debug things
> > > that slow things down.
> > >
> > > Because we keep getting these reports, that mysteriously don't occur if we
> > > re-run (or hit other code paths), they seem to hit somewhat arbitrary parts
> > > of mm, and because CONFIG_DEBUG_VM_MAPLE_TREE is set we spend a _long_ time
> > > in mm validating trees (this config option is REALLY REALLY heavy-handed).
> > >
> > > I note we also set CONFIG_KCOV and CONFIG_KCOV_INSTRUMENT_ALL which isn't
> > > going to make anything quicker if the USB gets laggy.
> >
> > These are necessary for coverage-guided fuzzing. Though when we find
> > and run reproducers, we don't actually set up /dev/kcov, so I guess
> > the impact of coverage callbacks here is not that significant here.
> > CONFIG_KASAN is likely slowing down things much more.
> >
> > >
> > > I'm not sure if there's a human who can help tweak the config for these
> > > hardware-centric tests at Google? At least tweaking the RCU stall time
> > > anyway?
> >
> > We currently set:
> >
> > CONFIG_RCU_CPU_STALL_TIMEOUT=100
> > CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000
> >
> > The expedited RCU timeout was limited to 21 seconds up to some time
> > ago, but I guess now we can safely increase this number as well. I'll
> > send a PR with syzbot config updates.
>
> Ah thanks, just sent a reply immediately prior to receiving this :)
>
> Maybe worth setting CONFIG_RCU_EXP_CPU_STALL_TIMEOUT to 0 so it matches the CPU
> stall timeout? At least for builds that also set heavy debug options like
> CONFIG_DEBUG_VM_MAPLE_TREE?

Oh, that's even simpler than I thought it would be, thank you!
I've sent https://github.com/google/syzkaller/pull/5455, it will come
into effect in a few days.

-- 
Aleksandr

>
> Cheers, Lorenzo
>
> >
> > --
> > Aleksandr
> >
> > >
> > > In any case this continues not to look likely to be an actual mm issue as
> > > far as I can see.
> > >
> > > In [0] we were stalled in a validate call which would align with the idea
> > > that perhaps we were just dealing with a very very big tree and getting
> > > slow down that way.
> > >
> > > Cheers, Lorenzo
> > >
> > > [0]:https://lore.kernel.org/all/967f3aa0-447a-4121-b80b-299c926a33f5@rowland.harvard.edu/
> > >