linux-kernel - RE: rcu_preempt caused oom

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CD6925E8781EFD4D8E11882D20FC406D52A1509A@SHSMSX104.ccr.corp.intel.com>
Date:   Tue, 4 Dec 2018 07:50:04 +0000
From:   "He, Bo" <bo.he@...el.com>
To:     "paulmck@...ux.ibm.com" <paulmck@...ux.ibm.com>
CC:     Steven Rostedt <rostedt@...dmis.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "josh@...htriplett.org" <josh@...htriplett.org>,
        "mathieu.desnoyers@...icios.com" <mathieu.desnoyers@...icios.com>,
        "jiangshanlai@...il.com" <jiangshanlai@...il.com>,
        "Zhang, Jun" <jun.zhang@...el.com>,
        "Xiao, Jin" <jin.xiao@...el.com>,
        "Zhang, Yanmin" <yanmin.zhang@...el.com>,
        "Bai, Jie A" <jie.a.bai@...el.com>
Subject: RE: rcu_preempt caused oom

Hi, Paul:
the enclosed is the log trigger the 120s hung_task_panic without other debug patches, the hung task is blocked at __wait_rcu_gp, it means the rcu_cpu_stall can't detect the scenario:
echo 1 > /proc/sys/kernel/panic_on_rcu_stall
echo 7 > /sys/module/rcupdate/parameters/rcu_cpu_stall_timeout


-----Original Message-----
From: Paul E. McKenney <paulmck@...ux.ibm.com> 
Sent: Monday, December 3, 2018 9:57 PM
To: He, Bo <bo.he@...el.com>
Cc: Steven Rostedt <rostedt@...dmis.org>; linux-kernel@...r.kernel.org; josh@...htriplett.org; mathieu.desnoyers@...icios.com; jiangshanlai@...il.com; Zhang, Jun <jun.zhang@...el.com>; Xiao, Jin <jin.xiao@...el.com>; Zhang, Yanmin <yanmin.zhang@...el.com>
Subject: Re: rcu_preempt caused oom

On Mon, Dec 03, 2018 at 07:44:03AM +0000, He, Bo wrote:
> Thanks, we have run the test for the whole weekend and not reproduce the issue,  so we confirm the CONFIG_RCU_BOOST can fix the issue.

Very good, that is encouraging.  Perhaps I should think about making CONFIG_RCU_BOOST=y the default for CONFIG_PREEMPT in mainline, at least for architectures for which rt_mutexes are implemented.

> We have enabled the rcupdate.rcu_cpu_stall_timeout=7 and also set panic on rcu stall and will see if we can see the panic, will keep you posed with the test results.
> echo 1 > /proc/sys/kernel/panic_on_rcu_stall

Looking forward to seeing what is going on!  Of course, to reproduce, you will need to again build with CONFIG_RCU_BOOST=n.

							Thanx, Paul

> -----Original Message-----
> From: Paul E. McKenney <paulmck@...ux.ibm.com>
> Sent: Saturday, December 1, 2018 12:49 AM
> To: He, Bo <bo.he@...el.com>
> Cc: Steven Rostedt <rostedt@...dmis.org>; 
> linux-kernel@...r.kernel.org; josh@...htriplett.org; 
> mathieu.desnoyers@...icios.com; jiangshanlai@...il.com; Zhang, Jun 
> <jun.zhang@...el.com>; Xiao, Jin <jin.xiao@...el.com>; Zhang, Yanmin 
> <yanmin.zhang@...el.com>
> Subject: Re: rcu_preempt caused oom
> 
> On Fri, Nov 30, 2018 at 03:18:58PM +0000, He, Bo wrote:
> > Here is the kernel cmdline:
> 
> Thank you!
> 
> > Kernel command line: androidboot.acpio_idx=0
> > androidboot.bootloader=efiwrapper-02_03-userdebug_kernelflinger-06_0
> > 3- userdebug androidboot.diskbus=00.0 
> > androidboot.verifiedbootstate=green
> > androidboot.bootreason=power-on androidboot.serialno=R1J56L6006a7bb
> > g_ffs.iSerialNumber=R1J56L6006a7bb no_timer_check noxsaves 
> > reboot_panic=p,w i915.hpd_sense_invert=0x7 mem=2G nokaslr nopti 
> > ftrace_dump_on_oops trace_buf_size=1024K intel_iommu=off gpt
> > loglevel=4 androidboot.hardware=gordon_peak 
> > firmware_class.path=/vendor/firmware relative_sleep_states=1
> > enforcing=0 androidboot.selinux=permissive cpu_init_udelay=10 
> > androidboot.android_dt_dir=/sys/bus/platform/devices/ANDR0001:00/pro
> > pe rties/android/ pstore.backend=ramoops memmap=0x1400000$0x50000000
> > ramoops.mem_address=0x50000000 ramoops.mem_size=0x1400000
> > ramoops.record_size=0x4000 ramoops.console_size=0x1000000
> > ramoops.ftrace_size=0x10000 ramoops.dump_oops=1 vga=current
> > i915.modeset=1 drm.atomic=1 i915.nuclear_pageflip=1 
> > drm.vblankoffdelay=
> 
> And no sign of any suppression of RCU CPU stall warnings.  Hmmm...
> It does take more than 21 seconds to OOM?  Or do things happen faster than that?  If they do happen faster than that, then on approach would be to add something like this to the kernel command line:
> 
> 	rcupdate.rcu_cpu_stall_timeout=7
> 
> This would set the stall timeout to seven seconds.  Note that timeouts less than three seconds are silently interpreted as three seconds.
> 
> 							Thanx, Paul
> 
> > -----Original Message-----
> > From: Steven Rostedt <rostedt@...dmis.org>
> > Sent: Friday, November 30, 2018 11:17 PM
> > To: Paul E. McKenney <paulmck@...ux.ibm.com>
> > Cc: He, Bo <bo.he@...el.com>; linux-kernel@...r.kernel.org; 
> > josh@...htriplett.org; mathieu.desnoyers@...icios.com; 
> > jiangshanlai@...il.com; Zhang, Jun <jun.zhang@...el.com>; Xiao, Jin 
> > <jin.xiao@...el.com>; Zhang, Yanmin <yanmin.zhang@...el.com>
> > Subject: Re: rcu_preempt caused oom
> > 
> > On Fri, 30 Nov 2018 06:43:17 -0800
> > "Paul E. McKenney" <paulmck@...ux.ibm.com> wrote:
> > 
> > > Could you please send me your list of kernel boot parameters?  
> > > They usually appear near the start of your console output.
> > 
> > Or just: cat /proc/cmdline
> > 
> > -- Steve
> > 
> 


Download attachment "apanic_console" of type "application/octet-stream" (27309 bytes)