lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Fri, 18 Nov 2016 04:56:27 -0800 From: "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com> To: Ding Tianhong <dingtianhong@...wei.com> Cc: josh@...htriplett.org, rostedt@...dmis.org, mathieu.desnoyers@...icios.com, jiangshanlai@...il.com, "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org> Subject: Re: [PATCH] rcu: Fix soft lockup for rcu_nocb_kthread On Fri, Nov 18, 2016 at 08:37:28PM +0800, Ding Tianhong wrote: > > > On 2016/8/10 9:59, Paul E. McKenney wrote: > > On Wed, Aug 10, 2016 at 09:13:14AM +0800, Ding Tianhong wrote: > >> On 2016/6/16 22:19, Paul E. McKenney wrote: > >>> On Thu, Jun 16, 2016 at 02:09:47PM +0800, Ding Tianhong wrote: > >>>> On 2016/6/15 23:49, Paul E. McKenney wrote: > >>>>> On Wed, Jun 15, 2016 at 03:27:36PM +0800, Ding Tianhong wrote: > >>>>>> I met this problem when using the Testgine to send package to ixgbevf nic > >>>>>> by this steps: > >>>>>> 1. Connect to ixgbevf, and set the speed to 10Gb/s, it could work fine. > >>>>>> 2. Then use ifconfig to down the nic and up again, loop for several times. > >>>>>> 3. The system panic by soft lockup. > >>>>> > >>>>> Good catch, queued for review and testing. But what .config was your > >>>>> kernel built with? > >>>>> > >>>> > >>>> I use the redhat7.1 defconfig to build my kernel, and the RCU config is this: > >>>> 120 # > >>>> 121 # RCU Subsystem > >>>> 122 # > >>>> 123 CONFIG_TREE_RCU=y > >>>> 124 # CONFIG_PREEMPT_RCU is not set > >>>> 125 CONFIG_RCU_STALL_COMMON=y > >>>> 126 CONFIG_CONTEXT_TRACKING=y > >>>> 127 CONFIG_RCU_USER_QS=y > >>>> 128 # CONFIG_CONTEXT_TRACKING_FORCE is not set > >>>> 129 CONFIG_RCU_FANOUT=64 > >>>> 130 CONFIG_RCU_FANOUT_LEAF=16 > >>>> 131 # CONFIG_RCU_FANOUT_EXACT is not set > >>>> 132 # CONFIG_RCU_FAST_NO_HZ is not set > >>>> 133 # CONFIG_TREE_RCU_TRACE is not set > >>>> 134 CONFIG_RCU_NOCB_CPU=y > >>>> 135 CONFIG_RCU_NOCB_CPU_ALL=y > >>>> 136 CONFIG_BUILD_BIN2C=y > >>> > >>> Thank you! You were running with preemption disabled, so your system > >>> would indeed be very susceptible to this problem. > >>> > >>>>> Also, I did tweak both the commit log and the patch. Your cond_resched() > >>>>> would prevent soft lockups, but not RCU stalls, so I substituted > >>>>> cond_resched_rcu_qs(). Please let me know if either of those changes > >>>>> causes problems at your end. > >>>> > >>>> Looks fine to me, I will apply this to my branch and test it, thanks. > >>> > >>> Please let me know how it goes! > >>> > >>> Thanx, Paul > >>> > >> > >> Hi Paul: > >> > >> It has been a long time after applying this patch, and didn't found any problem, I believe this patch is fine, thanks. > > > > Very good! I will push this one upstream during the next merge window. > > > > Thanx, Paul > > > > Hi Paul: > > Sorry to say that I have found this patch will introduce an OOM problem, it will be triggered by huge IP abnormal packet > arrived, it looks that avoid process any pending softirqs in the rcuos kthread is the best way to fix this problem, I will > send a new patch to revert this and fix the problem. Interesting... Could you please let me know exactly how the added cond_resched_rcu_qs() leads to an OOM? Is it that the softirqs prevent the grace-period kthread from making progress? Thanx, Paul
Powered by blists - more mailing lists