linux-kernel - Re: [PATCH -tip/core/rcu 1/6] Cleanups and fixes for RCU in face of heavy CPU-hotplug stress

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20090819115952.GC32317@elte.hu>
Date:	Wed, 19 Aug 2009 13:59:52 +0200
From:	Ingo Molnar <mingo@...e.hu>
To:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
Cc:	Josh Triplett <josht@...ux.vnet.ibm.com>,
	linux-kernel@...r.kernel.org, laijs@...fujitsu.com,
	dipankar@...ibm.com, akpm@...ux-foundation.org,
	mathieu.desnoyers@...ymtl.ca, dvhltc@...ibm.com, niv@...ibm.com,
	tglx@...utronix.de, peterz@...radead.org, rostedt@...dmis.org,
	hugh.dickins@...cali.co.uk, benh@...nel.crashing.org
Subject: Re: [PATCH -tip/core/rcu 1/6] Cleanups and fixes for RCU in face
	of heavy CPU-hotplug stress


* Paul E. McKenney <paulmck@...ux.vnet.ibm.com> wrote:

> On Tue, Aug 18, 2009 at 01:07:01PM -0700, Paul E. McKenney wrote:
> > On Tue, Aug 18, 2009 at 05:26:43PM +0200, Ingo Molnar wrote:
> > > 
> > > FYI, i've started triggering hangs in -tip testing recently, during 
> > > CPU hotplug tests:
> > > 
> > > [   57.632003] eth0: no IPv6 routers present
> > > [  103.564010] kmemleak: 29 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
> > > [  200.380003] Hangcheck: hangcheck value past margin!
> > > [  248.192003] INFO: task S99local:2974 blocked for more than 120 seconds.
> > > [  248.194532] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [  248.202330] S99local      D 0000000c  6256  2974   2687 0x00000000
> > > [  248.208929]  9c7ebe90 00000086 6b67ef8b 0000000c 9f25a610 81a69869 00000001 820b6990
> > > [  248.216123]  820b6990 820b6990 9c6e4c20 9c6e4eb4 82c78990 00000000 6b993559 0000000c
> > > [  248.220616]  9c7ebe90 8105f22a 9c6e4eb4 9c6e4c20 00000001 9c7ebe98 9c7ebeb4 81a65cb3
> > > [  248.229990] Call Trace:
> > > [  248.234049]  [<81a69869>] ? _spin_unlock_irqrestore+0x22/0x37
> > > [  248.239769]  [<8105f22a>] ? prepare_to_wait+0x48/0x4e
> > > [  248.244796]  [<81a65cb3>] rcu_barrier_cpu_hotplug+0xaa/0xc9
> > > [  248.250343]  [<8105f029>] ? autoremove_wake_function+0x0/0x38
> > > [  248.256063]  [<81062cf2>] notifier_call_chain+0x49/0x71
> > > [  248.261263]  [<81062da0>] raw_notifier_call_chain+0x11/0x13
> > > [  248.266809]  [<81a0b475>] _cpu_down+0x272/0x288
> > > [  248.271316]  [<81a0b4d5>] cpu_down+0x4a/0xa2
> > > [  248.275563]  [<81a0c48a>] store_online+0x2a/0x5e
> > > [  248.280156]  [<81a0c460>] ? store_online+0x0/0x5e
> > > [  248.284836]  [<814ddc35>] sysdev_store+0x20/0x28
> > > [  248.289429]  [<8112e403>] sysfs_write_file+0xb8/0xe3
> > > [  248.294369]  [<8112e34b>] ? sysfs_write_file+0x0/0xe3
> > > [  248.299396]  [<810e4c8f>] vfs_write+0x91/0x120
> > > [  248.303817]  [<810e4dc1>] sys_write+0x40/0x65
> > > [  248.308150]  [<81002d73>] sysenter_do_call+0x12/0x28
> > > 
> > > config and bootlog attached. I'd suspect one of these patches:
> > > 
> > > 684ca5c: rcu: Fix typo in rcu_irq_exit() comment header
> > > b612ba8: rcu: Make rcupreempt_trace.c look at offline CPUs
> > > 8064d54: rcu: Make preemptable RCU scan all CPUs when summing RCU counters
> > > 2e59755: rcu: Simplify RCU CPU-hotplug notification
> > > 799e64f: cpu hotplug: Introduce cpu_notifier() to handle !HOTPLUG_CPU case
> > > 2756962: rcu: Split hierarchical RCU initialization into boot-time and CPU-online piece
> > > 
> > > Any ideas?
> > 
> > Gah...  I thought I had fixed that one!!!  I was seeing a deadlock
> > where rcu_barrier_cpu_hotplug() would register the three RCU callbacks,
> > then wait for them.  But in some situations, it would wait for them in
> > a state such that grace period could not complete.  I convinced myself
> > that moving the wait back from CPU_DEAD to CPU_POST_DEAD solved the
> > problem.
> > 
> > I am going to take a more bullet-proof approach, switching from the
> > wait_completion() form to wait_event(), which will allow me to wait
> > for the previous hotplug operation's callbacks at the beginning of the
> > subsequent hotplug operation.
> > 
> > I reserve the right to insert a short delay in the CPU-hotplug path
> > outside of any locks, but would imagine that people would prefer that
> > I avoid that sort of thing, at least until we have bulk CPU-hotplug
> > operations.
> 
> And here is a patch that is doing well in testing thus far.  (On 
> the other hand tip/core/rcu did fine in my testing.)  I am not 
> 100% confident that this new patch hitting the core 
> RCU/CPU-hotplug issue, but this is in any case helpful in getting 
> an RCU grace period off of the CPU hotunplug critical path.
> 
> Feel free to test if convenient.  The other thing I am 
> considering is moving the registering of the three 
> rcu_migrate_head callbacks from the CPU_DYING notifier to the 
> CPU_POST_DEAD notifier.

Thanks Paul, i've queued it up and started testing it. It took 
quite some time to trigger it so the test period needed will be 
measured in days (perhaps up to a week), not minutes or hours.

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/