Message-ID: <20120914114752.GA20349@localhost>
Date:	Fri, 14 Sep 2012 19:47:52 +0800
From:	Fengguang Wu <fengguang.wu@...el.com>
To:	"Srivatsa S. Bhat" <srivatsa.bhat@...ux.vnet.ibm.com>
Cc:	paulmck@...ux.vnet.ibm.com, jack@...e.cz,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	"rusty@...tcorp.com.au" <rusty@...tcorp.com.au>,
	Peter Zijlstra <peterz@...radead.org>,
	Tejun Heo <tj@...nel.org>
Subject: Re: WARNING: at kernel/rcutree.c:1558 rcu_do_batch+0x386/0x3a0(),
 during CPU hotplug

On Thu, Sep 13, 2012 at 02:05:17PM +0530, Srivatsa S. Bhat wrote:
> On 09/12/2012 06:06 PM, Srivatsa S. Bhat wrote:
> > On 07/19/2012 10:45 PM, Paul E. McKenney wrote:
> >> On Thu, Jul 19, 2012 at 05:39:30PM +0530, Srivatsa S. Bhat wrote:
> >>> Hi Paul,
> >>>
> >>> While running a CPU hotplug stress test on v3.5-rc7+
> >>> (mainline commit 8a7298b7805ab) I hit this warning.
> >>> I haven't tried to debug this yet...
> >>>
> >>> Line number 1550 maps to:
> >>>
> >>> WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));
> >>>
> >>> inside rcu_do_batch().
> >>
> >> Hello, Srivatsa,
> >>
> >> I believe that you need commit a16b7a69 (Prevent __call_rcu() from
> >> invoking RCU core on offline CPUs), which is currently in -tip, queued
> >> for 3.6.  Please see below for the patch.
> >>
> >> Does this help?
> >>
> > 
> > Hi Paul,
> > 
> > I am hitting the cpu_is_offline() warning in rcu_do_batch() (see 2 of the
> > examples below) occasionally while testing CPU hotplug on Thomas' smp/hotplug
> > branch in -tip. It does contain the commit that you had mentioned above.
> > 
> 
> I also hit some writeback related problems during some of these runs. But I was
> not able to reproduce them after that occurrence. (Adding relevant people to CC.)
>
> I hit the divide error shown below during the CPU hotplug test run, and the general

I've hit the divide error before. And I'm not luckier than you in
reproducing the bug. I tempt to add a test as the workaround:

+       if (WARN_ON(!denominator))
+               return dirty;

btw, any chance you may share the CPU hotplug test scripts?
It'd be a valuable addition to my 0day boot test system.

> protection fault subsequently, while trying to shutdown the machine after the test.

Thanks,
Fengguang

> [  522.987310] SMP alternatives: switching to SMP code
> [  522.999101] smpboot: Booting Node 1 Processor 7 APIC 0x16
> [  524.083872] SMP alternatives: lockdep: fixing up alternatives
> [  524.090053] smpboot: Booting Node 0 Processor 8 APIC 0x1
> [  525.148720] SMP alternatives: lockdep: fixing up alternatives
> [  525.154970] smpboot: Booting Node 0 Processor 9 APIC 0x3
> [  526.024180] divide error: 0000 [#1] SMP 
> [  526.028144] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf fuse loop dm_mod iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm cdc_ether pcspkr usbnet shpchp pci_hotplug i2c_i801 i2c_core ioatdma mii crc32c_intel serio_raw microcode lpc_ich mfd_core i7core_edac bnx2 dca edac_core tpm_tis tpm sg tpm_bios rtc_cmos button uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon
> [  526.028145] CPU 9 
> [  526.028145] Pid: 2235, comm: flush-8:0 Not tainted 3.6.0-rc1-tglx-hotplug-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033     
> [  526.028145] RIP: 0010:[<ffffffff811276f6>]  [<ffffffff811276f6>] bdi_dirty_limit+0x66/0xc0
> [  526.028145] RSP: 0018:ffff8811530bfcc0  EFLAGS: 00010206
> [  526.028145] RAX: 0000000000b9877e RBX: 00000000001a8112 RCX: 28f5c28f5c28f5c3
> [  526.028145] RDX: 0000000000000000 RSI: 0000000000b9877e RDI: 0000000000000000
> [  526.028145] RBP: ffff8811530bfce0 R08: 0000000000000010 R09: 0000000000000000
> [  526.028145] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d4408e20
> [  526.028145] R13: ffff8808d4408e20 R14: ffff8808d44091a0 R15: 0000000000000000
> [  526.028145] FS:  0000000000000000(0000) GS:ffff8808ddd40000(0000) knlGS:0000000000000000
> [  526.028145] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  526.028145] CR2: 00007fa35dd4eb60 CR3: 0000000001a0c000 CR4: 00000000000007e0
> [  526.028145] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  526.028145] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [  526.028145] Process flush-8:0 (pid: 2235, threadinfo ffff8811530be000, task ffff88115315c5e0)
> [  526.028145] Stack:
> [  526.028145]  0400000000000000 0000000000000007 ffff8808d4408e20 ffffffffffffffee
> [  526.028145]  ffff8811530bfd10 ffffffff811ae95c 0000000000350225 00000000001a8112
> [  526.028145]  0000000000000000 0000000000000002 ffff8811530bfdc0 ffffffff811b0620
> [  526.028145] Call Trace:
> [  526.209272] SMP alternatives: lockdep: fixing up alternatives
> [  526.209275] smpboot: Booting Node 0 Processor 10 APIC 0x5
> [  526.220012]  [<ffffffff811ae95c>] over_bground_thresh+0x7c/0x90
> [  526.220012]  [<ffffffff811b0620>] wb_do_writeback+0x170/0x310
> [  526.220012]  [<ffffffff811b08eb>] bdi_writeback_thread+0x12b/0x420
> [  526.220012]  [<ffffffff811b07c0>] ? wb_do_writeback+0x310/0x310
> [  526.220012]  [<ffffffff8106deae>] kthread+0xde/0xf0
> [  526.220012]  [<ffffffff814c6184>] kernel_thread_helper+0x4/0x10
> [  526.220012]  [<ffffffff814bc1f0>] ? retint_restore_args+0x13/0x13
> [  526.220012]  [<ffffffff8106ddd0>] ? __init_kthread_worker+0x70/0x70
> [  526.220012]  [<ffffffff814c6180>] ? gs_change+0x13/0x13
> [  526.220012] Code: 28 5c 8f c2 f5 28 8b 7d e0 48 89 c6 48 0f af f3 48 c1 ee 02 48 89 f0 48 f7 e1 48 89 d6 31 d2 48 c1 ee 02 48 0f af 75 e8 48 89 f0 <48> f7 f7 41 8b 94 24 74 02 00 00 48 0f af d3 48 89 c7 48 c1 ea 
> [  526.220012] RIP  [<ffffffff811276f6>] bdi_dirty_limit+0x66/0xc0
> [  526.220012]  RSP <ffff8811530bfcc0>
> [  526.304469] ---[ end trace bcfc7ab74bdb11a5 ]---
> [  527.330948] SMP alternatives: lockdep: fixing up alternatives
> 
> 
> ----
> 
> [ 1941.614775] SMP alternatives: lockdep: fixing up alternatives
> [ 1941.620614] smpboot: Booting Node 1 Processor 5 APIC 0x12
> [ 1941.657424] SMP alternatives: lockdep: fixing up alternatives
> [ 1941.663215] smpboot: Booting Node 1 Processor 6 APIC 0x14
> 
> [ 1992.721819] general protection fault: 0000 [#2] SMP 
> [ 1992.724844] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf fuse loop dm_mod iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm cdc_ether pcspkr usbnet shpchp pci_hotplug i2c_i801 i2c_core ioatdma mii crc32c_intel serio_raw microcode lpc_ich mfd_core i7core_edac bnx2 dca edac_core tpm_tis tpm sg tpm_bios rtc_cmos button uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase
> scsi_transport_sas scsi_mod thermal thermal_sys hwmon
> [ 1992.726995] CPU 8 
> [ 1992.726995] Pid: 19654, comm: shutdown Tainted: G      D      3.6.0-rc1-tglx-hotplug-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033     
> [ 1992.726995] RIP: 0010:[<ffffffff810843d7>]  [<ffffffff810843d7>] try_to_wake_up+0x57/0x2f0
> [ 1992.726995] RSP: 0018:ffff8808d47e5e58  EFLAGS: 00010002
> [ 1992.726995] RAX: 6b6b6b6b6b6b6b6b RBX: 000000000000000f RCX: 000000006b6b6b6b
> [ 1992.726995] RDX: 000000006b6c6b6b RSI: ffffffff817a7fbf RDI: ffff88115315cdd0
> [ 1992.726995] RBP: ffff8808d47e5e98 R08: 0000000000000002 R09: 0000000000000001
> [ 1992.726995] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88115315c5e0
> [ 1992.726995] R13: 000000006b6b6b6b R14: ffff8808d44091a0 R15: 0000000000000000
> [ 1992.726995] FS:  00007f32b1beb700(0000) GS:ffff8808ddd00000(0000) knlGS:0000000000000000
> [ 1992.726995] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 1992.726995] CR2: 00007f32b173d1a0 CR3: 0000001153bd0000 CR4: 00000000000007e0
> [ 1992.726995] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 1992.726995] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 1992.726995] Process shutdown (pid: 19654, threadinfo ffff8808d47e4000, task ffff8808d4b045e0)
> [ 1992.726995] Stack:
> [ 1992.726995]  0000000000000246 ffff88115315cdd0 0000000000000286 0000000000000000
> [ 1992.726995]  ffff8808d4408e20 ffff8808d5433288 ffff8808d44091a0 00746c6168206d65
> [ 1992.726995]  ffff8808d47e5ea8 ffffffff810846a0 ffff8808d47e5ed8 ffffffff811adce1
> [ 1992.726995] Call Trace:
> [ 1992.726995]  [<ffffffff810846a0>] wake_up_process+0x10/0x20
> [ 1992.726995]  [<ffffffff811adce1>] bdi_queue_work+0xd1/0x1f0
> [ 1992.726995]  [<ffffffff811ae7d9>] __bdi_start_writeback+0x79/0x160
> [ 1992.726995]  [<ffffffff811af1b0>] wakeup_flusher_threads+0x120/0x1e0
> [ 1992.726995]  [<ffffffff811af0ca>] ? wakeup_flusher_threads+0x3a/0x1e0
> [ 1992.726995]  [<ffffffff811b45b2>] sys_sync+0x22/0x90
> [ 1992.726995]  [<ffffffff814c4fb9>] system_call_fastpath+0x16/0x1b
> [ 1992.726995] Code: 31 ed 48 89 c7 48 89 45 c8 e8 46 72 43 00 48 89 45 d0 49 8b 04 24 85 c3 0f 84 02 02 00 00 45 8b 6c 24 2c 49 8b 44 24 08 45 85 ed <44> 8b 70 18 74 75 8b 1d dd ea 9d 00 85 db 0f 85 2d 02 00 00 49  
> [ 1992.726995] RIP  [<ffffffff810843d7>] try_to_wake_up+0x57/0x2f0
> [ 1992.726995]  RSP <ffff8808d47e5e58>
> [ 1992.726995] ---[ end trace bcfc7ab74bdb11a6 ]---
> [ 1992.726995] Kernel panic - not syncing: Fatal exception in interrupt
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists