linux-kernel - Re: BUG: soft lockup - CPU#3 stuck for 61s! , while running cpu controller latency testcase on two containers parallaly

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <4AAFA0B3.4050805@in.ibm.com>
Date:	Tue, 15 Sep 2009 19:42:03 +0530
From:	Sharyathi Nagesh <sharyath@...ibm.com>
To:	Dhaval Giani <dhaval@...ux.vnet.ibm.com>
CC:	Rishikesh <risrajak@...ux.vnet.ibm.com>, containers@...ts.osdl.org,
	linux-kernel@...r.kernel.org, iranna.ankad@...ibm.com,
	bharanga@...ibm.com, sharyath@...ux.vnet.ibm.com,
	risrajak@...ibm.com, bharata.rao@...ibm.com, mbeeraka@...ibm.com,
	svishuku@...ibm.com, santwana.samantray@...ibm.com,
	Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>
Subject: Re: BUG: soft lockup - CPU#3 stuck for 61s! , while running cpu controller
 latency testcase on two containers parallaly


The same problem was observed while testing with 2.6.29 kernel, this is what I could gather 
from looking into the error message,

Dhaval Giani wrote:
> [Adding the scheduler maintainers to the cc]
> 
>> BUG: soft lockup - CPU#3 stuck for 61s! [cpuctl_latency_:17174]
>> Modules linked in: bridge stp llc bnep sco l2cap bluetooth sunrpc ipv6 
>> p4_clockmod dm_multipath uinput qla2xxx ata_generic pata_acpi 
>> usb_storage e1000 scsi_transport_fc joydev scsi_tgt i2c_piix4 
>> pata_serverworks pcspkr serio_raw mptspi mptscsih mptbase 
>> scsi_transport_spi radeon ttm drm i2c_algo_bit i2c_core [last unloaded: 
>> scsi_wait_scan]
>>
>> Pid: 17174, comm: cpuctl_latency_ Tainted: G        W  (2.6.31-rc7 #1) 
>> IBM eServer BladeCenter HS40 -[883961X]-                    
>> EIP: 0060:[<c058aded>] EFLAGS: 00000283 CPU: 3
>> EIP is at find_next_bit+0x9/0x79
>> EAX: c2c437a0 EBX: f3d433c0 ECX: 00000000 EDX: 00000020
>> ESI: c2c436bc EDI: 00000000 EBP: f063be6c ESP: f063be64
>>  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
>> CR0: 80050033 CR2: 008765a4 CR3: 314d7000 CR4: 000006d0
>> DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
>> DR6: ffff0ff0 DR7: 00000400
>> Call Trace:
>>  [<c0427b6e>] cpumask_next+0x17/0x19


EIP matches to 'tg_shares_up+0xd9/0x149',
-----------------------
   This maps to address location 5c37 in the objdump
/root/rpmbuild/BUILD/kernel-2.6.29/linux-2.6.29.i686/kernel/sched.c:1511

         if (abs(shares - tg->se[cpu]->load.weight) >
     5c33:       89 d0                   mov    %edx,%eax
     5c35:       2b 01                   sub    (%ecx),%eax
     5c37:       89 c1                   mov    %eax,%ecx
     5c39:       c1 f9 1f                sar    $0x1f,%ecx
     5c3c:       31 c8                   xor    %ecx,%eax
     5c3e:       29 c8                   sub    %ecx,%eax
     5c40:       3b 05 00 00 00 00       cmp    0x0,%eax
     5c46:       76 3d                   jbe    5c85 <tg_shares_up+0x127>
/root/rpmbuild/BUILD/kernel-2.6.29/linux-2.6.29.i686/kernel/sched.c:1516
                         sysctl_sched_shares_thresh) {
                 struct rq *rq = cpu_rq(cpu);
-----------------------
This matches to line 1511 - 1516 in the kernel code

1511                if (abs(shares - tg->se[cpu]->load.weight) >
1512                         sysctl_sched_shares_thresh) {
1513                 struct rq *rq = cpu_rq(cpu);
1514                 unsigned long flags;
1515
1516                 spin_lock_irqsave(&rq->lock, flags);
-----------------------
I suspect this soft lock ups are due to rq->lock contention.
Thanks
Sharyathi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/