linux-kernel - BUG: soft lockup detected on CPU + cifs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:	Sun, 17 Sep 2006 18:13:57 -0400
From:	Russell Leighton <russ@...gant-software.com>
To:	Linux Kernel list <linux-kernel@...r.kernel.org>
Subject: BUG: soft lockup detected on CPU + cifs_getattr


For the last few months we have had "hangs" where the machine
will respond to pings , but nothing else. After upgrading the kernel
we now get a useful message when it hangs, so I thought that
with the message and an example program perhaps someone can track this down.

The nature of the issue is with a program that execs many processes
quickly in succession on an SMP machine and these processes read the 
some of the same
files over a CIFS share. At random times the machine hangs under this load.
I created a shell script (below) that does a similar thing and can 
reliably cause the
machine to hang.

We are running:

    Linux prodeng2 2.6.17-1.2142_FC4smp #1 SMP Tue Jul 11 22:57:02 EDT
    2006 i686 i686 i386 GNU/Linux

we get the following error:

Sep 16 00:04:30 prodeng2 kernel: BUG: soft lockup detected on CPU#2!
Sep 16 00:04:30 prodeng2 kernel:  <c0447823> softlockup_tick+0x9f/0xb2  <c042b497> update_process_times+0x39/0x5c
Sep 16 00:04:30 prodeng2 kernel:  <c041737d> smp_apic_timer_interrupt+0x54/0x5a  <c0404800> apic_timer_interrupt+0x1c/0x24
Sep 16 00:04:30 prodeng2 kernel:  <c046eed9> generic_fillattr+0x71/0x9f  <f8ff18db> cifs_getattr+0x1e/0x27 [cifs]
Sep 16 00:04:30 prodeng2 kernel:  <f8ff18bd> cifs_getattr+0x0/0x27 [cifs]  <c046ef45> vfs_getattr+0x3e/0x97
Sep 16 00:04:30 prodeng2 kernel:  <c046f051> vfs_fstat+0x22/0x31  <c046f724> sys_fstat64+0xf/0x23
Sep 16 00:04:30 prodeng2 kernel:  <c0403d2f> syscall_call+0x7/0xb 
Sep 16 00:04:30 prodeng2 kernel: BUG: soft lockup detected on CPU#1!
Sep 16 00:04:30 prodeng2 kernel:  <c0447823> softlockup_tick+0x9f/0xb2  <c042b497> update_process_times+0x39/0x5c
Sep 16 00:04:30 prodeng2 kernel:  <c041737d> smp_apic_timer_interrupt+0x54/0x5a  <c0404800> apic_timer_interrupt+0x1c/0x24
Sep 16 00:04:30 prodeng2 kernel:  <c046eecd> generic_fillattr+0x65/0x9f  <f8ff18db> cifs_getattr+0x1e/0x27 [cifs]
Sep 16 00:04:30 prodeng2 kernel:  <f8ff18bd> cifs_getattr+0x0/0x27 [cifs]  <c046ef45> vfs_getattr+0x3e/0x97
Sep 16 00:04:30 prodeng2 kernel:  <c046f051> vfs_fstat+0x22/0x31  <c046f724> sys_fstat64+0xf/0x23
Sep 16 00:04:30 prodeng2 kernel:  <c0403d2f> syscall_call+0x7/0xb 

  

...after googling a little, the closest similar report I find is:

    http://blog.gmane.org/gmane.comp.emulators.xen.devel/day=20060916

To tickle this bug do:

   1. Create a CIFS  share, ideally on a  real Windows server since that
      is the current context of the issue I am reporting
   2. Run this shell script using a path to a file on the share:

file_on_cifs_mnt=$1

while true
do
# limited fork bomb...repeat until hang
c=1;
while [ $c -lt 10000 ]
do
 # cif mounted file will cause hang
 (cat $file_on_cifs_mnt > /dev/null ; ) &
 # local file won't hang me
 # (cat /etc/passwd > /dev/null ; ) &
 let c=c+1
done
wait
done

NOTE: sometimes it would hang immediately, sometimes after a time, 
sometimes I had to run 2 instances of the script concurrently to get the 
BUG.

If anyone has a fix OR a work around I would very much appreciate the help.

Thx.

Russ


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/