linux-kernel - PROBLEM: All CPUs in soft lockup

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130327015540.GA27623@pyro.melbourne.osa>
Date:	Wed, 27 Mar 2013 12:55:41 +1100
From:	Robert Norris <robn@...tmail.fm>
To:	linux-kernel@...r.kernel.org
Subject: PROBLEM: All CPUs in soft lockup

In the last two weeks we've had three servers (identical hardware,
software and load) hang. The details in this report are from one that
hung last night.

They're all IMAP servers servicing many hundreds of users, so several
thousand processes and active connections. There's been two major
application level changes in the last couple of weeks, corresponding to
the time where these hangs started. One is that we now do mail event
notifications directly to user clients, so more TCP connections. The
other is that we're now maintaining live search indexes, so a lot more
disk and tmpfs IO.

All that said, we're not under what we'd consider to be heavy load. When
they're running, the servers are fast and responsive.

During the hang itself, the machine responds to pings, and TCP
connections can be established, but the servicing processes never
respond. The console shows a new "BUG: soft lockup" line every few
seconds, and will not respond to keyboard input. It is a virtual console
though, which may or may not make a difference, I'm not sure.

The kernel is 3.4.33 with AUFS patches applied. However there are no
AUFS mounts on this machine; we use this elsewhere. If you think that's
a problem I can rebuild for this machine without it.

Attached are various bits of information requested in REPORTING-BUGS.
I'm not entirely sure what else is relevant. I'm happy to supply any
other information and test things, just let me know.

Thanks,
Rob.

View attachment "4.1.version" of type "text/plain" (116 bytes)

View attachment "4.2.config" of type "text/plain" (119602 bytes)

Download attachment "6.messages.gz" of type "application/octet-stream" (185688 bytes)

View attachment "8.1.software" of type "text/plain" (1441 bytes)

View attachment "8.2.cpuinfo" of type "text/plain" (14114 bytes)

View attachment "8.3.modules" of type "text/plain" (4095 bytes)

View attachment "8.4.ioports-iomem" of type "text/plain" (3190 bytes)

View attachment "8.5.pci" of type "text/plain" (65535 bytes)

View attachment "8.6.scsi" of type "text/plain" (1132 bytes)