linux-kernel - Re: PROBLEM: All CPUs in soft lockup

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Thu, 28 Mar 2013 11:22:31 +1100
From:	Robert Norris <robn@...ra.com>
To:	linux-kernel@...r.kernel.org
Subject: Re: PROBLEM: All CPUs in soft lockup

On Wed, Mar 27, 2013 at 12:55:41PM +1100, Robert Norris wrote:
> The console shows a new "BUG: soft lockup" line every few seconds

Looking closer, the whole thing starts with a _hard_ lockup.

  2013-03-26T08:33:39.921834-04:00 imap30 kernel: [185090.090328] Watchdog detected hard LOCKUP on cpu 3

(also in the logs of the other two servers I mentioned).

Looking down to where the watchdog interrupt comes in:

  2013-03-26T08:33:39.921870-04:00 imap30 kernel: [185090.090426] <<EOE>>  <IRQ>  [<ffffffff8112a5b1>] ? end_buffer_async_read+0x79/0xff

Disassembling:

  0xffffffff8112a57a <+66>:    mov    %rbx,%rdi
  0xffffffff8112a57d <+69>:    callq  0xffffffff81129265 <buffer_io_error>
  0xffffffff8112a582 <+74>:    lock orb $0x2,0x0(%rbp)
  0xffffffff8112a587 <+79>:    mov    0x0(%rbp),%rax
  0xffffffff8112a58b <+83>:    test   $0x8,%ah
  0xffffffff8112a58e <+86>:    jne    0xffffffff8112a594 <end_buffer_async_read+92>
  0xffffffff8112a590 <+88>:    ud2    
  0xffffffff8112a592 <+90>:    jmp    0xffffffff8112a592 <end_buffer_async_read+90>

That lock at +74 is presumably the offender here. Which is line 275 of fs/buffer.c:

  275                 SetPageError(page);

So another CPU has these page flags locked right now, and isn't keen to
release that lock?

I don't know how to debug this further. What's the next step?

Thanks,
Rob.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/