linux-kernel - Re: Crash in rbd, need advice

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CALFYKtAGBizs8gDariQH2NJACjoRD=2-1u-zwnPfo7D5bVvy3Q@mail.gmail.com>
Date:	Fri, 4 Apr 2014 15:08:41 +0400
From:	Ilya Dryomov <ilya.dryomov@...tank.com>
To:	Hannes Landeholm <hannes@...pstarter.io>
Cc:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Ceph Development <ceph-devel@...r.kernel.org>,
	Sage Weil <sage@...tank.com>, Sage Weil <sage@...dream.net>,
	Alex Elder <elder@...tank.com>,
	Yehuda Sadeh <yehuda@...newdream.net>,
	Thorwald Lundqvist <thorwald@...pstarter.io>
Subject: Re: Crash in rbd, need advice

On Wed, Apr 2, 2014 at 12:18 AM, Hannes Landeholm <hannes@...pstarter.io> wrote:
> Hello,
>
> We're running a couple of Arch Linux servers of version 3.13.5-1 in
> production and suddenly one of them had a strange problem after
> running for a few days. One process (pid 319) was running with a few
> threads, one of those threads (pid 322) was eating 100% cpu. I assumed
> it was stuck in an infinite loop (this was our own software so I
> assumed we had a bug) so I sent a SIGKILL to 319 which caused all
> other threads to exit and it turning into a zombie, but thread 322 was
> still running. After trying to stop some other services and failing I
> realized that sending any signals to any process now didn't work at
> all in the system.
>
> This was the process stack output:
>
> $ cat /proc/319/stack
> [<ffffffff810642fa>] do_exit+0x73a/0xa80
> [<ffffffff810646bf>] do_group_exit+0x3f/0xa0
> [<ffffffff81073295>] get_signal_to_deliver+0x295/0x5f0
> [<ffffffff810144a8>] do_signal+0x48/0x950
> [<ffffffff81014e18>] do_notify_resume+0x68/0xa0
> [<ffffffff8152326a>] int_signal+0x12/0x17
> [<ffffffffffffffff>] 0xffffffffffffffff
> $ cat /proc/319/task/322/stack
> [<ffffffff8151c11a>] error_exit+0x2a/0x60
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> We're using ceph + rbd and this happened right after doing a rbd
> mapping (mounting it) or during the mapping itself, so we suspected
> rbd.
>
> A few days later (today) we had a server crash in another server, same
> version+distro and it had also just been running a few days as well.
> After starting it again we found the following in the system log:
>
> hostname kernel: BUG: unable to handle kernel paging request at ffff87fff75ad450
> hostname kernel: IP: [<ffffffffa018c196>] rbd_img_request_fill+0x126/0x930 [rbd]

Can you try gdb'ing that exact rbd.ko and

(gdb) list *rbd_img_request_fill+0x126

Also, the entire stack trace pertaining to rbd_img_request_fill would
help.

Thanks,

                Ilya
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/