linux-kernel - Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CALCETrU7SOhTi0g0=tvcirn_C4s9oTkhaV8BfbDf2F90+himJA@mail.gmail.com>
Date:	Mon, 7 Dec 2015 17:59:41 -0800
From:	Andy Lutomirski <luto@...capital.net>
To:	Dominique Martinet <dominique.martinet@....fr>
Cc:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	V9FS Developers <v9fs-developer@...ts.sourceforge.net>,
	Linux FS Devel <linux-fsdevel@...r.kernel.org>
Subject: Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race

On Mon, Dec 7, 2015 at 2:46 PM, Dominique Martinet
<dominique.martinet@....fr> wrote:
> Andy Lutomirski wrote on Mon, Dec 07, 2015:
>> On Thu, Dec 3, 2015 at 9:52 PM, Andy Lutomirski <luto@...capital.net> wrote:
>> > Sometimes udevadm trigger --action=add hangs the system, and the splat
>> > below happens.  This seems to be timing dependent, and I haven't been
>> > able to trigger it yet with lockdep enabled, sadly.
>> >
>> > Any ideas?  I not, I'll try to instrument it better tomorrow.
>>
>> More details: this is caused by a storm of /sbin/hotplug UMH calls
>> (yes, misconfigured kernel, but still).  /sbin is a symlink to
>> /usr/sbin, /usr/sbin/hotplug doesn't exist, and all of the above is on
>> rootfs, which is 9p over virtio.
>>
>> Pointing uevent_helper at /usr/sbin/hotplug (which still doesn't
>> exist) seems to work around it.
>
> Can you reproduce it on a booted system with something like
> `seq 1 1000000 | xargs -P 1024 -I{} cat /sbin/foo >&/dev/null` ?

This doesn't reproduce it.

This doesn't either:

seq 1 1000000 | xargs -P 1024 -I{} bash -c 'exec /sbin/foo' &>/dev/null

>
> (trying execs might be closer to your workload, not sure how much this
> or using umh might change)
>
>
> Also, what qemu version please just to try to match your environment ?

$ qemu-system-x86_64 --version
QEMU emulator version 2.4.1 (qemu-2.4.1-1.fc23), Copyright (c)
2003-2008 Fabrice Bellard

My reproducer is:

$ virtme-run --kdir . --pwd

using this virtme version:

https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git/commit/?id=17363c2900e8b796c80c920c6fcdcc6747784ef7

Bad kernel config attached.  This config with v4.4-rc3 (and no
additional patches) reproduces it reliably for me.

With the latest virtme, I don't reproduce it -- the latest virtme
turns off uevent_helper early in boot, which suppresses the bug for
me, at least most of the time.


If I dump all task states (see attached typescript), I see a bunch of
things blocked in 9p rpc.  This makes me think it could be a QEMU bug,
not a kernel bug.

--Andy



-- 
Andy Lutomirski
AMA Capital Management, LLC

Download attachment "bad-config.xz" of type "application/x-xz" (21352 bytes)

Download attachment "typescript.xz" of type "application/x-xz" (18584 bytes)