netdev - Re: Linux 4.14 - regression: broken tun/tap / bridge network with virtio

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <5e2eb425-ef66-40c8-a23d-b926593fa230@01019freenet.de>
Date:   Mon, 27 Nov 2017 20:09:28 +0100
From:   Andreas Hartmann <andihartmann@...19freenet.de>
To:     Michal Kubecek <mkubecek@...e.cz>
Cc:     netdev@...r.kernel.org, john.fastabend@...il.com
Subject: Re: Linux 4.14 - regression: broken tun/tap / bridge network with
 virtio - bisected

On 11/27/2017 at 05:55 PM Michal Kubecek wrote:
> On Mon, Nov 27, 2017 at 05:46:14PM +0100, Andreas Hartmann wrote:
>>
>> Using virtio not just breaks the network completely as described above,
>> it even leaves a never stoppable or restartable qemu process (even kill
>> -9 doesn't work). It's absolutely necessary to *force* a reboot to exit
>> or restart the VM.
>>
>> I switched back to linux 4.13 as 4.14 virtualization is quite unusable.
>>
>> I'm not the only one affected:
>> https://bugzilla.kernel.org/show_bug.cgi?id=197861
> 
> What does stack trace of that process look like (/proc/$pid/stack)? Is
> it similar to the stack trace from kernel.org bugzilla?

Unfortunately, I didn't look at /proc/$pid/stack at this moment (I had
two hanging VMs). I didn't get any log entry neither on the host nor in
the VM. I tried to strace one - but strace didn't show anything, two.

As I could see at the moment now, according to the logfiles of the VMs,
they have been shutdown completely - but qemu can't be killed:

Nov 26 20:43:08 proxy systemd: Stopping Remount Root and Kernel File
Systems...
Nov 26 20:43:08 proxy systemd: Stopping Monitoring of LVM2 mirrors,
snapshots etc. using dmeventd or progress polling...
Nov 26 20:43:08 proxy systemd: Stopped Create Static Device Nodes in /dev.
Nov 26 20:43:08 proxy systemd: Stopping Create Static Device Nodes in
/dev...
Nov 26 20:43:08 proxy systemd: Reached target Shutdown.
Nov 26 20:43:08 proxy systemd: Starting Shutdown.
Nov 26 20:43:08 proxy systemd: Reached target Final Step.
Nov 26 20:43:08 proxy systemd: Starting Final Step.
Nov 26 20:43:08 proxy systemd: Starting Power-Off...
Nov 26 20:43:08 proxy systemd: Shutting down.
Nov 26 20:43:08 proxy lvm: 3 logical volume(s) in volume group "centos"
unmonitored
Nov 26 20:43:08 proxy systemd-shutdown[1]: Sending SIGTERM to remaining
processes...
Nov 26 20:43:08 proxy lvmetad[500]: Failed to accept connection errno 11.
Nov 26 20:43:08 proxy journal: Journal stopped

Anyway, I reproduced the situation at the moment to get a stack while
qemu hangs forever. It looks like this:

[<ffffffffc0d0e3c5>] vhost_net_ubuf_put_and_wait+0x35/0x60 [vhost_net]
[<ffffffffc0d0f264>] vhost_net_ioctl+0x304/0x870 [vhost_net]
[<ffffffff9b25460f>] do_vfs_ioctl+0x8f/0x5c0
[<ffffffff9b254bb4>] SyS_ioctl+0x74/0x80
[<ffffffff9b00365b>] do_syscall_64+0x5b/0x100
[<ffffffff9b78e7ab>] entry_SYSCALL64_slow_path+0x25/0x25
[<ffffffffffffffff>] 0xffffffffffffffff

Regards,
Andreas