linux-kernel - Re: [PATCH] virtio: Remove virtio device during shutdown

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150312233552.GB32054@ad.nay.redhat.com>
Date:	Fri, 13 Mar 2015 07:35:52 +0800
From:	Fam Zheng <famz@...hat.com>
To:	"Michael S. Tsirkin" <mst@...hat.com>
Cc:	linux-kernel@...r.kernel.org,
	Rusty Russell <rusty@...tcorp.com.au>,
	virtualization@...ts.linux-foundation.org,
	Paolo Bonzini <pbonzini@...hat.com>,
	Jason Wang <jasowang@...hat.com>
Subject: Re: [PATCH] virtio: Remove virtio device during shutdown

On Thu, 03/12 17:22, Michael S. Tsirkin wrote:
> On Wed, Mar 11, 2015 at 06:11:35PM +0800, Fam Zheng wrote:
> > On Wed, 03/11 10:06, Michael S. Tsirkin wrote:
> > > On Wed, Mar 11, 2015 at 04:09:17PM +0800, Fam Zheng wrote:
> > > > Currently shutdown is nop for virtio devices, but the core code could
> > > > remove things behind us such as MSI-X handler etc. For example in the
> > > > case of virtio-scsi-pci, the device may still try to send interupts,
> > > > which will be on IRQ lines seeing MSI-X disabled. Those interrupts will
> > > > be unhandled, and may cause flood.
> > 
> > Here is the problem I want to solve - file system driver hang:
> > 
> > If a fs code happen to hit __wait_on_buffer right after pci pci_device_shutdown
> > disabled msix, it will never make progress because the requests it waits for
> > will never be completed. So the system hangs.
> 
> Paolo says that pci reset of virtio scsi device guarantees
> that all outstanding requests complete.
> 
> If true and implemented correctly, I don't see what else
> needs to be done.
> 
> You will need to debug this some more.

First of all I was wrong about the fs driver above, scratch that, I'm sorry for
the misleading.

Regarding the hang in shutdown, Ulrich Obergfell has already pointed out that
the vcpu is "busy/stuck in interrupt processing":

https://bugzilla.redhat.com/attachment.cgi?id=998391 (RHBZ 1199155)

Summary: The reason it is stuck is that an IRQ from virtio-scsi-pci is not
handled. Why is there that IRQ? Because pci core code disabled msix. Why is it
not handled?  Because it's done behind virtio-scsi, who still is waiting for
msix.

"Hence, the interrupt will not be acknowledged and the guest becomes flooded
with IRQ 11 interrupt."

Fortunately it's not a livelock for upstream, because of:

    commit 184564efae4d775225c8fe3b762a56956fb1f827
    Author: Zhang Haoyu <zhanghy@...gfor.com>
    Date:   Thu Sep 11 16:47:04 2014 +0800

    kvm: ioapic: conditionally delay irq delivery duringeoi broadcast

But we still should do the shutdown right.

I also propose to not shutdown msix from pci core shutdown if the device
doesn't have shutdown function:

http://www.spinics.net/lists/kernel/msg1944041.html

With that patch is applied, the "nop" .shutdown in virtio-pci shouldn't hurt
much.

Regarding handing the requests, now I don't know if we really care about them
at shutdown. As you said, waiting for requests may cause more hang.

Ideas?

Fam
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/