[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <20190102145652.02b1046c@oc2783563651>
Date: Wed, 2 Jan 2019 14:56:52 +0100
From: Halil Pasic <pasic@...ux.ibm.com>
To: Cornelia Huck <cohuck@...hat.com>
Cc: "Wang, Wei W" <wei.w.wang@...el.com>,
Christian Borntraeger <borntraeger@...ibm.com>,
"virtio-dev@...ts.oasis-open.org" <virtio-dev@...ts.oasis-open.org>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"virtualization@...ts.linux-foundation.org"
<virtualization@...ts.linux-foundation.org>,
"kvm@...r.kernel.org" <kvm@...r.kernel.org>,
"mst@...hat.com" <mst@...hat.com>,
"pbonzini@...hat.com" <pbonzini@...hat.com>,
"dgilbert@...hat.com" <dgilbert@...hat.com>
Subject: Re: [virtio-dev] RE: [PATCH v1 0/2] Virtio: fix some vq allocation
issues
On Wed, 2 Jan 2019 10:53:14 +0100
Cornelia Huck <cohuck@...hat.com> wrote:
> On Tue, 1 Jan 2019 00:40:19 +0100
> Halil Pasic <pasic@...ux.ibm.com> wrote:
>
> > On Mon, 31 Dec 2018 06:03:51 +0000
> > "Wang, Wei W" <wei.w.wang@...el.com> wrote:
> >
> > > On Sunday, December 30, 2018 2:06 PM, Halil Pasic wrote:
> > > >
> > > > I guess you are the first one trying to read virtio config from within interrupt
> > > > context. AFAICT this never worked.
> > >
> > > I'm not sure about "never worked". It seems to work well with virtio-pci.
> > > But looking forward to hearing a solid reason why reading config inside
> > > the handler is forbidden (if that's true).
> >
> > By "never worked" I meant "never worked with virtio-ccw". Sorry
> > about the misunderstanding. Seems I've also failed to convey that I don't
> > know if reading config inside the handler is forbidden or not. So please
> > don't expect me providing the solid reasons you are looking forward to.
>
> It won't work with the current code, and this is all a bit ugly :( More
> verbose explanation below.
>
> >
> > >
> > > > About what happens. The apidoc of ccw_device_start() says it needs to be
> > > > called with the ccw device lock held, so ccw_io_helper() tries to take it (since
> > > > forever I guess). OTOH do_cio_interrupt() takes the subchannel lock and
> > > > io_subchannel_initialize_dev() makes the ccw device lock be the subchannel
> > > > lock. That means when one tries to get virtio config form within a cio
> > > > interrupt context we deadlock, because we try to take a lock we already have.
> > > >
> > > > That said, I don't think this limitation is by design (i.e. intended).
> > > > Maybe Connie can help us with that question. AFAIK we have nothing
> > > > documented regarding this (neither that can nor can't).
>
> The main problem is that channel I/O is a fundamentally asynchronous
> mechanism. As channel devices don't have the concept of config spaces
> (or some other things that virtio needs), I decided to map
> reading/writing the config space to channel commands. Starting I/O on a
> subchannel always needs the lock (to avoid races on the subchannel),
> and the asynchronous interrupt for that I/O needs the lock as well (for
> the same reason; things like the scsw contain state that you want to
> access without races). A config change also means that the subchannel
> becomes state pending (and an interrupt is made pending), so the
> subchannel lock is taken for that path as well. (Virtqueue
> notifications are handled differently on modern QEMU, but that does not
> come into play here.)
>
Besides locking (thinking along the lines that we work around the
lock problem somehow) there is also the new PSW which masks IO
interrupts. As I said, doing something about this seems non-trivial at
least.
> > > >
> > > > Obviously, there are multiple ways around this problem, and at the moment
> > > > I can't tell which would be my preferred one.
> > >
> > > Yes, it's also not difficult to tweak the virtio-balloon code to avoid that issue.
> > > But if that's just an issue with ccw itself, I think it's better to tweak ccw and
> > > remain virtio-balloon unchanged.
> > >
> >
> > As I said, at the moment I don't have a preference regarding the fix,
> > partly because I'm not sure if "reading config inside the handler" is OK
> > or not. Maybe Connie or Michael can help us here. I'm however sure that
> > commit 86a5597 "virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT"
> > breaks virtio-balloon with the ccw transport (i.e. effectively breaks
> > virtio-balloon on s390): it used to work before and does not work
> > after.
>
> Yes, that's unfortunate.
>
> >
> > AFAICT tweaking the balloon code may be simpler than tweaking the
> > virtio-ccw (transport code). ccw_io_helper() relies on getting
> > an interrupt when the issued IO is done. If virtio-ccw is buggy, it
> > needs to be fixed, but I'm not sure it is.
>
> I would not call virtio-ccw buggy, but it has some constraints that
> virtio-pci apparently doesn't have (and which did not show up so far;
> e.g. virtio-blk schedules a work item on config change, so there's no
> deadlock there.)
IMHO it is an internal API design thing. From the spirit of the virtio
standard perspective a virtio-ccw device is a ccw device, and acts like
one. We don't support new IO form ccw device interrupt handler. So that's
quite OK. OTOH we probably do want a coherent in kernel virtio
interface. And if that one needs to account for all the quirks of any
transport, that is quite ugly.
>
> One way to get out of that constraint (don't interact with the config
> space directly in the config changed handler) would be to schedule a
> work item in virtio-ccw that calls virtio_config_changed() for the
> device. My understanding is that delaying the notification to a work
> queue would be fine.
>
That would get us out of irq context, but I read you found other
problems.
[..]
Regards,
Halil
Powered by blists - more mailing lists