netdev - Re: [PATCH V2 mlx5-next 12/14] vfio/mlx5: Implement vfio

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20211102155420.GK2744544@nvidia.com>
Date:   Tue, 2 Nov 2021 12:54:20 -0300
From:   Jason Gunthorpe <jgg@...dia.com>
To:     Alex Williamson <alex.williamson@...hat.com>
Cc:     Shameerali Kolothum Thodi <shameerali.kolothum.thodi@...wei.com>,
        Cornelia Huck <cohuck@...hat.com>,
        Yishai Hadas <yishaih@...dia.com>, bhelgaas@...gle.com,
        saeedm@...dia.com, linux-pci@...r.kernel.org, kvm@...r.kernel.org,
        netdev@...r.kernel.org, kuba@...nel.org, leonro@...dia.com,
        kwankhede@...dia.com, mgurtovoy@...dia.com, maorg@...dia.com,
        "Dr. David Alan Gilbert" <dgilbert@...hat.com>
Subject: Re: [PATCH V2 mlx5-next 12/14] vfio/mlx5: Implement vfio_pci driver
 for mlx5 devices

On Tue, Nov 02, 2021 at 08:56:51AM -0600, Alex Williamson wrote:

> > Still, this is something that needs clear definition, I would expect
> > the SET_IRQS to happen after resuming clears but before running sets
> > to give maximum HW flexibility and symmetry with saving.
> 
> There's no requirement that the device enters a null state (!_RESUMING
> | !_SAVING | !_RUNNING), the uAPI even species the flows as _RESUMING
> transitioning to _RUNNING.  

If the device saves the MSI-X state inside it's migration data (as
apparently would be convenient for other OSs) then when RESUMING
clears and the migration data is de-serialized the device will
overwrite the MSI-X data.

Since Linux as an OS wants to control the MSI-X it needs to load it
after RESUMING, but before RUNNING.

> There's no point at which we can do SET_IRQS other than in the
> _RESUMING state.  Generally SET_IRQS ioctls are coordinated with the
> guest driver based on actions to the device, we can't be mucking
> with IRQs while the device is presumed running and already
> generating interrupt conditions.

We need to do it in state 000

ie resume should go 

  000 -> 100 -> 000 -> 001

With SET_IRQS and any other fixing done during the 2nd 000, after the
migration data has been loaded into the device.

> > And we should really define clearly what a device is supposed to do
> > with the interrupt vectors during migration. Obviously there are races
> > here.
> 
> The device should not be generating interrupts while !_RUNNING, pending
> interrupts should be held until the device is _RUNNING.  To me this
> means the sequence must be that INTx/MSI/MSI-X are restored while in
> the !_RUNNING state.

Yes

> > > In any case, it requires that the device cannot be absolutely static
> > > while !_RUNNING.  Does (_RESUMING) have different rules than
> > > (_SAVING)?  
> > 
> > I'd prever to avoid all device touches during both resuming and
> > saving, and do them during !RUNNING
> 
> There's no such state required by the uAPI.

The uAPI comment does not define when to do the SET_IRQS, it seems
this has been missed.

We really should fix it, unless you feel strongly that the
experimental API in qemu shouldn't be changed.

Jason