Message-ID: <YqpnQ63gx/3HUWOA@Asurada-Nvidia>
Date:   Wed, 15 Jun 2022 16:12:03 -0700
From:   Nicolin Chen <nicolinc@...dia.com>
To:     "Tian, Kevin" <kevin.tian@...el.com>
CC:     Jason Gunthorpe <jgg@...dia.com>,
        "joro@...tes.org" <joro@...tes.org>,
        "will@...nel.org" <will@...nel.org>,
        "marcan@...can.st" <marcan@...can.st>,
        "sven@...npeter.dev" <sven@...npeter.dev>,
        "robin.murphy@....com" <robin.murphy@....com>,
        "robdclark@...il.com" <robdclark@...il.com>,
        "m.szyprowski@...sung.com" <m.szyprowski@...sung.com>,
        "krzysztof.kozlowski@...aro.org" <krzysztof.kozlowski@...aro.org>,
        "baolu.lu@...ux.intel.com" <baolu.lu@...ux.intel.com>,
        "agross@...nel.org" <agross@...nel.org>,
        "bjorn.andersson@...aro.org" <bjorn.andersson@...aro.org>,
        "matthias.bgg@...il.com" <matthias.bgg@...il.com>,
        "heiko@...ech.de" <heiko@...ech.de>,
        "orsonzhai@...il.com" <orsonzhai@...il.com>,
        "baolin.wang7@...il.com" <baolin.wang7@...il.com>,
        "zhang.lyra@...il.com" <zhang.lyra@...il.com>,
        "wens@...e.org" <wens@...e.org>,
        "jernej.skrabec@...il.com" <jernej.skrabec@...il.com>,
        "samuel@...lland.org" <samuel@...lland.org>,
        "jean-philippe@...aro.org" <jean-philippe@...aro.org>,
        "alex.williamson@...hat.com" <alex.williamson@...hat.com>,
        "virtualization@...ts.linux-foundation.org" 
        <virtualization@...ts.linux-foundation.org>,
        "thierry.reding@...il.com" <thierry.reding@...il.com>,
        "alim.akhtar@...sung.com" <alim.akhtar@...sung.com>,
        "alyssa@...enzweig.io" <alyssa@...enzweig.io>,
        "linux-s390@...r.kernel.org" <linux-s390@...r.kernel.org>,
        "linux-samsung-soc@...r.kernel.org" 
        <linux-samsung-soc@...r.kernel.org>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "jonathanh@...dia.com" <jonathanh@...dia.com>,
        "linux-rockchip@...ts.infradead.org" 
        <linux-rockchip@...ts.infradead.org>,
        "gerald.schaefer@...ux.ibm.com" <gerald.schaefer@...ux.ibm.com>,
        "linux-sunxi@...ts.linux.dev" <linux-sunxi@...ts.linux.dev>,
        "linux-arm-msm@...r.kernel.org" <linux-arm-msm@...r.kernel.org>,
        "linux-mediatek@...ts.infradead.org" 
        <linux-mediatek@...ts.infradead.org>,
        "linux-tegra@...r.kernel.org" <linux-tegra@...r.kernel.org>,
        "linux-arm-kernel@...ts.infradead.org" 
        <linux-arm-kernel@...ts.infradead.org>,
        "cohuck@...hat.com" <cohuck@...hat.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "iommu@...ts.linux-foundation.org" <iommu@...ts.linux-foundation.org>,
        "dwmw2@...radead.org" <dwmw2@...radead.org>
Subject: Re: [PATCH 3/5] vfio/iommu_type1: Prefer to reuse domains vs match
 enforced cache coherency

On Wed, Jun 15, 2022 at 07:35:00AM +0000, Tian, Kevin wrote:
> External email: Use caution opening links or attachments
> 
> 
> > From: Nicolin Chen <nicolinc@...dia.com>
> > Sent: Wednesday, June 15, 2022 4:45 AM
> >
> > Hi Kevin,
> >
> > On Wed, Jun 08, 2022 at 11:48:27PM +0000, Tian, Kevin wrote:
> > > > > > The KVM mechanism for controlling wbinvd is only triggered during
> > > > > > kvm_vfio_group_add(), meaning it is a one-shot test done once the
> > > > devices
> > > > > > are setup.
> > > > >
> > > > > It's not one-shot. kvm_vfio_update_coherency() is called in both
> > > > > group_add() and group_del(). Then the coherency property is
> > > > > checked dynamically in wbinvd emulation:
> > > >
> > > > From the perspective of managing the domains that is still
> > > > one-shot. It doesn't get updated when individual devices are
> > > > added/removed to domains.
> > >
> > > It's unchanged per-domain but dynamic per-vm when multiple
> > > domains are added/removed (i.e. kvm->arch.noncoherent_dma_count).
> > > It's the latter being checked in the kvm.
> >
> > I am going to send a v2, yet not quite getting the point here.
> > Meanwhile, Jason is on leave.
> >
> > What, in your opinion, would be an accurate description here?
> >
> 
> Something like below:
> --
> The KVM mechanism for controlling wbinvd is based on OR of
> the coherency property of all devices attached to a guest, no matter
> those devices  are attached to a single domain or multiple domains.
> 
> So, there is no value in trying to push a device that could do enforced
> cache coherency to a dedicated domain vs re-using an existing domain
> which is non-coherent since KVM won't be able to take advantage of it.
> This just wastes domain memory.
> 
> Simplify this code and eliminate the test. This removes the only logic
> that needed to have a dummy domain attached prior to searching for a
> matching domain and simplifies the next patches.
> 
> It's unclear whether we want to further optimize the Intel driver to
> update the domain coherency after a device is detached from it, at
> least not before KVM can be verified to handle such dynamics in related
> emulation paths (wbinvd, vcpu load, write_cr0, ept, etc.). In reality
> we don't see an usage requiring such optimization as the only device
> which imposes such non-coherency is Intel GPU which even doesn't
> support hotplug/hot remove.

Thanks! I just updated that and will send v2.

Powered by blists - more mailing lists