Message-ID: <3858dd8c-ee55-b0d7-96cc-3c047ba8f652@redhat.com>
Date:   Wed, 13 May 2020 15:28:41 +0200
From:   Auger Eric <eric.auger@...hat.com>
To:     Shameerali Kolothum Thodi <shameerali.kolothum.thodi@...wei.com>,
        Zhangfei Gao <zhangfei.gao@...aro.org>,
        "eric.auger.pro@...il.com" <eric.auger.pro@...il.com>,
        "iommu@...ts.linux-foundation.org" <iommu@...ts.linux-foundation.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "kvmarm@...ts.cs.columbia.edu" <kvmarm@...ts.cs.columbia.edu>,
        "will@...nel.org" <will@...nel.org>,
        "joro@...tes.org" <joro@...tes.org>,
        "maz@...nel.org" <maz@...nel.org>,
        "robin.murphy@....com" <robin.murphy@....com>
Cc:     "jean-philippe@...aro.org" <jean-philippe@...aro.org>,
        "alex.williamson@...hat.com" <alex.williamson@...hat.com>,
        "jacob.jun.pan@...ux.intel.com" <jacob.jun.pan@...ux.intel.com>,
        "yi.l.liu@...el.com" <yi.l.liu@...el.com>,
        "peter.maydell@...aro.org" <peter.maydell@...aro.org>,
        "tn@...ihalf.com" <tn@...ihalf.com>,
        "bbhushan2@...vell.com" <bbhushan2@...vell.com>
Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)

Hi Shameer,

On 5/7/20 8:59 AM, Shameerali Kolothum Thodi wrote:
> Hi Eric,
> 
>> -----Original Message-----
>> From: Shameerali Kolothum Thodi
>> Sent: 30 April 2020 10:38
>> To: 'Auger Eric' <eric.auger@...hat.com>; Zhangfei Gao
>> <zhangfei.gao@...aro.org>; eric.auger.pro@...il.com;
>> iommu@...ts.linux-foundation.org; linux-kernel@...r.kernel.org;
>> kvm@...r.kernel.org; kvmarm@...ts.cs.columbia.edu; will@...nel.org;
>> joro@...tes.org; maz@...nel.org; robin.murphy@....com
>> Cc: jean-philippe@...aro.org; alex.williamson@...hat.com;
>> jacob.jun.pan@...ux.intel.com; yi.l.liu@...el.com; peter.maydell@...aro.org;
>> tn@...ihalf.com; bbhushan2@...vell.com
>> Subject: RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
>>
>> Hi Eric,
>>
>>> -----Original Message-----
>>> From: Auger Eric [mailto:eric.auger@...hat.com]
>>> Sent: 16 April 2020 08:45
>>> To: Zhangfei Gao <zhangfei.gao@...aro.org>; eric.auger.pro@...il.com;
>>> iommu@...ts.linux-foundation.org; linux-kernel@...r.kernel.org;
>>> kvm@...r.kernel.org; kvmarm@...ts.cs.columbia.edu; will@...nel.org;
>>> joro@...tes.org; maz@...nel.org; robin.murphy@....com
>>> Cc: jean-philippe@...aro.org; Shameerali Kolothum Thodi
>>> <shameerali.kolothum.thodi@...wei.com>; alex.williamson@...hat.com;
>>> jacob.jun.pan@...ux.intel.com; yi.l.liu@...el.com; peter.maydell@...aro.org;
>>> tn@...ihalf.com; bbhushan2@...vell.com
>>> Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
>>>
>>> Hi Zhangfei,
>>>
>>> On 4/16/20 6:25 AM, Zhangfei Gao wrote:
>>>>
>>>>
>>>> On 2020/4/14 下午11:05, Eric Auger wrote:
>>>>> This version fixes an issue observed by Shameer on an SMMU 3.2,
>>>>> when moving from dual stage config to stage 1 only config.
>>>>> The 2 high 64b of the STE now get reset. Otherwise, leaving the
>>>>> S2TTB set may cause a C_BAD_STE error.
>>>>>
>>>>> This series can be found at:
>>>>> https://github.com/eauger/linux/tree/v5.6-2stage-v11_10.1
>>>>> (including the VFIO part)
>>>>> The QEMU fellow series still can be found at:
>>>>> https://github.com/eauger/qemu/tree/v4.2.0-2stage-rfcv6
>>>>>
>>>>> Users have expressed interest in that work and tested v9/v10:
>>>>> - https://patchwork.kernel.org/cover/11039995/#23012381
>>>>> - https://patchwork.kernel.org/cover/11039995/#23197235
>>>>>
>>>>> Background:
>>>>>
>>>>> This series brings the IOMMU part of HW nested paging support
>>>>> in the SMMUv3. The VFIO part is submitted separately.
>>>>>
>>>>> The IOMMU API is extended to support 2 new API functionalities:
>>>>> 1) pass the guest stage 1 configuration
>>>>> 2) pass stage 1 MSI bindings
>>>>>
>>>>> Then those capabilities gets implemented in the SMMUv3 driver.
>>>>>
>>>>> The virtualizer passes information through the VFIO user API
>>>>> which cascades them to the iommu subsystem. This allows the guest
>>>>> to own stage 1 tables and context descriptors (so-called PASID
>>>>> table) while the host owns stage 2 tables and main configuration
>>>>> structures (STE).
>>>>>
>>>>>
>>>>
>>>> Thanks Eric
>>>>
>>>> Tested v11 on Hisilicon kunpeng920 board via hardware zip accelerator.
>>>> 1. no-sva works, where guest app directly use physical address via ioctl.
>>> Thank you for the testing. Glad it works for you.
>>>> 2. vSVA still not work, same as v10,
>>> Yes that's normal this series is not meant to support vSVM at this stage.
>>>
>>> I intend to add the missing pieces during the next weeks.
>>
>> Thanks for that. I have made an attempt to add the vSVA based on
>> your v10 + JPBs sva patches. The host kernel and Qemu changes can
>> be found here[1][2].
>>
>> This basically adds multiple pasid support on top of your changes.
>> I have done some basic sanity testing and we have some initial success
>> with the zip vf dev on our D06 platform. Please note that the STALL event is
>> not yet supported though, but works fine if we mlock() guest usr mem.
> 
> I have added STALL support for our vSVA prototype and it seems to be
> working(on our hardware). I have updated the kernel and qemu branches with
> the same[1][2]. I should warn you though that these are prototype code and I am pretty
> much re-using the VFIO_IOMMU_SET_PASID_TABLE interface for almost everything.
> But thought of sharing, in case if it is useful somehow!.

Thank you again for sharing the POC. I looked at the kernel and QEMU
branches.

Here are some preliminary comments:
- "arm-smmu-v3: Reset S2TTB while switching back from nested stage":  as
you mentionned S2TTB reset now is featured in v11
- "arm-smmu-v3: Add support for multiple pasid in nested mode": I could
easily integrate this into my series. Update the iommu api first and
pass multiple CD info in a separate patch
- "arm-smmu-v3: Add support to Invalidate CD": CD invalidation should be
cascaded to host through the PASID cache invalidation uapi (no pb you
warned us for the POC you simply used VFIO_IOMMU_SET_PASID_TABLE). I
think I should add this support in my original series although it does
not seem to trigger any issue up to now.
- "arm-smmu-v3: Remove duplication of fault propagation". I understand
the transcode is done somewhere else with SVA but we still need to do it
if a single CD is used, right? I will review the SVA code to better
understand.
- for the STALL response injection I would tend to use a new VFIO region
for responses. At the moment there is a single VFIO region for reporting
the fault.

On QEMU side:
- I am currently working on 3.2 range invalidation support which is
needed for DPDK/VFIO
- While at it I will look at how to incrementally introduce some of the
features you need in this series.

Thanks

Eric



> 
> Thanks,
> Shameer
> 
> [1]https://github.com/hisilicon/kernel-dev/commits/vsva-prototype-host-v1
> 
> [2]https://github.com/hisilicon/qemu/tree/v4.2.0-2stage-rfcv6-vsva-prototype-v1
> 

Powered by blists - more mailing lists