linux-kernel - Re: [PATCH v12 0/6] iommu/dma: s390 DMA API conversion and optimized IOTLB flushing

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <068aeebd-df24-5097-96e3-ebaaa7a763fd@arm.com>
Date:   Wed, 27 Sep 2023 10:26:14 +0100
From:   Robin Murphy <robin.murphy@....com>
To:     Niklas Schnelle <schnelle@...ux.ibm.com>,
        Jason Gunthorpe <jgg@...pe.ca>, Joerg Roedel <joro@...tes.org>
Cc:     Matthew Rosato <mjrosato@...ux.ibm.com>,
        Will Deacon <will@...nel.org>,
        Wenjia Zhang <wenjia@...ux.ibm.com>,
        Gerd Bayer <gbayer@...ux.ibm.com>,
        Julian Ruess <julianr@...ux.ibm.com>,
        Pierre Morel <pmorel@...ux.ibm.com>,
        Alexandra Winter <wintera@...ux.ibm.com>,
        Heiko Carstens <hca@...ux.ibm.com>,
        Vasily Gorbik <gor@...ux.ibm.com>,
        Alexander Gordeev <agordeev@...ux.ibm.com>,
        Christian Borntraeger <borntraeger@...ux.ibm.com>,
        Sven Schnelle <svens@...ux.ibm.com>,
        Suravee Suthikulpanit <suravee.suthikulpanit@....com>,
        Hector Martin <marcan@...can.st>,
        Sven Peter <sven@...npeter.dev>,
        Alyssa Rosenzweig <alyssa@...enzweig.io>,
        David Woodhouse <dwmw2@...radead.org>,
        Lu Baolu <baolu.lu@...ux.intel.com>,
        Andy Gross <agross@...nel.org>,
        Bjorn Andersson <andersson@...nel.org>,
        Konrad Dybcio <konrad.dybcio@...aro.org>,
        Yong Wu <yong.wu@...iatek.com>,
        Matthias Brugger <matthias.bgg@...il.com>,
        AngeloGioacchino Del Regno 
        <angelogioacchino.delregno@...labora.com>,
        Gerald Schaefer <gerald.schaefer@...ux.ibm.com>,
        Orson Zhai <orsonzhai@...il.com>,
        Baolin Wang <baolin.wang@...ux.alibaba.com>,
        Chunyan Zhang <zhang.lyra@...il.com>,
        Chen-Yu Tsai <wens@...e.org>,
        Jernej Skrabec <jernej.skrabec@...il.com>,
        Samuel Holland <samuel@...lland.org>,
        Thierry Reding <thierry.reding@...il.com>,
        Krishna Reddy <vdumpa@...dia.com>,
        Jonathan Hunter <jonathanh@...dia.com>,
        Jonathan Corbet <corbet@....net>, linux-s390@...r.kernel.org,
        netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
        iommu@...ts.linux.dev, asahi@...ts.linux.dev,
        linux-arm-kernel@...ts.infradead.org,
        linux-arm-msm@...r.kernel.org, linux-mediatek@...ts.infradead.org,
        linux-sunxi@...ts.linux.dev, linux-tegra@...r.kernel.org,
        linux-doc@...r.kernel.org
Subject: Re: [PATCH v12 0/6] iommu/dma: s390 DMA API conversion and optimized
 IOTLB flushing

On 2023-09-27 09:55, Niklas Schnelle wrote:
> On Tue, 2023-09-26 at 13:08 -0300, Jason Gunthorpe wrote:
>> On Tue, Sep 26, 2023 at 05:04:28PM +0200, Joerg Roedel wrote:
>>> Hi Niklas,
>>>
>>> On Fri, Aug 25, 2023 at 12:11:15PM +0200, Niklas Schnelle wrote:
>>>> Niklas Schnelle (6):
>>>>        iommu: Allow .iotlb_sync_map to fail and handle s390's -ENOMEM return
>>>>        s390/pci: prepare is_passed_through() for dma-iommu
>>>>        s390/pci: Use dma-iommu layer
>>>>        iommu/s390: Disable deferred flush for ISM devices
>>>>        iommu/dma: Allow a single FQ in addition to per-CPU FQs
>>>>        iommu/dma: Use a large flush queue and timeout for shadow_on_flush
>>>
>>> Turned out this series has non-trivial conflicts with Jasons
>>> default-domain work so I had to remove it from the IOMMU tree for now.
>>> Can you please rebase it to the latest iommu/core branch and re-send? I
>>> will take it into the tree again then.
>>
>> Niklas, I think you just 'take yours' to resolve this. All the
>> IOMMU_DOMAIN_PLATFORM related and .default_domain = parts should be
>> removed. Let me know if you need anything
>>
>> Thanks,
>> Jason
> 
> Hi Joerg, Hi Jason,
> 
> I've run into an unfortunate problem, not with the rebase itself but
> with the iommu/core branch.
> 
> Jason is right, I basically need to just remove the platform ops and
> .default_domain ops. This seems to work fine for an NVMe both in the
> host and also when using the IOMMU with vfio-pci + KVM. I've already
> pushed the result of that to my git.kernel.org:
> https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/log/?h=b4/dma_iommu
> 
> The problem is that something seems to  be broken in the iommu/core
> branch. Regardless of whether I have my DMA API conversion on top or
> with the base iommu/core branch I can not use ConnectX-4 VFs.
> 
> # lspci
> 111a:00:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
> # dmesg | grep mlx
> [    3.189749] mlx5_core 111a:00:00.0: mlx5_mdev_init:1802:(pid 464): Failed initializing cmdif SW structs, aborting
> [    3.189783] mlx5_core: probe of 111a:00:00.0 failed with error -12
> 
> This same card works on v6.6-rc3 both with and without my DMA API
> conversion patch series applied. Looking at mlx5_mdev_init() ->
> mlx5_cmd_init(). The -ENOMEM seems to come from the following
> dma_pool_create():
> 
> cmd->pool = dma_pool_create("mlx5_cmd", mlx5_core_dma_dev(dev), size, align, 0);
> 
> I'll try to debug this further but wanted to let you know already in
> case you have some ideas.

I could imagine that potentially something in the initial default domain 
conversion somehow interferes with the DMA ops in a way that ends up 
causing alloc_cmd_page() to fail (maybe calling zpci_dma_init_device() 
at the wrong point, or too many times?). FWIW I see nothing that would 
obviously affect dma_pool_create() itself.

Robin.

> Either way as it doesn't seem to be related
> to the DMA API conversion I can sent that out again regardless if you
> want, really don't want to miss another cycle.
> 
> Thanks,
> Niklas