lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <6dab29f58ac1ccd58caaee031f98f4d0d382cbcd.camel@linux.ibm.com>
Date: Wed, 27 Sep 2023 16:31:11 +0200
From: Niklas Schnelle <schnelle@...ux.ibm.com>
To: Joerg Roedel <joro@...tes.org>
Cc: Jason Gunthorpe <jgg@...pe.ca>, Matthew Rosato <mjrosato@...ux.ibm.com>,
        Will Deacon <will@...nel.org>, Wenjia Zhang <wenjia@...ux.ibm.com>,
        Robin
 Murphy <robin.murphy@....com>,
        Gerd Bayer <gbayer@...ux.ibm.com>,
        Julian
 Ruess <julianr@...ux.ibm.com>,
        Pierre Morel <pmorel@...ux.ibm.com>,
        Alexandra Winter <wintera@...ux.ibm.com>,
        Heiko Carstens
 <hca@...ux.ibm.com>, Vasily Gorbik <gor@...ux.ibm.com>,
        Alexander Gordeev
 <agordeev@...ux.ibm.com>,
        Christian Borntraeger
 <borntraeger@...ux.ibm.com>,
        Sven Schnelle <svens@...ux.ibm.com>,
        Suravee
 Suthikulpanit <suravee.suthikulpanit@....com>,
        Hector Martin
 <marcan@...can.st>, Sven Peter <sven@...npeter.dev>,
        Alyssa Rosenzweig
 <alyssa@...enzweig.io>,
        David Woodhouse <dwmw2@...radead.org>,
        Lu Baolu
 <baolu.lu@...ux.intel.com>, Andy Gross <agross@...nel.org>,
        Bjorn Andersson
 <andersson@...nel.org>,
        Konrad Dybcio <konrad.dybcio@...aro.org>,
        Yong Wu
 <yong.wu@...iatek.com>,
        Matthias Brugger <matthias.bgg@...il.com>,
        AngeloGioacchino Del Regno <angelogioacchino.delregno@...labora.com>,
        Gerald Schaefer <gerald.schaefer@...ux.ibm.com>,
        Orson Zhai
 <orsonzhai@...il.com>,
        Baolin Wang <baolin.wang@...ux.alibaba.com>,
        Chunyan
 Zhang <zhang.lyra@...il.com>, Chen-Yu Tsai <wens@...e.org>,
        Jernej Skrabec
 <jernej.skrabec@...il.com>,
        Samuel Holland <samuel@...lland.org>,
        Thierry
 Reding <thierry.reding@...il.com>,
        Krishna Reddy <vdumpa@...dia.com>,
        Jonathan Hunter <jonathanh@...dia.com>,
        Jonathan Corbet <corbet@....net>, linux-s390@...r.kernel.org,
        netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
        iommu@...ts.linux.dev, asahi@...ts.linux.dev,
        linux-arm-kernel@...ts.infradead.org, linux-arm-msm@...r.kernel.org,
        linux-mediatek@...ts.infradead.org, linux-sunxi@...ts.linux.dev,
        linux-tegra@...r.kernel.org, linux-doc@...r.kernel.org
Subject: Re: [PATCH v12 0/6] iommu/dma: s390 DMA API conversion and
 optimized IOTLB flushing

On Wed, 2023-09-27 at 15:20 +0200, Niklas Schnelle wrote:
> On Wed, 2023-09-27 at 13:24 +0200, Niklas Schnelle wrote:
> > On Wed, 2023-09-27 at 11:55 +0200, Joerg Roedel wrote:
> > > Hi Niklas,
> > > 
> > > On Wed, Sep 27, 2023 at 10:55:23AM +0200, Niklas Schnelle wrote:
> > > > The problem is that something seems to  be broken in the iommu/core
> > > > branch. Regardless of whether I have my DMA API conversion on top or
> > > > with the base iommu/core branch I can not use ConnectX-4 VFs.
> > > 
> > > Have you already tried to bisect the issue in the iommu/core branch?
> > > The result might sched some light on the issue.
> > > 
> > > Regards,
> > > 
> > > 	Joerg
> > 
> > Hi Joerg,
> > 
> > Working on it, somehow I must have messed up earlier. It now looks like
> > it might in fact be caused by my DMA API conversion rebase and the
> > "s390/pci: Use dma-iommu layer" commit. Maybe there is some interaction
> > with Jason's patches that I haven't thought about. So sorry for any
> > wrong blame.
> > 
> > Thanks,
> > Niklas
> 
> Hi,
> 
> I tracked the problem down from mlx5_core's alloc_cmd_page() via
> dma_alloc_coherent(), ops->alloc, iommu_dma_alloc_remap(), and
> __iommu_dma_alloc_noncontiguous() to a failed iommu_dma_alloc_iova().
> The allocation here is for 4K so nothing crazy.
> 
> On second look I also noticed:
> 
> nvme 2007:00:00.0: Using 42-bit DMA addresses
> 
> for the NVMe that is working. The problem here seems to be that we set
> iommu_dma_forcedac = true in s390_iommu_probe_finalize() because we
> have currently have a reserved region over the first 4 GiB anyway so
> will always use IOVAs larger than that. That however is too late since
> iommu_dma_set_pci_32bit_workaround() is already checked in
> __iommu_probe_device() which is called just before ops-
> > probe_finalize(). So I moved setting iommu_dma_forcedac = true to
> zpci_init_iommu() and that gets rid of the notice for the NVMe but I
> still get a failure of iommu_dma_alloc_iova() in
> __iommu_dma_alloc_noncontiguous(). So I'll keep digging.
> 
> Thanks,
> Niklas


Ok I think I got it and this doesn't seem strictly s390x specific but
I'd think should happen with iommu.forcedac=1 everywhere.

The reason iommu_dma_alloc_iova() fails seems to be that mlx5_core does
dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)) in 
mlx5_pci_init()->set_dma_caps() which happens after it already called
mlx5_mdev_init()->mlx5_cmd_init()->alloc_cmd_page() so for the
dma_alloc_coherent() in there the dev->coherent_dma_mask is still
DMA_BIT_MASK(32) for which we can't find an IOVA because well we don't
have IOVAs below 4 GiB. Not entirely sure what caused this not to be
enforced before.

Thanks,
Niklas

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ