netdev - Re: [PATCH 4/6] arm64/io: Provide a WC friendly __iowriteXX

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <eea8b03c20fc49cd88b159959a589f78521ff53b.camel@linux.ibm.com>
Date: Fri, 23 Feb 2024 17:35:42 +0100
From: Niklas Schnelle <schnelle@...ux.ibm.com>
To: Jason Gunthorpe <jgg@...dia.com>
Cc: David Laight <David.Laight@...lab.com>,
        Alexander Gordeev
 <agordeev@...ux.ibm.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Christian Borntraeger <borntraeger@...ux.ibm.com>,
        Borislav Petkov
 <bp@...en8.de>,
        Dave Hansen <dave.hansen@...ux.intel.com>,
        "David S.
 Miller" <davem@...emloft.net>,
        Eric Dumazet <edumazet@...gle.com>,
        Gerald
 Schaefer <gerald.schaefer@...ux.ibm.com>,
        Vasily Gorbik
 <gor@...ux.ibm.com>, Heiko Carstens <hca@...ux.ibm.com>,
        "H. Peter Anvin"
 <hpa@...or.com>,
        Justin Stitt <justinstitt@...gle.com>,
        Jakub Kicinski
 <kuba@...nel.org>, Leon Romanovsky <leon@...nel.org>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        "linux-s390@...r.kernel.org" <linux-s390@...r.kernel.org>,
        "llvm@...ts.linux.dev" <llvm@...ts.linux.dev>,
        Ingo Molnar
 <mingo@...hat.com>, Bill Wendling <morbo@...gle.com>,
        Nathan Chancellor
 <nathan@...nel.org>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Paolo Abeni
 <pabeni@...hat.com>, Salil Mehta <salil.mehta@...wei.com>,
        Jijie Shao
 <shaojijie@...wei.com>,
        Sven Schnelle <svens@...ux.ibm.com>,
        Thomas
 Gleixner <tglx@...utronix.de>,
        "x86@...nel.org" <x86@...nel.org>,
        Yisen
 Zhuang <yisen.zhuang@...wei.com>, Arnd Bergmann <arnd@...db.de>,
        Catalin
 Marinas <catalin.marinas@....com>,
        Leon Romanovsky <leonro@...lanox.com>,
        "linux-arch@...r.kernel.org" <linux-arch@...r.kernel.org>,
        "linux-arm-kernel@...ts.infradead.org"
 <linux-arm-kernel@...ts.infradead.org>,
        Mark Rutland
 <mark.rutland@....com>,
        Michael Guralnik <michaelgur@...lanox.com>,
        "patches@...ts.linux.dev" <patches@...ts.linux.dev>,
        Will Deacon
 <will@...nel.org>
Subject: Re: [PATCH 4/6] arm64/io: Provide a WC friendly __iowriteXX_copy()

On Fri, 2024-02-23 at 08:58 -0400, Jason Gunthorpe wrote:
> On Fri, Feb 23, 2024 at 12:38:18PM +0100, Niklas Schnelle wrote:
> > > Although I doubt that generating long TLP from byte writes is
> > > really necessary.
> > 
> > I might have gotten confused but I think these are not byte writes.
> > Remember that the count is in terms of the number of bits sized
> > quantities to copy so "count == 1" is 4/8 bytes here.
> 
> Right.
> 
> There seem to be two callers of this API in the kernel, one is calling
> with a constant size and wants a large TLP
> 
> Another seems to want memcpy_to_io with a guarenteed 32/64 bit store.

I don't really understand how that works together with the order not
being guaranteed. Do they use normal ioremap() and then require 32/64
bit TLPs and don't care about the order? But then the generic and ARM
variants do things in order so who knows if they actually rely on that.

> 
> > > IIRC you were merging at most 4 writes.
> > > So better to do a single 32bit write instead.
> > > (Unless you have misaligned source data - unlikely.)
> > > 
> > > While write-combining to generate long TLP is probably mostly
> > > safe for PCIe targets, there are some that will only handle
> > > TLP for single 32bit data items.
> > > Which might be why the code is explicitly requesting 4 byte copies.
> > > So it may be entirely wrong to write-combine anything except
> > > the generic memcpy_toio().
> > 
> > On anything other than s390x this should only do write-combine if the
> > memory mapping allows it, no? Meaning a driver that can't handle larger
> > TLPs really shouldn't use ioremap_wc() then.
> 
> Right.
> 
> > On s390x one could argue that our version of __iowriteXX_copy() is
> > strictly speaking not correct in that zpci_memcpy_toio() doesn't really
> > use XX bit writes which is why for us memcpy_toio() was actually a
> > better fit indeed. On the other hand doing 32 bit PCI stores (an s390x
> > thing) can't combine multiple stores into a single TLP which these
> > functions are used for and which has much more use cases than forcing a
> > copy loop with 32/64 bit sized writes which would also be a lot slower
> > on s390x than an aligned zpci_memcpy_toio().
> 
> mlx5 will definitely not work right if __iowrite64_copy() results in
> anything smaller than 32/64 bit PCIe TLPs.
> 
> Jason

Yes and we do actually have mlx5 on s390x so this is my priority.