linux-kernel - RE: [PATCH] locking/memory-barriers.txt: Improve documentation for writel() usage

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <PH0PR12MB548166865BF446C7F232DCE1DC519@PH0PR12MB5481.namprd12.prod.outlook.com>
Date:   Fri, 23 Sep 2022 15:55:51 +0000
From:   Parav Pandit <parav@...dia.com>
To:     Arnd Bergmann <arnd@...db.de>,
        "stern@...land.harvard.edu" <stern@...land.harvard.edu>,
        "parri.andrea@...il.com" <parri.andrea@...il.com>,
        Will Deacon <will@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        "boqun.feng@...il.com" <boqun.feng@...il.com>,
        Nicholas Piggin <npiggin@...il.com>,
        "dhowells@...hat.com" <dhowells@...hat.com>,
        "j.alglave@....ac.uk" <j.alglave@....ac.uk>,
        "luc.maranget@...ia.fr" <luc.maranget@...ia.fr>,
        "Paul E. McKenney" <paulmck@...nel.org>,
        "akiyks@...il.com" <akiyks@...il.com>,
        Dan Lustig <dlustig@...dia.com>,
        "joel@...lfernandes.org" <joel@...lfernandes.org>,
        Jonathan Corbet <corbet@....net>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Linux-Arch <linux-arch@...r.kernel.org>,
        "linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>
Subject: RE: [PATCH] locking/memory-barriers.txt: Improve documentation for
 writel() usage

> From: Arnd Bergmann <arnd@...db.de>
> Sent: Friday, September 16, 2022 12:09 AM

[..]
> >>
> > I think it is worth because current documentation, indirectly (or
> > incorrectly) indicate that
> > "writel() does wmb() internally, so those drivers, who has difficulty
> > in using writel() can do, wmb() + raw write".
> 
> I don't think it's wrong from a barrier perspective though:
> if a driver uses writel_relaxed(), then the only way to guarantee ordering is
> to have a full wmb() before it.
> 
Sorry for the late response.

Yes. Idea is to avoid wmb() whenever it is not necessary.

I will update the example description to reflect it.

> > And I sort of see above pattern in two drivers, and it is not good.
> > It ends up doing dsb(st) on arm64, while needed barrier is only
> > dmb(oshst).
> >
> > So to fix those two drivers, it is better to first avoid wmb()
> > documentation reference when referring to writel().
> 
> Yes, this suggestion is correct. On x86 and a few others, I think it's even
> worse when wmb() is an expensive barrier, while writel() is the same as
> writel_relaxed() and the barrier is implied by the MMIO access.
> 
> It might help to spell this out and say that writel() is always preferred over
> wmb()+writel_relaxed().
> 
True.

> Site note: there are several other problems with wmb()+__raw_writel(),
> which on many architectures does not guarantee any atomicity of the access
> (a word store could get split into four byte stores), breaks endianess
> assumptions and may still not provide the correct barrier semantics.
>
Hmm. So far didn't observe this on arm64, x86_64, ppc64 yet.
May be because the address is aligned to 8 bytes, we don't see the byte stores?
 
> >> I see that there is more going on with that function, at least the
> >> loop in
> >> post_send_nop() probably just wants to use __iowrite64_copy(), but
> >> that also has no barrier in it, while changing mlx5_write64() to use
> >> iowrite64be() or similar would of course add excessive barriers inside of
> the loop.
> >
> > True. All other conversion seems possible.
> > For post_send_nop(), __iowmb() needs to be exposed, which is not
> > available today and it is only one-off user, I am inclined to keep
> > post_send_nop()  as-is, but want to improve/correct rest of the
> > callers in these two drivers.
> 
> __iowmb() is architecture-specific and does not have a well-defined
> behavior. wmb() is probably the best choice for post_send_nop().
Yes.

> Alternatively, one could use __iowrite64_copy() for the first few fields
> followed by a single writel64be for the last one.
> 
__iowrite64_copy() () seems right fit for post_send_nop() compare t current code.

> If you think we need something better than that, maybe having an
> iowrite64_copy() (without leading __) that includes a barrier would work.

It is only one-off user, and not so critical path, so we can differ iowrite64_copy() for now.

mlx5_write64() variant to use writeX() and avoid wmb() post the documentation update is good start.