linux-kernel - [PATCH v2 9/9] refcount: Optimize __refcount_add_not

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20211210162313.951869207@infradead.org>
Date:   Fri, 10 Dec 2021 17:16:27 +0100
From:   Peter Zijlstra <peterz@...radead.org>
To:     will@...nel.org, boqun.feng@...il.com
Cc:     linux-kernel@...r.kernel.org, x86@...nel.org, peterz@...radead.org,
        mark.rutland@....com, elver@...gle.com, keescook@...omium.org,
        hch@...radead.org, torvalds@...ux-foundation.org, axboe@...nel.dk
Subject: [PATCH v2 9/9] refcount: Optimize __refcount_add_not_zero(.i=1)

Allow the same off-by-one on the target range that refcount_inc()
already has by only testing the new value for overflow when the
increment is not constant-1.

Improves code-gen, for the common case from:

    a887:       41 8b 14 24             mov    (%r12),%edx
    a88b:       83 fa ff                cmp    $0xffffffff,%edx
    a88e:       74 1f                   je     a8af <ring_buffer_get+0x3f>
    a890:       8d 4a 01                lea    0x1(%rdx),%ecx
    a893:       89 d0                   mov    %edx,%eax
    a895:       f0 41 0f b1 0c 24       lock cmpxchg %ecx,(%r12)
    a89b:       75 35                   jne    a8d2 <ring_buffer_get+0x62>
    a89d:       83 c2 02                add    $0x2,%edx
    a8a0:       09 ca                   or     %ecx,%edx
    a8a2:       78 19                   js     a8bd <ring_buffer_get+0x4d>

to:

    a887:       41 8b 04 24             mov    (%r12),%eax
    a88b:       83 f8 ff                cmp    $0xffffffff,%eax
    a88e:       74 1a                   je     a8aa <ring_buffer_get+0x3a>
    a890:       8d 50 01                lea    0x1(%rax),%edx
    a893:       f0 41 0f b1 14 24       lock cmpxchg %edx,(%r12)
    a899:       75 f0                   jne    a88b <ring_buffer_get+0x1b>
    a89b:       85 d2                   test   %edx,%edx
    a89d:       78 19                   js     a8b8 <ring_buffer_get+0x48>

Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
---
 include/linux/refcount.h |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

--- a/include/linux/refcount.h
+++ b/include/linux/refcount.h
@@ -161,7 +161,7 @@ static inline __must_check bool __refcou
 	if (oldp)
 		*oldp = old;
 
-	if (unlikely(old < 0 || old + i < 0))
+	if (unlikely(old < 0 || (!(__builtin_constant_p(i) && i == 1) && old + i < 0)))
 		refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF);
 
 	return old;