linux-kernel - [PATCH 13/16] mm/percpu: micro-optimize pcpu_is

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20211005054059.475634-14-yury.norov@gmail.com>
Date:   Mon,  4 Oct 2021 22:40:56 -0700
From:   Yury Norov <yury.norov@...il.com>
To:     Stephen Rothwell <sfr@...b.auug.org.au>
Cc:     Yury Norov <yury.norov@...il.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        linux-kernel@...r.kernel.org, linux-mm@...ck.org,
        linux-arch@...r.kernel.org, linux-kselftest@...r.kernel.org,
        linux-mmc@...r.kernel.org, linux-perf-users@...r.kernel.org,
        kvm@...r.kernel.org,
        "James E.J. Bottomley" <James.Bottomley@...senPartnership.com>,
        Alexander Lobakin <alobakin@...me>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Alexey Klimov <aklimov@...hat.com>,
        Andrea Merello <andrea.merello@...il.com>,
        Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
        Arnaldo Carvalho de Melo <acme@...hat.com>,
        Arnd Bergmann <arnd@...db.de>, Ben Gardon <bgardon@...gle.com>,
        Benjamin Herrenschmidt <benh@...nel.crashing.org>,
        Brian Cain <bcain@...eaurora.org>,
        Catalin Marinas <catalin.marinas@....com>,
        Christoph Lameter <cl@...ux.com>,
        Daniel Bristot de Oliveira <bristot@...hat.com>,
        David Hildenbrand <david@...hat.com>,
        Dennis Zhou <dennis@...nel.org>,
        Geert Uytterhoeven <geert@...ux-m68k.org>,
        Heiko Carstens <hca@...ux.ibm.com>,
        Ian Rogers <irogers@...gle.com>,
        Ingo Molnar <mingo@...hat.com>,
        Jaegeuk Kim <jaegeuk@...nel.org>,
        Jakub Kicinski <kuba@...nel.org>, Jiri Olsa <jolsa@...hat.com>,
        Joe Perches <joe@...ches.com>, Jonas Bonn <jonas@...thpole.se>,
        Leo Yan <leo.yan@...aro.org>,
        Mark Rutland <mark.rutland@....com>,
        Namhyung Kim <namhyung@...nel.org>,
        Palmer Dabbelt <palmer@...belt.com>,
        Paolo Bonzini <pbonzini@...hat.com>,
        Peter Xu <peterx@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Petr Mladek <pmladek@...e.com>,
        Rasmus Villemoes <linux@...musvillemoes.dk>,
        Rich Felker <dalias@...c.org>,
        Samuel Mendoza-Jonas <sam@...dozajonas.com>,
        Sean Christopherson <seanjc@...gle.com>,
        Sergey Senozhatsky <senozhatsky@...omium.org>,
        Shuah Khan <shuah@...nel.org>,
        Stefan Kristiansson <stefan.kristiansson@...nalahti.fi>,
        Steven Rostedt <rostedt@...dmis.org>,
        Tejun Heo <tj@...nel.org>,
        Thomas Bogendoerfer <tsbogend@...ha.franken.de>,
        Ulf Hansson <ulf.hansson@...aro.org>,
        Will Deacon <will@...nel.org>,
        Wolfram Sang <wsa+renesas@...g-engineering.com>,
        Yoshinori Sato <ysato@...rs.sourceforge.jp>
Subject: [PATCH 13/16] mm/percpu: micro-optimize pcpu_is_populated()

bitmap_next_clear_region() calls find_next_zero_bit() and find_next_bit()
sequentially to find a range of clear bits. In case of pcpu_is_populated()
there's a chance to return earlier if bitmap has all bits set.

Signed-off-by: Yury Norov <yury.norov@...il.com>
Tested-by: Wolfram Sang <wsa+renesas@...g-engineering.com>
Acked-by: Dennis Zhou <dennis@...nel.org>
---
 mm/percpu.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index e0a986818903..1cf0bb904b1d 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -1070,17 +1070,18 @@ static void pcpu_block_update_hint_free(struct pcpu_chunk *chunk, int bit_off,
 static bool pcpu_is_populated(struct pcpu_chunk *chunk, int bit_off, int bits,
 			      int *next_off)
 {
-	unsigned int page_start, page_end, rs, re;
+	unsigned int start, end;
 
-	page_start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE);
-	page_end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE);
+	start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE);
+	end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE);
 
-	rs = page_start;
-	bitmap_next_clear_region(chunk->populated, &rs, &re, page_end);
-	if (rs >= page_end)
+	start = find_next_zero_bit(chunk->populated, end, start);
+	if (start >= end)
 		return true;
 
-	*next_off = re * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE;
+	end = find_next_bit(chunk->populated, end, start + 1);
+
+	*next_off = end * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE;
 	return false;
 }
 
-- 
2.30.2