linux-kernel - Re: [PATCHv6 15/17] mm: Remove the branch from compound

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ac8ea0ef-4175-4563-b312-d888b31b83a9@kernel.org>
Date: Fri, 6 Feb 2026 11:23:48 +0100
From: "David Hildenbrand (Arm)" <david@...nel.org>
To: Kiryl Shutsemau <kas@...nel.org>,
 Andrew Morton <akpm@...ux-foundation.org>,
 Muchun Song <muchun.song@...ux.dev>, Matthew Wilcox <willy@...radead.org>,
 Usama Arif <usamaarif642@...il.com>, Frank van der Linden <fvdl@...gle.com>
Cc: Oscar Salvador <osalvador@...e.de>, Mike Rapoport <rppt@...nel.org>,
 Vlastimil Babka <vbabka@...e.cz>,
 Lorenzo Stoakes <lorenzo.stoakes@...cle.com>, Zi Yan <ziy@...dia.com>,
 Baoquan He <bhe@...hat.com>, Michal Hocko <mhocko@...e.com>,
 Johannes Weiner <hannes@...xchg.org>, Jonathan Corbet <corbet@....net>,
 Huacai Chen <chenhuacai@...nel.org>, WANG Xuerui <kernel@...0n.name>,
 Palmer Dabbelt <palmer@...belt.com>, Paul Walmsley
 <paul.walmsley@...ive.com>, Albert Ou <aou@...s.berkeley.edu>,
 Alexandre Ghiti <alex@...ti.fr>, kernel-team@...a.com, linux-mm@...ck.org,
 linux-kernel@...r.kernel.org, linux-doc@...r.kernel.org,
 loongarch@...ts.linux.dev, linux-riscv@...ts.infradead.org
Subject: Re: [PATCHv6 15/17] mm: Remove the branch from compound_head()

On 2/2/26 16:56, Kiryl Shutsemau wrote:
> The compound_head() function is a hot path. For example, the zap path
> calls it for every leaf page table entry.
> 
> Rewrite the helper function in a branchless manner to eliminate the risk
> of CPU branch misprediction.
> 
> Signed-off-by: Kiryl Shutsemau <kas@...nel.org>
> Reviewed-by: Muchun Song <muchun.song@...ux.dev>
> Reviewed-by: Zi Yan <ziy@...dia.com>
> ---
>   include/linux/page-flags.h | 27 +++++++++++++++++----------
>   1 file changed, 17 insertions(+), 10 deletions(-)
> 
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index 1aaa604f4b9b..16384cb6f962 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -224,25 +224,32 @@ static __always_inline bool compound_info_has_mask(void)
>   static __always_inline unsigned long _compound_head(const struct page *page)
>   {
>   	unsigned long info = READ_ONCE(page->compound_info);
> +	unsigned long mask;
> +
> +	if (!compound_info_has_mask()) {
> +		/* Bit 0 encodes PageTail() */
> +		if (info & 1)
> +			return info - 1;
>   
> -	/* Bit 0 encodes PageTail() */
> -	if (!(info & 1))
>   		return (unsigned long)page;
> -
> -	/*
> -	 * If compound_info_has_mask() is false, the rest of compound_info is
> -	 * the pointer to the head page.
> -	 */
> -	if (!compound_info_has_mask())
> -		return info - 1;
> +	}
>   
>   	/*
>   	 * If compoun_info_has_mask() is true the rest of the info encodes
>   	 * the mask that converts the address of the tail page to the head page.
>   	 *
>   	 * No need to clear bit 0 in the mask as 'page' always has it clear.
> +	 *
> +	 * Let's do it in a branchless manner.
>   	 */
> -	return (unsigned long)page & info;
> +
> +	/* Non-tail: -1UL, Tail: 0 */
> +	mask = (info & 1) - 1;
> +
> +	/* Non-tail: -1UL, Tail: info */
> +	mask |= info;
> +
> +	return (unsigned long)page & mask;
>   }
>   
>   #define compound_head(page)	((typeof(page))_compound_head(page))

Nice!

Acked-by: David Hildenbrand (Arm) <david@...nel.org>

-- 
Cheers,

David