linux-kernel - Re: [RFC PATCH] crypto: riscv: scalar accelerated GHASH

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250417170902.GC800@quark.localdomain>
Date: Thu, 17 Apr 2025 10:09:02 -0700
From: Eric Biggers <ebiggers@...nel.org>
To: Qingfang Deng <dqfext@...il.com>
Cc: Herbert Xu <herbert@...dor.apana.org.au>,
	"David S. Miller" <davem@...emloft.net>,
	Paul Walmsley <paul.walmsley@...ive.com>,
	Palmer Dabbelt <palmer@...belt.com>,
	Albert Ou <aou@...s.berkeley.edu>, Alexandre Ghiti <alex@...ti.fr>,
	linux-crypto@...r.kernel.org, linux-riscv@...ts.infradead.org,
	linux-kernel@...r.kernel.org,
	Christoph Müllner <christoph.muellner@...ll.eu>,
	Heiko Stuebner <heiko.stuebner@...ll.eu>,
	Qingfang Deng <qingfang.deng@...lower.com.cn>
Subject: Re: [RFC PATCH] crypto: riscv: scalar accelerated GHASH

On Thu, Apr 17, 2025 at 02:49:38PM +0800, Qingfang Deng wrote:
> +static __always_inline u64 riscv_zbb_swab64(u64 val)
> +{
> +	asm (".option push\n"
> +	     ".option arch,+zbb\n"
> +	     "rev8 %0, %1\n"
> +	     ".option pop\n"
> +	     : "=r" (val) : "r" (val));
> +	return val;
> +}
> +
> +static __always_inline __uint128_t get_unaligned_be128(const u8 *p)
> +{
> +	__uint128_t val;
> +#ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> +	val = *(__uint128_t *)p;
> +	val = riscv_zbb_swab64(val >> 64) | (__uint128_t)riscv_zbb_swab64(val) << 64;
> +#else
> +	val = (__uint128_t)p[0] << 120;
> +	val |= (__uint128_t)p[1] << 112;
> +	val |= (__uint128_t)p[2] << 104;
> +	val |= (__uint128_t)p[3] << 96;
> +	val |= (__uint128_t)p[4] << 88;
> +	val |= (__uint128_t)p[5] << 80;
> +	val |= (__uint128_t)p[6] << 72;
> +	val |= (__uint128_t)p[7] << 64;
> +	val |= (__uint128_t)p[8] << 56;
> +	val |= (__uint128_t)p[9] << 48;
> +	val |= (__uint128_t)p[10] << 40;
> +	val |= (__uint128_t)p[11] << 32;
> +	val |= (__uint128_t)p[12] << 24;
> +	val |= (__uint128_t)p[13] << 16;
> +	val |= (__uint128_t)p[14] << 8;
> +	val |= (__uint128_t)p[15];
> +#endif
> +	return val;
> +}
> +
> +static __always_inline void put_unaligned_be128(__uint128_t val, u8 *p)
> +{
> +#ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> +	*(__uint128_t *)p = riscv_zbb_swab64(val >> 64) | (__uint128_t)riscv_zbb_swab64(val) << 64;
> +#else
> +	p[0] = val >> 120;
> +	p[1] = val >> 112;
> +	p[2] = val >> 104;
> +	p[3] = val >> 96;
> +	p[4] = val >> 88;
> +	p[5] = val >> 80;
> +	p[6] = val >> 72;
> +	p[7] = val >> 64;
> +	p[8] = val >> 56;
> +	p[9] = val >> 48;
> +	p[10] = val >> 40;
> +	p[11] = val >> 32;
> +	p[12] = val >> 24;
> +	p[13] = val >> 16;
> +	p[14] = val >> 8;
> +	p[15] = val;
> +#endif
> +}

Please help properly optimize swab*() and {get,put}_unaligned_* for RISC-V
first, before considering random hacks like this.

https://lore.kernel.org/r/20250403-riscv-swab-v3-0-3bf705d80e33@iencinas.com
is working on swab*().

> +		/* Multiplication (without Karatsuba) */
> +		t0 = clmul128(p_lo, k_lo);
> +		t1 = clmul128(p_lo, k_hi);
> +		t2 = clmul128(p_hi, k_lo);
> +		t3 = clmul128(p_hi, k_hi);
> +		mid = t1 ^ t2;
> +		lo = t0 ^ (mid << 64);
> +		hi = t3 ^ (mid >> 64);

There is no need to explicitly XOR 'mid << 64' into lo and 'mid >> 64' into hi.
Take a look at how arch/x86/crypto/aes-gcm-*.S do it.

Also, since this is only doing one block at a time and does not use Karatsuba
multiplication, the single-step reduction would work well here.  See
aes-gcm-aesni-x86_64.S.

- Eric