netdev - Re: [PATCH v2] x86/csum: rewrite csum

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Fri, 12 Nov 2021 17:45:29 +0100
From:   Peter Zijlstra <peterz@...radead.org>
To:     Eric Dumazet <eric.dumazet@...il.com>
Cc:     "David S . Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        netdev <netdev@...r.kernel.org>,
        Eric Dumazet <edumazet@...gle.com>, x86@...nel.org,
        Alexander Duyck <alexander.duyck@...il.com>,
        Andrew Cooper <andrew.cooper3@...rix.com>
Subject: Re: [PATCH v2] x86/csum: rewrite csum_partial()

On Fri, Nov 12, 2021 at 08:19:50AM -0800, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@...gle.com>
> 
> With more NIC supporting CHECKSUM_COMPLETE, and IPv6 being widely used.
> csum_partial() is heavily used with small amount of bytes,
> and is consuming many cycles.
> 
> IPv6 header size for instance is 40 bytes.
> 
> Another thing to consider is that NET_IP_ALIGN is 0 on x86,
> meaning that network headers are not word-aligned, unless
> the driver forces this.
> 
> This means that csum_partial() fetches one u16
> to 'align the buffer', then perform three u64 additions
> with carry in a loop, then a remaining u32, then a remaining u16.
> 
> With this new version, we perform a loop only for the 64 bytes blocks,
> then the remaining is bisected.
> 
> Tested on various cpus, all of them show a big reduction in
> csum_partial() cost (by 50 to 80 %)
> 
> v3: - use "+r" (temp64) asm constraints (Andrew).
>     - fold do_csum() in csum_partial(), as gcc does not inline it.
>     - fix bug added in v2 for the "odd" case.
>     - back using addcq, as Andrew pointed the clang bug that was adding
> 	  a stall on my hosts.
>       (separate patch to add32_with_carry() will follow)
>     - use load_unaligned_zeropad() for final 1-7 bytes (Peter & Alexander).
> 
> v2: - removed the hard-coded switch(), as it was not RETPOLINE aware.
>     - removed the final add32_with_carry() that we were doing
>       in csum_partial(), we can simply pass @sum to do_csum().
> 
> Signed-off-by: Eric Dumazet <edumazet@...gle.com>

Looks nice, happen to have shiny perf numbers to show how awesome it it?
:-)