linux-kernel - Re: [PATCH] net: make SK_MEMORY_PCPU

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iJAKEUu_Fdh0OC-+BJ+iVY0D2y0nAakGLxWZ8TywDu=BA@mail.gmail.com>
Date: Wed, 28 Feb 2024 14:46:39 +0100
From: Eric Dumazet <edumazet@...gle.com>
To: Adam Li <adamli@...amperecomputing.com>
Cc: corbet@....net, davem@...emloft.net, kuba@...nel.org, pabeni@...hat.com, 
	willemb@...gle.com, yangtiezhu@...ngson.cn, atenart@...nel.org, 
	kuniyu@...zon.com, wuyun.abel@...edance.com, leitao@...ian.org, 
	alexander@...alicyn.com, dhowells@...hat.com, paulmck@...nel.org, 
	joel.granados@...il.com, urezki@...il.com, joel@...lfernandes.org, 
	linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org, 
	netdev@...r.kernel.org, patches@...erecomputing.com, 
	cl@...amperecomputing.com, shijie@...amperecomputing.com
Subject: Re: [PATCH] net: make SK_MEMORY_PCPU_RESERV tunable

On Wed, Feb 28, 2024 at 2:21 PM Adam Li <adamli@...amperecomputing.com> wrote:
>
> On 2/28/2024 4:38 AM, Eric Dumazet wrote:
>
> >>
> >> sk_prot->memory_allocated points to global atomic variable:
> >> atomic_long_t tcp_memory_allocated ____cacheline_aligned_in_smp;
> >>
> >> If increasing the per-cpu cache size from 1MB to e.g. 16MB,
> >> changes to sk->sk_prot->memory_allocated can be further reduced.
> >> Performance may be improved on system with many cores.
> >
> > This looks good, do you have any performance numbers to share ?
>
> I ran localhost memcached test on system with 320 CPU threads,
> perf shows 4% cycles spent in __sk_mem_raise_allocated() -->sk_memory_allocated().
> If increasing SK_MEMORY_PCPU_RESERV to 16MB, perf cycles spent in
> __sk_mem_raise_allocated() drops to 0.4%.

I suspect some kind of flow/cpu steering issues then.
Also maybe SO_RESERVE_MEM would be better for this workload.