linux-kernel - Re: [RFC v2 00/24] Per queue configs and large rx buffer support for zcrx

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <rekaw6n6fsfkh46tbkyms47wjf7cdllwsihpsvp46bcrhxsxp4@ytymllzp72u6>
Date: Fri, 15 Aug 2025 16:44:20 +0000
From: Dragos Tatulea <dtatulea@...dia.com>
To: Pavel Begunkov <asml.silence@...il.com>, 
	Jakub Kicinski <kuba@...nel.org>, netdev@...r.kernel.org
Cc: Eric Dumazet <edumazet@...gle.com>, 
	Willem de Bruijn <willemb@...gle.com>, Paolo Abeni <pabeni@...hat.com>, andrew+netdev@...n.ch, 
	horms@...nel.org, davem@...emloft.net, sdf@...ichev.me, almasrymina@...gle.com, 
	dw@...idwei.uk, michael.chan@...adcom.com, ap420073@...il.com, 
	linux-kernel@...r.kernel.org
Subject: Re: [RFC v2 00/24] Per queue configs and large rx buffer support for
 zcrx

On Thu, Aug 14, 2025 at 11:46:35AM +0100, Pavel Begunkov wrote:
> On 8/13/25 16:39, Dragos Tatulea wrote:
> > Hi Pavel,
> > 
> > On Fri, Aug 08, 2025 at 03:54:23PM +0100, Pavel Begunkov wrote:
> > > [...]
> > > For example, for 200Gbit broadcom NIC, 4K vs 32K buffers, and napi and
> > > userspace pinned to the same CPU:
> > > 
> > > packets=23987040 (MB=2745098), rps=199559 (MB/s=22837)
> > > CPU    %usr   %nice    %sys %iowait    %irq   %soft   %idle
> > >    0    1.53    0.00   27.78    2.72    1.31   66.45    0.22
> > > packets=24078368 (MB=2755550), rps=200319 (MB/s=22924)
> > > CPU    %usr   %nice    %sys %iowait    %irq   %soft   %idle
> > >    0    0.69    0.00    8.26   31.65    1.83   57.00    0.57
> > > 
> > > And for napi and userspace on different CPUs:
> > > 
> > > packets=10725082 (MB=1227388), rps=198285 (MB/s=22692)
> > > CPU    %usr   %nice    %sys %iowait    %irq   %soft   %idle
> > >    0    0.10    0.00    0.50    0.00    0.50   74.50    24.40
> > >    1    4.51    0.00   44.33   47.22    2.08    1.85    0.00
> > > packets=14026235 (MB=1605175), rps=198388 (MB/s=22703)
> > > CPU    %usr   %nice    %sys %iowait    %irq   %soft   %idle
> > >    0    0.10    0.00    0.70    0.00    1.00   43.78   54.42
> > >    1    1.09    0.00   31.95   62.91    1.42    2.63    0.00
> > > 
I forgot to ask: what is the MTU here?

> > What did you use for this benchmark, send-zerocopy? Could you share a
> > branch and how you ran it please?
> > 
> > I have added some initial support to mlx5 for rx-buf-len and would like
> > to benchmark it and compare it to what you posted.
> 
> You can use this branch:
> https://github.com/isilence/liburing.git zcrx/rx-buf-len
> 
> # server
> examples/zcrx -p <port> -q <queue_idx> -i <interface_name> -A1 \
>              -B <rx_buf_len> -S <area size / memory provided>
>
> "-A1" here is for using huge pages, so don't forget to configure
> /proc/sys/vm/nr_hugepages.
> 
> # client
> examples/send-zerocopy -6 tcp -D <ip addr> -p <port>
>                        -t <runtime secs>
>                        -l -b1 -n1 -z1 -d -s<send size>
>
Thanks a lot for the branch and the instructions Pavel! I am playing
with them now and seeing some preliminary good results. Will post
them once we share the patches.

> I had to play with the client a bit for it to keep up with
> the server. "-l" enables huge pages, and had to bump up the
> send size. You can also add -v to both for a basic payload
> verification.
>
I see what you mean. I also had to make the rx memory larger once
rx-buf-len >= 32K. Otherwise the traffic was hanging after a second or
so. This is probably related to the currently known issue where if a
page_pool is too small due to incorrect sizing of the buffer, mlx5 hangs
on first refill. That still needs fixing.

Thanks,
Dragos