lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Mon, 9 May 2022 10:00:31 -0700
From:   Eric Dumazet <edumazet@...gle.com>
To:     David Laight <David.Laight@...lab.com>
Cc:     netdev <netdev@...r.kernel.org>,
        "David S . Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>
Subject: Re: High packet rate udp receive

On Mon, May 9, 2022 at 9:55 AM David Laight <David.Laight@...lab.com> wrote:
>
> I'm testing some high channel count RTP (UDP audio).
> When I get much over 250000 receive packets/second the
> network receive softint processing seems to overload
> one cpu and then packets are silently discarded somewhere.
>
> I (probably) can see all the packets in /sys/class/net/em2/statistics/rx_packets
> but the counts from 'netstat -s' are a lot lower.
>
> The packets are destined for a lot of UDP sockets - each gets 50/sec.
> These can't be 'connected' because the source address is allowed to change.
> For testing the source IP is pretty fixed.
> But I've not tried to look for the actual bottleneck.
>
> Are we stuck with one cpu doing all the ethernet, IP and UDP
> receive processing?

RPS can help, if your NIC has a single queue.

Please look at Documentation/networking/scaling.rst for details.

You might have to tune the number of cpus you want to be able to
process these packets.

> (and the end of transmit reaping).
> Or is there a way to get another cpu to do some of the work?
>
> Since this is UDP things like gro can't help.
> We do have to handle very large numbers of packets.
>
> Would a multi-queue ethernet adapter help?

Might be, at least by having a more efficient bus.

> This system has a BCM5720 (tg3 driver) which I don't think is multi-Q.
>
> OTOH I've also had issues with a similar packet rate on an intel
> nic that would be multi-q because the interrupt mitigation logic
> is completely broken for high packet rates.
> Only increasing the ring size to 4096 stopped it dropping packets.
>
>         David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)
>

Powered by blists - more mailing lists