netdev - Re: High packet rate udp receive

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iLUUKCG0FbRU=xC2Snm3aQcMikEFwFv=DSTt_Ti7VvJsg@mail.gmail.com>
Date:   Mon, 9 May 2022 10:00:31 -0700
From:   Eric Dumazet <edumazet@...gle.com>
To:     David Laight <David.Laight@...lab.com>
Cc:     netdev <netdev@...r.kernel.org>,
        "David S . Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>
Subject: Re: High packet rate udp receive

On Mon, May 9, 2022 at 9:55 AM David Laight <David.Laight@...lab.com> wrote:
>
> I'm testing some high channel count RTP (UDP audio).
> When I get much over 250000 receive packets/second the
> network receive softint processing seems to overload
> one cpu and then packets are silently discarded somewhere.
>
> I (probably) can see all the packets in /sys/class/net/em2/statistics/rx_packets
> but the counts from 'netstat -s' are a lot lower.
>
> The packets are destined for a lot of UDP sockets - each gets 50/sec.
> These can't be 'connected' because the source address is allowed to change.
> For testing the source IP is pretty fixed.
> But I've not tried to look for the actual bottleneck.
>
> Are we stuck with one cpu doing all the ethernet, IP and UDP
> receive processing?

RPS can help, if your NIC has a single queue.

Please look at Documentation/networking/scaling.rst for details.

You might have to tune the number of cpus you want to be able to
process these packets.

> (and the end of transmit reaping).
> Or is there a way to get another cpu to do some of the work?
>
> Since this is UDP things like gro can't help.
> We do have to handle very large numbers of packets.
>
> Would a multi-queue ethernet adapter help?

Might be, at least by having a more efficient bus.

> This system has a BCM5720 (tg3 driver) which I don't think is multi-Q.
>
> OTOH I've also had issues with a similar packet rate on an intel
> nic that would be multi-q because the interrupt mitigation logic
> is completely broken for high packet rates.
> Only increasing the ring size to 4096 stopped it dropping packets.
>
>         David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)
>