netdev - Re: High packet rate udp receive

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAA93jw5pYtaPuzbVm-sFF5_pWup7PmzE+4aV+hm04_K00nE3kQ@mail.gmail.com>
Date:   Mon, 9 May 2022 10:09:59 -0700
From:   Dave Taht <dave.taht@...il.com>
To:     David Laight <David.Laight@...lab.com>
Cc:     netdev <netdev@...r.kernel.org>,
        "David S . Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Eric Dumazet <edumazet@...gle.com>
Subject: Re: High packet rate udp receive

On Mon, May 9, 2022 at 10:01 AM David Laight <David.Laight@...lab.com> wrote:
>
> I'm testing some high channel count RTP (UDP audio).
> When I get much over 250000 receive packets/second the
> network receive softint processing seems to overload
> one cpu and then packets are silently discarded somewhere.
>
> I (probably) can see all the packets in /sys/class/net/em2/statistics/rx_packets
> but the counts from 'netstat -s' are a lot lower.
>
> The packets are destined for a lot of UDP sockets - each gets 50/sec.
> These can't be 'connected' because the source address is allowed to change.
> For testing the source IP is pretty fixed.
> But I've not tried to look for the actual bottleneck.
>
> Are we stuck with one cpu doing all the ethernet, IP and UDP
> receive processing?
> (and the end of transmit reaping).
> Or is there a way to get another cpu to do some of the work?
>
> Since this is UDP things like gro can't help.
> We do have to handle very large numbers of packets.
>
> Would a multi-queue ethernet adapter help?
> This system has a BCM5720 (tg3 driver) which I don't think is multi-Q.
>
> OTOH I've also had issues with a similar packet rate on an intel
> nic that would be multi-q because the interrupt mitigation logic
> is completely broken for high packet rates.
> Only increasing the ring size to 4096 stopped it dropping packets.
>
>         David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)
>

In general I favor larger rx rings these days as linux presently seems
to be very batchy on rx.
-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC