netdev - tg3 dropping packets at high packet rates

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 18 May 2022 16:08:47 +0000
From:   David Laight <David.Laight@...LAB.COM>
To:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
CC:     "'mchan@...adcom.com'" <mchan@...adcom.com>,
        David Miller <davem@...emloft.net>
Subject: tg3 dropping packets at high packet rates

I'm trying to see why the tg3 driver is dropping a lot of
receive packets.

(This driver is making my head hurt...)

I think that the rx_packets count (sum of rx_[umb]cast_packets)
is all the packets, but a smaller number are actually processed
by the tg3_rx()
But none of the error counts get increased.

It is almost as if it has lost almost all the receive buffers.

If I read /sys/class/net/em2/statistics/rx_packets every second
delaying with:
  syscall(SYS_clock_nanosleep, CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL);
about every 43 seconds I get a zero increment.
This really doesn't help!
I've put a count into tg3_rx() that seems to match what IP/UDP
and the application see.

The traffic flow is pretty horrid (but could be worse).
There are 8000 small UDP packets every 20ms.
These are reasonably spread through the 20ms (not back to back).
All the destination ports are different (8000 receiving sockets).
(The receiving application handles this fine (now).)
The packets come from two different systems.

Firstly RSS doesn't seem to work very well.
With the current driver I think everything hits 2 rings.
With the 3.10 RHEL driver it all ends up in one.

Anyway after a hint from Eric I enabled RPS.
This offloads the IP and UDP processing enough to stop
any of the cpu (only 40 of them) from reporting even 50% busy.

I've also increased the rx ring size to 2047.
Changing the coalescing parameters seems to have no effect.

I think there should be 2047 receive buffers.
So 4 interrupts every 20ms or 200/sec might be enough
to receive all the frames.
The actual interrupt rate (deltas on /proc/interrupts)
is actual over 80000/sec.
So it doesn't look as though the driver is ever processing
many packets/interrupt.
If the driver were getting behind I'd expect a smaller number
of interrupts.

This would be consistent with there only being (say) 8 active
receive buffers.

The device in question identifies as:

tg3 0000:02:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address xx
tg3 0000:02:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
tg3 0000:02:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
tg3 0000:02:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit]

Any idea where to look?

Or should I just use different ethernet hardware!
(Although the interrupt coalescing parameters for igb are
also completely broken for this traffic flow.)

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)