netdev - rawip: delayed and mis-sequenced transmits

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <433be56da42f4ab2b7722c1caed3a747@AcuMS.aculab.com>
Date:   Wed, 6 Jul 2022 15:54:18 +0000
From:   David Laight <David.Laight@...LAB.COM>
To:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: rawip: delayed and mis-sequenced transmits

I'm seeing some UDP packets being significantly delayed (even 200ms)
and then being received immediately after another packet.
During the delay other packets are received on the same UDP socket.
The receiving program is very simple - so the problem must be
with the sender.

The sender is designed to send a lot of packets, but in this case
two packets are sent every 20ms.
While the two packets have the same UDP port numbers, they are
different application data streams.

The sender has a lot of threads that can send packets, each uses
a single 'rawip' socket (to avoid contention on the socket lock).
It is pretty random (but heavily biased) which threads actually send
the packets.
However it is pretty much certain that the two sends will be done
by different threads at almost exactly the same time.

I've not yet tried to find where the packets get queued, never
mind why. But my best guess is they are stuck on a queue
associated with the sending socket.

But the only reason I can see for a queue on the socket is to
allow the socket lock be dropped while a packet is passed to
the ethernet driver.
So a send from a second thread would queue a packet and it would
be picked up when the earlier transmit completes.
This is actually consistent with what I'm seeing - the end of tx
picks up a packet that was queued earlier.
But in my case the packet was queued much, much earlier.
And the earlier send was done by the same thread.

So what I think must be happening is that contention further
down the protocol stack is causing the packet be queued on the
sending socket instead of (probably) the qdisc layer.

Usually tx packets (seem to) get queued in the qdisc layer and
collected when the thread currently doing a tx setup returns
from the ethernet driver.
So I'm not at all sure why, at least sometimes, packets are
being queued on the socket.

I have got 'threaded napi' enabled (to avoid losing rx packets)
but nothing unusual is enabled on the tx side.
Ethernet is tg3 - single tx ring.

Anyone any ideas before I start digging through the kernel code?

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)