netdev - Re: [regression] UDP recv data corruption

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20210702163148.Horde.OPbmfqdhfEMDjt3Q2A7ru7c@mail.your-server.de>
Date:   Fri, 02 Jul 2021 16:31:48 +0200
From:   Matthias Treydte <mt@...dheinz.de>
To:     Paolo Abeni <pabeni@...hat.com>
Cc:     stable@...r.kernel.org, netdev@...r.kernel.org,
        regressions@...ts.linux.dev, davem@...emloft.net,
        yoshfuji@...ux-ipv6.org, dsahern@...nel.org,
        Willem de Bruijn <willemdebruijn.kernel@...il.com>,
        David Ahern <dsahern@...il.com>
Subject: Re: [regression] UDP recv data corruption

Quoting Paolo Abeni <pabeni@...hat.com>:

Finally got your mail, seems it was delayed, sorry.

> - check, if possibly, how exactly the pkts are corrupted. Wrong size?
> bad csum? what else?
>
> - ideally a short pcap trace comprising the problematic packets would
> be great!

While trying to produce something I came up with an interesting  
observation which seems to indicate that the problem lies in the  
combination of GRO and fragmented UDP packets:

Tuning the sending side (for my testing purposes also FFMpeg) to send  
UDP packets of 1316 bytes tops makes the problem go away in the  
receiver. The value must be an exact multiple of 188 (the MPEG TS  
frame size) to cause FFMpeg not to send fragmented packets at all.

Using this we were able to do the following on our normal desktop machines:

The sending side uses an command like this:

ffmpeg \
   -stream_loop -1 \
   -re -i "$video" \
   -c:v copy -c:a copy \
   -f mpegts "udp://239.12.23.0:1935"

and the receiver (in our case using Linux 5.12.14) "mpv  
udp://239.12.23.0:1935" to see the stream. For our test $video was  
just some h264 encoded MKV I had laying around. The receiver sees  
compression artifacts and the "Packet corrupt" messages in the  
console. Now there are two ways to improve this situation:

1) The receiver uses ethtool to disable GRO
2) The sender changes the URL to be "udp://239.12.23.0:1935?pkt_size=1316"

At this point I assume there are better ways to reproduce this using  
netcat or the like. But being a video guy, well, here we are. ;-)

My knowledge about the inner workings of Linux' IP stack are lacking,  
but because tcpdump "sees" packets before they are reassembled and the  
problem seems to exist only with packets that were fragmented and  
reassembled (as they are presented to libav), I have the feeling that  
a pcap file would not be that helpful with this, right?

Regards,
-Matthias