netdev - Re: Packets get stuck on RTL8111H using R8169 driver

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date: Thu, 28 Sep 2023 12:39:54 +0200
From: Heiner Kallweit <hkallweit1@...il.com>
To: Alexander Merkle <alexander.merkle@...terbach.com>, nic_swsd@...ltek.com,
 romieu@...zoreil.com
Cc: netdev@...r.kernel.org
Subject: Re: Packets get stuck on RTL8111H using R8169 driver

On 28.09.2023 11:28, Alexander Merkle wrote:
> Hi,
> 
> I want to report a problem seen on my quite recent desktop machine using an onboard RTL8111H ethernet controller mounted on a ASUS X570-P motherboard.
> I problem can be reproduced using
>   - Debian Bookworm using Kernel `6.1.0-12-amd64` / `6.1.52-1`
>   - Fedora Desktop 39 Beta using Kernel `6.5.5`
> . In both cases we see that small ethernet packets e.g. ICMP/UDP seem to get stuck in the controller and are send out when there is other activity on the interface.
> 

Is there an actual problem? Or are you just curious why the numbers are different?

> The simplest scenario to use is using `ping` in our office environment (active network). We used a quite powerful company core switch as ping target.
> Using the R8169 driver the ping times are 2-3 times as high as using the r8168-dkms driver from debian (non-free). In numbers
>   r8169: ~800-900us
>   r8168: ~200-300us
> .

In a home network with current linux-next and same chip version,
other side being my DSL router:

PING 192.168.178.1 (192.168.178.1) 56(84) bytes of data.
64 bytes from 192.168.178.1: icmp_seq=1 ttl=64 time=0.418 ms
64 bytes from 192.168.178.1: icmp_seq=2 ttl=64 time=0.485 ms
64 bytes from 192.168.178.1: icmp_seq=3 ttl=64 time=0.366 ms
64 bytes from 192.168.178.1: icmp_seq=4 ttl=64 time=0.363 ms
64 bytes from 192.168.178.1: icmp_seq=5 ttl=64 time=0.365 ms
64 bytes from 192.168.178.1: icmp_seq=6 ttl=64 time=0.361 ms
64 bytes from 192.168.178.1: icmp_seq=7 ttl=64 time=0.366 ms
64 bytes from 192.168.178.1: icmp_seq=8 ttl=64 time=0.549 ms
64 bytes from 192.168.178.1: icmp_seq=9 ttl=64 time=0.368 ms
64 bytes from 192.168.178.1: icmp_seq=10 ttl=64 time=0.474 ms
64 bytes from 192.168.178.1: icmp_seq=11 ttl=64 time=0.362 ms
64 bytes from 192.168.178.1: icmp_seq=12 ttl=64 time=0.365 ms
64 bytes from 192.168.178.1: icmp_seq=13 ttl=64 time=0.370 ms
64 bytes from 192.168.178.1: icmp_seq=14 ttl=64 time=0.352 ms

Hard to say why you see higher ping times.
1. Please test with a mainline kernel. Vendor kernels may include whatever changes.
2. Test different interrupt coalescing and GRO settings.
   For the latter see gro_flush_timeout and napi_defer_hard_irqs sysfs attributes

My setup uses the driver defaults.

> 
> Using our UDP based communication between host and device we see that UDP packets (especially small ones) are not send out and reach the device only when there is other activity on the ethernet link.
> Using the r8169 driver we did a cross-check to evaluate our theory and using a `ping -f <powerful company core switch>`  as root running in background. With that cross-check applied we see that the delayed packets / UDP protocol resends are gone.
> 
> I will try to collect as much information as possible for you:
> ```
> $ lspci
> 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
> ```
> Device is labeled as: REALTEK 8111H L31ZY23GL15
> ```
> # DMESG output using r8168 driver
> pci 0000:05:00.0: [10ec:8168] type 00 class 0x020000
> r8168: loading out-of-tree module taints kernel.
> r8168: module verification failed: signature and/or required key missing - tainting kernel
> r8168 Gigabit Ethernet driver 8.051.02-NAPI loaded
> r8168: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.
> r8168  Copyright (C) 2022 Realtek NIC software team <nicfae@...ltek.com>
> r8168 0000:05:00.0 enp5s0: renamed from eth0
> r8168: enp5s0: link up
> # DMESG output using r8169 driver - mac address is removed
> r8169 0000:05:00.0 eth0: RTL8168h/8111h, <mac address removed>, XID 541, IRQ 145
> r8169 0000:05:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
> r8169 0000:05:00.0 enp5s0: renamed from eth0
> r8169 0000:05:00.0: firmware: direct-loading firmware rtl_nic/rtl8168h-2.fw
> Generic FE-GE Realtek PHY r8169-0-500:00: attached PHY driver (mii_bus:phy_addr=r8169-0-500:00, irq=MAC)
> r8169 0000:05:00.0 enp5s0: Link is Down
> r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control off
> ```
> 
> Regards,
> 
> Alex
>