| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <c4c5aec7-d7d9-2e1e-3dd5-d0c2433a5601@alu.unizg.hr>
Date: Mon, 1 May 2023 02:06:28 +0200
From: Mirsad Goran Todorovac <mirsad.todorovac@....unizg.hr>
To: Heiner Kallweit <hkallweit1@...il.com>, netdev@...r.kernel.org,
linux-kernel@...r.kernel.org
Cc: nic_swsd@...ltek.com, "David S. Miller" <davem@...emloft.net>,
Eric Dumazet <edumazet@...gle.com>,
Jakub Kicinski <kuba@...nel.org>,
Paolo Abeni <pabeni@...hat.com>
Subject: Re: [WITHDRAW PATCH v1 1/1] net: r8169: fix the pci setup so the
Realtek RTL8111/8168/8411 ethernet speeds up
On 01. 05. 2023. 00:49, Heiner Kallweit wrote:
> On 30.04.2023 01:09, Mirsad Goran Todorovac wrote:
>> On 25. 04. 2023. 13:44, Mirsad Goran Todorovac wrote:
>>> It was noticed that Ookla Speedtest had shown only 250 Mbps download and
>>> 310 Mbps upload where Windows 10 on the same box showed 440/310 Mbps, which
>>> is the link capacity.
>>>
>>> This article: https://www.phoronix.com/news/Intel-i219-LM-Linux-60p-Fix
>>> inspired to check our speeds. (Previously I used to think it was a network
>>> congestion, or reduction on our ISP, but now each time Windows 10 downlink
>>> speed is 440 compared to 250 Mbps in Linuxes Linux is performing at 60% of
>>> the speed.)
>>>
>>> The latest 6.3 kernel shows 95% speed up with this patch as compared to the
>>> same commit without it:
>>>
>>> ::::::::::::::
>>> speedtest/6.3.0-00436-g173ea743bf7a-dirty-1
>>> ::::::::::::::
>>> [marvin@...mtodorov ~]$ speedtest -s 41437
>>>
>>> Speedtest by Ookla
>>>
>>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>>> ISP: Croatian Academic and Research Network
>>> Idle Latency: 1.53 ms (jitter: 0.15ms, low: 1.30ms, high: 1.71ms)
>>> Download: 225.13 Mbps (data used: 199.3 MB)
>>> 1.65 ms (jitter: 20.15ms, low: 0.81ms, high: 418.27ms)
>>> Upload: 350.00 Mbps (data used: 157.9 MB)
>>> 3.35 ms (jitter: 19.46ms, low: 1.61ms, high: 474.55ms)
>>> Packet Loss: 0.0%
>>> Result URL: https://www.speedtest.net/result/c/a0084fd8-c275-4019-899a-a1590e49a34b
>>> [marvin@...mtodorov ~]$ speedtest -s 41437
>>>
>>> Speedtest by Ookla
>>>
>>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>>> ISP: Croatian Academic and Research Network
>>> Idle Latency: 1.54 ms (jitter: 0.28ms, low: 1.17ms, high: 1.64ms)
>>> Download: 222.88 Mbps (data used: 207.9 MB)
>>> 10.23 ms (jitter: 31.76ms, low: 0.75ms, high: 353.79ms)
>>> Upload: 349.91 Mbps (data used: 157.7 MB)
>>> 3.27 ms (jitter: 13.05ms, low: 1.67ms, high: 236.76ms)
>>> Packet Loss: 0.0%
>>> Result URL: https://www.speedtest.net/result/c/f4c663ba-830d-44c6-8033-ce3b3b818c42
>>> [marvin@...mtodorov ~]$
>>> ::::::::::::::
>>> speedtest/6.3.0-r8169-00437-g323fe5352af6-dirty-2
>>> ::::::::::::::
>>> [marvin@...mtodorov ~]$ speedtest -s 41437
>>>
>>> Speedtest by Ookla
>>>
>>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>>> ISP: Croatian Academic and Research Network
>>> Idle Latency: 0.84 ms (jitter: 0.05ms, low: 0.82ms, high: 0.93ms)
>>> Download: 432.37 Mbps (data used: 360.5 MB)
>>> 142.43 ms (jitter: 76.45ms, low: 1.02ms, high: 1105.19ms)
>>> Upload: 346.29 Mbps (data used: 164.6 MB)
>>> 7.72 ms (jitter: 29.80ms, low: 0.92ms, high: 283.48ms)
>>> Packet Loss: 12.8%
>>> Result URL: https://www.speedtest.net/result/c/e473359e-c37e-4f29-aa9f-4b008210cf7c
>>> [marvin@...mtodorov ~]$ speedtest -s 41437
>>>
>>> Speedtest by Ookla
>>>
>>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>>> ISP: Croatian Academic and Research Network
>>> Idle Latency: 0.82 ms (jitter: 0.16ms, low: 0.75ms, high: 1.05ms)
>>> Download: 440.97 Mbps (data used: 427.5 MB)
>>> 72.50 ms (jitter: 52.89ms, low: 0.91ms, high: 865.08ms)
>>> Upload: 342.75 Mbps (data used: 166.6 MB)
>>> 3.26 ms (jitter: 22.93ms, low: 1.07ms, high: 239.41ms)
>>> Packet Loss: 13.4%
>>> Result URL: https://www.speedtest.net/result/c/f393e149-38d4-4a34-acc4-5cf81ff13708
>>>
>>> 440 Mbps is the speed achieved in Windows 10, and Linux 6.3 with
>>> the patch, while 225 Mbps without this patch is running at 51% of
>>> the nominal speed with the same hardware and Linux kernel commit.
>>>
>>> Cc: David S. Miller <davem@...emloft.net>
>>> Cc: Eric Dumazet <edumazet@...gle.com>
>>> Cc: Jakub Kicinski <kuba@...nel.org>
>>> Cc: Paolo Abeni <pabeni@...hat.com>
>>> Cc: nic_swsd@...ltek.com
>>> Cc: netdev@...r.kernel.org
>>> Link: https://bugzilla.redhat.com/show_bug.cgi?id=1671958#c60
>>> Suggested-by: Heiner Kallweit <hkallweit1@...il.com>
>>> Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac@....unizg.hr>
>>> ---
>>> drivers/net/ethernet/realtek/r8169_main.c | 1 +
>>> 1 file changed, 1 insertion(+)
>>>
>>> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
>>> index 45147a1016be..b8a04301d130 100644
>>> --- a/drivers/net/ethernet/realtek/r8169_main.c
>>> +++ b/drivers/net/ethernet/realtek/r8169_main.c
>>> @@ -3239,6 +3239,7 @@ static void rtl_hw_start_8168h_1(struct rtl8169_private *tp)
>>> r8168_mac_ocp_write(tp, 0xc094, 0x0000);
>>> r8168_mac_ocp_write(tp, 0xc09e, 0x0000);
>>>
>>> + pci_disable_link_state(tp->pci_dev, PCIE_LINK_STATE_CLKPM);
>>> rtl_hw_aspm_clkreq_enable(tp, true);
>>> }
>>>
>>
>> After some additional research, I came to the obvious realisation, reading more
>> thoroughly the discussion at the link - that the above patch did not work for
>> all Realtek RTL819x cards back then.
>>
>> My version, the RTL8168h/8111h indeed works 95% faster on the 6.3 Linux kernel,
>> but I cannot speak for the people with the power management problems and
>> battery life issues ... and the concerns explained here: https://github.com/KastB/r8169
>>
>> [root@...mtodorov marvin]# dmesg | grep RTL
>> [ 7.304130] r8169 0000:01:00.0 eth0: RTL8168h/8111h, f4:93:9f:f0:a5:f5, XID 541, IRQ 123
>>
>> Currently there seem to be at least 43 revisions of the RTL816x cards and firmware,
>> and I really cannot test on all of them.
>>
>> I will test the other Heiner's experimental patch, but it seems to disable ASPM completely,
>> while for my Lenovo desktop with RTL8168h/8111h disabling CLKPM alone.
>>
>> However, further homework revealed that the kernel patch is unnecessary, as the same
>> effect can be achieved in runtime by the sysfs parm introduced with THIS PATCH:
>> https://patchwork.kernel.org/project/linux-pci/patch/b1c83f8a-9bf6-eac5-82d0-cf5b90128fbf@gmail.com/
>> which was solved 3 1/2 years ago, but the default on my AlmaLinux 8.7 and Lenovo desktop
>> box 10TX000VCR was the 53% of the link capacity and speed.
>>
>> (I don't know if the card would operate with 220 Mbps on a Gigabit link, it was
>> not tested.)
>>
>> [marvin@...mtodorov ~]$ speedtest -s 41437
>>
>> Speedtest by Ookla
>>
>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>> ISP: Croatian Academic and Research Network
>> Idle Latency: 1.44 ms (jitter: 0.23ms, low: 1.20ms, high: 1.65ms)
>> Download: 220.62 Mbps (data used: 214.2 MB)
>> 22.01 ms (jitter: 36.04ms, low: 0.84ms, high: 817.47ms)
>> Upload: 346.86 Mbps (data used: 169.1 MB)
>> 3.32 ms (jitter: 12.12ms, low: 0.87ms, high: 221.69ms)
>> Packet Loss: 0.6%
>> Result URL: https://www.speedtest.net/result/c/20c546e7-0b8f-4a2e-a669-a597bb5aee36
>> [marvin@...mtodorov ~]$ sudo bash
>> [sudo] password for marvin:
>> [root@...mtodorov marvin]# cat /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
>> 1
>> [root@...mtodorov marvin]# echo 0 > /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
>> [root@...mtodorov marvin]# cat /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
>> 0
>> [root@...mtodorov marvin]# speedtest -s 41437
>>
>> Speedtest by Ookla
>>
>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>> ISP: Croatian Academic and Research Network
>> Idle Latency: 0.85 ms (jitter: 0.06ms, low: 0.78ms, high: 0.92ms)
>> Download: 431.13 Mbps (data used: 341.0 MB)
>> 157.40 ms (jitter: 68.09ms, low: 0.88ms, high: 823.19ms)
>> Upload: 351.36 Mbps (data used: 158.3 MB)
>> 2.88 ms (jitter: 6.24ms, low: 1.41ms, high: 209.74ms)
>> Packet Loss: 13.4%
>> Result URL: https://www.speedtest.net/result/c/ff695466-3ac7-405e-8cae-0a85c2c3d5cd
>> [root@...mtodorov marvin]#
>>
>> The clkpm setting can be reversed back to 1, causing the RTL speed to drop again.
>>
>> So, the patch is withdrawn as unnecessary, even when the majority of RTL8168h/8111h
>> and possibly other Realtek Gigabit cards will by default run at sub-Gigabit speeds.
>>
>
> RTL8168h doesn't need the CLKPM quirk in general. E.g. my test system runs fine w/o it
> at 950Mbps. Seems that ASPM is broken on your system.
> Alternatively you can test with latest linux-next, it disables ASPM during NAPI poll.
My system is basically a Lenovo minitower desktop with a SSD SATA disk.
Nothing peculiar. Please find attached lshw.txt and lspci -vv, if that could
help.
ASPM is reported as OK for all PCI devices (PCI-E root and RTL8198h). I guess
for a desktop ASPM is not all that critical.
All my RFC PATCH actually disabled CLKPM, however the lack of intel made me
recompile rather than do the runtime switching off. Kernel docs are unfortunately
around 2010 on PCI Express. :-(
I think that the problem might be in our Mikrotik RB2011 UiAS-RM which still
has Fast Ethernet ports and must be 15 years old and most likely doesn't support
CLKPM so odd things happen ...
But Windows 10 default is faster as I said, and I can't tell how many Linux
users will figure out to turn off CLKPM via sysfs ...
I saw that you had regressions with the fix I proposed (which is in fact
almost entirely based on your patch and applied to the different file as
the code was somewhat reorganised). I can simply run it at boot in a unit or
from cron, but sort of feel that the most sensible setting should be the default.
I wasn't lucky with finding RTL8168h specs, so I guess I can only keep my fingers
crossed for your zero copy and io_uring implementation of r8169.
We are really out of luck, but RTL is so ubiquitous and I think higher powers
would appreciate this driver debugged. ;-)
Mirsad
--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
The European Union
"I see something approaching fast ... Will it be friends with me?"
View attachment "lspci-vv-lenovo.txt" of type "text/plain" (22079 bytes)
View attachment "lshw-pc-lenovo.txt" of type "text/plain" (23812 bytes)
Powered by blists - more mailing lists