Message-ID: <c224bee7-7056-4c2a-a234-b8cb79900d40@gmail.com>
Date: Sun, 3 Nov 2024 23:21:54 +0100
From: Heiner Kallweit <hkallweit1@...il.com>
To: Felix Braun <f.braun@...ix.de>, nic_swsd@...ltek.com
Cc: netdev@...r.kernel.org
Subject: Re: r8169: regression in connection speed with kernels 6.2+
 (interrupt coalescing)

On 03.11.2024 17:08, Felix Braun wrote:
> Hi there,
> 
> commit 42f66a44d83715bef810a543dfd66008b883a7a5 to Linus' kernel tree ("r8169: enable GRO software interrupt coalescing per default") introduces a speed regression on my hardware. With that commit applied I get net throughput of 10.5 MB/s, without that commit I get around 100 MB/s on my setup.
> 
Thanks for the report. 6.2 has been out for quite some time, and this is
the first such report. So I don't think there's a general problem.
Can you please provide a full dmesg log and elaborate on the type of traffic
and how you measure the speed? BTW: With 100MB/s you refer to 100MBit/s?

Also interesting would be whether there are any errors or missed packets
in the ethtool -S <if> output.

Instead of commenting out this line you can also adjust the values from userspace:
/sys/class/net/<if>/gro_flush_timeout
/sys/class/net/<if>/napi_defer_hard_irqs
Does increasing the gro_flush_timeout value change something for you?

Somewhat strange is that lspci shows ASPM as disabled in LnkCtl, but
L1 sub-states are enabled in L1SubCtl1. Do you have any downstream kernel code
changes or any specific ASPM settings?

> I've verified that just commenting out the one line in r8169_main.c
> 
> ```
> --- a/drivers/net/ethernet/realtek/r8169_main.c  
> +++ b/drivers/net/ethernet/realtek/r8169_main.c  
> @@ -5505,6 +5505,8 @@ static int rtl_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)  
> dev->hw_features |= NETIF_F_RXALL;  
> dev->hw_features |= NETIF_F_RXFCS;
> 
> dev->pcpu_stat_type = NETDEV_PCPU_STAT_TSTATS;
> 
> - netdev_sw_irq_coalesce_default_on(dev);  
> + //netdev_sq_irq_coalesc_default_on(dev);
> 
> /* configure chip for default features */  
> rtl8169_set_features(dev, dev->features);
> ```
> 
> restores the speed with kernel 6.11.6. Is there perhaps a more elegant way to fix this regression for other people too?
> 
> Regards  
> Felix
> 
> This is my lspci -vvxx output:
> 
> ```
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15)
> DeviceName: Onboard - RTK Ethernet
> Subsystem: ASRock Incorporation Motherboard (one of many)
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 20
> IOMMU group: 11
> Region 0: I/O ports at e000 [size=256]
> Region 2: Memory at a1204000 (64-bit, non-prefetchable) [size=4K]
> Region 4: Memory at a1200000 (64-bit, non-prefetchable) [size=16K]
> Capabilities: [40] Power Management version 3
> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
> Address: 0000000000000000  Data: 0000
> Capabilities: [70] Express (v2) Endpoint, IntMsgNum 1
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10W TEE-IO-
> DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
> MaxPayload 128 bytes, MaxReadReq 4096 bytes
> DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x1
> TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
> 10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
> EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
> FRS- TPHComp- ExtTPHComp-
> AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
> AtomicOpsCtl: ReqEn-
> IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
> 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
> LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
> LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
> EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
> Retimer- 2Retimers- CrosslinkRes: unsupported
> Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
> Vector table: BAR=4 offset=00000000
> PBA: BAR=4 offset=00000800
> Capabilities: [100 v2] Advanced Error Reporting
> UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
> ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
> PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
> UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
> ECRC- UnsupReq+ ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
> PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
> ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
> PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
> CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
> CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
> AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> HeaderLog: 00000000 00000000 00000000 00000000
> Capabilities: [140 v1] Virtual Channel
> Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
> Arb:    Fixed- WRR32- WRR64- WRR128-
> Ctrl:   ArbSelect=Fixed
> Status: InProgress-
> VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> Status: NegoPending- InProgress-
> Capabilities: [160 v1] Device Serial Number 39-79-71-c2-85-70-00-00
> Capabilities: [170 v1] Latency Tolerance Reporting
> Max snoop latency: 3145728ns
> Max no snoop latency: 3145728ns
> Capabilities: [178 v1] L1 PM Substates
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
> PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
> L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> T_CommonMode=0us LTR1.2_Threshold=163840ns
> L1SubCtl2: T_PwrOn=150us
> Kernel driver in use: r8169
> Kernel modules: r8169
> 00: ec 10 68 81 07 04 10 00 15 00 00 02 10 00 00 00
> 10: 01 e0 00 00 00 00 00 00 04 40 20 a1 00 00 00 00
> 20: 04 00 20 a1 00 00 00 00 00 00 00 00 49 18 68 81
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
> ```

Powered by blists - more mailing lists