Date:   Tue, 29 Jan 2019 07:16:25 +0100
From:   Heiner Kallweit <hkallweit1@...il.com>
To:     Peter Ceiley <peter@...ley.net>,
        Realtek linux nic maintainers <nic_swsd@...ltek.com>
Cc:     netdev@...r.kernel.org
Subject: Re: r8169 Driver - Poor Network Performance Since Kernel 4.19

Hi Peter,

at a first glance it doesn't look like a typical driver issue.
What you could do:

- Test the r8169.c from 4.18 on top of 4.19.

- Check whether disabling ASPM (/sys/module/pcie_aspm) has an effect.

- Bisect between 4.18 and 4.19 to find the offending commit.

Any specific reason why you think root cause is in the driver and not
elsewhere in the network subsystem?

Heiner


On 28.01.2019 23:10, Peter Ceiley wrote:
> Hi Heiner,
> 
> Thanks for getting back to me.
> 
> No, I don't use jumbo packets.
> 
> Bandwidth is *generally* good, and iperf results to my NAS provide
> over 900 Mbits/s in both circumstances. The issue seems to appear when
> establishing a connection and is most notable, for example, on my
> mounted NFS shares where it takes seconds (up to 10's of seconds on
> larger directories) to list the contents of each directory. Once a
> transfer begins on a file, I appear to get good bandwidth.
> 
> I'm unsure of the best scientific data to provide you in order to
> troubleshoot this issue. Running the following
> 
>     netstat -s |grep retransmitted
> 
> shows a steady increase in retransmitted segments each time I list the
> contents of a remote directory, for example, running 'ls' on a
> directory containing 345 media files did the following using kernel
> 4.19.18:
> 
> increased retransmitted segments by 21 and the 'time' command showed
> the following:
>     real    0m19.867s
>     user    0m0.012s
>     sys    0m0.036s
> 
> The same command shows no retransmitted segments running kernel
> 4.18.16 and 'time' showed:
>     real    0m0.300s
>     user    0m0.004s
>     sys    0m0.007s
> 
> ifconfig does not show any RX/TX errors nor dropped packets in either case.
> 
> dmesg XID:
> [    2.979984] r8169 0000:03:00.0 eth0: RTL8168g/8111g,
> f8:b1:56:fe:67:e0, XID 4c000800, IRQ 32
> 
> # lspci -vv
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
>     Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>     Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>     Latency: 0, Cache Line Size: 64 bytes
>     Interrupt: pin A routed to IRQ 19
>     Region 0: I/O ports at d000 [size=256]
>     Region 2: Memory at f7b00000 (64-bit, non-prefetchable) [size=4K]
>     Region 4: Memory at f2100000 (64-bit, prefetchable) [size=16K]
>     Capabilities: [40] Power Management version 3
>         Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>     Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Address: 0000000000000000  Data: 0000
>     Capabilities: [70] Express (v2) Endpoint, MSI 01
>         DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <512ns, L1 <64us
>             ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> SlotPowerLimit 10.000W
>         DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
>             RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>             MaxPayload 128 bytes, MaxReadReq 4096 bytes
>         DevSta:    CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
>         LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit
> Latency L0s unlimited, L1 <64us
>             ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>         LnkCtl:    ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
>             ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
>         LnkSta:    Speed 2.5GT/s (ok), Width x1 (ok)
>             TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>         DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
> OBFF Via message/WAKE#
>              AtomicOpsCap: 32bit- 64bit- 128bitCAS-
>         DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+,
> OBFF Disabled
>              AtomicOpsCtl: ReqEn-
>         LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>              Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>              Compliance De-emphasis: -6dB
>         LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>              EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>     Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
>         Vector table: BAR=4 offset=00000000
>         PBA: BAR=4 offset=00000800
>     Capabilities: [d0] Vital Product Data
> pcilib: sysfs_read_vpd: read failed: Input/output error
>         Not readable
>     Capabilities: [100 v1] Advanced Error Reporting
>         UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>         UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>         UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>         CESta:    RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ AdvNonFatalErr-
>         CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
>         AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn-
> ECRCChkCap+ ECRCChkEn-
>             MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
>         HeaderLog: 00000000 00000000 00000000 00000000
>     Capabilities: [140 v1] Virtual Channel
>         Caps:    LPEVC=0 RefClk=100ns PATEntryBits=1
>         Arb:    Fixed- WRR32- WRR64- WRR128-
>         Ctrl:    ArbSelect=Fixed
>         Status:    InProgress-
>         VC0:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>             Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>             Ctrl:    Enable+ ID=0 ArbSelect=Fixed TC/VC=01
>             Status:    NegoPending- InProgress-
>     Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
>     Capabilities: [170 v1] Latency Tolerance Reporting
>         Max snoop latency: 71680ns
>         Max no snoop latency: 71680ns
>     Kernel driver in use: r8169
>     Kernel modules: r8169
> 
> Please let me know if you have any other ideas in terms of testing.
> 
> Thanks!
> 
> Peter.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Tue, 29 Jan 2019 at 05:28, Heiner Kallweit <hkallweit1@...il.com> wrote:
>>
>> On 28.01.2019 12:13, Peter Ceiley wrote:
>>> Hi,
>>>
>>> I have been experiencing very poor network performance since Kernel
>>> 4.19 and I'm confident it's related to the r8169 driver.
>>>
>>> I have no issue with kernel versions 4.18 and prior. I am experiencing
>>> this issue in kernels 4.19 and 4.20 (currently running/testing with
>>> 4.20.4 & 4.19.18).
>>>
>>> If someone could guide me in the right direction, I'm happy to help
>>> troubleshoot this issue. Note that I have been keeping an eye on one
>>> issue related to loading of the PHY driver, however, my symptoms
>>> differ in that I still have a network connection. I have attempted to
>>> reload the driver on a running system, but this does not improve the
>>> situation.
>>>
>>> Using the proprietary r8168 driver returns my device to proper working order.
>>>
>>> lshw shows:
>>>        description: Ethernet interface
>>>        product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>>>        vendor: Realtek Semiconductor Co., Ltd.
>>>        physical id: 0
>>>        bus info: pci@...0:03:00.0
>>>        logical name: enp3s0
>>>        version: 0c
>>>        serial:
>>>        size: 1Gbit/s
>>>        capacity: 1Gbit/s
>>>        width: 64 bits
>>>        clock: 33MHz
>>>        capabilities: pm msi pciexpress msix vpd bus_master cap_list
>>> ethernet physical tp aui bnc mii fibre 10bt 10bt-fd 100bt 100bt-fd
>>> 1000bt-fd autonegotiation
>>>        configuration: autonegotiation=on broadcast=yes driver=r8169
>>> duplex=full firmware=rtl8168g-2_0.0.1 02/06/13 ip=192.168.1.25
>>> latency=0 link=yes multicast=yes port=MII speed=1Gbit/s
>>>        resources: irq:19 ioport:d000(size=256)
>>> memory:f7b00000-f7b00fff memory:f2100000-f2103fff
>>>
>>> Kind Regards,
>>>
>>> Peter.
>>>
>> Hi Peter,
>>
>> the description "poor network performance" is quite vague, therefore:
>>
>> - Can you provide any measurements?
>> - iperf results before and after
>> - statistics about dropped packets (rx and/or tx)
>> - Do you use jumbo packets?
>>
>> Also help would be a "lspci -vv" output for the network card and
>> the dmesg output line with the chip XID.
>>
>> Heiner
> 

Powered by blists - more mailing lists