Date: Wed, 9 Aug 2023 12:58:05 -0700
From: Jakub Kicinski <kuba@...nel.org>
To: Martin Kjær Jørgensen <me@...y.org>
Cc: netdev@...r.kernel.org, Heiner Kallweit <hkallweit1@...il.com>,
 nic_swsd@...ltek.com
Subject: Re: r8169 link up but no traffic, and watchdog error

CC: Heiner

On Wed, 09 Aug 2023 13:50:31 +0200 Martin Kjær Jørgensen wrote:
> Hello netdev,
> 
> My machine have 2 x PCI-E card both having 2 x "Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE
> Controller" devices giving me 4 ethernet ports in addition to the onboard
> ethernet controller.
> 
> When a cable is plugged to on of the ports and the port in the other end
> suddenly goes down and up again, like if power cycled, all traffic on the wire
> stops even though the port LEDs light up, and kernel produces klog
> entry, indicating a link.
> 
> I need to issue a commands like:
> 
> ip link set enp3s0 down
> ip link set enp3s0 up
> 
> to have the link running again. This sometimes produces the error seen below,
> but not all the time. After commands are issued the traffic flows again like
> normal, until the remote port goes down and up again.
> 
> Have you guys got any ideas about this?

There were some fix in r8169 for power management changes recently.
Could you try the latest stable kernel? 6.4.9 ?

> [   36.324381] logitech-hidpp-device 0003:046D:4055.0009: HID++ 4.5 device connected.
> [  107.354641] r8169 0000:03:00.0 enp3s0: Link is Down
> [  169.142879] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx
> [  180.198458] ------------[ cut here ]------------
> [  180.198469] NETDEV WATCHDOG: enp3s0 (r8169): transmit queue 0 timed out 9028 ms
> [  180.198496] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x232/0x240
> [  180.198509] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq nf_conntrack_netlink xt_addrtype br_netfilter xt_policy jitterentropy_rng drbg ansi_cprng authenc echainiv esp4 xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat xfrm_interface xfrm6_tunnel tunnel6 tunnel4 xfrm_user xfrm_algo twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cmac cast5_avx_x86_64 nls_utf8 cast5_generic cast_common cifs ctr ecb des_generic libdes algif_skcipher camellia_generic cifs_arc4 cifs_md4 dns_resolver camellia_aesni_avx2 fscache netfs camellia_aesni_avx_x86_64 camellia_x86_64 xcbc md4 algif_hash af_alg nvme_fabrics overlay sunrpc binfmt_misc nft_fib_ipv6 nft_nat nft_fib_ipv4 nft_fib nls_ascii nls_cp437 vfat fat intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp bridge coretemp stp llc kvm_intel cfg80211 kvm rfkill
> [  180.198634]  snd_usb_audio uvcvideo videobuf2_vmalloc uvc videobuf2_memops videobuf2_v4l2 snd_usbmidi_lib irqbypass rtsx_usb_ms snd_hwdep iTCO_wdt intel_cstate memstick mei_hdcp videodev mei_wdt mei_pxp intel_pmc_bxt snd_rawmidi snd_seq_device iTCO_vendor_support snd_pcm ftdi_sio videobuf2_common intel_uncore usbserial snd_timer mei_me intel_wmi_thunderbolt snd mc think_lmi watchdog ee1004 firmware_attributes_class wmi_bmof mei soundcore joydev intel_pmc_core int3400_thermal acpi_thermal_rel acpi_tad acpi_pad button evdev nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables msr parport_pc ppdev lp nfnetlink parport fuse loop efi_pstore configfs ip_tables x_tables autofs4 btrfs blake2b_generic hid_logitech_hidpp hid_logitech_dj rtsx_usb_sdmmc mmc_core rtsx_usb hid_jabra hid_generic dm_crypt dm_mod efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear md_mod usbhid hid ext4 crc16 mbcache jbd2 crc32c_generic
> [  180.198772]  i915 i2c_algo_bit drm_buddy crc32_pclmul drm_display_helper crc32c_intel nvme drm_kms_helper nvme_core cec ghash_clmulni_intel sha512_ssse3 rc_core r8169 sha512_generic ahci t10_pi xhci_pci libahci ttm realtek crc64_rocksoft_generic libata xhci_hcd crc64_rocksoft crc_t10dif drm mdio_devres aesni_intel e1000e crct10dif_generic usbcore scsi_mod libphy crypto_simd cryptd crct10dif_pclmul i2c_i801 crc64 i2c_smbus crct10dif_common scsi_common usb_common fan video wmi
> [  180.198837] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.4.0-1-amd64 #1  Debian 6.4.4-2
> [  180.198843] Hardware name: LENOVO 30E30051UK/1052, BIOS S0AKT3AA 04/25/2023
> [  180.198846] RIP: 0010:dev_watchdog+0x232/0x240
> [  180.198852] Code: ff ff ff 48 89 df c6 05 36 03 05 01 01 e8 d6 3d fa ff 45 89 f8 44 89 f1 48 89 de 48 89 c2 48 c7 c7 20 85 cf a9 e8 3e 17 70 ff <0f> 0b e9 2d ff ff ff 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90
> [  180.198857] RSP: 0018:ffffa264c026ce70 EFLAGS: 00010286
> [  180.198862] RAX: 0000000000000000 RBX: ffff948781f58000 RCX: 0000000000000000
> [  180.198866] RDX: 0000000000000104 RSI: 0000000000000027 RDI: 00000000ffffffff
> [  180.198870] RBP: ffff948781f584c8 R08: 0000000000000000 R09: ffffa264c026cd00
> [  180.198873] R10: 0000000000000003 R11: ffffffffaa2d26a8 R12: ffff948781f4f400
> [  180.198875] R13: ffff948781f5841c R14: 0000000000000000 R15: 0000000000002344
> [  180.198879] FS:  0000000000000000(0000) GS:ffff9496b5680000(0000) knlGS:0000000000000000
> [  180.198883] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  180.198886] CR2: 0000561606a0b170 CR3: 0000000146c82002 CR4: 0000000000770ee0
> [  180.198890] PKRU: 55555554
> [  180.198892] Call Trace:
> [  180.198896]  <IRQ>
> [  180.198899]  ? dev_watchdog+0x232/0x240
> [  180.198904]  ? __warn+0x81/0x130
> [  180.198916]  ? dev_watchdog+0x232/0x240
> [  180.198920]  ? report_bug+0x191/0x1c0
> [  180.198929]  ? prb_read_valid+0x1b/0x30
> [  180.198938]  ? handle_bug+0x3c/0x80
> [  180.198945]  ? exc_invalid_op+0x17/0x70
> [  180.198951]  ? asm_exc_invalid_op+0x1a/0x20
> [  180.198960]  ? dev_watchdog+0x232/0x240
> [  180.198965]  ? __pfx_dev_watchdog+0x10/0x10
> [  180.198969]  call_timer_fn+0x24/0x130
> [  180.198979]  ? __pfx_dev_watchdog+0x10/0x10
> [  180.198982]  __run_timers+0x222/0x2c0
> [  180.198992]  run_timer_softirq+0x2f/0x50
> [  180.199000]  __do_softirq+0xf1/0x301
> [  180.199007]  __irq_exit_rcu+0xb5/0x130
> [  180.199016]  sysvec_apic_timer_interrupt+0xa2/0xd0
> [  180.199027]  </IRQ>
> [  180.199029]  <TASK>
> [  180.199031]  asm_sysvec_apic_timer_interrupt+0x1a/0x20
> [  180.199037] RIP: 0010:cpuidle_enter_state+0xcc/0x440
> [  180.199044] Code: da fa 58 ff e8 b5 f0 ff ff 8b 53 04 49 89 c5 0f 1f 44 00 00 31 ff e8 73 06 58 ff 45 84 ff 0f 85 56 02 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 85 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d
> [  180.199048] RSP: 0018:ffffa264c0193e90 EFLAGS: 00000246
> [  180.199052] RAX: ffff9496b5680000 RBX: ffffc264bfaa8e00 RCX: 0000000000000000
> [  180.199055] RDX: 0000000000000002 RSI: ffffffffa9c3fd02 RDI: ffffffffa9c2c5ed
> [  180.199058] RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000033483483
> [  180.199061] R10: ffff9496b56b1d84 R11: 00000000000006b1 R12: ffffffffaa398300
> [  180.199064] R13: 00000029f4a9f550 R14: 0000000000000002 R15: 0000000000000000
> [  180.199071]  cpuidle_enter+0x2d/0x40
> [  180.199081]  do_idle+0x217/0x270
> [  180.199089]  cpu_startup_entry+0x1d/0x20
> [  180.199095]  start_secondary+0x134/0x160
> [  180.199104]  secondary_startup_64_no_verify+0x10b/0x10b
> [  180.199115]  </TASK>
> [  180.199117] ---[ end trace 0000000000000000 ]---
> [  226.051135] r8169 0000:03:00.0 enp3s0: Link is Down
> [  275.843212] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx
> 

Powered by blists - more mailing lists