Message-ID: <20230215091103.17c8addd@hermes.local>
Date:   Wed, 15 Feb 2023 09:11:03 -0800
From:   Stephen Hemminger <stephen@...workplumber.org>
To:     netdev@...r.kernel.org
Subject: Fw: [Bug 217040] New: TXDCTL.ENABLE for one or more queues not
 cleared within the polling period



Begin forwarded message:

Date: Wed, 15 Feb 2023 03:46:21 +0000
From: bugzilla-daemon@...nel.org
To: stephen@...workplumber.org
Subject: [Bug 217040] New: TXDCTL.ENABLE for one or more queues not cleared within the polling period


https://bugzilla.kernel.org/show_bug.cgi?id=217040

            Bug ID: 217040
           Summary: TXDCTL.ENABLE for one or more queues not cleared
                    within the polling period
           Product: Networking
           Version: 2.5
    Kernel Version: 5.4.0-100-generic
          Hardware: Intel
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
          Assignee: stephen@...workplumber.org
          Reporter: satish.txt@...il.com
        Regression: No

I am running Ubuntu 20.04.4 with kernel version 5.4.0-100-generic. Recently in
my server rack one of server had memory failure caused all the servers in that
rack received following kernel trace and they failed to connect. 

Server vendor: HP DL460 Gen9 blades

# lspci | grep -i eth
06:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port
Backplane Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port
Backplane Connection (rev 01)

~# ethtool -i eno50
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x800008f0, 1.2836.0
expansion-rom-version:
bus-info: 0000:06:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


Kernel version: 5.4.0-100-generic #113-Ubuntu


# brctl show
bridge name     bridge id               STP enabled     interfaces
br-mgmt         8000.38eaa733b589       no              eno50.51
br-vlan         8000.38eaa733b589       no              eno50
br-vxlan        8000.38eaa733b589       no              eno50.29


Kernel trace logs, what could be the issue here. Does it related to STP loop or
some kind of failure. 

[Thu Feb  9 07:17:07 2023] ------------[ cut here ]------------
[Thu Feb  9 07:17:07 2023] NETDEV WATCHDOG: eno50 (ixgbe): transmit queue 19
timed out
[Thu Feb  9 07:17:07 2023] WARNING: CPU: 16 PID: 0 at
net/sched/sch_generic.c:472 dev_watchdog+0x258/0x260
[Thu Feb  9 07:17:07 2023] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs
msdos jfs xfs cpuid xt_CHECKSUM xt_conntrack xt_tcpudp ip6table_mangle
ip6table_nat binfmt_misc nf_tables nfnetlink iptable_raw bpfilter vhost_net
vhost tap nbd iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_vs
iptable_nat iptable_mangle iptable_filter ipt_REJECT nf_reject_ipv4
xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter
ip6_tables ebtables dm_snapshot dm_bufio br_netfilter 8021q garp mrp bridge stp
llc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ipmi_ssif ixgbevf
intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp
coretemp joydev input_leds kvm_intel kvm rapl intel_cstate serio_raw hpilo
ioatdma ipmi_si ipmi_devintf ipmi_msghandler acpi_tad mac_hid acpi_power_meter
sch_fq_codel msr ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
libcrc32c raid1 raid0 multipath
[Thu Feb  9 07:17:07 2023]  linear mgag200 drm_vram_helper i2c_algo_bit ttm
drm_kms_helper crct10dif_pclmul syscopyarea crc32_pclmul sysfillrect
ghash_clmulni_intel sysimgblt hid_generic aesni_intel fb_sys_fops crypto_simd
ixgbe usbhid cryptd psmouse hpsa glue_helper xfrm_algo drm hid i2c_i801 lpc_ich
scsi_transport_sas dca mdio wmi
[Thu Feb  9 07:17:07 2023] CPU: 16 PID: 0 Comm: swapper/16 Not tainted
5.4.0-100-generic #113-Ubuntu
[Thu Feb  9 07:17:07 2023] Hardware name: HP ProLiant BL460c Gen9, BIOS I36
03/25/2019
[Thu Feb  9 07:17:07 2023] RIP: 0010:dev_watchdog+0x258/0x260
[Thu Feb  9 07:17:07 2023] Code: 85 c0 75 e5 eb 9f 4c 89 ff c6 05 84 a2 2c 01
01 e8 ad bf fa ff 44 89 e9 4c 89 fe 48 c7 c7 f8 dc 63 8c 48 89 c2 e8 34 e2 13
00 <0f> 0b eb 80 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 d7
[Thu Feb  9 07:17:07 2023] RSP: 0018:ffffa133467c0e30 EFLAGS: 00010286
[Thu Feb  9 07:17:07 2023] RAX: 0000000000000000 RBX: ffff8f9deef6cec0 RCX:
000000000000083f
[Thu Feb  9 07:17:07 2023] RDX: 0000000000000000 RSI: 00000000000000f6 RDI:
000000000000083f
[Thu Feb  9 07:17:07 2023] RBP: ffffa133467c0e60 R08: ffff8fae1f71c8c8 R09:
0000000000000004
[Thu Feb  9 07:17:07 2023] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000040
[Thu Feb  9 07:17:07 2023] R13: 0000000000000013 R14: ffff8f9df16a0480 R15:
ffff8f9df16a0000
[Thu Feb  9 07:17:07 2023] FS:  0000000000000000(0000)
GS:ffff8fae1f700000(0000) knlGS:0000000000000000
[Thu Feb  9 07:17:07 2023] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu Feb  9 07:17:07 2023] CR2: 00007f0840382000 CR3: 0000000acea0a006 CR4:
00000000001626e0
[Thu Feb  9 07:17:07 2023] Call Trace:
[Thu Feb  9 07:17:07 2023]  <IRQ>
[Thu Feb  9 07:17:07 2023]  ? pfifo_fast_enqueue+0x150/0x150
[Thu Feb  9 07:17:07 2023]  call_timer_fn+0x32/0x130
[Thu Feb  9 07:17:07 2023]  __run_timers.part.0+0x180/0x280
[Thu Feb  9 07:17:07 2023]  ? tick_sched_handle+0x33/0x60
[Thu Feb  9 07:17:07 2023]  ? tick_sched_timer+0x3d/0x80
[Thu Feb  9 07:17:07 2023]  ? ktime_get+0x3e/0xa0
[Thu Feb  9 07:17:07 2023]  run_timer_softirq+0x2a/0x50
[Thu Feb  9 07:17:07 2023]  __do_softirq+0xe1/0x2d6
[Thu Feb  9 07:17:07 2023]  ? hrtimer_interrupt+0x136/0x220
[Thu Feb  9 07:17:07 2023]  irq_exit+0xae/0xb0
[Thu Feb  9 07:17:07 2023]  smp_apic_timer_interrupt+0x7b/0x140
[Thu Feb  9 07:17:07 2023]  apic_timer_interrupt+0xf/0x20
[Thu Feb  9 07:17:07 2023]  </IRQ>
[Thu Feb  9 07:17:07 2023] RIP: 0010:cpuidle_enter_state+0xc5/0x450
[Thu Feb  9 07:17:07 2023] Code: ff e8 ef b8 84 ff 80 7d c7 00 74 17 9c 58 0f
1f 44 00 00 f6 c4 02 0f 85 65 03 00 00 31 ff e8 62 c0 8a ff fb 66 0f 1f 44 00
00 <45> 85 ed 0f 88 8f 02 00 00 49 63 cd 4c 8b 7d d0 4c 2b 7d c8 48 8d
[Thu Feb  9 07:17:07 2023] RSP: 0018:ffffa13346377e38 EFLAGS: 00000246
ORIG_RAX: ffffffffffffff13
[Thu Feb  9 07:17:07 2023] RAX: ffff8fae1f72fe00 RBX: ffffffff8cd59fe0 RCX:
000000000000001f
[Thu Feb  9 07:17:07 2023] RDX: 0000000000000000 RSI: 000000003342629e RDI:
0000000000000000
[Thu Feb  9 07:17:07 2023] RBP: ffffa13346377e78 R08: 00285439725dde3a R09:
00285454394d7639
[Thu Feb  9 07:17:07 2023] R10: ffff8fae1f72eb00 R11: ffff8fae1f72eae0 R12:
ffffc1333f9021c0
[Thu Feb  9 07:17:07 2023] R13: 0000000000000004 R14: 0000000000000004 R15:
ffffc1333f9021c0
[Thu Feb  9 07:17:07 2023]  ? cpuidle_enter_state+0xa1/0x450
[Thu Feb  9 07:17:07 2023]  cpuidle_enter+0x2e/0x40
[Thu Feb  9 07:17:07 2023]  call_cpuidle+0x23/0x40
[Thu Feb  9 07:17:07 2023]  do_idle+0x1dd/0x270
[Thu Feb  9 07:17:07 2023]  cpu_startup_entry+0x20/0x30
[Thu Feb  9 07:17:07 2023]  start_secondary+0x167/0x1c0
[Thu Feb  9 07:17:07 2023]  secondary_startup_64+0xa4/0xb0
[Thu Feb  9 07:17:07 2023] ---[ end trace ca7e91fa58c34325 ]---
[Thu Feb  9 07:17:07 2023] ixgbe 0000:06:00.1 eno50: initiating reset due to tx
timeout
[Thu Feb  9 07:17:07 2023] ixgbe 0000:06:00.1 eno50: Reset adapter
[Thu Feb  9 07:17:07 2023] ixgbe 0000:06:00.1 eno50: TXDCTL.ENABLE for one or
more queues not cleared within the polling period
[Thu Feb  9 07:17:08 2023] br-vlan: port 1(eno50) entered disabled state
[Thu Feb  9 07:17:08 2023] br-vxlan: port 1(eno50.29) entered disabled state
[Thu Feb  9 07:17:08 2023] br-mgmt: port 1(eno50.51) entered disabled state
[Thu Feb  9 07:17:08 2023] ixgbe 0000:06:00.1 eno50: NIC Link is Up 10 Gbps,
Flow Control: RX/TX
[Thu Feb  9 07:17:08 2023] br-vlan: port 1(eno50) entered blocking state
[Thu Feb  9 07:17:08 2023] br-vlan: port 1(eno50) entered forwarding state
[Thu Feb  9 07:17:08 2023] br-vxlan: port 1(eno50.29) entered blocking state
[Thu Feb  9 07:17:08 2023] br-vxlan: port 1(eno50.29) entered forwarding state
[Thu Feb  9 07:17:08 2023] br-mgmt: port 1(eno50.51) entered blocking state
[Thu Feb  9 07:17:08 2023] br-mgmt: port 1(eno50.51) entered forwarding state
[Thu Feb  9 07:17:09 2023] ixgbe 0000:06:00.1 eno50: NIC Link is Down
[Thu Feb  9 07:17:10 2023] br-vlan: port 1(eno50) entered disabled state
[Thu Feb  9 07:17:10 2023] br-vxlan: port 1(eno50.29) entered disabled state
[Thu Feb  9 07:17:10 2023] br-mgmt: port 1(eno50.51) entered disabled state
[Thu Feb  9 07:17:10 2023] ixgbe 0000:06:00.1 eno50: NIC Link is Up 10 Gbps,
Flow Control: RX/TX
[Thu Feb  9 07:17:10 2023] br-vlan: port 1(eno50) entered blocking state
[Thu Feb  9 07:17:10 2023] br-vlan: port 1(eno50) entered forwarding state
[Thu Feb  9 07:17:10 2023] br-vxlan: port 1(eno50.29) entered blocking state
[Thu Feb  9 07:17:10 2023] br-vxlan: port 1(eno50.29) entered forwarding state
[Thu Feb  9 07:17:10 2023] br-mgmt: port 1(eno50.51) entered blocking state
[Thu Feb  9 07:17:10 2023] br-mgmt: port 1(eno50.51) entered forwarding state
[Thu Feb  9 07:27:34 2023] ixgbe 0000:06:00.1 eno50: initiating reset due to tx
timeout
[Thu Feb  9 07:27:34 2023] ixgbe 0000:06:00.1 eno50: Reset adapter
[Thu Feb  9 07:27:35 2023] br-vlan: port 1(eno50) entered disabled state
[Thu Feb  9 07:27:35 2023] br-vxlan: port 1(eno50.29) entered disabled state
[Thu Feb  9 07:27:35 2023] br-mgmt: port 1(eno50.51) entered disabled state
[Thu Feb  9 07:27:35 2023] ixgbe 0000:06:00.1 eno50: NIC Link is Up 10 Gbps,
Flow Control: RX/TX
[Thu Feb  9 07:27:35 2023] br-vlan: port 1(eno50) entered blocking state
[Thu Feb  9 07:27:35 2023] br-vlan: port 1(eno50) entered forwarding state
[Thu Feb  9 07:27:35 2023] br-vxlan: port 1(eno50.29) entered blocking state
[Thu Feb  9 07:27:35 2023] br-vxlan: port 1(eno50.29) entered forwarding state
[Thu Feb  9 07:27:35 2023] br-mgmt: port 1(eno50.51) entered blocking state
[Thu Feb  9 07:27:35 2023] br-mgmt: port 1(eno50.51) entered forwarding state
[Thu Feb  9 07:27:36 2023] ixgbe 0000:06:00.1 eno50: NIC Link is Down
[Thu Feb  9 07:27:37 2023] br-vlan: port 1(eno50) entered disabled state
[Thu Feb  9 07:27:37 2023] br-vxlan: port 1(eno50.29) entered disabled state
[Thu Feb  9 07:27:37 2023] br-mgmt: port 1(eno50.51) entered disabled state
[Thu Feb  9 07:27:37 2023] ixgbe 0000:06:00.1 eno50: NIC Link is Up 10 Gbps,
Flow Control: RX/TX
[Thu Feb  9 07:27:37 2023] br-vlan: port 1(eno50) entered blocking state
[Thu Feb  9 07:27:37 2023] br-vlan: port 1(eno50) entered forwarding state
[Thu Feb  9 07:27:37 2023] br-vxlan: port 1(eno50.29) entered blocking state
[Thu Feb  9 07:27:37 2023] br-vxlan: port 1(eno50.29) entered forwarding state
[Thu Feb  9 07:27:37 2023] br-mgmt: port 1(eno50.51) entered blocking state
[Thu Feb  9 07:27:37 2023] br-mgmt: port 1(eno50.51) entered forwarding state
[Thu Feb  9 07:38:14 2023] ixgbe 0000:06:00.1 eno50: initiating reset due to tx
timeout
[Thu Feb  9 07:38:14 2023] ixgbe 0000:06:00.1 eno50: Reset adapter

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

Powered by blists - more mailing lists