netdev - [BUG] possible issue with ixgbe

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAA85sZsn2oG4wUHPYOPTPW8j6jbHe=_0UiLntZmXjvaf0Cu9PA@mail.gmail.com>
Date:   Mon, 12 Apr 2021 20:41:47 +0200
From:   Ian Kumlien <ian.kumlien@...il.com>
To:     Jesse Brandeburg <jesse.brandeburg@...el.com>,
        Tony Nguyen <anthony.l.nguyen@...el.com>
Cc:     Linux Kernel Network Developers <netdev@...r.kernel.org>
Subject: [BUG] possible issue with ixgbe

Hi,

I've encountered this a few times now, been moving kernels since
there's been things fixed that
looked related... but...

[959642.297143] ------------[ cut here ]------------
[959642.297149] NETDEV WATCHDOG: eno1 (ixgbe): transmit queue 2 timed out
[959642.297189] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:442
dev_watchdog+0x21f/0x230
[959642.297199] Modules linked in: chaoskey
[959642.297205] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.11.11 #272
[959642.297209] Hardware name: Supermicro Super
Server/A2SDi-12C-HLN4F, BIOS 1.2 11/05/2019
[959642.297211] RIP: 0010:dev_watchdog+0x21f/0x230
[959642.297216] Code: 27 1a fd ff eb ab 4c 89 ef c6 05 c2 27 1a 01 01
e8 46 12 fa ff 44 89 e1 4c 89 ee 48 c7 c7 90 37 22 a5 48 89 c2 e8 a8
f2 32 00 <0f> 0b eb 8c 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 f9
48 8d
[959642.297220] RSP: 0018:ffffb1d20015cec8 EFLAGS: 00010282
[959642.297223] RAX: 0000000000000000 RBX: ffff8dda45c34ec0 RCX:
ffff8dddafad78d8
[959642.297225] RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI:
ffff8dddafad78d0
[959642.297228] RBP: ffff8dda45c28480 R08: 0000000000000000 R09:
ffffb1d20015cd00
[959642.297230] R10: ffffb1d20015ccf8 R11: ffffffffa553e8c8 R12:
0000000000000002
[959642.297232] R13: ffff8dda45c28000 R14: 0000000000000001 R15:
ffff8dddafadbb40
[959642.297234] FS:  0000000000000000(0000) GS:ffff8dddafac0000(0000)
knlGS:0000000000000000
[959642.297237] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[959642.297240] CR2: 00007f0fdc76d5a8 CR3: 000000013debe000 CR4:
00000000003526e0
[959642.297242] Call Trace:
[959642.297246]  <IRQ>
[959642.297250]  ? pfifo_fast_init+0x100/0x100
[959642.297254]  call_timer_fn+0x24/0xf0
[959642.297259]  __run_timers.part.0+0x1b8/0x220
[959642.297263]  ? recalibrate_cpu_khz+0x10/0x10
[959642.297268]  ? ktime_get+0x33/0x90
[959642.297272]  ? lapic_timer_set_periodic+0x20/0x20
[959642.297276]  ? clockevents_program_event+0x88/0xe0
[959642.297280]  run_timer_softirq+0x21/0x50
[959642.297284]  __do_softirq+0xba/0x264
[959642.297290]  asm_call_irq_on_stack+0x12/0x20
[959642.297294]  </IRQ>
[959642.297296]  do_softirq_own_stack+0x32/0x40
[959642.297300]  irq_exit_rcu+0x83/0xb0
[959642.297304]  sysvec_apic_timer_interrupt+0x36/0x80
[959642.297309]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[959642.297314] RIP: 0010:cpuidle_enter_state+0xcd/0x340
[959642.297320] Code: 49 89 c5 0f 1f 44 00 00 31 ff e8 8e 96 5b ff 45
84 ff 74 12 9c 58 f6 c4 02 0f 85 56 02 00 00 31 ff e8 b7 3b 60 ff fb
45 85 f6 <0f> 88 e9 00 00 00 49 63 c6 4c 2b 2c 24 48 8d 14 40 48 8d 0c
90 48
[959642.297322] RSP: 0018:ffffb1d20008fea8 EFLAGS: 00000202
[959642.297326] RAX: ffff8dddafae9a40 RBX: ffffd1d1ffaeb500 RCX:
000000000000001f
[959642.297328] RDX: 0000000000000000 RSI: 0000000040000000 RDI:
0000000000000000
[959642.297330] RBP: 0000000000000002 R08: 000368ca223b0fae R09:
0000000000000008
[959642.297332] R10: 00000000000003dc R11: 00000000000003da R12:
ffffffffa55ecf40
[959642.297334] R13: 000368ca223b0fae R14: 0000000000000002 R15:
0000000000000000
[959642.297338]  ? cpuidle_enter_state+0xb2/0x340
[959642.297342]  cpuidle_enter+0x24/0x40
[959642.297346]  do_idle+0x1ba/0x210
[959642.297351]  cpu_startup_entry+0x14/0x20
[959642.297354]  secondary_startup_64_no_verify+0xb0/0xbb
[959642.297359] ---[ end trace 9b29a940f734a412 ]---
[959642.297365] ixgbe 0000:06:00.0 eno1: initiating reset due to tx timeout
[959647.929975] ixgbe 0000:06:00.0 eno1: initiating reset due to tx timeout
[959649.130161] ixgbe 0000:06:00.0 eno1: Reset adapter
[959649.135186] ixgbe 0000:06:00.0 eno1: NIC Link is Down
[959652.566949] ixgbe 0000:06:00.0 eno1: NIC Link is Up 1 Gbps, Flow
Control: RX/TX
[959669.944390] ixgbe 0000:06:00.0 eno1: initiating reset due to tx timeout
[959670.096838] ixgbe 0000:06:00.0 eno1: Reset adapter
[959670.101836] ixgbe 0000:06:00.0 eno1: NIC Link is Down
[959673.405387] ixgbe 0000:06:00.0 eno1: NIC Link is Up 1 Gbps, Flow
Control: RX/TX
[959856.307490] ixgbe 0000:06:00.0 eno1: initiating reset due to tx timeout
[959861.536015] ixgbe 0000:06:00.0 eno1: Reset adapter
[959861.541031] ixgbe 0000:06:00.0 eno1: NIC Link is Down
[959865.387649] ixgbe 0000:06:00.0 eno1: NIC Link is Up 1 Gbps, Flow
Control: RX/TX
----

It always seems to happen under cpu load - I'm mad enough to run
gentoo on a Atom (Intel(R) Atom(TM) CPU C3858 @ 2.00GHz) machine ;)

When first triggered it takes a while - eventually any spike in cpu
load will crash the machine... (I suspect that the hw-watchdog reboots
it but there is no logs)

Any clues? I did run with flow offload in nftables - but I have since
disabled that as well...