| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <5226cedc180a1126ac5cdb48ee9aa9ef8b594452.camel@nvidia.com> Date: Fri, 19 Apr 2024 15:09:32 +0000 From: Dragos Tatulea <dtatulea@...dia.com> To: Shay Drori <shayd@...dia.com>, "netdev@...r.kernel.org" <netdev@...r.kernel.org>, "oxana@...udflare.com" <oxana@...udflare.com> CC: "davem@...emloft.net" <davem@...emloft.net>, Rahul Rameshbabu <rrameshbabu@...dia.com>, "kernel-team@...udflare.com" <kernel-team@...udflare.com>, "kuba@...nel.org" <kuba@...nel.org>, "Anatoli.Chechelnickiy@...nterpipe.biz" <Anatoli.Chechelnickiy@...nterpipe.biz>, "edumazet@...gle.com" <edumazet@...gle.com>, Saeed Mahameed <saeedm@...dia.com>, "leon@...nel.org" <leon@...nel.org>, "pabeni@...hat.com" <pabeni@...hat.com> Subject: Re: mlx5 driver fails to detect NIC in 6.6.28 On Fri, 2024-04-19 at 14:06 +0000, Dragos Tatulea wrote: > On Fri, 2024-04-19 at 13:45 +0100, Oxana Kharitonova wrote: > > Hello, > > > > NIC stopped being detected in Linux 6.6.28. The problem was observed on > > two servers, after reverting kernel to 6.6.25 (our current stable version) > > everything returned to normal. > > > > We suspect commit "net/mlx5e: Do not produce metadata freelist entries in > > Tx port ts WQE xmit", but we haven't done bisect yet. > > > The issue appears only in 6.6.28. Bisectet down to commit "net/mlx5: Register > devlink first under devlink lock" which was added to 6.6.28. Note that latest > stable 6.8 doesn't have this issue. > > Tried quickly looking around and found that "net/mlx5: Restore mistakenly > dropped parts in register devlink flow" was missing. But it didn't help > unfortunately. > > Adding Shay as he probably has more context. > Was tipped by Shay that the missing commit from stable is 0553e753ea9e "net/mlx5: E-switch, store eswitch pointer before registering devlink_param". Tested on my side and it works. Oxana, would it be a tall ask to get this patch tested on your end as well before we ask for inclusion in 6.6.x stable? Thanks, Dragos > > > > The kernel log is below. > > > > root@...alhost:~# ifconfig > > lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 > > inet 127.0.0.1 netmask 255.0.0.0 > > inet6 ::1 prefixlen 128 scopeid 0x10<host> > > loop txqueuelen 1000 (Local Loopback) > > RX packets 80 bytes 6480 (6.3 KiB) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 80 bytes 6480 (6.3 KiB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > root@...alhost:~# lspci | grep Eth > > c1:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] > > c1:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] > > > > [ 23.519113] RIP: 0010:esw_port_metadata_get (drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c:4095 drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c:2442) mlx5_core > > [ 23.524293] usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd > > [ 23.528602] Code: eb 8e 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 53 48 89 d3 e8 f2 5d ea c9 48 8b 80 b0 09 00 00 <8b> 80 18 11 00 00 88 03 31 c0 80 23 01 5b e9 38 1f f3 c9 0f 1f 84 > > All code > > ======== > > 0: eb 8e jmp 0xffffffffffffff90 > > 2: 0f 1f 00 nopl (%rax) > > 5: 90 nop > > 6: 90 nop > > 7: 90 nop > > 8: 90 nop > > 9: 90 nop > > a: 90 nop > > b: 90 nop > > c: 90 nop > > d: 90 nop > > e: 90 nop > > f: 90 nop > > 10: 90 nop > > 11: 90 nop > > 12: 90 nop > > 13: 90 nop > > 14: 90 nop > > 15: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) > > 1a: 53 push %rbx > > 1b: 48 89 d3 mov %rdx,%rbx > > 1e: e8 f2 5d ea c9 call 0xffffffffc9ea5e15 > > 23: 48 8b 80 b0 09 00 00 mov 0x9b0(%rax),%rax > > 2a:* 8b 80 18 11 00 00 mov 0x1118(%rax),%eax <-- trapping instruction > > 30: 88 03 mov %al,(%rbx) > > 32: 31 c0 xor %eax,%eax > > 34: 80 23 01 andb $0x1,(%rbx) > > 37: 5b pop %rbx > > 38: e9 38 1f f3 c9 jmp 0xffffffffc9f31f75 > > 3d: 0f .byte 0xf > > 3e: 1f (bad) > > 3f: 84 .byte 0x84 > > > > Code starting with the faulting instruction > > =========================================== > > 0: 8b 80 18 11 00 00 mov 0x1118(%rax),%eax > > 6: 88 03 mov %al,(%rbx) > > 8: 31 c0 xor %eax,%eax > > a: 80 23 01 andb $0x1,(%rbx) > > d: 5b pop %rbx > > e: e9 38 1f f3 c9 jmp 0xffffffffc9f31f4b > > 13: 0f .byte 0xf > > 14: 1f (bad) > > 15: 84 .byte 0x84 > > [ 23.528604] RSP: 0018:ffffc9000dbbfba8 EFLAGS: 00010282 > > [ 23.530802] hub 1-1:1.0: 4 ports detected > > [ 23.537574] RAX: 0000000000000000 RBX: ffffc9000dbbfbfc RCX: 0000000000000028 > > [ 23.537576] RDX: ffffc9000dbbfbfc RSI: 0000000000000013 RDI: ffff88811ec38000 > > [ 23.537578] RBP: ffffffffc23fa560 R08: 0000000000000000 R09: 0000000000000000 > > [ 23.537580] R10: 0000000000036ea0 R11: 0000000000000dc0 R12: ffff889850104f00 > > [ 23.547568] usb 2-1: New USB device found, idVendor=05e3, idProduct=0620, bcdDevice=93.03 > > [ 23.564222] R13: ffff8881075ca840 R14: ffff88811ec38000 R15: 0000000000000000 > > [ 23.564224] FS: 0000000000000000(0000) GS:ffff88843fa00000(0000) knlGS:0000000000000000 > > [ 23.564226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 23.570143] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 > > [ 23.574839] CR2: 0000000000001118 CR3: 0000000c7af5c000 CR4: 0000000000350ef0 > > [ 23.574841] Call Trace: > > [ 23.574844] <TASK> > > [ 23.574846] ? __die (arch/x86/kernel/dumpstack.c:421 arch/x86/kernel/dumpstack.c:434) > > [ 23.590485] ? page_fault_oops (arch/x86/mm/fault.c:707) > > [ 23.590490] ? get_page_from_freelist (mm/page_alloc.c:1553 mm/page_alloc.c:3177) > > [ 23.606129] ? exc_page_fault (arch/x86/include/asm/irqflags.h:37 arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1504 arch/x86/mm/fault.c:1552) > > [ 23.655856] ? asm_exc_page_fault (arch/x86/include/asm/idtentry.h:570) > > [ 23.671501] ? esw_port_metadata_get (drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c:4095 drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c:2442) mlx5_core > > [ 23.790812] ? esw_port_metadata_get (drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c:4095 drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c:2442) mlx5_core > > [ 23.955180] devlink_nl_param_fill.constprop.0 (net/devlink/param.c:268) > > [ 23.961276] ? __alloc_skb (net/core/skbuff.c:651 (discriminator 1)) > > [ 23.974490] ? srso_return_thunk (arch/x86/lib/retpoline.S:217) > > [ 23.987013] ? __kmalloc_node_track_caller (mm/slab_common.c:1025 mm/slab_common.c:1046) > > [ 23.987018] ? srso_return_thunk (arch/x86/lib/retpoline.S:217) > > [ 23.997812] ? kmalloc_reserve (net/core/skbuff.c:584) > > [ 23.997816] ? srso_return_thunk (arch/x86/lib/retpoline.S:217) > > [ 24.007217] ? __alloc_skb (net/core/skbuff.c:666) > > [ 24.007222] devlink_param_notify.constprop.0 (net/devlink/param.c:354 net/devlink/param.c:330) > > [ 24.055512] devl_params_register (net/devlink/param.c:686 (discriminator 1)) > > [ 24.055516] esw_offloads_init (drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c:2482) mlx5_core > > [ 24.065009] mlx5_eswitch_init (drivers/net/ethernet/mellanox/mlx5/core/eswitch.c:1872) mlx5_core > > [ 24.165229] mlx5_init_one_devl_locked (drivers/net/ethernet/mellanox/mlx5/core/main.c:1022 drivers/net/ethernet/mellanox/mlx5/core/main.c:1447) mlx5_core > > [ 24.177007] probe_one (drivers/net/ethernet/mellanox/mlx5/core/main.c:1507 drivers/net/ethernet/mellanox/mlx5/core/main.c:1947) mlx5_core > > [ 24.187296] local_pci_probe (drivers/pci/pci-driver.c:325) > > [ 24.196698] work_for_cpu_fn (kernel/workqueue.c:5618 (discriminator 1)) > > [ 24.205988] process_one_work (kernel/workqueue.c:2632) > > [ 24.215612] worker_thread (kernel/workqueue.c:2694 (discriminator 2) kernel/workqueue.c:2781 (discriminator 2)) > > [ 24.224999] ? __pfx_worker_thread (kernel/workqueue.c:2727) > > [ 24.234912] kthread (kernel/kthread.c:388) > > [ 24.243647] ? __pfx_kthread (kernel/kthread.c:341) > > [ 24.252988] ret_from_fork (arch/x86/kernel/process.c:153) > > [ 24.262103] ? __pfx_kthread (kernel/kthread.c:341) > > [ 24.271347] ret_from_fork_asm (arch/x86/entry/entry_64.S:314) > > [ 24.280754] </TASK> > > > > >
Powered by blists - more mailing lists