netdev - Kernel panic triggered while removing mlx5

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <PH0PR11MB515991D1E1AB73AFB7DCBD03E6F52@PH0PR11MB5159.namprd11.prod.outlook.com>
Date: Fri, 24 May 2024 08:07:23 +0000
From: "Berger, Michal" <michal.berger@...el.com>
To: "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Kernel panic triggered while removing mlx5_core devices from the pci
 bus

Kernel: 6.7.0, 6.8.8 (fedora builds)
Devices: MT27710 Family [ConnectX-4 Lx] (0x1015), fw_ver: 14.23.1020
rdma-core: 44.0

We have a small test which performs a somewhat controlled hotplug of the net device on the pci bus (via sysfs). The affected device is part of the nvmf-rdma setup running in SPDK context (i.e. https://github.com/spdk/spdk/blob/master/test/nvmf/target/device_removal.sh)  Sometimes (it's not reproducible at each run unfortunately) when the device is removed, kernel hits
Oops - with our panic setup it's then followed by a kernel reboot, but if we allow the kernel to continue it eventually deadlocks itself. 

This happens across different systems using the same set of NICs. Example of these oops attached. 

Just to note, we previously had the same issue under older kernels (e.g. 6.1), all reported here https://bugzilla.kernel.org/show_bug.cgi?id=218288. Bump to 6.7.0 helped to reduce the frequency
of this issue but unfortunately it's still there. 

Any hints on how to tackle this issue would be appreciated.

Regards,
Michal
---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Spolka oswiadcza, ze posiada status duzego przedsiebiorcy w rozumieniu ustawy z dnia 8 marca 2013 r. o przeciwdzialaniu nadmiernym opoznieniom w transakcjach handlowych.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.

View attachment "6.7.0_6.8.8_panics.txt" of type "text/plain" (48149 bytes)