linux-kernel - Re: modprobe mlx5_core on OCI bare-metal instance causes unrecoverable hang and I/O error

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <Z6ZgAbyCsJzoqbZc@x130>
Date: Fri, 7 Feb 2025 11:33:21 -0800
From: Saeed Mahameed <saeedm@...dia.com>
To: Mitchell Augustin <mitchell.augustin@...onical.com>
Cc: Jason Gunthorpe <jgg@...pe.ca>, leon@...nel.org, tariqt@...dia.com,
	andrew+netdev@...n.ch, "David S. Miller" <davem@...emloft.net>,
	Eric Dumazet <edumazet@...gle.com>,
	Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
	netdev@...r.kernel.org, linux-rdma@...r.kernel.org,
	linux-kernel@...r.kernel.org, Talat Batheesh <talatb@...dia.com>,
	Feras Daoud <ferasda@...dia.com>
Subject: Re: modprobe mlx5_core on OCI bare-metal instance causes
 unrecoverable hang and I/O error

On 07 Feb 13:24, Mitchell Augustin wrote:
>*facepalm*
>
>Thanks, I can't believe that wasn't my first thought as soon as I
>learned these instances were using iSCSI. That's almost certainly what
>is happening on this OCI instance, since the host adapter for its
>iSCSI transport is a ConnectX card.
>
>The fact that I was able to see similar behavior once on a machine
>booted from a local disk (in the A100 test I mentioned) is still
>confusing though. I'll update this thread if I can figure out a
>reliable way to reproduce that behavior.
>

BTW I saw this happening in few instances of virtualized environments as
well, where the VM storage is network/RDMA packed by the host driver/network,
when the host driver restarts (which is a normal behavior for a PV setup), 
the VM also get storage related timeouts and soft lockups.  Graceful
shutdown needs to be handled inside of the network backed block devices
IMHO.

-Saeed.