linux-kernel - Re: modprobe mlx5_core on OCI bare-metal instance causes unrecoverable hang and I/O error

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAHTA-uZMZ6qQZf_n55gNaTjQQ0j8nXdt1Yi_+8+-YUNhxcrs_A@mail.gmail.com>
Date: Fri, 7 Feb 2025 13:24:24 -0600
From: Mitchell Augustin <mitchell.augustin@...onical.com>
To: Jason Gunthorpe <jgg@...pe.ca>
Cc: saeedm@...dia.com, leon@...nel.org, tariqt@...dia.com, 
	andrew+netdev@...n.ch, "David S. Miller" <davem@...emloft.net>, 
	Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>, 
	netdev@...r.kernel.org, linux-rdma@...r.kernel.org, 
	linux-kernel@...r.kernel.org, Talat Batheesh <talatb@...dia.com>, 
	Feras Daoud <ferasda@...dia.com>
Subject: Re: modprobe mlx5_core on OCI bare-metal instance causes
 unrecoverable hang and I/O error

*facepalm*

Thanks, I can't believe that wasn't my first thought as soon as I
learned these instances were using iSCSI. That's almost certainly what
is happening on this OCI instance, since the host adapter for its
iSCSI transport is a ConnectX card.

The fact that I was able to see similar behavior once on a machine
booted from a local disk (in the A100 test I mentioned) is still
confusing though. I'll update this thread if I can figure out a
reliable way to reproduce that behavior.


On Fri, Feb 7, 2025 at 1:01 PM Jason Gunthorpe <jgg@...pe.ca> wrote:
>
> On Fri, Feb 07, 2025 at 10:02:46AM -0600, Mitchell Augustin wrote:
> > > Is it using iscsi/srp/nfs/etc for any filesystems?
> >
> > Yes, dev sda is using iSCSI:
>
> If you remove the driver that is providing transport for your
> filesystem the system will hang like you showed.
>
> It can be done, but the process sequencing the load/unload has to be
> entirely contained to a tmpfs so it doesn't become blocked on IO that
> cannot complete.
>
> Jason



-- 
Mitchell Augustin
Software Engineer - Ubuntu Partner Engineering