lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Message-ID: <0a361ac2-c6bd-2b18-4841-b1b991f0635e@suse.com> Date: Thu, 15 Dec 2022 10:51:15 +0100 From: Petr Pavlu <petr.pavlu@...e.com> To: tariqt@...dia.com, yishaih@...dia.com, netdev@...r.kernel.org Cc: linux-kernel@...r.kernel.org Subject: Part of devices not initialized with mlx4 Hello, We have seen an issue when some of ConnectX-3 devices are not initialized when mlx4 drivers are a part of initrd. The basic scenario looks as follows: * A machine has multiple ConnectX-3 devices, they can be VF ones. The system uses an initrd driven by dracut+systemd. The initrd is built as no-hostonly (think of a VM image) and includes the mlx4 drivers. * The machine boots. The initrd invokes udevd to start inserting device drivers until the root disk is available. * The udev daemon inserts the mlx4_core driver, which asynchronously requests a load of mlx4_en. This is done by calling request_module_nowait() from mlx4_request_modules(). The kernel spawns a modprobe userspace task to handle this request. * The modprobe task finds the mlx4_en module and asks the kernel to load it. The module loader runs the init function of the module which starts iterating over mlx4_core devices and initializing their eth support. * The root disk becomes available in the meantime and the initrd logic starts the switch root process. * Systemd stops running services and then sends SIGTERM to "unmanaged" tasks on the system to terminate them too. This includes the modprobe task. * Initialization of mlx4_en is interrupted in the middle of its init function. The module remains inserted but only some eth devices are initialized and operational. The modprobe task uses the default SIGTERM handling and so this signal becomes fatal. Specifically, it causes the create_singlethread_workqueue() call in mlx4_en_add() to error out. The workqueue requires a rescuer thread and a wait on the new thread fails because a fatal signal is pending. As mentioned, this can result in only a part of all devices being initialized. It could also likely happen that the modprobe task fails in some other obscure way as it has its root switched under its hands. It is a task that is completely asynchronous from any systemd control. Has anyone else seen this issue before too? Note that some parts of the problem are not fully clear to me yet. In particular, systemd also sends SIGSTOP before and SIGCONT after the mentioned SIGTERM signal, which can actually in some cases prevent the kernel from treating SIGTERM immediately as a fatal signal. I'm waiting on some additional test machine to analyze this part more. One idea how to address this issue is to model the mlx4 drivers using an auxiliary bus, similar to how the same conversion was already done in mlx5. This leaves all module loads to udevd which better integrates with the systemd processing and a load of mlx4_en doesn't get interrupted. My incomplete patches implementing this idea are available at: https://github.com/petrpavlu/linux/commits/bsc1187236-wip-v1 The rework turned out to be not exactly straightforward and would need more effort. I realize mlx4 is only used for ConnectX-3 and older hardware. I wonder then if this kind of rework would be suitable and something to proceed with, or if some simpler idea how to address the described issue would be better and preferred. Thank you, Petr
Powered by blists - more mailing lists