linux-kernel - Re: [PATCH v2] iommu/vt-d: fix system hang on reboot -f

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <f77cc212-deb0-47a7-90fa-8d76bc891564@gmail.com>
Date: Wed, 26 Feb 2025 08:35:17 +0800
From: Ethan Zhao <etzhao1900@...il.com>
To: Jason Gunthorpe <jgg@...pe.ca>, Ethan Zhao <haifeng.zhao@...ux.intel.com>
Cc: Baolu Lu <baolu.lu@...ux.intel.com>, Yunhui Cui
 <cuiyunhui@...edance.com>, dwmw2@...radead.org, joro@...tes.org,
 will@...nel.org, robin.murphy@....com, iommu@...ts.linux.dev,
 linux-kernel@...r.kernel.org
Subject: Re: [PATCH v2] iommu/vt-d: fix system hang on reboot -f

On 2/25/2025 10:26 PM, Jason Gunthorpe wrote:
> On Tue, Feb 25, 2025 at 04:54:54PM +0800, Ethan Zhao wrote:
>>> On 2025/2/25 14:48, Yunhui Cui wrote:
>>>> We found that executing the command ./a.out &;reboot -f (where a.out
>>>> is a
>>>> program that only executes a while(1) infinite loop) can
>>>> probabilistically
>>>> cause the system to hang in the intel_iommu_shutdown() function,
>>>> rendering
>>>> it unresponsive. Through analysis, we identified that the factors
>>>> contributing to this issue are as follows:
>>>>
>>>> 1. The reboot -f command does not prompt the kernel to notify the
>>>> application layer to perform cleanup actions, allowing the
>>>> application to
>>>> continue running.
>>>>
>>>> 2. When the kernel reaches the intel_iommu_shutdown() function, only the
>>>> BSP (Bootstrap Processor) CPU is operational in the system.
>>>>
>>>> 3. During the execution of intel_iommu_shutdown(), the function
>>>> down_write
>>>> (&dmar_global_lock) causes the process to sleep and be scheduled out.
> Why does this happen? If the kernel has shutdown other CPUs then what
> thread is holding the other side of this lock and why?

The down_write() actually executes might_sleep()->might_resched()->__cond_resched()->

__schedule() first before acquiring the lock, thus there is change to got scheduled out.

The caller is scheduled out due to voluntary sleep or because its time slice is exhausted,

not because the lock is held by other processes here.

>>>> 4. At this point, though the processor's interrupt flag is not cleared,
>>>>    allowing interrupts to be accepted. However, only legacy devices
>>>> and NMI
>>>> (Non-Maskable Interrupt) interrupts could come in, as other interrupts
>>>> routing have already been disabled. If no legacy or NMI interrupts occur
>>>> at this stage, the scheduler will not be able to run.
>>>> 5. If the application got scheduled at this time is executing a
>>>> while(1)-
>>>> type loop, it will be unable to be preempted, leading to an infinite
>>>> loop
>>>> and causing the system to become unresponsive.
> If the schedular doesn't run how did we get from 4 -> 5?

We got from 4-->5 because caller thread's voluntary invocation of the scheduler.

>
> Maybe the issue is the shutdown handler here is running in the wrong
> time and it should not be running after the scheduler has been shut
> down.

Move thex86_platform.iommu_shutdown() before hpet_disable() ?

I didn't figure out why we need scheduler is active when we execute

iommu_shutdown(), or the reason to keep this pair of down_write()/

up_write(). could you help to shed light ?

Thanks,
Ethan

>
> I don't think removing the lock is a great idea without more
> explanation.
>
> Jason
>