linux-kernel - Re: hung tasks on shutdown in linux-next-202409{20,23,24,25}

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <f4547877-8aa2-45a0-b05d-624eb4e2d296@gmail.com>
Date: Mon, 30 Sep 2024 16:11:55 -0500
From: stuart hayes <stuart.w.hayes@...il.com>
To: Bert Karwatzki <spasswolf@....de>
Cc: Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
 linux-kernel@...r.kernel.org, linux-next@...r.kernel.org,
 Tejun Heo <tj@...nel.org>
Subject: Re: hung tasks on shutdown in linux-next-202409{20,23,24,25}

On 9/29/2024 5:53 AM, Bert Karwatzki wrote:
> Summary: The introduction of async reboot in commit 8064952c6504
> ("driver core: shut down devices asynchronously") leads to frequent hangs on
> shutdown even after commit 4f2c346e6216 ("driver core: fix async device shutdown hang")
> is introduced.
> 
> I did some further experimenting (and lots of reboots ...) and found out that
> the bug is preemption related, for me it only occurs when using CONFIG_PREEMPT=y
> or CONFIG_PREEMPT_RT=y. When using CONFIG_PREEMPT_NONE=y or
> CONFIG_PREEMPT_VOLUNTARY=y everything works fine.
> 
> Test results (linux-next-20240925):
> PREEMPT_NONE		20 reboots, no fail
> PREEMPT_VOLUNTARY	20 reboots, no fail
> PREEMPT			3 reboots, 4th reboot failed
> PREEMPT_RT		2 reboots, 3rd reboot failed
> 
> The behaviour can be improved by increasing the number of min_active items
> in the async workqueue:
> 

Thank you for continuing to look at this! That is interesting data.

I see from an earlier message that drm_atomic_helper_dirtyfb is holding a lock when
the hang occurs:

> T115;4 locks held by kworker/7:2/343:
> T115; #0: ffff91ea00050d48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x4a4/0x580
> T115; #1: ffffbaf182e07e58 ((work_completion)(&helper->damage_work)){+.+.}-{0:0}, at: process_one_work+0x1c7/0x580
> T115; #2: ffffbaf182e07d00 (crtc_ww_class_acquire){+.+.}-{0:0}, at: drm_atomic_helper_dirtyfb+0x47/0x280
> T115; #3: ffff91ea13b80528 (crtc_ww_class_mutex){+.+.}-{3:3}, at: modeset_lock+0xbf/0x1b0

Except for NVMe drives, the shutdown process with the async shutdown patches should be
the same as the shutdown process without the patch--that is, the devices should be shut
down one after the other, in the same order... the only difference is that the individual
device shutdowns are scheduled in a workqueue where they wait for the previous device
shutdown to finish, instead of being shut down one at a time in a loop in the systemd
task.  So I'm wondering if the async shutdown could somehow exposing some sort of race in
a display device driver's shutdown function.

A full CPU backtrace (which you could get from setting /proc/sys/kernel/hung_task_all_cpu_backtrace
before reproducing the error) would be extremely helpful if you have the inclination... :)