Message-ID: <ff528ebb-2c09-4b03-a641-4a306b31ff62@notapiano>
Date: Sat, 24 Aug 2024 14:45:03 -0400
From: Nícolas F. R. A. Prado <nfraprado@...labora.com>
To: "Rafael J. Wysocki" <rjw@...ysocki.net>
Cc: Linux PM <linux-pm@...r.kernel.org>,
	LKML <linux-kernel@...r.kernel.org>,
	Daniel Lezcano <daniel.lezcano@...aro.org>,
	Lukasz Luba <lukasz.luba@....com>, Zhang Rui <rui.zhang@...el.com>,
	regressions@...ts.linux.dev, kernelci@...ts.linux.dev,
	kernel@...labora.com
Subject: Re: [PATCH v3 00/14] thermal: Rework binding cooling devices to trip
 points

On Mon, Aug 19, 2024 at 05:49:07PM +0200, Rafael J. Wysocki wrote:
> Hi Everyone,
> 
> This is one more update of
> 
> https://lore.kernel.org/linux-pm/3134863.CbtlEUcBR6@rjwysocki.net/#r
> 
> the cover letter of which was sent separately by mistake:
> 
> https://lore.kernel.org/linux-pm/CAJZ5v0jo5vh2uD5t4GqBnN0qukMBG_ty33PB=NiEqigqxzBcsw@mail.gmail.com/
> 
> and it has been updated once already:
> 
> https://lore.kernel.org/linux-pm/114901234.nniJfEyVGO@rjwysocki.net/
> 
> Relative to the v2 above it drops 3 patches, one because it was broken ([04/17
> in the v2), and two more that would need to be rebased significantly, either
> because of dropping the other broken patch or because of the recent Bang-bang
> governor fixes:
> 
> https://lore.kernel.org/linux-pm/1903691.tdWV9SEqCh@rjwysocki.net/
> 
> The remaining 14 patches, 2 of which have been slightly rebased and the rest
> is mostly unchanged (except for some very minor subject and changelog fixes),
> is not expected to be controversial and are targeting 6.12, on top of the
> current linux-next material.
> 
> The original motivation for this series quoted below has not changed:
> 
>  The code for binding cooling devices to trip points (and unbinding them from
>  trip point) is one of the murkiest pieces of the thermal subsystem.  It is
>  convoluted, bloated with unnecessary code doing questionable things, and it
>  works backwards.
> 
>  The idea is to bind cooling devices to trip points in accordance with some
>  information known to the thermal zone owner (thermal driver).  This information
>  is not known to the thermal core when the thermal zone is registered, so the
>  driver needs to be involved, but instead of just asking the driver whether
>  or not the given cooling device should be bound to a given trip point, the
>  thermal core expects the driver to carry out all of the binding process
>  including calling functions specifically provided by the core for this
>  purpose which is cumbersome and counter-intuitive.
> 
>  Because the driver has no information regarding the representation of the trip
>  points at the core level, it is forced to walk them (and it has to avoid some
>  locking traps while doing this), or it needs to make questionable assumptions
>  regarding the ordering of the trips in the core.  There are drivers doing both
>  these things.
> 
> The first 5 patches in the series are preliminary.
> 
> Patch [06/14] introduces a new .should_bind() callback for thermal zones and
> patches [07,09-12/14] modifies drivers to use it instead of the .bind() and
> .unbind() callbacks which allows them to be simplified quite a bit.
> 
> The other patches [08,13-14/14] get rid of code that becomes unused after the
> previous changes and do some cleanups on top of that.
> 
> The entire series along with 2 patches on top of it (that were present in the
> v2 of this set of patches) is available in the thermal-core-testing git branch:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git/log/?h=thermal-core-testing
> 
> (note that this branch is going to be rebased shortly on top of 6.11-rc4
> and the thermal control material in linux-next).
> 
> Thanks!

Hi,

KernelCI has identified a boot regression originating from this series. I've
verified that reverting the series fixes the issue.

Affected platforms:
* mt8195-cherry-tomato-r2
* mt8192-asurada-spherion-r0
* mt8183-kukui-jacuzzi-juniper-sku16
* mt8186-corsola-steelix-sku131072
* sc7180-trogdor-kingoftown
* sc7180-trogdor-lazor-limozeen

Relevant log from mt8195-cherry-tomato-r2 (with additional debug configs
enabled):

[   11.326726] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:1578
[   11.335294] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 165, name: udevd
[   11.342944] preempt_count: 1, expected: 0
[   11.346943] RCU nest depth: 0, expected: 0
[   11.351028] 4 locks held by udevd/165:
[   11.354766]  #0: ffff4dc8825db0f8 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x130/0x4a8
[   11.363207]  #1: ffffc208f386c3c8 (thermal_list_lock){+.+.}-{3:3}, at: thermal_zone_device_register_with_trips+0x85c/0xcd8
[   11.374248]  #2: ffff4dc7dc3586f0 (&tz->lock){+.+.}-{3:3}, at: thermal_zone_cdev_binding.part.0+0x98/0x280
[   11.383896]  #3: ffffc208f39b7b78 (devtree_lock){....}-{2:2}, at: of_get_next_child+0x2c/0xc4
[   11.392418] irq event stamp: 173740
[   11.395895] hardirqs last  enabled at (173739): [<ffffc208ecde804c>] _raw_spin_unlock_irqrestore+0x84/0x90
[   11.405537] hardirqs last disabled at (173740): [<ffffc208ecde6f7c>] _raw_spin_lock_irqsave+0xe0/0xf4
[   11.414742] softirqs last  enabled at (172404): [<ffffc208e978bb20>] handle_softirqs+0x534/0x874
[   11.423517] softirqs last disabled at (172393): [<ffffc208e961097c>] __do_softirq+0x14/0x20
[   11.431857] CPU: 5 UID: 0 PID: 165 Comm: udevd Not tainted 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628
[   11.441670] Hardware name: Acer Tomato (rev2) board (DT)
[   11.446970] Call trace:
[   11.449407]  dump_backtrace+0x98/0xf0
[   11.453059]  show_stack+0x18/0x24
[   11.456364]  dump_stack_lvl+0x90/0xd0
[   11.460018]  dump_stack+0x1c/0x28
[   11.463322]  __might_resched+0x358/0x570
[   11.467234]  __might_sleep+0xa4/0x16c
[   11.470885]  down_write+0x8c/0x21c
[   11.474277]  kernfs_remove+0x64/0x98
[   11.477844]  sysfs_remove_dir+0xa8/0xe8
[   11.481669]  __kobject_del+0xb0/0x27c
[   11.485321]  kobject_release+0xfc/0x134
[   11.489146]  kobject_put+0xb0/0x130
[   11.492624]  of_node_put+0x18/0x28
[   11.496016]  of_get_next_child+0x64/0xc4
[   11.499929]  thermal_of_should_bind+0x154/0x390
[   11.504449]  thermal_zone_cdev_binding.part.0+0x174/0x280
[   11.509836]  thermal_zone_device_register_with_trips+0x914/0xcd8
[   11.515831]  thermal_of_zone_register+0x284/0x464
[   11.520523]  devm_thermal_of_zone_register+0x80/0xf4
[   11.525476]  lvts_domain_init+0x500/0x760 [lvts_thermal]
[   11.530785]  lvts_probe+0x1b4/0x3ac [lvts_thermal]
[   11.535565]  platform_probe+0xc4/0x214
[   11.539303]  really_probe+0x188/0x5d0
[   11.542954]  __driver_probe_device+0x160/0x2e8
[   11.547386]  driver_probe_device+0x5c/0x298
[   11.551558]  __driver_attach+0x13c/0x4a8
[   11.555470]  bus_for_each_dev+0xf8/0x180
[   11.559383]  driver_attach+0x3c/0x58
[   11.562947]  bus_add_driver+0x1c4/0x458
[   11.566772]  driver_register+0xf4/0x3c0
[   11.570598]  __platform_driver_register+0x60/0x88
[   11.575291]  lvts_driver_init+0x20/0x1000 [lvts_thermal]
[   11.580593]  do_one_initcall+0xcc/0x284
[   11.584418]  do_init_module+0x278/0x740
[   11.588244]  load_module+0xed8/0x1434
[   11.591897]  init_module_from_file+0xdc/0x1fc
[   11.596243]  idempotent_init_module+0x2bc/0x604
[   11.600762]  __arm64_sys_finit_module+0xac/0x100
[   11.605368]  invoke_syscall+0x6c/0x258
[   11.609107]  el0_svc_common.constprop.0+0xac/0x230
[   11.613886]  do_el0_svc+0x40/0x58
[   11.617190]  el0_svc+0x48/0xb8
[   11.620234]  el0t_64_sync_handler+0x100/0x12c
[   11.624580]  el0t_64_sync+0x190/0x194
[   11.628233]
[   11.629713] =============================
[   11.633708] [ BUG: Invalid wait context ]
[   11.637705] 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628 Tainted: G        W
[   11.645953] -----------------------------
[   11.649950] udevd/165 is trying to lock:
[   11.653859] ffff4dc880881148 (&root->kernfs_rwsem){++++}-{3:3}, at: kernfs_remove+0x64/0x98
[   11.662200] other info that might help us debug this:
[   11.667238] context-{4:4}
[   11.669846] 4 locks held by udevd/165:
[   11.673582]  #0: ffff4dc8825db0f8 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x130/0x4a8
[   11.682009]  #1: ffffc208f386c3c8 (thermal_list_lock){+.+.}-{3:3}, at: thermal_zone_device_register_with_trips+0x85c/0xcd8
[   11.693041]  #2: ffff4dc7dc3586f0 (&tz->lock){+.+.}-{3:3}, at: thermal_zone_cdev_binding.part.0+0x98/0x280
[   11.702684]  #3: ffffc208f39b7b78 (devtree_lock){....}-{2:2}, at: of_get_next_child+0x2c/0xc4
[   11.711199] stack backtrace:
[   11.714067] CPU: 5 UID: 0 PID: 165 Comm: udevd Tainted: G        W          6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628
[   11.725355] Tainted: [W]=WARN
[   11.728310] Hardware name: Acer Tomato (rev2) board (DT)
[   11.733608] Call trace:
[   11.736041]  dump_backtrace+0x98/0xf0
[   11.739692]  show_stack+0x18/0x24
[   11.742994]  dump_stack_lvl+0x90/0xd0
[   11.746645]  dump_stack+0x1c/0x28
[   11.749948]  __lock_acquire+0x10f8/0x2710
[   11.753948]  lock_acquire.part.0+0x218/0x518
[   11.758206]  lock_acquire+0x90/0xb4
[   11.761683]  down_write+0xb4/0x21c
[   11.765074]  kernfs_remove+0x64/0x98
[   11.768637]  sysfs_remove_dir+0xa8/0xe8
[   11.772461]  __kobject_del+0xb0/0x27c
[   11.776111]  kobject_release+0xfc/0x134
[   11.779935]  kobject_put+0xb0/0x130
[   11.783413]  of_node_put+0x18/0x28
[   11.786803]  of_get_next_child+0x64/0xc4
[   11.790714]  thermal_of_should_bind+0x154/0x390
[   11.795231]  thermal_zone_cdev_binding.part.0+0x174/0x280
[   11.800617]  thermal_zone_device_register_with_trips+0x914/0xcd8
[   11.806609]  thermal_of_zone_register+0x284/0x464
[   11.811301]  devm_thermal_of_zone_register+0x80/0xf4
[   11.816253]  lvts_domain_init+0x500/0x760 [lvts_thermal]
[   11.821553]  lvts_probe+0x1b4/0x3ac [lvts_thermal]
[   11.826332]  platform_probe+0xc4/0x214
[   11.830069]  really_probe+0x188/0x5d0
[   11.833719]  __driver_probe_device+0x160/0x2e8
[   11.838150]  driver_probe_device+0x5c/0x298
[   11.842320]  __driver_attach+0x13c/0x4a8
[   11.846230]  bus_for_each_dev+0xf8/0x180
[   11.850141]  driver_attach+0x3c/0x58
[   11.853704]  bus_add_driver+0x1c4/0x458
[   11.857529]  driver_register+0xf4/0x3c0
[   11.861352]  __platform_driver_register+0x60/0x88
[   11.866043]  lvts_driver_init+0x20/0x1000 [lvts_thermal]
[   11.871342]  do_one_initcall+0xcc/0x284
[   11.875166]  do_init_module+0x278/0x740
[   11.878990]  load_module+0xed8/0x1434
[   11.882641]  init_module_from_file+0xdc/0x1fc
[   11.886986]  idempotent_init_module+0x2bc/0x604
[   11.891504]  __arm64_sys_finit_module+0xac/0x100
[   11.896109]  invoke_syscall+0x6c/0x258
[   11.899846]  el0_svc_common.constprop.0+0xac/0x230
[   11.904624]  do_el0_svc+0x40/0x58
[   11.907927]  el0_svc+0x48/0xb8
[   11.910969]  el0t_64_sync_handler+0x100/0x12c
[   11.915314]  el0t_64_sync+0x190/0x194
[   36.261761] watchdog: Watchdog detected hard LOCKUP on cpu 0
[   36.267414] Modules linked in: cbmem cros_ec_lid_angle cros_ec_sensors(+) cros_ec_sensors_core pcie_mediatek_gen3 sbs_battery cros_kbd_led_backlight industrialio_triggered_buffer kfifo_buf cros_ec_chardev cros_ec_rpmsg lvts_thermal(+) cros_ec_typec leds_cros_ec mtk_svs snd_sof_mt8195 mtk_adsp_common snd_sof_xtensa_dsp snd_sof_of mt6577_auxadc snd_soc_mt8195_afe snd_sof snd_sof_utils mtk_scp mtk_rpmsg mtk_scp_ipi pwm_bl mtk_wdt coreboot_table backlight mt8195_mt6359 ramoops reed_solomon
[   36.310414] irq event stamp: 197347
[   36.313890] hardirqs last  enabled at (197347): [<ffffc208ecdc994c>] exit_to_kernel_mode+0x38/0x118
[   36.322923] hardirqs last disabled at (197346): [<ffffc208ecdcac44>] el1_interrupt+0x24/0x54
[   36.331347] softirqs last  enabled at (197268): [<ffffc208e978bb20>] handle_softirqs+0x534/0x874
[   36.340117] softirqs last disabled at (197263): [<ffffc208e961097c>] __do_softirq+0x14/0x20

Full log at http://0x0.st/XyID.txt

Let me know if you need any more information.

#regzbot introduced: next-20240821..next-20240822
#regzbot title: Hang during boot in sysfs_remove_dir() called by thermal_of_zone_register()

Thanks,
Nícolas

Powered by blists - more mailing lists