Message-ID: <4ac55e2c-54a9-4fab-b0c5-2a928faef33e@amd.com>
Date: Tue, 15 Jul 2025 23:01:23 -0700
From: "Koralahalli Channabasappa, Smita"
 <Smita.KoralahalliChannabasappa@....com>
To: Alison Schofield <alison.schofield@...el.com>
Cc: linux-cxl@...r.kernel.org, linux-kernel@...r.kernel.org,
 nvdimm@...ts.linux.dev, linux-fsdevel@...r.kernel.org,
 linux-pm@...r.kernel.org, Davidlohr Bueso <dave@...olabs.net>,
 Jonathan Cameron <jonathan.cameron@...wei.com>,
 Dave Jiang <dave.jiang@...el.com>, Vishal Verma <vishal.l.verma@...el.com>,
 Ira Weiny <ira.weiny@...el.com>, Dan Williams <dan.j.williams@...el.com>,
 Matthew Wilcox <willy@...radead.org>, Jan Kara <jack@...e.cz>,
 "Rafael J . Wysocki" <rafael@...nel.org>, Len Brown <len.brown@...el.com>,
 Pavel Machek <pavel@...nel.org>, Li Ming <ming.li@...omail.com>,
 Jeff Johnson <jeff.johnson@....qualcomm.com>,
 Ying Huang <huang.ying.caritas@...il.com>,
 Yao Xingtao <yaoxt.fnst@...itsu.com>, Peter Zijlstra <peterz@...radead.org>,
 Greg KH <gregkh@...uxfoundation.org>,
 Nathan Fontenot <nathan.fontenot@....com>,
 Terry Bowman <terry.bowman@....com>, Robert Richter <rrichter@....com>,
 Benjamin Cheatham <benjamin.cheatham@....com>,
 PradeepVineshReddy Kodamati <PradeepVineshReddy.Kodamati@....com>,
 Zhijian Li <lizhijian@...itsu.com>
Subject: Re: [PATCH v5 0/7] Add managed SOFT RESERVE resource handling

Hi Alison,

On 7/15/2025 2:07 PM, Alison Schofield wrote:
> On Tue, Jul 15, 2025 at 06:04:00PM +0000, Smita Koralahalli wrote:
>> This series introduces the ability to manage SOFT RESERVED iomem
>> resources, enabling the CXL driver to remove any portions that
>> intersect with created CXL regions.
> 
> Hi Smita,
> 
> This set applied cleanly to todays cxl-next but fails like appended
> before region probe.
> 
> BTW - there were sparse warnings in the build that look related:
>    CHECK   drivers/dax/hmem/hmem_notify.c
> drivers/dax/hmem/hmem_notify.c:10:6: warning: context imbalance in 'hmem_register_fallback_handler' - wrong count at exit
> drivers/dax/hmem/hmem_notify.c:24:9: warning: context imbalance in 'hmem_fallback_register_device' - wrong count at exit

Thanks for pointing this bug. I failed to release the spinlock before 
calling hmem_register_device(), which internally calls 
platform_device_add() and can sleep. The following fix addresses that 
bug. I’ll incorporate this into v6:

diff --git a/drivers/dax/hmem/hmem_notify.c b/drivers/dax/hmem/hmem_notify.c
index 6c276c5bd51d..8f411f3fe7bd 100644
--- a/drivers/dax/hmem/hmem_notify.c
+++ b/drivers/dax/hmem/hmem_notify.c
@@ -18,8 +18,9 @@ void hmem_fallback_register_device(int target_nid, 
const struct resource *res)
  {
         walk_hmem_fn hmem_fn;

-       guard(spinlock)(&hmem_notify_lock);
+       spin_lock(&hmem_notify_lock);
         hmem_fn = hmem_fallback_fn;
+       spin_unlock(&hmem_notify_lock);

         if (hmem_fn)
                 hmem_fn(target_nid, res);
--

As for the log:
[   53.652454] cxl_acpi:cxl_softreserv_mem_work_fn:888: Timeout waiting 
for cxl_mem probing

I’m still analyzing that. Here's what was my thought process so far.

- This occurs when cxl_acpi_probe() runs significantly earlier than 
cxl_mem_probe(), so CXL region creation (which happens in 
cxl_port_endpoint_probe()) may or may not have completed by the time 
trimming is attempted.

- Both cxl_acpi and cxl_mem have MODULE_SOFTDEPs on cxl_port. This does 
guarantee load order when all components are built as modules. So even 
if the timeout occurs and cxl_mem_probe() hasn’t run within the wait 
window, MODULE_SOFTDEP ensures that cxl_port is loaded before both 
cxl_acpi and cxl_mem in modular configurations. As a result, region 
creation is eventually guaranteed, and wait_for_device_probe() will 
succeed once the relevant probes complete.

- However, when both CONFIG_CXL_PORT=y and CONFIG_CXL_ACPI=y, there's no 
guarantee of probe ordering. In such cases, cxl_acpi_probe() may finish 
before cxl_port_probe() even begins, which can cause 
wait_for_device_probe() to return prematurely and trigger the timeout.

- In my local setup, I observed that a 30-second timeout was generally 
sufficient to catch this race, allowing cxl_port_probe() to load while 
cxl_acpi_probe() is still active. Since we cannot mix built-in and 
modular components (i.e., have cxl_acpi=y and cxl_port=m), the timeout 
serves as a best-effort mechanism. After the timeout, 
wait_for_device_probe() ensures cxl_port_probe() has completed before 
trimming proceeds, making the logic good enough to most boot-time races.

One possible improvement I’m considering is to schedule a 
delayed_workqueue() from cxl_acpi_probe(). This deferred work could wait 
slightly longer for cxl_mem_probe() to complete (which itself softdeps 
on cxl_port) before initiating the soft reserve trimming.

That said, I'm still evaluating better options to more robustly 
coordinate probe ordering between cxl_acpi, cxl_port, cxl_mem and 
cxl_region and looking for suggestions here.

Thanks
Smita

> 
> 
> This isn't all the logs, I trimmed. Let me know if you need more or
> other info to reproduce.
> 
> [   53.652454] cxl_acpi:cxl_softreserv_mem_work_fn:888: Timeout waiting for cxl_mem probing
> [   53.653293] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:321
> [   53.653513] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1875, name: kworker/46:1
> [   53.653540] preempt_count: 1, expected: 0
> [   53.653554] RCU nest depth: 0, expected: 0
> [   53.653568] 3 locks held by kworker/46:1/1875:
> [   53.653569]  #0: ff37d78240041548 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x578/0x630
> [   53.653583]  #1: ff6b0385dedf3e38 (cxl_sr_work){+.+.}-{0:0}, at: process_one_work+0x1bd/0x630
> [   53.653589]  #2: ffffffffb33476d8 (hmem_notify_lock){+.+.}-{3:3}, at: hmem_fallback_register_device+0x23/0x60
> [   53.653598] Preemption disabled at:
> [   53.653599] [<ffffffffb1e23993>] hmem_fallback_register_device+0x23/0x60
> [   53.653640] CPU: 46 UID: 0 PID: 1875 Comm: kworker/46:1 Not tainted 6.16.0CXL-NEXT-ALISON-SR-V5+ #5 PREEMPT(voluntary)
> [   53.653643] Workqueue: events cxl_softreserv_mem_work_fn [cxl_acpi]
> [   53.653648] Call Trace:
> [   53.653649]  <TASK>
> [   53.653652]  dump_stack_lvl+0xa8/0xd0
> [   53.653658]  dump_stack+0x14/0x20
> [   53.653659]  __might_resched+0x1ae/0x2d0
> [   53.653666]  __might_sleep+0x48/0x70
> [   53.653668]  __kmalloc_node_track_caller_noprof+0x349/0x510
> [   53.653674]  ? __devm_add_action+0x3d/0x160
> [   53.653685]  ? __pfx_devm_action_release+0x10/0x10
> [   53.653688]  __devres_alloc_node+0x4a/0x90
> [   53.653689]  ? __devres_alloc_node+0x4a/0x90
> [   53.653691]  ? __pfx_release_memregion+0x10/0x10 [dax_hmem]
> [   53.653693]  __devm_add_action+0x3d/0x160
> [   53.653696]  hmem_register_device+0xea/0x230 [dax_hmem]
> [   53.653700]  hmem_fallback_register_device+0x37/0x60
> [   53.653703]  cxl_softreserv_mem_register+0x24/0x30 [cxl_core]
> [   53.653739]  walk_iomem_res_desc+0x55/0xb0
> [   53.653744]  ? __pfx_cxl_softreserv_mem_register+0x10/0x10 [cxl_core]
> [   53.653755]  cxl_region_softreserv_update+0x46/0x50 [cxl_core]
> [   53.653761]  cxl_softreserv_mem_work_fn+0x4a/0x110 [cxl_acpi]
> [   53.653763]  ? __pfx_autoremove_wake_function+0x10/0x10
> [   53.653768]  process_one_work+0x1fa/0x630
> [   53.653774]  worker_thread+0x1b2/0x360
> [   53.653777]  kthread+0x128/0x250
> [   53.653781]  ? __pfx_worker_thread+0x10/0x10
> [   53.653784]  ? __pfx_kthread+0x10/0x10
> [   53.653786]  ret_from_fork+0x139/0x1e0
> [   53.653790]  ? __pfx_kthread+0x10/0x10
> [   53.653792]  ret_from_fork_asm+0x1a/0x30
> [   53.653801]  </TASK>
> 
> [   53.654193] =============================
> [   53.654203] [ BUG: Invalid wait context ]
> [   53.654451] 6.16.0CXL-NEXT-ALISON-SR-V5+ #5 Tainted: G        W
> [   53.654623] -----------------------------
> [   53.654785] kworker/46:1/1875 is trying to lock:
> [   53.654946] ff37d7824096d588 (&root->kernfs_rwsem){++++}-{4:4}, at: kernfs_add_one+0x34/0x390
> [   53.655115] other info that might help us debug this:
> [   53.655273] context-{5:5}
> [   53.655428] 3 locks held by kworker/46:1/1875:
> [   53.655579]  #0: ff37d78240041548 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x578/0x630
> [   53.655739]  #1: ff6b0385dedf3e38 (cxl_sr_work){+.+.}-{0:0}, at: process_one_work+0x1bd/0x630
> [   53.655900]  #2: ffffffffb33476d8 (hmem_notify_lock){+.+.}-{3:3}, at: hmem_fallback_register_device+0x23/0x60
> [   53.656062] stack backtrace:
> [   53.656224] CPU: 46 UID: 0 PID: 1875 Comm: kworker/46:1 Tainted: G        W           6.16.0CXL-NEXT-ALISON-SR-V5+ #5 PREEMPT(voluntary)
> [   53.656227] Tainted: [W]=WARN
> [   53.656228] Workqueue: events cxl_softreserv_mem_work_fn [cxl_acpi]
> [   53.656232] Call Trace:
> [   53.656232]  <TASK>
> [   53.656234]  dump_stack_lvl+0x85/0xd0
> [   53.656238]  dump_stack+0x14/0x20
> [   53.656239]  __lock_acquire+0xaf4/0x2200
> [   53.656246]  lock_acquire+0xd8/0x300
> [   53.656248]  ? kernfs_add_one+0x34/0x390
> [   53.656252]  ? __might_resched+0x208/0x2d0
> [   53.656257]  down_write+0x44/0xe0
> [   53.656262]  ? kernfs_add_one+0x34/0x390
> [   53.656263]  kernfs_add_one+0x34/0x390
> [   53.656265]  kernfs_create_dir_ns+0x5a/0xa0
> [   53.656268]  sysfs_create_dir_ns+0x74/0xd0
> [   53.656270]  kobject_add_internal+0xb1/0x2f0
> [   53.656273]  kobject_add+0x7d/0xf0
> [   53.656275]  ? get_device_parent+0x28/0x1e0
> [   53.656280]  ? __pfx_klist_children_get+0x10/0x10
> [   53.656282]  device_add+0x124/0x8b0
> [   53.656285]  ? dev_set_name+0x56/0x70
> [   53.656287]  platform_device_add+0x102/0x260
> [   53.656289]  hmem_register_device+0x160/0x230 [dax_hmem]
> [   53.656291]  hmem_fallback_register_device+0x37/0x60
> [   53.656294]  cxl_softreserv_mem_register+0x24/0x30 [cxl_core]
> [   53.656323]  walk_iomem_res_desc+0x55/0xb0
> [   53.656326]  ? __pfx_cxl_softreserv_mem_register+0x10/0x10 [cxl_core]
> [   53.656335]  cxl_region_softreserv_update+0x46/0x50 [cxl_core]
> [   53.656342]  cxl_softreserv_mem_work_fn+0x4a/0x110 [cxl_acpi]
> [   53.656343]  ? __pfx_autoremove_wake_function+0x10/0x10
> [   53.656346]  process_one_work+0x1fa/0x630
> [   53.656350]  worker_thread+0x1b2/0x360
> [   53.656352]  kthread+0x128/0x250
> [   53.656354]  ? __pfx_worker_thread+0x10/0x10
> [   53.656356]  ? __pfx_kthread+0x10/0x10
> [   53.656357]  ret_from_fork+0x139/0x1e0
> [   53.656360]  ? __pfx_kthread+0x10/0x10
> [   53.656361]  ret_from_fork_asm+0x1a/0x30
> [   53.656366]  </TASK>
> [   53.662274] BUG: scheduling while atomic: kworker/46:1/1875/0x00000002
> [   53.663552]  schedule+0x4a/0x160
> [   53.663553]  schedule_timeout+0x10a/0x120
> [   53.663555]  ? debug_smp_processor_id+0x1b/0x30
> [   53.663556]  ? trace_hardirqs_on+0x5f/0xd0
> [   53.663558]  __wait_for_common+0xb9/0x1c0
> [   53.663559]  ? __pfx_schedule_timeout+0x10/0x10
> [   53.663561]  wait_for_completion+0x28/0x30
> [   53.663562]  __synchronize_srcu+0xbf/0x180
> [   53.663566]  ? __pfx_wakeme_after_rcu+0x10/0x10
> [   53.663571]  ? i2c_repstart+0x30/0x80
> [   53.663576]  synchronize_srcu+0x46/0x120
> [   53.663577]  kill_dax+0x47/0x70
> [   53.663580]  __devm_create_dev_dax+0x112/0x470
> [   53.663582]  devm_create_dev_dax+0x26/0x50
> [   53.663584]  dax_hmem_probe+0x87/0xd0 [dax_hmem]
> [   53.663585]  platform_probe+0x61/0xd0
> [   53.663589]  really_probe+0xe2/0x390
> [   53.663591]  ? __pfx___device_attach_driver+0x10/0x10
> [   53.663593]  __driver_probe_device+0x7e/0x160
> [   53.663594]  driver_probe_device+0x23/0xa0
> [   53.663596]  __device_attach_driver+0x92/0x120
> [   53.663597]  bus_for_each_drv+0x8c/0xf0
> [   53.663599]  __device_attach+0xc2/0x1f0
> [   53.663601]  device_initial_probe+0x17/0x20
> [   53.663603]  bus_probe_device+0xa8/0xb0
> [   53.663604]  device_add+0x687/0x8b0
> [   53.663607]  ? dev_set_name+0x56/0x70
> [   53.663609]  platform_device_add+0x102/0x260
> [   53.663610]  hmem_register_device+0x160/0x230 [dax_hmem]
> [   53.663612]  hmem_fallback_register_device+0x37/0x60
> [   53.663614]  cxl_softreserv_mem_register+0x24/0x30 [cxl_core]
> [   53.663637]  walk_iomem_res_desc+0x55/0xb0
> [   53.663640]  ? __pfx_cxl_softreserv_mem_register+0x10/0x10 [cxl_core]
> [   53.663647]  cxl_region_softreserv_update+0x46/0x50 [cxl_core]
> [   53.663654]  cxl_softreserv_mem_work_fn+0x4a/0x110 [cxl_acpi]
> [   53.663655]  ? __pfx_autoremove_wake_function+0x10/0x10
> [   53.663658]  process_one_work+0x1fa/0x630
> [   53.663662]  worker_thread+0x1b2/0x360
> [   53.663664]  kthread+0x128/0x250
> [   53.663666]  ? __pfx_worker_thread+0x10/0x10
> [   53.663668]  ? __pfx_kthread+0x10/0x10
> [   53.663670]  ret_from_fork+0x139/0x1e0
> [   53.663672]  ? __pfx_kthread+0x10/0x10
> [   53.663673]  ret_from_fork_asm+0x1a/0x30
> [   53.663677]  </TASK>
> [   53.700107] BUG: scheduling while atomic: kworker/46:1/1875/0x00000002
> [   53.700264] INFO: lockdep is turned off.
> [   53.701315] Preemption disabled at:
> [   53.701316] [<ffffffffb1e23993>] hmem_fallback_register_device+0x23/0x60
> [   53.701631] CPU: 46 UID: 0 PID: 1875 Comm: kworker/46:1 Tainted: G        W           6.16.0CXL-NEXT-ALISON-SR-V5+ #5 PREEMPT(voluntary)
> [   53.701633] Tainted: [W]=WARN
> [   53.701635] Workqueue: events cxl_softreserv_mem_work_fn [cxl_acpi]
> [   53.701638] Call Trace:
> [   53.701638]  <TASK>
> [   53.701640]  dump_stack_lvl+0xa8/0xd0
> [   53.701644]  dump_stack+0x14/0x20
> [   53.701645]  __schedule_bug+0xa2/0xd0
> [   53.701649]  __schedule+0xe6f/0x10d0
> [   53.701652]  ? debug_smp_processor_id+0x1b/0x30
> [   53.701655]  ? lock_release+0x1e6/0x2b0
> [   53.701658]  ? trace_hardirqs_on+0x5f/0xd0
> [   53.701661]  schedule+0x4a/0x160
> [   53.701662]  schedule_timeout+0x10a/0x120
> [   53.701664]  ? debug_smp_processor_id+0x1b/0x30
> [   53.701666]  ? trace_hardirqs_on+0x5f/0xd0
> [   53.701667]  __wait_for_common+0xb9/0x1c0
> [   53.701668]  ? __pfx_schedule_timeout+0x10/0x10
> [   53.701670]  wait_for_completion+0x28/0x30
> [   53.701671]  __synchronize_srcu+0xbf/0x180
> [   53.701677]  ? __pfx_wakeme_after_rcu+0x10/0x10
> [   53.701682]  ? i2c_repstart+0x30/0x80
> [   53.701685]  synchronize_srcu+0x46/0x120
> [   53.701687]  kill_dax+0x47/0x70
> [   53.701689]  __devm_create_dev_dax+0x112/0x470
> [   53.701691]  devm_create_dev_dax+0x26/0x50
> [   53.701693]  dax_hmem_probe+0x87/0xd0 [dax_hmem]
> [   53.701695]  platform_probe+0x61/0xd0
> [   53.701698]  really_probe+0xe2/0x390
> [   53.701700]  ? __pfx___device_attach_driver+0x10/0x10
> [   53.701701]  __driver_probe_device+0x7e/0x160
> [   53.701703]  driver_probe_device+0x23/0xa0
> [   53.701704]  __device_attach_driver+0x92/0x120
> [   53.701706]  bus_for_each_drv+0x8c/0xf0
> [   53.701708]  __device_attach+0xc2/0x1f0
> [   53.701710]  device_initial_probe+0x17/0x20
> [   53.701711]  bus_probe_device+0xa8/0xb0
> [   53.701712]  device_add+0x687/0x8b0
> [   53.701715]  ? dev_set_name+0x56/0x70
> [   53.701717]  platform_device_add+0x102/0x260
> [   53.701718]  hmem_register_device+0x160/0x230 [dax_hmem]
> [   53.701720]  hmem_fallback_register_device+0x37/0x60
> [   53.701722]  cxl_softreserv_mem_register+0x24/0x30 [cxl_core]
> [   53.701734]  walk_iomem_res_desc+0x55/0xb0
> [   53.701738]  ? __pfx_cxl_softreserv_mem_register+0x10/0x10 [cxl_core]
> [   53.701745]  cxl_region_softreserv_update+0x46/0x50 [cxl_core]
> [   53.701751]  cxl_softreserv_mem_work_fn+0x4a/0x110 [cxl_acpi]
> [   53.701752]  ? __pfx_autoremove_wake_function+0x10/0x10
> [   53.701756]  process_one_work+0x1fa/0x630
> [   53.701760]  worker_thread+0x1b2/0x360
> [   53.701762]  kthread+0x128/0x250
> [   53.701765]  ? __pfx_worker_thread+0x10/0x10
> [   53.701766]  ? __pfx_kthread+0x10/0x10
> [   53.701768]  ret_from_fork+0x139/0x1e0
> [   53.701771]  ? __pfx_kthread+0x10/0x10
> [   53.701772]  ret_from_fork_asm+0x1a/0x30
> [   53.701777]  </TASK>
> 

Powered by blists - more mailing lists