| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <aac2caec-0909-40ef-978e-97fdb34e1937@nvidia.com>
Date: Thu, 20 Nov 2025 17:03:36 +1100
From: Balbir Singh <balbirs@...dia.com>
To: Matthew Brost <matthew.brost@...el.com>
Cc: Andrew Morton <akpm@...ux-foundation.org>, linux-kernel@...r.kernel.org,
dri-devel@...ts.freedesktop.org, linux-mm@...ck.org,
David Hildenbrand <david@...hat.com>, Zi Yan <ziy@...dia.com>,
Joshua Hahn <joshua.hahnjy@...il.com>, Rakie Kim <rakie.kim@...com>,
Byungchul Park <byungchul@...com>, Gregory Price <gourry@...rry.net>,
Ying Huang <ying.huang@...ux.alibaba.com>,
Alistair Popple <apopple@...dia.com>, Oscar Salvador <osalvador@...e.de>,
Lorenzo Stoakes <lorenzo.stoakes@...cle.com>,
Baolin Wang <baolin.wang@...ux.alibaba.com>,
"Liam R. Howlett" <Liam.Howlett@...cle.com>, Nico Pache <npache@...hat.com>,
Ryan Roberts <ryan.roberts@....com>, Dev Jain <dev.jain@....com>,
Barry Song <baohua@...nel.org>, Lyude Paul <lyude@...hat.com>,
Danilo Krummrich <dakr@...nel.org>, David Airlie <airlied@...il.com>,
Simona Vetter <simona@...ll.ch>, Ralph Campbell <rcampbell@...dia.com>,
Mika Penttilä <mpenttil@...hat.com>,
Francois Dugast <francois.dugast@...el.com>
Subject: Re: [v7 00/16] mm: support device-private THP
On 11/20/25 16:53, Matthew Brost wrote:
> On Thu, Nov 20, 2025 at 02:58:58PM +1100, Balbir Singh wrote:
>> On 11/20/25 14:15, Matthew Brost wrote:
>>> On Thu, Nov 20, 2025 at 01:59:09PM +1100, Balbir Singh wrote:
>>>> On 11/20/25 13:50, Balbir Singh wrote:
>>>>> On 11/20/25 13:40, Matthew Brost wrote:
>>>>>> On Wed, Nov 12, 2025 at 10:52:43AM +1100, Balbir Singh wrote:
>>>>>>> On 11/12/25 10:43, Andrew Morton wrote:
>>>>>>>> On Thu, 9 Oct 2025 03:33:33 -0700 Matthew Brost <matthew.brost@...el.com> wrote:
>>>>>>>>
>>>>>>>>>>>> This patch series introduces support for Transparent Huge Page
>>>>>>>>>>>> (THP) migration in zone device-private memory. The implementation enables
>>>>>>>>>>>> efficient migration of large folios between system memory and
>>>>>>>>>>>> device-private memory
>>>>>>>>>>>
>>>>>>>>>>> Lots of chatter for the v6 series, but none for v7. I hope that's a
>>>>>>>>>>> good sign.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I hope so too, I've tried to address the comments in v6.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Circling back to this series, we will itegrate and test this version.
>>>>>>>>
>>>>>>>> How'd it go?
>>>>>>>>
>>>>>>
>>>>>> My apologies for the delay—I got distracted by other tasks in Xe (my
>>>>>> driver) and was out for a bit. Unfortunately, this series breaks
>>>>>> something in the existing core MM code for the Xe SVM implementation. I
>>>>>> have an extensive test case that hammers on SVM, which fully passes
>>>>>> prior to applying this series, but fails randomly with the series
>>>>>> applied (to drm-tip-rc6) due to the below kernel lockup.
>>>>>>
>>>>>> I've tried to trace where the migration PTE gets installed but not
>>>>>> removed or isolate a test case which causes this failure but no luck so
>>>>>> far. I'll keep digging as I have time.
>>>>>>
>>>>>> Beyond that, if I enable Xe SVM + THP, it seems to mostly work (though
>>>>>> the same issue as above eventually occurs), but I do need two additional
>>>>>> core MM patches—one is new code required for Xe, and the other could be
>>>>>> considered a bug fix. Those patches can included when Xe merges SVM THP
>>>>>> support but we need at least not break Xe SVM before this series merges.
>>>>>>
>>>>>> Stack trace:
>>>>>>
>>>>>> INFO: task kworker/u65:2:1642 blocked for more than 30
>>>>>> seconds.
>>>>>> [ 212.624286] Tainted: G S W 6.18.0-rc6-xe+ #1719
>>>>>> [ 212.630561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 212.638285] task:kworker/u65:2 state:D stack:0 pid:1642
>>>>>> tgid:1642 ppid:2 task_flags:0x4208060 flags:0x00080000
>>>>>> [ 212.638288] Workqueue: xe_page_fault_work_queue
>>>>>> xe_pagefault_queue_work [xe]
>>>>>> [ 212.638323] Call Trace:
>>>>>> [ 212.638324] <TASK>
>>>>>> [ 212.638325] __schedule+0x4b0/0x990
>>>>>> [ 212.638330] schedule+0x22/0xd0
>>>>>> [ 212.638331] io_schedule+0x41/0x60
>>>>>> [ 212.638333] migration_entry_wait_on_locked+0x1d8/0x2d0
>>>>>> [ 212.638336] ? __pfx_wake_page_function+0x10/0x10
>>>>>> [ 212.638339] migration_entry_wait+0xd2/0xe0
>>>>>> [ 212.638341] hmm_vma_walk_pmd+0x7c9/0x8d0
>>>>>> [ 212.638343] walk_pgd_range+0x51d/0xa40
>>>>>> [ 212.638345] __walk_page_range+0x75/0x1e0
>>>>>> [ 212.638347] walk_page_range_mm+0x138/0x1f0
>>>>>> [ 212.638349] hmm_range_fault+0x59/0xa0
>>>>>> [ 212.638351] drm_gpusvm_get_pages+0x194/0x7b0 [drm_gpusvm_helper]
>>>>>> [ 212.638354] drm_gpusvm_range_get_pages+0x2d/0x40 [drm_gpusvm_helper]
>>>>>> [ 212.638355] __xe_svm_handle_pagefault+0x259/0x900 [xe]
>>>>>> [ 212.638375] ? update_load_avg+0x7f/0x6c0
>>>>>> [ 212.638377] ? update_curr+0x13d/0x170
>>>>>> [ 212.638379] xe_svm_handle_pagefault+0x37/0x90 [xe]
>>>>>> [ 212.638396] xe_pagefault_queue_work+0x2da/0x3c0 [xe]
>>>>>> [ 212.638420] process_one_work+0x16e/0x2e0
>>>>>> [ 212.638422] worker_thread+0x284/0x410
>>>>>> [ 212.638423] ? __pfx_worker_thread+0x10/0x10
>>>>>> [ 212.638425] kthread+0xec/0x210
>>>>>> [ 212.638427] ? __pfx_kthread+0x10/0x10
>>>>>> [ 212.638428] ? __pfx_kthread+0x10/0x10
>>>>>> [ 212.638430] ret_from_fork+0xbd/0x100
>>>>>> [ 212.638433] ? __pfx_kthread+0x10/0x10
>>>>>> [ 212.638434] ret_from_fork_asm+0x1a/0x30
>>>>>> [ 212.638436] </TASK>
>>>>>>
>>>>>
>>>>> Hi, Matt
>>>>>
>>>>> Thanks for the report, two questions
>>>>>
>>>>> 1. Are you using mm/mm-unstable, we've got some fixes in there (including fixes to remove_migration_pmd())
>>>
>>> remove_migration_pmd - This is a PTE migration entry.
>>>
>>
>> I don't have your symbols, I thought we were hitting, the following condition in the walk
>>
>> if (thp_migration_supported() && pmd_is_migration_entry(pmd)) {
>>
>> But sounds like you are not, PMD/THP has not been enabled in this case
>>
>
> No, migration_entry_wait rather than pmd_migration_entry_wait.
>
>>
>>>>> - Generally a left behind migration entry is a symptom of a failed migration that did not clean up
>>>>> after itself.
>>>
>>> I'm on drm-tip as I generally need the latest version of my driver
>>> because of the speed we move at.
>>>
>>> Yes, I agree it looks like somehow a migration PTE is not getting
>>> properly removed.
>>>
>>> I'm happy to cherry pick any patches that you think might be helpful
>>> into my tree.
>>>
>>
>> Could you try the mm/mm-new tree with the current xe driver?
>>
>
> Unfortunately, this is a tough one. We land a lot of patches in Xe/DRM,
> so bringing the driver up to date with an MM branch is difficult, and
> I’m not an expert at merging branches. It would be nice if, in the DRM
> flow, we could merge patches from outside our subsystem into a
> bleeding-edge kernel for the things we typically care about—but we’d
> need a maintainer to sign up for that.
>
>> In general, w.r.t failure, I would check for the following
>>
>> 1. Are the dst_pfns in migrate_vma_pages() setup correctly by the device driver?
>> 2. Any failures in folio_migrate_mapping()?
>> 3. In migrate_vma_finalize() check to see if remove_migration_ptes() failed
>>
>> If (3) fails that will explain the left over migration entries
>>
>
> Good tips, but think I got it via biscet.
>
> Offending patch is:
>
> 'mm/migrate_device: handle partially mapped folios during collection'
>
> The failing test case involves some remap-related issue. It’s a
> parameterized test, so I honestly couldn’t tell you exactly what it’s
> doing beyond the fact that it seems nonsensical but stresses remap. I
> thought commit '66d81853fa3d selftests/mm/hmm-tests: partial unmap,
> mremap and anon_write tests' would catch this, but it looks like I need
> to make the remap HMM test cases a bit more robust—similar to my
> driver-side tests. I can take an action item to follow up on this.
>
> Good news, I can tell you how to fix this...
>
> In 'mm/migrate_device: handle partially mapped folios during collection':
>
> 109 +#if 0
> 110 + folio = page ? page_folio(page) : NULL;
> 111 + if (folio && folio_test_large(folio)) {
> 112 + int ret;
> 113 +
> 114 + pte_unmap_unlock(ptep, ptl);
> 115 + ret = migrate_vma_split_folio(folio,
> 116 + migrate->fault_page);
> 117 +
> 118 + if (ret) {
> 119 + ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl);
> 120 + goto next;
> 121 + }
> 122 +
> 123 + addr = start;
> 124 + goto again;
> 125 + }
> 126 +#endif
>
> You can probably just delete this and use my patch below, but if you
> want to try fixing it with a quick look: if migrate_vma_split_folio
> fails, you probably need to collect a hole. On success, you likely want
> to continue executing the remainder of the loop. I can try playing with
> this tomorrow, but it’s late here.
>
> I had privately sent you a version of this patch as a fix for Xe, and
> this one seems to work:
>
> [PATCH] mm/migrate: Split THP found in middle of PMD during page collection
>
> The migrate layer is not coded to handle a THP found in the middle of a
> PMD. This can occur if a user manipulates mappings with mremap(). If a
> THP is found mid-PMD during page collection, split it.
>
> Cc: Balbir Singh <balbirs@...dia.com>
> Signed-off-by: Matthew Brost <matthew.brost@...el.com>
> ---
> mm/migrate_device.c | 37 +++++++++++++++++++++++++++++++++++--
> 1 file changed, 35 insertions(+), 2 deletions(-)
>
> diff --git a/mm/migrate_device.c b/mm/migrate_device.c
> index abd9f6850db6..9ffc025bad50 100644
> --- a/mm/migrate_device.c
> +++ b/mm/migrate_device.c
> @@ -65,6 +65,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
> struct vm_area_struct *vma = walk->vma;
> struct mm_struct *mm = vma->vm_mm;
> unsigned long addr = start, unmapped = 0;
> + struct folio *split_folio = NULL;
> spinlock_t *ptl;
> pte_t *ptep;
>
> @@ -107,10 +108,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
> }
> }
>
> - ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl);
> + ptep = pte_offset_map_lock(mm, pmdp, start, &ptl);
> if (!ptep)
> goto again;
> arch_enter_lazy_mmu_mode();
> + ptep += (addr - start) / PAGE_SIZE;
>
> for (; addr < end; addr += PAGE_SIZE, ptep++) {
> struct dev_pagemap *pgmap;
> @@ -209,6 +211,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
> bool anon_exclusive;
> pte_t swp_pte;
>
> + if (folio_order(folio)) {
> + split_folio = folio;
> + goto split;
> + }
> +
> flush_cache_page(vma, addr, pte_pfn(pte));
> anon_exclusive = folio_test_anon(folio) &&
> PageAnonExclusive(page);
> @@ -287,8 +294,34 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
> if (unmapped)
> flush_tlb_range(walk->vma, start, end);
>
> +split:
> arch_leave_lazy_mmu_mode();
> - pte_unmap_unlock(ptep - 1, ptl);
> + pte_unmap_unlock(ptep - 1 + !!split_folio, ptl);
> +
> + /*
> + * XXX: No clean way to support higher-order folios that don't match PMD
> + * boundaries for now — split them instead. Once mTHP support lands, add
> + * proper support for this case.
> + *
> + * The test, which exposed this as problematic, remapped (memremap) a
> + * large folio to an unaligned address, resulting in the folio being
> + * found in the middle of the PTEs. The requested number of pages was
> + * less than the folio size. Likely to be handled gracefully by upper
> + * layers eventually, but not yet.
> + */
> + if (split_folio) {
> + int ret;
> +
> + ret = split_folio(split_folio);
> + if (fault_folio != split_folio)
> + folio_unlock(split_folio);
> + folio_put(split_folio);
> + if (ret)
> + return migrate_vma_collect_skip(addr, end, walk);
> +
> + split_folio = NULL;
> + goto again;
> + }
>
> return 0;
> }
>
> If I apply the #if 0 change along with my patch above (plus one core
> MM patch needed for Xe that adds a support function), Xe SVM fully
> passes our test cases with both THP enabled and disabled.
>
Excellent work! Since you found this, do you mind sending the fix to Andrew as a fixup
to the original patch. Since I don't have the test case, I have no way of validating the
change or any change on top of it would continue to work
FYI: The original code does something similar, I might be missing the
migrate_vma_collect_skip() bits.
Thanks!
Balbir
Powered by blists - more mailing lists