| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <aR9Phl4tsMHHJJwI@lstrano-desk.jf.intel.com>
Date: Thu, 20 Nov 2025 09:27:34 -0800
From: Matthew Brost <matthew.brost@...el.com>
To: Balbir Singh <balbirs@...dia.com>
CC: Andrew Morton <akpm@...ux-foundation.org>, <linux-kernel@...r.kernel.org>,
<dri-devel@...ts.freedesktop.org>, <linux-mm@...ck.org>, David Hildenbrand
<david@...hat.com>, Zi Yan <ziy@...dia.com>, Joshua Hahn
<joshua.hahnjy@...il.com>, Rakie Kim <rakie.kim@...com>, Byungchul Park
<byungchul@...com>, Gregory Price <gourry@...rry.net>, Ying Huang
<ying.huang@...ux.alibaba.com>, Alistair Popple <apopple@...dia.com>, "Oscar
Salvador" <osalvador@...e.de>, Lorenzo Stoakes <lorenzo.stoakes@...cle.com>,
Baolin Wang <baolin.wang@...ux.alibaba.com>, "Liam R. Howlett"
<Liam.Howlett@...cle.com>, Nico Pache <npache@...hat.com>, Ryan Roberts
<ryan.roberts@....com>, Dev Jain <dev.jain@....com>, Barry Song
<baohua@...nel.org>, Lyude Paul <lyude@...hat.com>, Danilo Krummrich
<dakr@...nel.org>, David Airlie <airlied@...il.com>, Simona Vetter
<simona@...ll.ch>, Ralph Campbell <rcampbell@...dia.com>, Mika
Penttilä <mpenttil@...hat.com>, Francois Dugast
<francois.dugast@...el.com>
Subject: Re: [v7 00/16] mm: support device-private THP
On Thu, Nov 20, 2025 at 05:03:36PM +1100, Balbir Singh wrote:
> On 11/20/25 16:53, Matthew Brost wrote:
> > On Thu, Nov 20, 2025 at 02:58:58PM +1100, Balbir Singh wrote:
> >> On 11/20/25 14:15, Matthew Brost wrote:
> >>> On Thu, Nov 20, 2025 at 01:59:09PM +1100, Balbir Singh wrote:
> >>>> On 11/20/25 13:50, Balbir Singh wrote:
> >>>>> On 11/20/25 13:40, Matthew Brost wrote:
> >>>>>> On Wed, Nov 12, 2025 at 10:52:43AM +1100, Balbir Singh wrote:
> >>>>>>> On 11/12/25 10:43, Andrew Morton wrote:
> >>>>>>>> On Thu, 9 Oct 2025 03:33:33 -0700 Matthew Brost <matthew.brost@...el.com> wrote:
> >>>>>>>>
> >>>>>>>>>>>> This patch series introduces support for Transparent Huge Page
> >>>>>>>>>>>> (THP) migration in zone device-private memory. The implementation enables
> >>>>>>>>>>>> efficient migration of large folios between system memory and
> >>>>>>>>>>>> device-private memory
> >>>>>>>>>>>
> >>>>>>>>>>> Lots of chatter for the v6 series, but none for v7. I hope that's a
> >>>>>>>>>>> good sign.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I hope so too, I've tried to address the comments in v6.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Circling back to this series, we will itegrate and test this version.
> >>>>>>>>
> >>>>>>>> How'd it go?
> >>>>>>>>
> >>>>>>
> >>>>>> My apologies for the delay—I got distracted by other tasks in Xe (my
> >>>>>> driver) and was out for a bit. Unfortunately, this series breaks
> >>>>>> something in the existing core MM code for the Xe SVM implementation. I
> >>>>>> have an extensive test case that hammers on SVM, which fully passes
> >>>>>> prior to applying this series, but fails randomly with the series
> >>>>>> applied (to drm-tip-rc6) due to the below kernel lockup.
> >>>>>>
> >>>>>> I've tried to trace where the migration PTE gets installed but not
> >>>>>> removed or isolate a test case which causes this failure but no luck so
> >>>>>> far. I'll keep digging as I have time.
> >>>>>>
> >>>>>> Beyond that, if I enable Xe SVM + THP, it seems to mostly work (though
> >>>>>> the same issue as above eventually occurs), but I do need two additional
> >>>>>> core MM patches—one is new code required for Xe, and the other could be
> >>>>>> considered a bug fix. Those patches can included when Xe merges SVM THP
> >>>>>> support but we need at least not break Xe SVM before this series merges.
> >>>>>>
> >>>>>> Stack trace:
> >>>>>>
> >>>>>> INFO: task kworker/u65:2:1642 blocked for more than 30
> >>>>>> seconds.
> >>>>>> [ 212.624286] Tainted: G S W 6.18.0-rc6-xe+ #1719
> >>>>>> [ 212.630561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>>>>> disables this message.
> >>>>>> [ 212.638285] task:kworker/u65:2 state:D stack:0 pid:1642
> >>>>>> tgid:1642 ppid:2 task_flags:0x4208060 flags:0x00080000
> >>>>>> [ 212.638288] Workqueue: xe_page_fault_work_queue
> >>>>>> xe_pagefault_queue_work [xe]
> >>>>>> [ 212.638323] Call Trace:
> >>>>>> [ 212.638324] <TASK>
> >>>>>> [ 212.638325] __schedule+0x4b0/0x990
> >>>>>> [ 212.638330] schedule+0x22/0xd0
> >>>>>> [ 212.638331] io_schedule+0x41/0x60
> >>>>>> [ 212.638333] migration_entry_wait_on_locked+0x1d8/0x2d0
> >>>>>> [ 212.638336] ? __pfx_wake_page_function+0x10/0x10
> >>>>>> [ 212.638339] migration_entry_wait+0xd2/0xe0
> >>>>>> [ 212.638341] hmm_vma_walk_pmd+0x7c9/0x8d0
> >>>>>> [ 212.638343] walk_pgd_range+0x51d/0xa40
> >>>>>> [ 212.638345] __walk_page_range+0x75/0x1e0
> >>>>>> [ 212.638347] walk_page_range_mm+0x138/0x1f0
> >>>>>> [ 212.638349] hmm_range_fault+0x59/0xa0
> >>>>>> [ 212.638351] drm_gpusvm_get_pages+0x194/0x7b0 [drm_gpusvm_helper]
> >>>>>> [ 212.638354] drm_gpusvm_range_get_pages+0x2d/0x40 [drm_gpusvm_helper]
> >>>>>> [ 212.638355] __xe_svm_handle_pagefault+0x259/0x900 [xe]
> >>>>>> [ 212.638375] ? update_load_avg+0x7f/0x6c0
> >>>>>> [ 212.638377] ? update_curr+0x13d/0x170
> >>>>>> [ 212.638379] xe_svm_handle_pagefault+0x37/0x90 [xe]
> >>>>>> [ 212.638396] xe_pagefault_queue_work+0x2da/0x3c0 [xe]
> >>>>>> [ 212.638420] process_one_work+0x16e/0x2e0
> >>>>>> [ 212.638422] worker_thread+0x284/0x410
> >>>>>> [ 212.638423] ? __pfx_worker_thread+0x10/0x10
> >>>>>> [ 212.638425] kthread+0xec/0x210
> >>>>>> [ 212.638427] ? __pfx_kthread+0x10/0x10
> >>>>>> [ 212.638428] ? __pfx_kthread+0x10/0x10
> >>>>>> [ 212.638430] ret_from_fork+0xbd/0x100
> >>>>>> [ 212.638433] ? __pfx_kthread+0x10/0x10
> >>>>>> [ 212.638434] ret_from_fork_asm+0x1a/0x30
> >>>>>> [ 212.638436] </TASK>
> >>>>>>
> >>>>>
> >>>>> Hi, Matt
> >>>>>
> >>>>> Thanks for the report, two questions
> >>>>>
> >>>>> 1. Are you using mm/mm-unstable, we've got some fixes in there (including fixes to remove_migration_pmd())
> >>>
> >>> remove_migration_pmd - This is a PTE migration entry.
> >>>
> >>
> >> I don't have your symbols, I thought we were hitting, the following condition in the walk
> >>
> >> if (thp_migration_supported() && pmd_is_migration_entry(pmd)) {
> >>
> >> But sounds like you are not, PMD/THP has not been enabled in this case
> >>
> >
> > No, migration_entry_wait rather than pmd_migration_entry_wait.
> >
> >>
> >>>>> - Generally a left behind migration entry is a symptom of a failed migration that did not clean up
> >>>>> after itself.
> >>>
> >>> I'm on drm-tip as I generally need the latest version of my driver
> >>> because of the speed we move at.
> >>>
> >>> Yes, I agree it looks like somehow a migration PTE is not getting
> >>> properly removed.
> >>>
> >>> I'm happy to cherry pick any patches that you think might be helpful
> >>> into my tree.
> >>>
> >>
> >> Could you try the mm/mm-new tree with the current xe driver?
> >>
> >
> > Unfortunately, this is a tough one. We land a lot of patches in Xe/DRM,
> > so bringing the driver up to date with an MM branch is difficult, and
> > I’m not an expert at merging branches. It would be nice if, in the DRM
> > flow, we could merge patches from outside our subsystem into a
> > bleeding-edge kernel for the things we typically care about—but we’d
> > need a maintainer to sign up for that.
> >
> >> In general, w.r.t failure, I would check for the following
> >>
> >> 1. Are the dst_pfns in migrate_vma_pages() setup correctly by the device driver?
> >> 2. Any failures in folio_migrate_mapping()?
> >> 3. In migrate_vma_finalize() check to see if remove_migration_ptes() failed
> >>
> >> If (3) fails that will explain the left over migration entries
> >>
> >
> > Good tips, but think I got it via biscet.
> >
> > Offending patch is:
> >
> > 'mm/migrate_device: handle partially mapped folios during collection'
> >
> > The failing test case involves some remap-related issue. It’s a
> > parameterized test, so I honestly couldn’t tell you exactly what it’s
> > doing beyond the fact that it seems nonsensical but stresses remap. I
> > thought commit '66d81853fa3d selftests/mm/hmm-tests: partial unmap,
> > mremap and anon_write tests' would catch this, but it looks like I need
> > to make the remap HMM test cases a bit more robust—similar to my
> > driver-side tests. I can take an action item to follow up on this.
> >
> > Good news, I can tell you how to fix this...
> >
> > In 'mm/migrate_device: handle partially mapped folios during collection':
> >
> > 109 +#if 0
> > 110 + folio = page ? page_folio(page) : NULL;
> > 111 + if (folio && folio_test_large(folio)) {
> > 112 + int ret;
> > 113 +
> > 114 + pte_unmap_unlock(ptep, ptl);
> > 115 + ret = migrate_vma_split_folio(folio,
> > 116 + migrate->fault_page);
> > 117 +
> > 118 + if (ret) {
> > 119 + ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl);
> > 120 + goto next;
> > 121 + }
> > 122 +
> > 123 + addr = start;
> > 124 + goto again;
> > 125 + }
> > 126 +#endif
> >
> > You can probably just delete this and use my patch below, but if you
> > want to try fixing it with a quick look: if migrate_vma_split_folio
> > fails, you probably need to collect a hole. On success, you likely want
> > to continue executing the remainder of the loop. I can try playing with
> > this tomorrow, but it’s late here.
> >
> > I had privately sent you a version of this patch as a fix for Xe, and
> > this one seems to work:
> >
> > [PATCH] mm/migrate: Split THP found in middle of PMD during page collection
> >
> > The migrate layer is not coded to handle a THP found in the middle of a
> > PMD. This can occur if a user manipulates mappings with mremap(). If a
> > THP is found mid-PMD during page collection, split it.
> >
> > Cc: Balbir Singh <balbirs@...dia.com>
> > Signed-off-by: Matthew Brost <matthew.brost@...el.com>
> > ---
> > mm/migrate_device.c | 37 +++++++++++++++++++++++++++++++++++--
> > 1 file changed, 35 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/migrate_device.c b/mm/migrate_device.c
> > index abd9f6850db6..9ffc025bad50 100644
> > --- a/mm/migrate_device.c
> > +++ b/mm/migrate_device.c
> > @@ -65,6 +65,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
> > struct vm_area_struct *vma = walk->vma;
> > struct mm_struct *mm = vma->vm_mm;
> > unsigned long addr = start, unmapped = 0;
> > + struct folio *split_folio = NULL;
> > spinlock_t *ptl;
> > pte_t *ptep;
> >
> > @@ -107,10 +108,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
> > }
> > }
> >
> > - ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl);
> > + ptep = pte_offset_map_lock(mm, pmdp, start, &ptl);
> > if (!ptep)
> > goto again;
> > arch_enter_lazy_mmu_mode();
> > + ptep += (addr - start) / PAGE_SIZE;
> >
> > for (; addr < end; addr += PAGE_SIZE, ptep++) {
> > struct dev_pagemap *pgmap;
> > @@ -209,6 +211,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
> > bool anon_exclusive;
> > pte_t swp_pte;
> >
> > + if (folio_order(folio)) {
> > + split_folio = folio;
> > + goto split;
> > + }
> > +
> > flush_cache_page(vma, addr, pte_pfn(pte));
> > anon_exclusive = folio_test_anon(folio) &&
> > PageAnonExclusive(page);
> > @@ -287,8 +294,34 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
> > if (unmapped)
> > flush_tlb_range(walk->vma, start, end);
> >
> > +split:
> > arch_leave_lazy_mmu_mode();
> > - pte_unmap_unlock(ptep - 1, ptl);
> > + pte_unmap_unlock(ptep - 1 + !!split_folio, ptl);
> > +
> > + /*
> > + * XXX: No clean way to support higher-order folios that don't match PMD
> > + * boundaries for now — split them instead. Once mTHP support lands, add
> > + * proper support for this case.
> > + *
> > + * The test, which exposed this as problematic, remapped (memremap) a
> > + * large folio to an unaligned address, resulting in the folio being
> > + * found in the middle of the PTEs. The requested number of pages was
> > + * less than the folio size. Likely to be handled gracefully by upper
> > + * layers eventually, but not yet.
> > + */
> > + if (split_folio) {
> > + int ret;
> > +
> > + ret = split_folio(split_folio);
> > + if (fault_folio != split_folio)
> > + folio_unlock(split_folio);
> > + folio_put(split_folio);
> > + if (ret)
> > + return migrate_vma_collect_skip(addr, end, walk);
> > +
> > + split_folio = NULL;
> > + goto again;
> > + }
> >
> > return 0;
> > }
> >
> > If I apply the #if 0 change along with my patch above (plus one core
> > MM patch needed for Xe that adds a support function), Xe SVM fully
> > passes our test cases with both THP enabled and disabled.
> >
> Excellent work! Since you found this, do you mind sending the fix to Andrew as a fixup
Done. Here is a dri-devel patchworks link [1] to the patch.
Matt
[1] https://patchwork.freedesktop.org/series/157859/
> to the original patch. Since I don't have the test case, I have no way of validating the
> change or any change on top of it would continue to work
>
> FYI: The original code does something similar, I might be missing the
> migrate_vma_collect_skip() bits.
>
> Thanks!
> Balbir
>
>
Powered by blists - more mailing lists