linux-kernel - Re: [LSF/MM/BPF TOPIC] Overhauling hot page detection and promotion based on PTE A bit scanning

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <c9c70938-24ac-4044-881e-6dffccc78126@amd.com>
Date: Mon, 27 Jan 2025 10:41:07 +0530
From: Bharata B Rao <bharata@....com>
To: "Huang, Ying" <ying.huang@...ux.alibaba.com>,
 Raghavendra K T <raghavendra.kt@....com>
Cc: linux-mm@...ck.org, akpm@...ux-foundation.org,
 lsf-pc@...ts.linux-foundation.org, gourry@...rry.net, nehagholkar@...a.com,
 abhishekd@...a.com, nphamcs@...il.com, hannes@...xchg.org,
 feng.tang@...el.com, kbusch@...a.com, Hasan.Maruf@....com, sj@...nel.org,
 david@...hat.com, willy@...radead.org, k.shutemov@...il.com,
 mgorman@...hsingularity.net, vbabka@...e.cz, hughd@...gle.com,
 rientjes@...gle.com, shy828301@...il.com, liam.howlett@...cle.com,
 peterz@...radead.org, mingo@...hat.com, nadav.amit@...il.com,
 shivankg@....com, ziy@...dia.com, jhubbard@...dia.com,
 AneeshKumar.KizhakeVeetil@....com, linux-kernel@...r.kernel.org,
 jon.grimm@....com, santosh.shukla@....com, Michael.Day@....com,
 riel@...riel.com, weixugc@...gle.com, leesuyeon0506@...il.com,
 honggyu.kim@...com, leillc@...gle.com, kmanaouil.dev@...il.com,
 rppt@...nel.org, dave.hansen@...el.com
Subject: Re: [LSF/MM/BPF TOPIC] Overhauling hot page detection and promotion
 based on PTE A bit scanning

On 26-Jan-25 7:57 AM, Huang, Ying wrote:
> Hi, Raghavendra,
> 
> Raghavendra K T <raghavendra.kt@....com> writes:
> 
>> Bharata and I would like to propose the following topic for LSFMM.
>>
>> Topic: Overhauling hot page detection and promotion based on PTE A bit scanning.
>>   
>> In the Linux kernel, hot page information can potentially be obtained from
>> multiple sources:
>>   
>> a. PROT_NONE faults (NUMA balancing)
>> b. PTE Access bit (LRU scanning)
>> c. Hardware provided page hotness info (like AMD IBS)
>>   
>> This information is further used to migrate (or promote) pages from slow memory
>> tier to top tier to increase performance.
>>
>> In the current hot page promotion mechanism, all the activities including the
>> process address space scanning, NUMA hint fault handling and page migration are
>> performed in the process context. i.e., scanning overhead is borne by the
>> applications.
>>   
>> I had recently posted a patch [1] to improve this in the context of slow-tier
>> page promotion. Here, Scanning is done by a global kernel thread which routinely
>> scans all the processes' address spaces and checks for accesses by reading the
>> PTE A bit. The hot pages thus identified are maintained in list and subsequently
>> are promoted to a default top-tier node. Thus, the approach pushes overhead of
>> scanning, NUMA hint faults and migrations off from process context.
> 
> This has been discussed before too.  For example, in the following thread
> 
> https://lore.kernel.org/all/20200417100633.GU20730@hirez.programming.kicks-ass.net/T/

Thanks for pointing to this discussion.

> 
> The drawbacks of asynchronous scanning including
> 
> - The CPU cycles used are not charged properly
> 
> - There may be no idle CPU cycles to use
> 
> - The scanning CPU may be not near the workload CPUs enough
> 
> It's better to involve Mel and Peter in the discussion for this.

They are CC'ed in this thread and hopefully have insights to share.

Charging CPU cycles to the right process has been brought up in other 
similar contexts. Recent one is from page migration batching and using 
multiple threads for migration - 
https://lore.kernel.org/all/CAHbLzkpoKP0fVZP5b10wdzAMDLWysDy7oH0qaUssiUXj80R6bw@mail.gmail.com/

Does it make sense to treat hot page promotion from slow tiers 
differently compared to locality based balancing? I mean couldn't the 
charging of this async thread be similar to the cycles spent by other 
system threads like kcompactd and khugepaged?

> 
>> The topic was presented in the MM alignment session hosted by David Rientjes [2].
>> The topic also finds a mention in S J Park's LSFMM proposal [3].
>>   
>> Here is the list of potential discussion points:
>> 1. Other improvements and enhancements to PTE A bit scanning approach. Use of
>> multiple kernel threads, throttling improvements, promotion policies, per-process
>> opt-in via prctl, virtual vs physical address based scanning, tuning hot page
>> detection algorithm etc.
> 
> One drawback of physical address based scanning is that it's hard to
> apply some workload specific policy.  For example, if a low priority
> workload has many relatively hot pages, while a high priority workload
> has many relative warm (not so hot) pages.  We need to promote the warm
> pages in the high priority workload, while physcial address based
> scanning may report the hot pages in the low priority workload.  Right?

Correct. I wonder if DAMON has already devised a scheme to address this. SJ?

Regards,
Bharata.