[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20250430143714.GA2020@cmpxchg.org>
Date: Wed, 30 Apr 2025 10:37:14 -0400
From: Johannes Weiner <hannes@...xchg.org>
To: Muchun Song <songmuchun@...edance.com>
Cc: mhocko@...nel.org, roman.gushchin@...ux.dev, shakeel.butt@...ux.dev,
muchun.song@...ux.dev, akpm@...ux-foundation.org,
david@...morbit.com, zhengqi.arch@...edance.com,
yosry.ahmed@...ux.dev, nphamcs@...il.com, chengming.zhou@...ux.dev,
linux-kernel@...r.kernel.org, cgroups@...r.kernel.org,
linux-mm@...ck.org, hamzamahfooz@...ux.microsoft.com,
apais@...ux.microsoft.com, Hugh Dickins <hughd@...gle.com>
Subject: Re: [PATCH RFC 07/28] mm: thp: use folio_batch to handle THP
splitting in deferred_split_scan()
On Tue, Apr 15, 2025 at 10:45:11AM +0800, Muchun Song wrote:
> The maintenance of the folio->_deferred_list is intricate because it's
> reused in a local list.
>
> Here are some peculiarities:
>
> 1) When a folio is removed from its split queue and added to a local
> on-stack list in deferred_split_scan(), the ->split_queue_len isn't
> updated, leading to an inconsistency between it and the actual
> number of folios in the split queue.
>
> 2) When the folio is split via split_folio() later, it's removed from
> the local list while holding the split queue lock. At this time,
> this lock protects the local list, not the split queue.
>
> 3) To handle the race condition with a third-party freeing or migrating
> the preceding folio, we must ensure there's always one safe (with
> raised refcount) folio before by delaying its folio_put(). More
> details can be found in commit e66f3185fa04. It's rather tricky.
>
> We can use the folio_batch infrastructure to handle this clearly. In this
> case, ->split_queue_len will be consistent with the real number of folios
> in the split queue. If list_empty(&folio->_deferred_list) returns false,
> it's clear the folio must be in its split queue (not in a local list
> anymore).
>
> In the future, we aim to reparent LRU folios during memcg offline to
> eliminate dying memory cgroups. This patch prepares for using
> folio_split_queue_lock_irqsave() as folio memcg may change then.
>
> Signed-off-by: Muchun Song <songmuchun@...edance.com>
This is a very nice simplification. And getting rid of the stack list
and its subtle implication on all the various current and future
list_empty(&folio->_deferred_list) checks should be much more robust.
However, I think there is one snag related to this:
> ---
> mm/huge_memory.c | 69 +++++++++++++++++++++---------------------------
> 1 file changed, 30 insertions(+), 39 deletions(-)
>
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 70820fa75c1f..d2bc943a40e8 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -4220,40 +4220,47 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
> struct pglist_data *pgdata = NODE_DATA(sc->nid);
> struct deferred_split *ds_queue = &pgdata->deferred_split_queue;
> unsigned long flags;
> - LIST_HEAD(list);
> - struct folio *folio, *next, *prev = NULL;
> - int split = 0, removed = 0;
> + struct folio *folio, *next;
> + int split = 0, i;
> + struct folio_batch fbatch;
> + bool done;
>
> #ifdef CONFIG_MEMCG
> if (sc->memcg)
> ds_queue = &sc->memcg->deferred_split_queue;
> #endif
> -
> + folio_batch_init(&fbatch);
> +retry:
> + done = true;
> spin_lock_irqsave(&ds_queue->split_queue_lock, flags);
> /* Take pin on all head pages to avoid freeing them under us */
> list_for_each_entry_safe(folio, next, &ds_queue->split_queue,
> _deferred_list) {
> if (folio_try_get(folio)) {
> - list_move(&folio->_deferred_list, &list);
> - } else {
> + folio_batch_add(&fbatch, folio);
> + } else if (folio_test_partially_mapped(folio)) {
> /* We lost race with folio_put() */
> - if (folio_test_partially_mapped(folio)) {
> - folio_clear_partially_mapped(folio);
> - mod_mthp_stat(folio_order(folio),
> - MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
> - }
> - list_del_init(&folio->_deferred_list);
> - ds_queue->split_queue_len--;
> + folio_clear_partially_mapped(folio);
> + mod_mthp_stat(folio_order(folio),
> + MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
> }
> + list_del_init(&folio->_deferred_list);
> + ds_queue->split_queue_len--;
> if (!--sc->nr_to_scan)
> break;
> + if (folio_batch_space(&fbatch) == 0) {
> + done = false;
> + break;
> + }
> }
> split_queue_unlock_irqrestore(ds_queue, flags);
>
> - list_for_each_entry_safe(folio, next, &list, _deferred_list) {
> + for (i = 0; i < folio_batch_count(&fbatch); i++) {
> bool did_split = false;
> bool underused = false;
> + struct deferred_split *fqueue;
>
> + folio = fbatch.folios[i];
> if (!folio_test_partially_mapped(folio)) {
> underused = thp_underused(folio);
> if (!underused)
> @@ -4269,39 +4276,23 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
> }
> folio_unlock(folio);
> next:
> + if (did_split || !folio_test_partially_mapped(folio))
> + continue;
There IS a list_empty() check in the splitting code that we actually
relied on, for cleaning up the partially_mapped state and counter:
!list_empty(&folio->_deferred_list)) {
ds_queue->split_queue_len--;
if (folio_test_partially_mapped(folio)) {
folio_clear_partially_mapped(folio);
mod_mthp_stat(folio_order(folio),
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
/*
* Reinitialize page_deferred_list after removing the
* page from the split_queue, otherwise a subsequent
* split will see list corruption when checking the
* page_deferred_list.
*/
list_del_init(&folio->_deferred_list);
With the folios isolated up front, it looks like you need to handle
this from the shrinker.
Otherwise this looks correct to me. But this code is subtle, I would
feel much better if Hugh (CC-ed) could take a look as well.
Thanks!
> /*
> - * split_folio() removes folio from list on success.
> * Only add back to the queue if folio is partially mapped.
> * If thp_underused returns false, or if split_folio fails
> * in the case it was underused, then consider it used and
> * don't add it back to split_queue.
> */
> - if (did_split) {
> - ; /* folio already removed from list */
> - } else if (!folio_test_partially_mapped(folio)) {
> - list_del_init(&folio->_deferred_list);
> - removed++;
> - } else {
> - /*
> - * That unlocked list_del_init() above would be unsafe,
> - * unless its folio is separated from any earlier folios
> - * left on the list (which may be concurrently unqueued)
> - * by one safe folio with refcount still raised.
> - */
> - swap(folio, prev);
> - }
> - if (folio)
> - folio_put(folio);
> + fqueue = folio_split_queue_lock_irqsave(folio, &flags);
> + list_add_tail(&folio->_deferred_list, &fqueue->split_queue);
> + fqueue->split_queue_len++;
> + split_queue_unlock_irqrestore(fqueue, flags);
> }
> + folios_put(&fbatch);
>
> - spin_lock_irqsave(&ds_queue->split_queue_lock, flags);
> - list_splice_tail(&list, &ds_queue->split_queue);
> - ds_queue->split_queue_len -= removed;
> - split_queue_unlock_irqrestore(ds_queue, flags);
> -
> - if (prev)
> - folio_put(prev);
> -
> + if (!done)
> + goto retry;
> /*
> * Stop shrinker if we didn't split any page, but the queue is empty.
> * This can happen if pages were freed under us.
> --
> 2.20.1
Powered by blists - more mailing lists