Message-ID: <CAA1CXcBwO77=4ki8kG93rNhB1BUpGqfvo3UbRt0WQ5pFMDyVWw@mail.gmail.com>
Date: Thu, 24 Apr 2025 12:56:31 -0600
From: Nico Pache <npache@...hat.com>
To: Mitchell Augustin <mitchell.augustin@...onical.com>
Cc: akpm@...ux-foundation.org, 
	20250211152341.3431089327c5e0ec6ba6064d@...ux-foundation.org, 
	21cnbao@...il.com, aneesh.kumar@...nel.org, anshuman.khandual@....com, 
	apopple@...dia.com, baohua@...nel.org, catalin.marinas@....com, cl@...two.org, 
	dave.hansen@...ux.intel.com, david@...hat.com, dev.jain@....com, 
	haowenchao22@...il.com, hughd@...gle.com, ioworker0@...il.com, jack@...e.cz, 
	jglisse@...gle.com, John Hubbard <jhubbard@...dia.com>, kirill.shutemov@...ux.intel.com, 
	linux-kernel@...r.kernel.org, linux-mm@...ck.org, mhocko@...e.com, 
	Peter Xu <peterx@...hat.com>, ryan.roberts@....com, srivatsa@...il.mit.edu, 
	surenb@...gle.com, vbabka@...e.cz, vishal.moola@...il.com, 
	wangkefeng.wang@...wei.com, will@...nel.org, willy@...radead.org, 
	yang@...amperecomputing.com, zhengqi.arch@...edance.com, 
	Zi Yan <ziy@...dia.com>, zokeefe@...gle.com, 
	Jacob Martin <jacob.martin@...onical.com>, 
	Vanda Hendrychová <vanda.hendrychova@...onical.com>
Subject: Re: [PATCH v2 00/17] khugepaged: Asynchronous mTHP collapse

On Thu, Apr 24, 2025 at 12:18 PM Mitchell Augustin
<mitchell.augustin@...onical.com> wrote:
>
> Hello,
>
> I realize this is an older version of the series, but @Vanda
> Hendrychová and I started on a benchmark effort of this version prior
> to the most recent revision's introduction and wanted to provide our
> results as feedback for this discussion.
>
> For context, my team and I previously identified that some of the
> benchmarks outlined in this phoronix benchmark suite [0] perform more
> poorly with thp=madvise than thp=always - so I suspected that the
> THP=defer and khugepaged collapse functionality outlined in this
> article [6] might yield performance in between madvise and always for
> the following benchmarks from that suite:
> - GraphicsMagick (all tests), which were substantially improved when
> switching from thp=madvise to thp=always
> - 7-Zip Compression rating, which was substantially improved when
> switching from thp=madvise to thp=always
> - Compilation time tests, which were slightly improved when switching
> from thp=madvise to thp=always
>
> There were more benchmarks in this suite, but these three were the
> ones we had previously identified as being significantly impacted by
> the thp setting, and thus are the primary focus of our results.
>
> To analyze this, we ran the benchmarks outlined in this article on the
> upstream 6.14 kernel with the following configurations:
> - linux v6.14 thp=defer-v1: Transparent Huge Pages: defer
> - linux v6.14 thp=defer-v2: Transparent Huge Pages: defer
> - linux v6.14 thp=always: Transparent Huge Pages: always
> - linux v6.14 thp=never: Transparent Huge Pages: never
> - linux v6.14 thp=madvise: Transparent Huge Pages: madvise
>
> "defer-v1" refers to the thp collapse implementation by Nico Pache
> [3], and "defer-v2" refers to the implementation in this thread [4].
> Both use defer as implemented by series [5].
>
>
> Ultimately, we did observe that some of the GraphicsMagick tests
> performed marginally better with Nico Pache's khugepaged collapse
> implementation and thp=defer than with just thp=madvise, which aligns
> a bit with my theory - however, these improvements unfortunately did
> not appear to be statistically significant and gained only marginal
> ground in the performance gap between thp=madvise and thp=always in
> our workloads of interest.
>
> Results for other benchmarks in this set also did not show any
> conclusive performance gains from mTHP=defer (however I was not
> expecting those to change significantly with this series, since they
> weren’t heavily impacted by thp settings in my prior tests).
>
> I can't speak for the impact of this series on other workloads - I
> just wanted to share results for the ones we were aware of and
> interested in.
Hi Mitchell,

Thank you very much for both testing and sharing the results! I'm glad
no major regressions were noted, and in some cases performance was
marginally better. Another good set of workloads to test for defer
would be latency tests... THP=always can increase PF latencies, while
"defer" should eliminate that penalty, with the hopes of regaining
some of the THP benefits after the khugepaged collapse.

I wanted to note one thing, with the default of max_ptes_none=511 and
no mTHP sizes configured, the khugepaged series' (both mine and Devs)
should have very little impact. This is a good test of the defer
feature, while confirming that neither me nor Dev regressed the legacy
PMD khugepaged case; however, this is not a good test of the actual
mTHP collapsing.

If you plan on testing the mTHP changes for performance changes, I
would suggest enabling all the mTHP orders and setting max_ptes_none=0
(Devs series requires 0 or 511 for mTHP collapse to work). Given this
is a new feature, it may be hard to find something to compare it to,
other than each other's series'. enabling defer during these tests has
the added benefit of pushing everything to khugepaged and really
stressing its mTHP collapse performance.

Once again thank you for taking the time to test these features :)
-- Nico


>
> Full results from our tests on the DGX A100 [1] and Lenovo SR670v2 [2]
> are linked below.
>
> [0]: https://www.phoronix.com/review/linux-os-ampereone/5
> [1]: https://pastebin.ubuntu.com/p/SDSSj8cr6k/
> [2]: https://pastebin.ubuntu.com/p/nqbWxyC33d/
> [3]: https://lwn.net/ml/all/20250211003028.213461-1-npache@redhat.com
> [4]: https://lwn.net/ml/all/20250211111326.14295-1-dev.jain@arm.com
> [5]: https://lwn.net/ml/all/20250211004054.222931-1-npache@redhat.com
> [6]: https://lwn.net/Articles/1009039/
> --
> Mitchell Augustin
> Software Engineer - Ubuntu Partner Engineering
>

Powered by blists - more mailing lists