linux-kernel - Re: [PATCH] thp: reduce usage of huge zero page's atomic counter

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160829141011.GA15819@aaronlu.sh.intel.com>
Date:   Mon, 29 Aug 2016 22:10:12 +0800
From:   Aaron Lu <aaron.lu@...el.com>
To:     Anshuman Khandual <khandual@...ux.vnet.ibm.com>
Cc:     Linux Memory Management List <linux-mm@...ck.org>,
        "'Kirill A. Shutemov'" <kirill.shutemov@...ux.intel.com>,
        Dave Hansen <dave.hansen@...el.com>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        Huang Ying <ying.huang@...el.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Vlastimil Babka <vbabka@...e.cz>,
        Jerome Marchand <jmarchan@...hat.com>,
        Andrea Arcangeli <aarcange@...hat.com>,
        Mel Gorman <mgorman@...hsingularity.net>,
        Ebru Akagunduz <ebru.akagunduz@...il.com>,
        linux-kernel@...r.kernel.org,
        "Aneesh Kumar K.V" <aneesh.kumar@...ux.vnet.ibm.com>
Subject: Re: [PATCH] thp: reduce usage of huge zero page's atomic counter

On Mon, Aug 29, 2016 at 07:17:58PM +0530, Anshuman Khandual wrote:
> On 08/29/2016 02:23 PM, Aaron Lu wrote:
> > On 08/29/2016 04:49 PM, Anshuman Khandual wrote:
> >> > On 08/29/2016 12:01 PM, Aaron Lu wrote:
> >>> >> The global zero page is used to satisfy an anonymous read fault. If
> >>> >> THP(Transparent HugePage) is enabled then the global huge zero page is used.
> >>> >> The global huge zero page uses an atomic counter for reference counting
> >>> >> and is allocated/freed dynamically according to its counter value.
> >>> >>
> >>> >> CPU time spent on that counter will greatly increase if there are
> >>> >> a lot of processes doing anonymous read faults. This patch proposes a
> >>> >> way to reduce the access to the global counter so that the CPU load
> >>> >> can be reduced accordingly.
> >>> >>
> >>> >> To do this, a new flag of the mm_struct is introduced: MMF_USED_HUGE_ZERO_PAGE.
> >>> >> With this flag, the process only need to touch the global counter in
> >>> >> two cases:
> >>> >> 1 The first time it uses the global huge zero page;
> >>> >> 2 The time when mm_user of its mm_struct reaches zero.
> >>> >>
> >>> >> Note that right now, the huge zero page is eligible to be freed as soon
> >>> >> as its last use goes away.  With this patch, the page will not be
> >>> >> eligible to be freed until the exit of the last process from which it
> >>> >> was ever used.
> >>> >>
> >>> >> And with the use of mm_user, the kthread is not eligible to use huge
> >>> >> zero page either. Since no kthread is using huge zero page today, there
> >>> >> is no difference after applying this patch. But if that is not desired,
> >>> >> I can change it to when mm_count reaches zero.
> >>> >>
> >>> >> Case used for test on Haswell EP:
> >>> >> usemem -n 72 --readonly -j 0x200000 100G
> >> > 
> >> > Is this benchmark publicly available ? Does not seem to be this one
> >> > https://github.com/gnubert/usemem.git, Does it ?
> > Sorry, forgot to attach its link.
> > It's this one:
> > https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git
> > 
> > And the above mentioned usemem is:
> > https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/tree/usemem.c
> 
> Hey Aaron,
> 
> Thanks for pointing out. I did ran similar test on a POWER8 box using 16M
> steps (huge page size is 16MB on it) instead of 2MB. But the perf profile
> looked different. The perf command line was like this on a 32 CPU system.
> 
> perf record ./usemem -n 256 --readonly -j 0x1000000 100G
> 
> But the relative weight of the above mentioned function came out to be
> pretty less compared to what you have reported from your experiment
> which is around 54.03%.
> 
> 0.07%  usemem  [kernel.vmlinux]  [k] get_huge_zero_page
> 
> Seems way out of the mark. Can you please confirm your exact perf record
> command line and how many CPUs you have on the system.

Haswell EP has 72 CPUs.

Since the huge page size is 16MB on your system, maybe you can try:
perf record ./usemem -n 32 --readonly -j 0x1000000 800G

Regards,
Aaron