linux-kernel - Re: [RFC PATCH v4 11/13] mm: parallelize deferred struct page initialization within each node

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20181127202359.biav42vbfchprmo5@ca-dmjordan1.us.oracle.com>
Date:   Tue, 27 Nov 2018 12:23:59 -0800
From:   Daniel Jordan <daniel.m.jordan@...cle.com>
To:     "Elliott, Robert (Persistent Memory)" <elliott@....com>
Cc:     "'Daniel Jordan'" <daniel.m.jordan@...cle.com>,
        "linux-mm@...ck.org" <linux-mm@...ck.org>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "aarcange@...hat.com" <aarcange@...hat.com>,
        "aaron.lu@...el.com" <aaron.lu@...el.com>,
        "akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
        "alex.williamson@...hat.com" <alex.williamson@...hat.com>,
        "bsd@...hat.com" <bsd@...hat.com>,
        "darrick.wong@...cle.com" <darrick.wong@...cle.com>,
        "dave.hansen@...ux.intel.com" <dave.hansen@...ux.intel.com>,
        "jgg@...lanox.com" <jgg@...lanox.com>,
        "jwadams@...gle.com" <jwadams@...gle.com>,
        "jiangshanlai@...il.com" <jiangshanlai@...il.com>,
        "mhocko@...nel.org" <mhocko@...nel.org>,
        "mike.kravetz@...cle.com" <mike.kravetz@...cle.com>,
        "Pavel.Tatashin@...rosoft.com" <Pavel.Tatashin@...rosoft.com>,
        "prasad.singamsetty@...cle.com" <prasad.singamsetty@...cle.com>,
        "rdunlap@...radead.org" <rdunlap@...radead.org>,
        "steven.sistare@...cle.com" <steven.sistare@...cle.com>,
        "tim.c.chen@...el.com" <tim.c.chen@...el.com>,
        "tj@...nel.org" <tj@...nel.org>, "vbabka@...e.cz" <vbabka@...e.cz>,
        peterz@...radead.org, dhaval.giani@...cle.com
Subject: Re: [RFC PATCH v4 11/13] mm: parallelize deferred struct page
 initialization within each node

On Tue, Nov 27, 2018 at 12:12:28AM +0000, Elliott, Robert (Persistent Memory) wrote:
> I ran a short test with:
> * HPE ProLiant DL360 Gen9 system
> * Intel Xeon E5-2699 CPU with 18 physical cores (0-17) and 
>   18 hyperthreaded cores (36-53)
> * DDR4 NVDIMM-Ns (which run at regular DRAM DIMM speeds)
> * fio workload generator
> * cores on one CPU socket talking to a pmem device on the same CPU
> * large (1 MiB) random writes (to minimize the threads getting CPU cache
>   hits from each other)
> 
> Results:
> * 31.7 GB/s    four threads, four physical cores (0,1,2,3)
> * 22.2 GB/s    four threads, two physical cores (0,1,36,37)
> * 21.4 GB/s    two threads, two physical cores (0,1)
> * 12.1 GB/s    two threads, one physical core (0,36)
> * 11.2 GB/s    one thread, one physical core (0)
> 
> So, I think it's important that the initialization threads run on
> separate physical cores.

Thanks for running this.  And fair enough, in this test using both siblings
gives only a 4-8% speedup over one, so it makes sense to use only cores in the
calculation.

As for how to actually do this, some arches have smp_num_siblings, but there
should be a generic interface to provide that.

It's also possible to calculate this from the existing
topology_sibling_cpumask, but the first option is better IMHO.  Open to
suggestions.

> For the number of cores to use, one approach is:
>     memory bandwidth (number of interleaved channels * speed)
> divided by 
>     CPU core max sustained write bandwidth
> 
> For example, this 2133 MT/s system is roughly:
>     68 GB/s    (4 * 17 GB/s nominal)
> divided by
>     11.2 GB/s  (one core's performance)
> which is 
>     6 cores
> 
> ACPI HMAT will report that 68 GB/s number.  I'm not sure of
> a good way to discover the 11.2 GB/s number.

Yes, this would be nice to do if we could know the per-core number, with the
caveat that a single number like this would be most useful for the CPU-memory
pair it was calculated for, so the kernel could at least calculate it for jobs
operating on local memory.

Some BogoMIPS-like calibration may work, but I'll wait for ACPI HMAT support in
the kernel.