lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Thu, 21 Dec 2017 16:23:23 +0800
From:   kemi <>
To:     Michal Hocko <>
Cc:     Greg Kroah-Hartman <>,
        Andrew Morton <>,
        Vlastimil Babka <>,
        Mel Gorman <>,
        Johannes Weiner <>,
        Christopher Lameter <>,
        Andrey Ryabinin <>,
        Nikolay Borisov <>,
        Pavel Tatashin <>,
        David Rientjes <>,
        Sebastian Andrzej Siewior <>,
        Dave <>,
        Andi Kleen <>,
        Tim Chen <>,
        Jesper Dangaard Brouer <>,
        Ying Huang <>,
        Aaron Lu <>, Aubrey Li <>,
        Linux MM <>,
        Linux Kernel <>
Subject: Re: [PATCH v2 3/5] mm: enlarge NUMA counters threshold size

On 2017年12月21日 16:17, Michal Hocko wrote:
> On Thu 21-12-17 16:06:50, kemi wrote:
>> On 2017年12月20日 18:12, Michal Hocko wrote:
>>> On Wed 20-12-17 13:52:14, kemi wrote:
>>>> On 2017年12月19日 20:40, Michal Hocko wrote:
>>>>> On Tue 19-12-17 14:39:24, Kemi Wang wrote:
>>>>>> We have seen significant overhead in cache bouncing caused by NUMA counters
>>>>>> update in multi-threaded page allocation. See 'commit 1d90ca897cb0 ("mm:
>>>>>> update NUMA counter threshold size")' for more details.
>>>>>> This patch updates NUMA counters to a fixed size of (MAX_S16 - 2) and deals
>>>>>> with global counter update using different threshold size for node page
>>>>>> stats.
>>>>> Again, no numbers.
>>>> Compare to vanilla kernel, I don't think it has performance improvement, so
>>>> I didn't post performance data here.
>>>> But, if you would like to see performance gain from enlarging threshold size
>>>> for NUMA stats (compare to the first patch), I will do that later. 
>>> Please do. I would also like to hear _why_ all counters cannot simply
>>> behave same. In other words why we cannot simply increase
>>> stat_threshold? Maybe calculate_normal_threshold needs a better scaling
>>> for larger machines.
>> I will add this performance data to changelog in V3 patch series.
>> Test machine: 2-sockets skylake platform (112 CPUs, 62G RAM)
>> Benchmark: page_bench03
>> Description: 112 threads do single page allocation/deallocation in parallel.
>>                before                           after
>>                                        (enlarge threshold size)       
>> CPU cycles     722                              379(-47.5%)
> Please describe the numbers some more. Is this an average?


> What is the std? 

I increase the loop times to 10m, so the std is quite slow (repeat 3 times)

> Can you see any difference with a more generic workload?

I didn't see obvious improvement for will-it-scale.page_fault1
Two reasons for that:
1) too long code path
2) server zone lock and lru lock contention (access to buddy system frequently) 

>> Some thinking about that:
>> a) the overhead due to cache bouncing caused by NUMA counter update in fast path 
>> severely increase with more and more CPUs cores
> What is an effect on a smaller system with fewer CPUs?

Several CPU cycles can be saved using single thread for that.

>> b) AFAIK, the typical usage scenario (similar at least)for which this optimization can 
>> benefit is 10/40G NIC used in high-speed data center network of cloud service providers.
> I would expect those would disable the numa accounting altogether.

Yes, but it is still worthy to do some optimization, isn't?

Powered by blists - more mailing lists