linux-kernel - Re: 5.11-rc4+git: Shortest NUMA path spans too many nodes

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <99ecee86-8d96-c1e2-5b38-8f3dfcc38bd1@arm.com>
Date:   Fri, 22 Jan 2021 11:05:08 +0100
From:   Dietmar Eggemann <dietmar.eggemann@....com>
To:     "Song Bao Hua (Barry Song)" <song.bao.hua@...ilicon.com>,
        Valentin Schneider <valentin.schneider@....com>,
        Meelis Roos <mroos@...ux.ee>,
        LKML <linux-kernel@...r.kernel.org>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Mel Gorman <mgorman@...e.de>
Subject: Re: 5.11-rc4+git: Shortest NUMA path spans too many nodes

On 21/01/2021 22:17, Song Bao Hua (Barry Song) wrote:
> 
> 
>> -----Original Message-----
>> From: Dietmar Eggemann [mailto:dietmar.eggemann@....com]
>> Sent: Friday, January 22, 2021 7:54 AM
>> To: Valentin Schneider <valentin.schneider@....com>; Meelis Roos
>> <mroos@...ux.ee>; LKML <linux-kernel@...r.kernel.org>
>> Cc: Peter Zijlstra <peterz@...radead.org>; Vincent Guittot
>> <vincent.guittot@...aro.org>; Song Bao Hua (Barry Song)
>> <song.bao.hua@...ilicon.com>; Mel Gorman <mgorman@...e.de>
>> Subject: Re: 5.11-rc4+git: Shortest NUMA path spans too many nodes
>>
>> On 21/01/2021 19:21, Valentin Schneider wrote:
>>> On 21/01/21 19:39, Meelis Roos wrote:

[...]

>> # cat /sys/devices/system/node/node*/distance
>> 10 12 12 14 14 14 14 16
>> 12 10 14 12 14 14 12 14
>> 12 14 10 14 12 12 14 14
>> 14 12 14 10 12 12 14 14
>> 14 14 12 12 10 14 12 14
>> 14 14 12 12 14 10 14 12
>> 14 12 14 14 12 14 10 12
>> 16 14 14 14 14 12 12 10
>>
>> The '16' seems to be the culprit. How does such a topo look like?

Maybe like this:

      _________
      |       |
    .-6   0   4-.
    |  \ / \ /  |
    |   1   2   |
    |   \    \  |
    --7  3----5 |
      |  |____|_|
      |_______|

> 
> Once we get a topology like this:
> 
> 
>          +------+         +------+        +-------+       +------+
>          | node |         |node  |        | node  |       |node  |
>          |      +---------+      +--------+       +-------+      |
>          +------+         +------+        +-------+       +------+
> 
> We can reproduce this issue. 
> For example, every cpu with the below numa_distance can have 
> "groups don't span domain->span":
> node   0   1   2   3
>   0:  10  12  20  22
>   1:  12  10  22  24
>   2:  20  22  10  12
>   3:  22  24  12  10
                             2     20     2
So this should look like: 1 --- 0 ---- 2 --- 3