linux-kernel - Re: [PATCH v4] sched/topology: fix the issue groups don't span domain->span for NUMA diameter

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <YD4Z1K6Vb0+u0JQz@hirez.programming.kicks-ass.net>
Date:   Tue, 2 Mar 2021 11:56:20 +0100
From:   Peter Zijlstra <peterz@...radead.org>
To:     Barry Song <song.bao.hua@...ilicon.com>
Cc:     valentin.schneider@....com, vincent.guittot@...aro.org,
        mgorman@...e.de, mingo@...nel.org, dietmar.eggemann@....com,
        morten.rasmussen@....com, linux-kernel@...r.kernel.org,
        linuxarm@...neuler.org, xuwei5@...wei.com, liguozhu@...ilicon.com,
        tiantao6@...ilicon.com, wanghuiqiang@...wei.com,
        prime.zeng@...ilicon.com, jonathan.cameron@...wei.com,
        guodong.xu@...aro.org, yangyicong@...wei.com,
        Meelis Roos <mroos@...ux.ee>
Subject: Re: [PATCH v4] sched/topology: fix the issue groups don't span
 domain->span for NUMA diameter > 2

On Wed, Feb 24, 2021 at 04:09:44PM +1300, Barry Song wrote:
> As long as NUMA diameter > 2, building sched_domain by sibling's child
> domain will definitely create a sched_domain with sched_group which will
> span out of the sched_domain:
> 
>                +------+         +------+        +-------+       +------+
>                | node |  12     |node  | 20     | node  |  12   |node  |
>                |  0   +---------+1     +--------+ 2     +-------+3     |
>                +------+         +------+        +-------+       +------+
> 
> domain0        node0            node1            node2          node3
> 
> domain1        node0+1          node0+1          node2+3        node2+3
>                                                  +
> domain2        node0+1+2                         |
>              group: node0+1                      |
>                group:node2+3 <-------------------+
> 
> when node2 is added into the domain2 of node0, kernel is using the child
> domain of node2's domain2, which is domain1(node2+3). Node 3 is outside
> the span of the domain including node0+1+2.
> 
> This will make load_balance() run based on screwed avg_load and group_type
> in the sched_group spanning out of the sched_domain, and it also makes
> select_task_rq_fair() pick an idle CPU outside the sched_domain.
> 
> Real servers which suffer from this problem include Kunpeng920 and 8-node
> Sun Fire X4600-M2, at least.
> 
> Here we move to use the *child* domain of the *child* domain of node2's
> domain2 as the new added sched_group. At the same, we re-use the lower
> level sgc directly.
>                +------+         +------+        +-------+       +------+
>                | node |  12     |node  | 20     | node  |  12   |node  |
>                |  0   +---------+1     +--------+ 2     +-------+3     |
>                +------+         +------+        +-------+       +------+
> 
> domain0        node0            node1          +- node2          node3
>                                                |
> domain1        node0+1          node0+1        | node2+3        node2+3
>                                                |
> domain2        node0+1+2                       |
>              group: node0+1                    |
>                group:node2 <-------------------+
> 
> While the lower level sgc is re-used, this patch only changes the remote
> sched_groups for those sched_domains playing grandchild trick, therefore,
> sgc->next_update is still safe since it's only touched by CPUs that have
> the group span as local group. And sgc->imbalance is also safe because
> sd_parent remains the same in load_balance and LB only tries other CPUs
> from the local group.
> Moreover, since local groups are not touched, they are still getting
> roughly equal size in a TL. And should_we_balance() only matters with
> local groups, so the pull probability of those groups are still roughly
> equal.
> 

> Reported-by: Valentin Schneider <valentin.schneider@....com>
> Tested-by: Meelis Roos <mroos@...ux.ee>
> Reviewed-by: Valentin Schneider <valentin.schneider@....com>
> Signed-off-by: Barry Song <song.bao.hua@...ilicon.com>

Thanks!