Date:   Wed, 12 Jan 2022 09:55:51 +0100
From:   Peter Zijlstra <peterz@...radead.org>
To:     王贇 <yun.wang@...ux.alibaba.com>
Cc:     Ingo Molnar <mingo@...hat.com>, Juri Lelli <juri.lelli@...hat.com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
        Daniel Bristot de Oliveira <bristot@...hat.com>,
        Masahiro Yamada <masahiroy@...nel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Nathan Chancellor <nathan@...nel.org>,
        Kees Cook <keescook@...omium.org>,
        Vlastimil Babka <vbabka@...e.cz>,
        Chris Down <chris@...isdown.name>,
        Vipin Sharma <vipinsh@...gle.com>,
        Rasmus Villemoes <linux@...musvillemoes.dk>,
        Daniel Borkmann <daniel@...earbox.net>,
        Arnd Bergmann <arnd@...db.de>,
        Cruz Zhao <cruzzhao@...ux.alibaba.com>,
        Tianchen Ding <dtcccc@...ux.alibaba.com>,
        linux-kernel@...r.kernel.org
Subject: Re: [RFC PATCH] sched: introduce group balancer

On Tue, Jan 04, 2022 at 03:33:57PM +0800, 王贇 wrote:
> Modern platform are growing fast on CPU numbers, multiple
> apps sharing one box are very common, they used to have
> exclusive cpu setting but nowadays things are changing.
> 
> To achieve better utility of CPU resource, multiple apps
> are starting to sharing the CPUs. The CPU resources usually
> overcommitted since app's workload are undulated.
> 
> This introduced problems on performance when share mode vs
> exclusive mode, for eg with cgroup A,B and C deployed in
> exclusive mode, it will be:
> 
>   CPU_X (100%)     CPU_Y (100%)     CPU_Z (50%)
>   T_1_CG_A         T_1_CG_B         T_1_CG_C
>   T_2_CG_A         T_2_CG_B         T_2_CG_C
>   T_3_CG_A         T_3_CG_B
>   T_4_CG_A         T_4_CG_B
> 
> while the share mode will be:
> 
>   CPU_X (100%)     CPU_Y (75%)     CPU_Z (75%)
>   T_1_CG_A         T_2_CG_A         T_1_CG_B
>   T_2_CG_B         T_3_CG_B         T_2_CG_C
>   T_4_CG_B         T_4_CG_A         T_3_CG_A
>   T_1_CG_C
> 
> As we can see, the confliction between groups on CPU
> resources are now happening all over the CPUs.
> 
> The testing on sysbench-memory show 30+% drop on share
> mode, and redis-benchmark show 10+% drop too, compared
> to the exclusive mode.
> 
> However, despite of the performance drop, in real world
> we still prefer share mode. The undulated workload can
> make the exclusive mode so unefficient on CPU utilization,
> for eg the next period, when CG_A become 'idle', exclusive
> mode will like:
> 
>   CPU_X (0%)     CPU_Y (100%)     CPU_Z (50%)
>                  T_1_CG_B         T_1_CG_C
>                  T_2_CG_B         T_2_CG_C
>                  T_3_CG_B
>                  T_4_CG_B
> 
> while share mode like:
> 
>   CPU_X (50%)     CPU_Y (50%)     CPU_Z (50%)
>   T_2_CG_B         T_1_CG_C        T_3_CG_B
>   T_4_CG_B         T_1_CG_B        T_2_CG_C
> 
> The CPU_X is totally wasted in exclusive mode, the resource
> efficiency are really poor.
> 
> Thus what we need, is a way to ease confliction in share mode,
> make groups as exclusive as possible, to gain both performance
> and resource efficiency.
> 
> The main idea of group balancer is to fulfill this requirement
> by balancing groups of tasks among groups of CPUs, consider this
> as a dynamic demi-exclusive mode.

Also look at the oracle soft affinity patches

> Just like balance the task among CPUs, now with GB a user can
> put CPU X,Y,Z into three partitions, and balance group A,B,C
> into these partition, to make them as exclusive as possible.
> 
> The design is very likely to the numa balancing, task trigger
> work to settle it's group into a proper partition (minimum
> predicted load), then try migrate itself into it. To gradually
> settle groups into the most exclusively partition.

No words on the interaction between this and numa balancing. Numa
balancing is already a bit tricky because it and the regular load
balancer will have conflicting goals, some of that is mitigated by
teaching the regular balancing about some of that.

I can't help but feel you're making the whole thing look like a 3 body
problem. Also, regular balancing in the face of affinities is already
somewhat dicy. All that needs exploring.

> 
> How To Use:
> 
> To create partition, for example run:
>   echo disable > /proc/gb_ctrl
>   echo "0-15;16-31;32-47;48-63;" > /proc/gb_ctrl
>   echo enable > /proc/gb_ctrl

That's just never going to happen; please look at the cpuset partition
stuff.

> 
> this will create 4 partitions contain CPUs 0-15,16-31,32-47 and
> 48-63 separately.
> 
> Then enable GB for your cgroup, run
>   $CPU_CGROUP_PATH/cpu.gb_period_ms
> 
> And you can check:
>   $CPU_CGROUP_PATH/cpu.gb_stat
> 
> which give output as:
>   PART-0 0-15 1008 1086  *
>   PART-1 16-31 0 2
>   PART-2 32-47 0 0
>   PART-3 48-63 0 1024
> 
> The partition ID followed by it's CPUs range, load of group, load
> of partition and a star mark as preferred.
> 
> Testing Results:
>   In order to enlarge the differences, we do testing on ARM platform
>   with 128 CPUs, create 8 partition according to cluster info.
> 
>   Since we pick benchmark which can gain benefit from exclusive mode,
>   this is more like a functional testing rather than performance, to
>   show that GB help winback the performance.
> 
>   Create 8 cgroup each running 'sysbench memory --threads=16 run',
>   the output of share mode is:
>     events/s (eps):                      4181233.4646
>     events/s (eps):                      3548328.2346
>     events/s (eps):                      4578816.2412
>     events/s (eps):                      4761797.3932
>     events/s (eps):                      3486703.0455
>     events/s (eps):                      3474920.9803
>     events/s (eps):                      3604632.7799
>     events/s (eps):                      3149506.7001
>   the output of gb mode is:
>     events/s (eps):                      5472334.9313
>     events/s (eps):                      4085399.1606
>     events/s (eps):                      4398122.2170
>     events/s (eps):                      6180233.6766
>     events/s (eps):                      4299784.2742
>     events/s (eps):                      4914813.6847
>     events/s (eps):                      3675395.1191
>     events/s (eps):                      6767666.6229
> 
>   Create 4 cgroup each running redis-server with 16 io threads,
>   4 redis-benchmark per each server show average rps as:
> 
>                     share mode       gb mode
> 
>   PING_INLINE     : 41154.84         42229.27               2.61%
>   PING_MBULK      : 43042.07         44907.10               4.33%
>   SET             : 34502.00         37374.58               8.33%
>   GET             : 41713.47         45257.68               8.50%
>   INCR            : 41533.26         44259.31               6.56%
>   LPUSH           : 36541.23         39417.84               7.87%
>   RPUSH           : 39059.26         42075.32               7.72%
>   LPOP            : 36978.73         39903.15               7.91%
>   RPOP            : 39553.32         42071.53               6.37%
>   SADD            : 40614.30         44693.33              10.04%
>   HSET            : 39101.93         42401.16               8.44%
>   SPOP            : 42838.90         46560.46               8.69%
>   ZADD            : 38346.80         41685.46               8.71%
>   ZPOPMIN         : 41952.26         46138.14               9.98%
>   LRANGE_100      : 19364.66         20251.56               4.58%
>   LRANGE_300      : 9699.57          9935.86                2.44%
>   LRANGE_500      : 6291.76          6512.48                3.51%
>   LRANGE_600      : 5619.13          5658.31                0.70%
>   MSET            : 24432.78         26517.63               8.53%
> 
> Signed-off-by: Cruz Zhao <cruzzhao@...ux.alibaba.com>
> Signed-off-by: Tianchen Ding <dtcccc@...ux.alibaba.com>
> Signed-off-by: Michael Wang <yun.wang@...ux.alibaba.com>

Invalid SoB chain.


I'll not really have much time at the moment to look at the code.
Hopefully in a few weeks, but I first need to recover from a 2 week
break and then finish the umcg bits I was working on before that.

Powered by blists - more mailing lists