| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <Yd6Xlw1qvEbWFSwU@hirez.programming.kicks-ass.net>
Date: Wed, 12 Jan 2022 09:55:51 +0100
From: Peter Zijlstra <peterz@...radead.org>
To: 王贇 <yun.wang@...ux.alibaba.com>
Cc: Ingo Molnar <mingo@...hat.com>, Juri Lelli <juri.lelli@...hat.com>,
Vincent Guittot <vincent.guittot@...aro.org>,
Dietmar Eggemann <dietmar.eggemann@....com>,
Steven Rostedt <rostedt@...dmis.org>,
Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
Daniel Bristot de Oliveira <bristot@...hat.com>,
Masahiro Yamada <masahiroy@...nel.org>,
Andrew Morton <akpm@...ux-foundation.org>,
Nathan Chancellor <nathan@...nel.org>,
Kees Cook <keescook@...omium.org>,
Vlastimil Babka <vbabka@...e.cz>,
Chris Down <chris@...isdown.name>,
Vipin Sharma <vipinsh@...gle.com>,
Rasmus Villemoes <linux@...musvillemoes.dk>,
Daniel Borkmann <daniel@...earbox.net>,
Arnd Bergmann <arnd@...db.de>,
Cruz Zhao <cruzzhao@...ux.alibaba.com>,
Tianchen Ding <dtcccc@...ux.alibaba.com>,
linux-kernel@...r.kernel.org
Subject: Re: [RFC PATCH] sched: introduce group balancer
On Tue, Jan 04, 2022 at 03:33:57PM +0800, 王贇 wrote:
> Modern platform are growing fast on CPU numbers, multiple
> apps sharing one box are very common, they used to have
> exclusive cpu setting but nowadays things are changing.
>
> To achieve better utility of CPU resource, multiple apps
> are starting to sharing the CPUs. The CPU resources usually
> overcommitted since app's workload are undulated.
>
> This introduced problems on performance when share mode vs
> exclusive mode, for eg with cgroup A,B and C deployed in
> exclusive mode, it will be:
>
> CPU_X (100%) CPU_Y (100%) CPU_Z (50%)
> T_1_CG_A T_1_CG_B T_1_CG_C
> T_2_CG_A T_2_CG_B T_2_CG_C
> T_3_CG_A T_3_CG_B
> T_4_CG_A T_4_CG_B
>
> while the share mode will be:
>
> CPU_X (100%) CPU_Y (75%) CPU_Z (75%)
> T_1_CG_A T_2_CG_A T_1_CG_B
> T_2_CG_B T_3_CG_B T_2_CG_C
> T_4_CG_B T_4_CG_A T_3_CG_A
> T_1_CG_C
>
> As we can see, the confliction between groups on CPU
> resources are now happening all over the CPUs.
>
> The testing on sysbench-memory show 30+% drop on share
> mode, and redis-benchmark show 10+% drop too, compared
> to the exclusive mode.
>
> However, despite of the performance drop, in real world
> we still prefer share mode. The undulated workload can
> make the exclusive mode so unefficient on CPU utilization,
> for eg the next period, when CG_A become 'idle', exclusive
> mode will like:
>
> CPU_X (0%) CPU_Y (100%) CPU_Z (50%)
> T_1_CG_B T_1_CG_C
> T_2_CG_B T_2_CG_C
> T_3_CG_B
> T_4_CG_B
>
> while share mode like:
>
> CPU_X (50%) CPU_Y (50%) CPU_Z (50%)
> T_2_CG_B T_1_CG_C T_3_CG_B
> T_4_CG_B T_1_CG_B T_2_CG_C
>
> The CPU_X is totally wasted in exclusive mode, the resource
> efficiency are really poor.
>
> Thus what we need, is a way to ease confliction in share mode,
> make groups as exclusive as possible, to gain both performance
> and resource efficiency.
>
> The main idea of group balancer is to fulfill this requirement
> by balancing groups of tasks among groups of CPUs, consider this
> as a dynamic demi-exclusive mode.
Also look at the oracle soft affinity patches
> Just like balance the task among CPUs, now with GB a user can
> put CPU X,Y,Z into three partitions, and balance group A,B,C
> into these partition, to make them as exclusive as possible.
>
> The design is very likely to the numa balancing, task trigger
> work to settle it's group into a proper partition (minimum
> predicted load), then try migrate itself into it. To gradually
> settle groups into the most exclusively partition.
No words on the interaction between this and numa balancing. Numa
balancing is already a bit tricky because it and the regular load
balancer will have conflicting goals, some of that is mitigated by
teaching the regular balancing about some of that.
I can't help but feel you're making the whole thing look like a 3 body
problem. Also, regular balancing in the face of affinities is already
somewhat dicy. All that needs exploring.
>
> How To Use:
>
> To create partition, for example run:
> echo disable > /proc/gb_ctrl
> echo "0-15;16-31;32-47;48-63;" > /proc/gb_ctrl
> echo enable > /proc/gb_ctrl
That's just never going to happen; please look at the cpuset partition
stuff.
>
> this will create 4 partitions contain CPUs 0-15,16-31,32-47 and
> 48-63 separately.
>
> Then enable GB for your cgroup, run
> $CPU_CGROUP_PATH/cpu.gb_period_ms
>
> And you can check:
> $CPU_CGROUP_PATH/cpu.gb_stat
>
> which give output as:
> PART-0 0-15 1008 1086 *
> PART-1 16-31 0 2
> PART-2 32-47 0 0
> PART-3 48-63 0 1024
>
> The partition ID followed by it's CPUs range, load of group, load
> of partition and a star mark as preferred.
>
> Testing Results:
> In order to enlarge the differences, we do testing on ARM platform
> with 128 CPUs, create 8 partition according to cluster info.
>
> Since we pick benchmark which can gain benefit from exclusive mode,
> this is more like a functional testing rather than performance, to
> show that GB help winback the performance.
>
> Create 8 cgroup each running 'sysbench memory --threads=16 run',
> the output of share mode is:
> events/s (eps): 4181233.4646
> events/s (eps): 3548328.2346
> events/s (eps): 4578816.2412
> events/s (eps): 4761797.3932
> events/s (eps): 3486703.0455
> events/s (eps): 3474920.9803
> events/s (eps): 3604632.7799
> events/s (eps): 3149506.7001
> the output of gb mode is:
> events/s (eps): 5472334.9313
> events/s (eps): 4085399.1606
> events/s (eps): 4398122.2170
> events/s (eps): 6180233.6766
> events/s (eps): 4299784.2742
> events/s (eps): 4914813.6847
> events/s (eps): 3675395.1191
> events/s (eps): 6767666.6229
>
> Create 4 cgroup each running redis-server with 16 io threads,
> 4 redis-benchmark per each server show average rps as:
>
> share mode gb mode
>
> PING_INLINE : 41154.84 42229.27 2.61%
> PING_MBULK : 43042.07 44907.10 4.33%
> SET : 34502.00 37374.58 8.33%
> GET : 41713.47 45257.68 8.50%
> INCR : 41533.26 44259.31 6.56%
> LPUSH : 36541.23 39417.84 7.87%
> RPUSH : 39059.26 42075.32 7.72%
> LPOP : 36978.73 39903.15 7.91%
> RPOP : 39553.32 42071.53 6.37%
> SADD : 40614.30 44693.33 10.04%
> HSET : 39101.93 42401.16 8.44%
> SPOP : 42838.90 46560.46 8.69%
> ZADD : 38346.80 41685.46 8.71%
> ZPOPMIN : 41952.26 46138.14 9.98%
> LRANGE_100 : 19364.66 20251.56 4.58%
> LRANGE_300 : 9699.57 9935.86 2.44%
> LRANGE_500 : 6291.76 6512.48 3.51%
> LRANGE_600 : 5619.13 5658.31 0.70%
> MSET : 24432.78 26517.63 8.53%
>
> Signed-off-by: Cruz Zhao <cruzzhao@...ux.alibaba.com>
> Signed-off-by: Tianchen Ding <dtcccc@...ux.alibaba.com>
> Signed-off-by: Michael Wang <yun.wang@...ux.alibaba.com>
Invalid SoB chain.
I'll not really have much time at the moment to look at the code.
Hopefully in a few weeks, but I first need to recover from a 2 week
break and then finish the umcg bits I was working on before that.
Powered by blists - more mailing lists