Message-ID: <YjjA+vyQuh5fNeLG@slm.duckdns.org>
Date:   Mon, 21 Mar 2022 08:16:26 -1000
From:   Tejun Heo <tj@...nel.org>
To:     Tianchen Ding <dtcccc@...ux.alibaba.com>
Cc:     Zefan Li <lizefan.x@...edance.com>, Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Juri Lelli <juri.lelli@...hat.com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
        Daniel Bristot de Oliveira <bristot@...hat.com>,
        Johannes Weiner <hannes@...xchg.org>,
        Michael Wang <yun.wang@...ux.alibaba.com>,
        Cruz Zhao <cruzzhao@...ux.alibaba.com>,
        Masahiro Yamada <masahiroy@...nel.org>,
        Nathan Chancellor <nathan@...nel.org>,
        Kees Cook <keescook@...omium.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Vlastimil Babka <vbabka@...e.cz>,
        "Gustavo A. R. Silva" <gustavoars@...nel.org>,
        Arnd Bergmann <arnd@...db.de>, Miguel Ojeda <ojeda@...nel.org>,
        Chris Down <chris@...isdown.name>,
        Vipin Sharma <vipinsh@...gle.com>,
        Daniel Borkmann <daniel@...earbox.net>,
        linux-kernel@...r.kernel.org, cgroups@...r.kernel.org
Subject: Re: [RFC PATCH v2 0/4] Introduce group balancer

Hello,

On Thu, Mar 10, 2022 at 01:47:34PM +0800, Tianchen Ding wrote:
> If we want to build group balancer in userspace, we need:
>   1) gather load info from each rq periodically
>   2) make decision to set cpuset.cpus of each cgroup
> 
> However, there're some problems about this way.
> 
> For 1), we need to consider how frequently collecting these info, which may
> impact performance and accuracy. If the load changes hugely right after we
> get it once, our data are expired and then the decision may be wrong. (If we
> are in kernel, faster action can be taken.)

We now have a pretty well established way to transport data to userspace at
really low overhead. If you piggy back on bpf interfaces, they can usually
be pretty unintrusive and low effort as long as you have the right kind of
data aggregated already, which shouldn't be that difficult here.

> We believe 2) is harder. The specific policy may be complex and alter
> according to different scenes. There's not a general method.
> e.g., with 16cpus and 4 cgroups, how to decide when we set one of them
> 0-3(when busy)or 0-7(when some of other cgroups are idle)? If there are much
> more threads in cgroupA than cgroupB/C/D , and we want to satisfy cgroupA as
> far as possible(on the premise of fairness of B/C/D)， dynamically
> enlarging(when B/C/D partly idle) and shrinking(when B/C/D busy) cpuset of
> cgroupA requires complex policy. In this example, fairness and performance
> can be provided by existing scheduler, but when it comes to grouping hot
> cache or decreasing competion, both scheduler in kernel and action in
> userspace are hard to solve.

So, I get that it's not easy. In fact, we don't even yet know how to
properly compare loads across groups of CPUs - simple sums that you're using
break down when there are big gaps in weight numbers across tasks and can
become meaningless in the presence of CPU affinities. They can still work
when the configuration is fairly homogeenous and controlled but the standard
should be far higher for something we bake into the kernel and expose
userland-visible interface for.

> What's more, in many cloud computing scenes, there may be hundreds or
> thousands of containers, which are much larger than partition number. These
> containers may be dynamically created and destroyed at any time. Making
> policy to manage them from userspace will not be practical.
> 
> These problems become easy when going to kernelspace. We get info directly
> from scheduler, and help revising its decision at some key points, or do
> some support work(e.g., task migration if possible).

I don't think they become necessarily easy. Sure, you can hack up something
which works for some cases by poking into existing code; however, the bar
for acceptance is also way higher for a kernel interface - it should be
generic, consistent with other interfaces (I won't go into cgroup interface
issues here), and work orthogonally with other kernel features (ie. task /
group weights should work in an explainable way). I don't think the proposed
patches are scoring high in those axes.

I'm not against the goal here. Given that cgroups express the logical
structure of applications running on the system, it does make sense to
factor that into scheduling decisions. However, what's proposed seems too
premature and I have a hard time seeing why this level of functionality
would be difficult to be implement from userspace with some additions in
terms of visibility which is really easy to do these days.

Thanks.

-- 
tejun

Powered by blists - more mailing lists