Message-ID: <20160407194555.GI7822@mtj.duckdns.org>
Date:	Thu, 7 Apr 2016 15:45:55 -0400
From:	Tejun Heo <tj@...nel.org>
To:	Peter Zijlstra <peterz@...radead.org>
Cc:	Johannes Weiner <hannes@...xchg.org>,
	torvalds@...ux-foundation.org, akpm@...ux-foundation.org,
	mingo@...hat.com, lizefan@...wei.com, pjt@...gle.com,
	linux-kernel@...r.kernel.org, cgroups@...r.kernel.org,
	linux-api@...r.kernel.org, kernel-team@...com
Subject: Re: [PATCHSET RFC cgroup/for-4.6] cgroup, sched: implement resource
 group and PRIO_RGRP

Hello, Peter.

On Thu, Apr 07, 2016 at 10:08:33AM +0200, Peter Zijlstra wrote:
> On Thu, Apr 07, 2016 at 03:35:47AM -0400, Johannes Weiner wrote:
> > So it was a nice cleanup for the memory controller and I believe the
> > IO controller as well. I'd be curious how it'd be a problem for CPU?
> 
> The full hierarchy took years to make work and is fully ingrained with
> how the thing words, changing it isn't going to be nice or easy.
> 
> So sure, go with a lowest common denominator, instead of fixing shit,
> yay for progress :/

It's easy to get fixated on what each subsystem can do and develop
towards different directions siloed in each subsystem.  That's what
we've had for quite a while in cgroup.  Expectedly, this sends off
controllers towards different directions.  Direct competion between
tasks and child cgroups was one of the main sources of balkanization.

The balkanization was no coincidence either.  Tasks and cgroups are
different types of entities and don't have the same control knobs or
follow the same lifetime rules.  For absolute limits, it isn't clear
how much of the parent's resources should be distributed to internal
children as opposed to child cgroups.  People end up depending on
specific implementation details and proposing one-off hacks and
interface additions.

Proportional weights aren't much better either.  CPU has internal
mapping between nice values and shares and treat them equally, which
can get confusing as the configured weights behave differently
depending on how many threads are in the parent cgroup which often is
opaque and can't be controlled from outside.  Widely diverging from
CPU's behavior, IO grouped all internal tasks into an internal leaf
node and used to assign a fixed weight to it.

Now, you might think that none of it matters and each subsystem
treating cgroup hierarchy as arbitrary and orthogonal collections of
bean counters is fine; however, that makes it impossible to account
for and control operations which span different types of resources.
This prevented us from implementing resource control over frigging
buffered writes, making the whole IO control thing a joke.  While CPU
currently doesn't directly tie into it, that is only because CPU
cycles spent during writeback isn't yet properly accounted.

The structural constraints and resulting consistency don't just
subtract from the abilities of each controller.  It establishes a
common base, the shared resource domains and consistent behaviors on
top of them, that further capabilities can be built upon, capabilities
as fundamental as comprehensive resource control over buffered
writeback.

It can be convenient to have subsystem-specific raw bean counters.  If
that's what the use case calls for, individual controllers can easily
be moved to a separate hierarchy although it would naturally lose the
capabilities coming from cooperating over shared resource domains.
However, please understand that there are a lot of use cases where
comprehensive and consistent resource accounting and control over all
major resources is useful and necessary.

Thanks.

-- 
tejun

Powered by blists - more mailing lists