Date: Thu, 20 Jun 2024 04:35:08 +0200
From: Thomas Gleixner <tglx@...utronix.de>
To: Linus Torvalds <torvalds@...ux-foundation.org>
Cc: Tejun Heo <tj@...nel.org>, mingo@...hat.com, peterz@...radead.org,
 juri.lelli@...hat.com, vincent.guittot@...aro.org,
 dietmar.eggemann@....com, rostedt@...dmis.org, bsegall@...gle.com,
 mgorman@...e.de, bristot@...hat.com, vschneid@...hat.com, ast@...nel.org,
 daniel@...earbox.net, andrii@...nel.org, martin.lau@...nel.org,
 joshdon@...gle.com, brho@...gle.com, pjt@...gle.com, derkling@...gle.com,
 haoluo@...gle.com, dvernet@...a.com, dschatzberg@...a.com,
 dskarlat@...cmu.edu, riel@...riel.com, changwoo@...lia.com,
 himadrics@...ia.fr, memxor@...il.com, andrea.righi@...onical.com,
 joel@...lfernandes.org, linux-kernel@...r.kernel.org, bpf@...r.kernel.org,
 kernel-team@...a.com
Subject: Re: [PATCHSET v6] sched: Implement BPF extensible scheduler class

Linus!

On Wed, Jun 19 2024 at 15:55, Linus Torvalds wrote:
> On Wed, 19 Jun 2024 at 15:27, Thomas Gleixner <tglx@...utronix.de> wrote:
>> Right, but that applies to both sides, no?
>
> But Thomas, this isn't a "both sides" issue.

It is whether you like it or not.

I clearly gave the sched_ext people deep technical feedback on the way
how they integrated this seven months ago in Richmond.

I sat down with them because _you_ asked me explicitely for it as _you_
did not want to deal with it.

That feedback got ignored completely. Thank you very much for wasting
_my_ time.

> This is a "people want to do new code and features, and the scheduler
> people ARE ACTIVELY HOLDING IT UP" issue.
>
> Yes, part of that "actively holding it up" is trying to make rules for
> "you need to do this other XYZ thing to make us happy".
>
> But no, then "not doing XYZ" does *NOT* make it some "but but other side" issue.
>
> This, btw, is not some new thing. It's something that has been
> discussed multiple times over the years at the maintainer summit for
> different maintainers. When people come in and propose feature X, it's
> not kosher to then say "you have to do Y first".

Seriously? You yourself requested this from me more than once in the
past 20 years. But let's not go there.

> And yes, maybe everybody even agrees that Y would be a good thing, and
> yes, wouldn't it be lovely if somebody did it. But the people who
> wanted X didn't care about Y, and trying to get Y done by then gating
> X is simply not ok.

I very well remember the discussions about fix X first before Y.

But you are completely ignoring the fact that in this case

    - problem X got introduced by the very same people who are now
      pushing for Y

    - the resolution to problem X was not rejected by the scheduler
      people at all

      It stayed unresolved because the very same parties which are now
      so cosily working together on Y (aka sched_ext0 could not agree on
      anything

      So the mess they created in the first place stays as is and both
      parties "solved" their problem with Y (aka sched_ext) and have
      their own implementations to work around it how they see fit.

IOW, what you are saying is:

    #1 Push for your solution X and get it merged

    #2 Ignore the resulting problems related to X for at least a decade

    #3 Come up with a half baken workaround Y to be able to deal with #2
       and leave everyone else behind to deal with the results of #1 and
       #2

    #4 Make Linus decide that Y is the right thing to do because other
       people suck

The past discussions about fix "X" before "Y" were distinctly different.

> Now, if there was some technical argument against X itself, that would
> be one thing. But the arguments I've heard have basically fallen into
> two camps: the political one ("We don't want to do X because we simply
> don't want an extensible scheduler, because we want people to work on
> _our_ scheduler") and the tying one ("X is ok but we want Y solved
> first").

There have been voiced a lot of technical arguments, which never got
addressed and at some point people gave up due to being ignored.

When I sat there in Richmond with the sched_ext people I gave them very
deep technical feedback especially on the way how they integrate it:

  Sprinkle hooks and callbacks all over the place until it works by some
  definition of works.

That's perfectly fine for a PoC, but not for something which gets merged
into the core of an OS. I clearly asked them to refactor the existing
code so that these warts go away and everything is contained into the
scheduler classes and at the very end sched_ext falls into place. That's
a basic engineering principle as far as I know.

They nodded, ignored my feedback and just continued to persue their way.

Are you still claiming with a straight face that this is a problem
rooted in one party?

It's a problem of one interest group to get this into the tree no matter
what.

I sat in the room at OSPM 2023 when the introduction of the sched ext
advertisement started with "This saves us (FB) millions and Google is
collaborating with us [unspoken - to save millions too by eliminating
their unmaintainable scheduler hacks]', which is clearly a technical
argument, right?

> I was hoping the tying argument would get solved. I saw a couple of
> half-hearted emails to that effect, and Rik at some point saying
> "maybe the problems are solvable", referring to his work from a couple
> of years ago, but again, nothing actually happened.

See above. It's not a problem of the scheduler people.

It's a problem created by the very same people who refuse to solve it
and at the same time push for their new magic cure.

> And I don't see the argument that the way to make something happen is
> to continue to do nothing.

I clearly offered you to try to resolve this amicably within a
reasonable time frame.

How exaclty is that equivalent to "continue to do nothing" ?

> Because if you are serious about making forward progress *with* the
> BPF extensions, why not merge them and actually work with that as the
> base?

There is a very simple argument to this:

   - The way how it is integrated sucks in a big way on purely technical
     grounds. If you want details, I'm happy to reiterate them again in
     a separate mail just in case you can't find them in your inbox or
     can't remember what I told you seven month ago.

   - This got pointed out by myself seven months ago to the sched_ext
     people

   - They sat with me at the table and nodded

   - Then they went off and ignored it for seven months and just added
     more warts.

Now you are asking me seriously why I don't want to see this merged in
the technical state which it is in right now?

Are you seriously expecting that this is resolved amicably just by
forcing this into the tree on the obvious expense of the people who were
working on the existing code for decades?

That aside, your whole argument about the so "very high" participation
barrier on the scheduler is just based on hearsay and not on facts,
which you could easily have gathered yourself. Let me do your homework.

In the past five years there have been:

   ~4000 mail (patch) threads related to scheduler issues
   ~1600 commits (i.e. 1.6 commits per work day on average)
    ~300 different authors

300 different authors is clearly not a sign of a problematic community
neither is the ratio of commits to mail threads is 1:2.5.

The fact tell clearly that this is a healthy community and not a sign of
a participation barrier problem.

The fact, that the contributions and contribution attempts from the
proponents of sched_ext are close to zero cannot be abused to claim that
there is a high bar to get patches into the scheduler subsystem.

If you don't try in the first place then you can't complain about it,
no?

Just for the record, the scheduler people and myself spent a lot of time
to help to get intrusive features like UMCG into mainline, but the
efforts were dropped by the submitters for no reason. Short time after
that sched_ext came around.

Can we please agree that the root of this has left the technical grounds
long ago?

If you think that the correct non-technical solution is to resolve this
brute force without giving those who are willing to work this out in the
proper way a completely irrelevant delay of three month, then I really
have to ask you what you are trying to achive.

If you pull that stuff as is then you create a patently bad precedent
and on top of that you slap everyone who worked and works
collaboratively and constructively with maintainers and the wider
community to get their features merged straight into their face.

You are obviously free to do so, but then please clearly state that this
is the new world order by merging an unreviewed patch against
Documentation/process/* which makes this as a general rule applicable
for everyone.

We've been there and done that during the 2.5 period and it took us
years to recover from it, but maybe you have forgotten about that
because you merely had to merge the fixes which were created by people
who cared and lost their sleep over the mess.

Whatever the outcome is, we definitely have to have a major discussion
about the underlying problem at the maintainer summit whether you like
it or not.

That said, my offer stands to work on an amicable and collaborative
solution for this nuisance, but that's obviously all I can do.

Up to you.

Thanks,

        Thomas

Powered by blists - more mailing lists