linux-kernel - Re: [RFC PATCH v3 00/16] Core scheduling v3

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <69cd9bca-da28-1d35-3913-1efefe0c1c22@linux.intel.com>
Date:   Wed, 7 Aug 2019 10:10:47 -0700
From:   Tim Chen <tim.c.chen@...ux.intel.com>
To:     Dario Faggioli <dfaggioli@...e.com>,
        Julien Desfossez <jdesfossez@...italocean.com>,
        "Li, Aubrey" <aubrey.li@...ux.intel.com>
Cc:     Aaron Lu <aaron.lu@...ux.alibaba.com>,
        Aubrey Li <aubrey.intel@...il.com>,
        Subhra Mazumdar <subhra.mazumdar@...cle.com>,
        Vineeth Remanan Pillai <vpillai@...italocean.com>,
        Nishanth Aravamudan <naravamudan@...italocean.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...nel.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Paul Turner <pjt@...gle.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Linux List Kernel Mailing <linux-kernel@...r.kernel.org>,
        Frédéric Weisbecker <fweisbec@...il.com>,
        Kees Cook <keescook@...omium.org>,
        Greg Kerr <kerrnel@...gle.com>, Phil Auld <pauld@...hat.com>,
        Valentin Schneider <valentin.schneider@....com>,
        Mel Gorman <mgorman@...hsingularity.net>,
        Pawan Gupta <pawan.kumar.gupta@...ux.intel.com>,
        Paolo Bonzini <pbonzini@...hat.com>
Subject: Re: [RFC PATCH v3 00/16] Core scheduling v3

On 8/7/19 1:58 AM, Dario Faggioli wrote:

> So, here comes my question: I've done a benchmarking campaign (yes,
> I'll post numbers soon) using this branch:
> 
> https://github.com/digitalocean/linux-coresched.git  vpillai/coresched-v3-v5.1.5-test
> https://github.com/digitalocean/linux-coresched/tree/vpillai/coresched-v3-v5.1.5-test
> 
> Last commit:
> 7feb1007f274 "Fix stalling of untagged processes competing with tagged
> processes"
> 
> Since I see that, in this thread, there are various patches being
> proposed and discussed... should I rerun my benchmarks with them
> applied? If yes, which ones? And is there, by any chance, one (or maybe
> more than one) updated git branch(es)?
> 
> Thanks in advance and Regards
> 

Hi Dario,

Having an extra set of eyes are certainly welcomed.
I'll give my 2 cents on the issues with v3.
Others feel free to chime in if my understanding is
incorrect or I'm missing something.

1) Unfairness between the sibling threads
-----------------------------------------
One sibling thread could be suppressing and force idling
the sibling thread over proportionally.  Resulting in
the force idled CPU not getting run and stall tasks on
suppressed CPU.

Status:
i) Aaron has proposed a patchset here based on using one
rq as a base reference for vruntime for task priority
comparison between siblings.

https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
It works well on fairness but has some initialization issues

ii) Tim has proposed a patchset here to account for forced
idle time in rq's min_vruntime
https://lore.kernel.org/lkml/f96350c1-25a9-0564-ff46-6658e96d726c@linux.intel.com/
It improves over v3 with simpler logic compared to
Aaron's patch, but does not work as well on fairness

iii) Tim has proposed yet another patch to maintain fairness
of forced idle time between CPU threads per Peter's suggestion.
https://lore.kernel.org/lkml/21933a50-f796-3d28-664c-030cb7c98431@linux.intel.com/
Its performance has yet to be tested.

2) Not rescheduling forced idled CPU
------------------------------------
The forced idled CPU does not get a chance to re-schedule
itself, and will stall for a long time even though it
has eligible tasks to run.

Status:
i) Aaron proposed a patch to fix this to check if there
are runnable tasks when scheduling tick comes in.
https://lore.kernel.org/lkml/20190725143344.GD992@aaronlu/

ii) Vineeth has patches to this issue and also issue 1, based
on scheduling in a new "forced idle task" when getting forced
idle, but has yet to post the patches.

3) Load balancing between CPU cores
-----------------------------------
Say if one CPU core's sibling threads get forced idled
a lot as it has mostly incompatible tasks between the siblings,
moving the incompatible load to other cores and pulling
compatible load to the core could help CPU utilization.

So just considering the load of a task is not enough during
load balancing, task compatibility also needs to be considered.
Peter has put in mechanisms to balance compatible tasks between
CPU thread siblings, but not across cores.

Status:
I have not seen patches on this issue.  This issue could lead to
large variance in workload performance based on your luck
in placing the workload among the cores.

Thanks.

Tim