Date:   Fri, 11 Dec 2020 10:08:09 -0800
From:   Yury Norov <yury.norov@...il.com>
To:     Thomas Gleixner <tglx@...utronix.de>
Cc:     Pavel Machek <pavel@....cz>, Alex Belits <abelits@...vell.com>,
        "nitesh@...hat.com" <nitesh@...hat.com>,
        "frederic@...nel.org" <frederic@...nel.org>,
        Prasun Kapoor <pkapoor@...vell.com>,
        "linux-api@...r.kernel.org" <linux-api@...r.kernel.org>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "trix@...hat.com" <trix@...hat.com>,
        "mingo@...nel.org" <mingo@...nel.org>,
        "catalin.marinas@....com" <catalin.marinas@....com>,
        "rostedt@...dmis.org" <rostedt@...dmis.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "peterx@...hat.com" <peterx@...hat.com>,
        "linux-arch@...r.kernel.org" <linux-arch@...r.kernel.org>,
        "mtosatti@...hat.com" <mtosatti@...hat.com>,
        "will@...nel.org" <will@...nel.org>,
        "peterz@...radead.org" <peterz@...radead.org>,
        "leon@...ebranch.com" <leon@...ebranch.com>,
        "linux-arm-kernel@...ts.infradead.org" 
        <linux-arm-kernel@...ts.infradead.org>,
        "pauld@...hat.com" <pauld@...hat.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: [PATCH v5 0/9] "Task_isolation" mode

On Sun, Dec 06, 2020 at 12:25:45AM +0100, Thomas Gleixner wrote:
> Pavel,
> 
> On Sat, Dec 05 2020 at 21:40, Pavel Machek wrote:
> > So... what kind of guarantees does this aim to provide / what tasks it
> > is useful for?
> >
> > For real time response, we have other approaches.
> 
> Depends on your requirements. Some problems are actually better solved
> with busy polling. See below.
> 
> > If you want to guarantee performnace of the "isolated" task... I don't
> > see how that works. Other tasks on the system still compete for DRAM
> > bandwidth, caches, etc...
> 
> Applications which want to run as undisturbed as possible. There is
> quite a range of those:
> 
>   - Hardware in the loop simulation is today often done with that crude
>     approach of "offlining" a CPU and then instead of playing dead
>     jumping to a preloaded bare metal executable. That's a horrible hack
>     and impossible to debug, but gives them the results they need to
>     achieve. These applications are well optimized vs. cache and memory
>     foot print, so they don't worry about these things too much and they
>     surely don't run on SMI and BIOS value add inflicted machines.
> 
>     Don't even think about waiting for an interrupt to achieve what
>     these folks are doing. So no, there are problems which a general
>     purpose realtime OS cannot solve ever.
> 
>   - HPC computations on large data sets. While the memory foot print is
>     large the access patterns are cache optimized. 
> 
>     The problem there is that any unnecessary IPI, tick interrupt or
>     whatever nuisance is disturbing the carefully optimized cache usage
>     and alone getting rid of the timer interrupt gained them measurable
>     performance. Even very low single digit percentage of runtime saving
>     is valuable for these folks because the compute time on such beasts
>     is expensive.
> 
>   - Realtime guests in KVM. With posted interrupts and a fully populated
>     host side page table there is no point in running host side
>     interrupts or IPIs for random accounting or whatever purposes as
>     they affect the latency in the guest. With all the side effects
>     mitigated and a properly set up guest and host it is possible to get
>     to a zero exit situation after the bootup phase which means pretty
>     much matching bare metal behaviour.
> 
>     Yes, you can do that with e.g. Jailhouse as well, but you lose lots
>     of the fancy things KVM provides. And people care about these not
>     just because they are fancy. They care because their application
>     scenario needs them.
> 
> There are more reasons why people want to be able to get as much
> isolation from the OS as possible but at the same time have a sane
> execution environment, debugging, performance monitoring and the OS
> provided protection mechanisms instead of horrible hacks.
> 
> Isolation makes sense for a range of applications and there is no reason
> why Linux should not support them. 

One good client for the task isolation is Open Data Plane. There are
even some code stubs supposed to enable isolation where needed.

> > If you want to guarantee performnace of the "isolated" task... I don't
> > see how that works. Other tasks on the system still compete for DRAM
> > bandwidth, caches, etc...

My experiments say that typical delay caused by dry IPI or syscall is
2000-20000 'ticks'. Typical delay caused by cache miss is 3-30 ticks.

To guarantee cache / memory bandwidth, one can use resctrl. Linux has
implementation of it for x86 only, but arm64 has support for for
resctrl on CPU side.

Thanks,
Yury

> Thanks,
> 
>         tglx

Powered by blists - more mailing lists