linux-ext4 - Re: CFQ idling kills I/O performance on ext4 with blkio cgroup controller

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 13 Jun 2019 12:03:59 -0700
From:   "Srivatsa S. Bhat" <srivatsa@...il.mit.edu>
To:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>
Cc:     Jan Kara <jack@...e.cz>, Paolo Valente <paolo.valente@...aro.org>,
        linux-fsdevel@...r.kernel.org,
        linux-block <linux-block@...r.kernel.org>,
        linux-ext4@...r.kernel.org, cgroups@...r.kernel.org,
        kernel list <linux-kernel@...r.kernel.org>,
        Jens Axboe <axboe@...nel.dk>, Jeff Moyer <jmoyer@...hat.com>,
        Theodore Ts'o <tytso@....edu>, amakhalov@...are.com,
        anishs@...are.com, srivatsab@...are.com,
        Ulf Hansson <ulf.hansson@...aro.org>,
        Linus Walleij <linus.walleij@...aro.org>,
        Stable <stable@...r.kernel.org>
Subject: Re: CFQ idling kills I/O performance on ext4 with blkio cgroup
 controller

On 6/12/19 11:02 PM, Greg Kroah-Hartman wrote:
> On Wed, Jun 12, 2019 at 12:36:53PM -0700, Srivatsa S. Bhat wrote:
>>
>> [ Adding Greg to CC ]
>>
>> On 6/12/19 6:04 AM, Jan Kara wrote:
>>> On Tue 11-06-19 15:34:48, Srivatsa S. Bhat wrote:
>>>> On 6/2/19 12:04 AM, Srivatsa S. Bhat wrote:
>>>>> On 5/30/19 3:45 AM, Paolo Valente wrote:
>>>>>>
>>>> [...]
>>>>>> At any rate, since you pointed out that you are interested in
>>>>>> out-of-the-box performance, let me complete the context: in case
>>>>>> low_latency is left set, one gets, in return for this 12% loss,
>>>>>> a) at least 1000% higher responsiveness, e.g., 1000% lower start-up
>>>>>> times of applications under load [1];
>>>>>> b) 500-1000% higher throughput in multi-client server workloads, as I
>>>>>> already pointed out [2].
>>>>>>
>>>>>
>>>>> I'm very happy that you could solve the problem without having to
>>>>> compromise on any of the performance characteristics/features of BFQ!
>>>>>
>>>>>
>>>>>> I'm going to prepare complete patches.  In addition, if ok for you,
>>>>>> I'll report these results on the bug you created.  Then I guess we can
>>>>>> close it.
>>>>>>
>>>>>
>>>>> Sounds great!
>>>>>
>>>>
>>>> Hi Paolo,
>>>>
>>>> Hope you are doing great!
>>>>
>>>> I was wondering if you got a chance to post these patches to LKML for
>>>> review and inclusion... (No hurry, of course!)
>>>>
>>>> Also, since your fixes address the performance issues in BFQ, do you
>>>> have any thoughts on whether they can be adapted to CFQ as well, to
>>>> benefit the older stable kernels that still support CFQ?
>>>
>>> Since CFQ doesn't exist in current upstream kernel anymore, I seriously
>>> doubt you'll be able to get any performance improvements for it in the
>>> stable kernels...
>>>
>>
>> I suspected as much, but that seems unfortunate though. The latest LTS
>> kernel is based on 4.19, which still supports CFQ. It would have been
>> great to have a process to address significant issues on older
>> kernels too.
>>
>> Greg, do you have any thoughts on this? The context is that both CFQ
>> and BFQ I/O schedulers have issues that cause I/O throughput to suffer
>> upto 10x - 30x on certain workloads and system configurations, as
>> reported in [1].
>>
>> In this thread, Paolo posted patches to fix BFQ performance on
>> mainline. However CFQ suffers from the same performance collapse, but
>> CFQ was removed from the kernel in v5.0. So obviously the usual stable
>> backporting path won't work here for several reasons:
>>
>>   1. There won't be a mainline commit to backport from, as CFQ no
>>      longer exists in mainline.
>>
>>   2. This is not a security/stability fix, and is likely to involve
>>      invasive changes.
>>
>> I was wondering if there was a way to address the performance issues
>> in CFQ in the older stable kernels (including the latest LTS 4.19),
>> despite the above constraints, since the performance drop is much too
>> significant. I guess not, but thought I'd ask :-)
> 
> If someone cares about something like this, then I strongly just
> recommend they move to the latest kernel version.  There should not be
> anything stoping them from doing that, right?  Nothing "forces" anyone
> to be on the 4.19.y release, especially when it really starts to show
> its age.
> 
> Don't ever treat the LTS releases as "the only thing someone can run, so
> we must backport huge things to it!"  Just use 5.1, and then move to 5.2
> when it is out and so on.  That's always the preferred way, you always
> get better support, faster kernels, newer features, better hardware
> support, and most importantly, more bugfixes.
> 

Thank you for the clarification!
 
Regards,
Srivatsa
VMware Photon OS