linux-kernel - Re: [REGRESSION] 5-10% increase in IO latencies with nohz balance patch

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <878rx6bia5.mognet@arm.com>
Date:   Tue, 30 Nov 2021 00:26:26 +0000
From:   Valentin Schneider <valentin.schneider@....com>
To:     Josef Bacik <josef@...icpanda.com>
Cc:     peterz@...radead.org, vincent.guittot@...aro.org,
        torvalds@...ux-foundation.org, linux-kernel@...r.kernel.org,
        linux-btrfs@...r.kernel.org
Subject: Re: [REGRESSION] 5-10% increase in IO latencies with nohz balance patch

On 29/11/21 14:49, Josef Bacik wrote:
> On Mon, Nov 29, 2021 at 06:31:17PM +0000, Valentin Schneider wrote:
>> On 29/11/21 13:15, Josef Bacik wrote:
>> > On Mon, Nov 29, 2021 at 06:03:24PM +0000, Valentin Schneider wrote:
>> >> Would you happen to have execution traces by any chance? If not I should be
>> >> able to get one out of that fsperf thingie.
>> >>
>> >
>> > I don't, if you want to tell me how I can do it right now.  I've disabled
>> > everything on this box for now so it's literally just sitting there waiting to
>> > have things done to it.  Thanks,
>> >
>>
>> I see you have Ftrace enabled in your config, so that ought to do it:
>>
>>   trace-cmd record -e 'sched:*' -e 'cpu_idle' $your_test_cmd
>>
>
> http://toxicpanda.com/performance/trace.dat
>
> it's like 16mib.  Enjoy,
>

Neat, thanks!

Runqueue depth seems to be very rarely greater than 1, tasks with ~1ms
runtime and lots of sleeping (also bursty kworker activity with activations
of tens of µs), and some cores (Internet tells me that Xeon Bronze 3204
doesn't have SMT) spend most of their time idling. Not the most apocalyptic
task placement vs ILB selection, but the task activation patterns roughly
look like what I was thinking of - there might be hope for me yet.

I'll continue the headscratching after tomorrow's round of thinking juice.

> Josef