[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAKfTPtA7wJBROfRkSQV7FzWaWqoaQjSO7iyqBt6AgGsv2OsNSw@mail.gmail.com>
Date: Fri, 24 Jun 2022 10:22:38 +0200
From: Vincent Guittot <vincent.guittot@...aro.org>
To: David Chen <david.chen@...anix.com>
Cc: "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
Ingo Molnar <mingo@...hat.com>
Subject: Re: Perf regression from scheduler load_balance rework in 5.5?
On Thu, 23 Jun 2022 at 21:50, David Chen <david.chen@...anix.com> wrote:
>
> Hi,
>
> I'm working on upgrading our kernel from 4.14 to 5.10
> However, I'm seeing performance regression when doing rand read from windows client through smbd
> with a well cached file.
>
> One thing I noticed is that on the new kernel, the smbd thread doing socket I/O tends to stay on
> the same cpu core as the net_rx softirq, where as in the old kernel it tends to be moved around
> more randomly. And when they are on the same cpu, it tends to saturate the cpu more and causes
> performance to drop.
>
> For example, here's the duration (ns) the thread spend on each cpu I captured using bpftrace
> On 4.14:
> @cputime[7]: 20741458382
> @cputime[0]: 25219285005
> @cputime[6]: 30892418441
> @cputime[5]: 31032404613
> @cputime[3]: 33511324691
> @cputime[1]: 35564174562
> @cputime[4]: 39313421965
> @cputime[2]: 55779811909 (net_rx cpu)
>
> On 5.10:
> @cputime[3]: 2150554823
> @cputime[5]: 3294276626
> @cputime[7]: 4277890448
> @cputime[4]: 5094586003
> @cputime[1]: 6058168291
> @cputime[0]: 14688093441
> @cputime[6]: 17578229533
> @cputime[2]: 223473400411 (net_rx cpu)
>
> I also tried setting the cpu affinity of the smbd thread away from the net_rx cpu and indeed that
> seems to bring the perf on par with old kernel.
>
> I noticed that there's scheduler load_balance rework in 5.5, so I did the test on 5.4 and 5.5 and
> it did show the behavior changed between 5.4 and 5.5.
Have you tested v5.18 ? several improvements happened since v5.5
>
> Anyone know how to work around this?
Have you enabled IRQ_TIME_ACCOUNTING ?
When the time spent under interrupt becomes significant, scheduler
migrate task on another cpu
Vincent
>
> Thanks,
> David
Powered by blists - more mailing lists