linux-kernel - Re: 6.10/regression/bisected - after f1d97e769152 I spotted increased execution time of the kswapd0 process and symptoms as if there is not enough memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAL3q7H5RC6dinsA2KLtus07jxDuY1PecPXbhYOWtW+nVyzXwuA@mail.gmail.com>
Date: Wed, 3 Jul 2024 11:31:27 +0100
From: Filipe Manana <fdmanana@...nel.org>
To: Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>
Cc: Linux List Kernel Mailing <linux-kernel@...r.kernel.org>, 
	Linux regressions mailing list <regressions@...ts.linux.dev>, Btrfs BTRFS <linux-btrfs@...r.kernel.org>, 
	dsterba@...e.com, josef@...icpanda.com
Subject: Re: 6.10/regression/bisected - after f1d97e769152 I spotted increased
 execution time of the kswapd0 process and symptoms as if there is not enough memory

On Tue, Jul 2, 2024 at 6:22 PM Filipe Manana <fdmanana@...nel.org> wrote:
>
> On Tue, Jul 2, 2024 at 3:13 PM Mikhail Gavrilov
> <mikhail.v.gavrilov@...il.com> wrote:
> >
> > On Mon, Jul 1, 2024 at 2:31 PM Filipe Manana <fdmanana@...nel.org> wrote:
> > >
> > > Try this:
> > >
> > > https://lore.kernel.org/linux-btrfs/cb12212b9c599817507f3978c9102767267625b2.1719825714.git.fdmanana@suse.com/
> > >
> > > That applies only to the "for-next", it will need conflict resolution
> > > for 6.10-rc, as noted in the commnets.
> > > For a version that cleanly applies to 6.10-rc:
> > >
> > > https://gist.githubusercontent.com/fdmanana/5262e608b3eecb9a3b2631f8dad49863/raw/1a82fe8eafbd5f6958dddf34d3c9648d7335018e/btrfs-don-t-loop-again-over-pinned-extent-maps-when-.patch
> >
> > I tested this patch on top of v6.10-rc6
> >
> > > Btw, besides the longer kswapd execution times, what else do you observe?
> > > Is it impacting performance of any applications?
> >
> > I observe that the system freezes under load.
> > Demonstration: https://youtu.be/1-gUrnEi2aU
> > The GNOME shell stops responding, and even the clock in the GNOME
> > status bar stops updating seconds.
> > And this didn't happen when the v6.9 kernel was running. Second, I
> > spotted high CPU usage by process kswapd0 when freezes occurred.
> > Therefore, I decided to find the commit that led to high CPU
> > consumption by the kswapd0 process.
> > As we found out, this commit turned out to be 956a17d9d050.
> >
> > > I think no matter what we do, it's likely that kswapd will take more
> > > time than before, because now there's extra work of going through
> > > extent maps and dropping them.
> > > We had to do it to prevent OOM situations because extent map creation
> > > was unbounded.
> >
> > Unfortunately, the patch didn't improve anything.
> > kswapd0 still consumes 100% CPU under load.
> > And my system continues to freeze.
>
> Ok, the concerning part is the freezing and high cpu usage.
>
> So besides that patch, try 2 other patches on top of it:
>
> 1) https://gist.githubusercontent.com/fdmanana/5262e608b3eecb9a3b2631f8dad49863/raw/aaf4c00fd40aaee0ee2788cd9fdfe2f083328c39/btrfs-don-t-loop-again-over-pinned-extent-maps-when-.patch
>     (this is the patch you tried before)
>
> 2) https://gist.githubusercontent.com/fdmanana/f2275050f04d1830adb811745bfd99d4/raw/1001d8154133d862e305959ee9eedebf55941669/gistfile1.txt
>
> 3) https://gist.githubusercontent.com/fdmanana/0a71b9e0fe71f38f67a50b7b53d520e6/raw/680cab70d2ef32337583bee6a4fb6519241b2faa/0003-btrfs-prevent-extent-map-shrinker-from-monopolizing-.patch
>
> Apply those patches on top of 6.10-rc in that order and let me know how it goes.

Also, a 4th one:

https://gist.githubusercontent.com/fdmanana/638d90142e4db7cd462121d812075de7/raw/acb90d92c1cab512414e0bd5461640c9015da4ec/0004-btrfs-use-delayed-iput-during-extent-map-shrinking.patch

This one should apply in any order. Try all those 4 together please.
Thanks!

> Thanks.
>
> >
> > 6.10.0-0.rc6.51.fc41.x86_64+debug with patch
> > up  1:00
> > root         269 13.1  0.0      0     0 ?        S    12:24   7:53 [kswapd0]
> > up  2:00
> > root         269 29.9  0.0      0     0 ?        R    12:24  36:02 [kswapd0]
> > up  3:00
> > root         269 37.8  0.0      0     0 ?        S    12:24  68:19 [kswapd0]
> > up  4:05
> > root         269 39.3  0.0      0     0 ?        R    12:24  96:40 [kswapd0]
> > up  5:01
> > root         269 38.8  0.0      0     0 ?        R    12:24 117:00 [kswapd0]
> > up  6:00
> > root         269 40.3  0.0      0     0 ?        S    12:24 145:24 [kswapd0]
> >
> > --
> > Best Regards,
> > Mike Gavrilov.