linux-kernel - Re: 6.10/regression/bisected - after f1d97e769152 I spotted increased execution time of the kswapd0 process and symptoms as if there is not enough memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAL3q7H4MiarsqxSMc0OzY2TNRk8J7Lg+89MaPHY2+NPO-EcDgQ@mail.gmail.com>
Date: Wed, 3 Jul 2024 11:44:32 +0100
From: Filipe Manana <fdmanana@...nel.org>
To: Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>
Cc: Linux List Kernel Mailing <linux-kernel@...r.kernel.org>, 
	Linux regressions mailing list <regressions@...ts.linux.dev>, Btrfs BTRFS <linux-btrfs@...r.kernel.org>, 
	dsterba@...e.com, josef@...icpanda.com
Subject: Re: 6.10/regression/bisected - after f1d97e769152 I spotted increased
 execution time of the kswapd0 process and symptoms as if there is not enough memory

On Wed, Jul 3, 2024 at 11:31 AM Filipe Manana <fdmanana@...nel.org> wrote:
>
> On Tue, Jul 2, 2024 at 6:22 PM Filipe Manana <fdmanana@...nel.org> wrote:
> >
> > On Tue, Jul 2, 2024 at 3:13 PM Mikhail Gavrilov
> > <mikhail.v.gavrilov@...il.com> wrote:
> > >
> > > On Mon, Jul 1, 2024 at 2:31 PM Filipe Manana <fdmanana@...nel.org> wrote:
> > > >
> > > > Try this:
> > > >
> > > > https://lore.kernel.org/linux-btrfs/cb12212b9c599817507f3978c9102767267625b2.1719825714.git.fdmanana@suse.com/
> > > >
> > > > That applies only to the "for-next", it will need conflict resolution
> > > > for 6.10-rc, as noted in the commnets.
> > > > For a version that cleanly applies to 6.10-rc:
> > > >
> > > > https://gist.githubusercontent.com/fdmanana/5262e608b3eecb9a3b2631f8dad49863/raw/1a82fe8eafbd5f6958dddf34d3c9648d7335018e/btrfs-don-t-loop-again-over-pinned-extent-maps-when-.patch
> > >
> > > I tested this patch on top of v6.10-rc6
> > >
> > > > Btw, besides the longer kswapd execution times, what else do you observe?
> > > > Is it impacting performance of any applications?
> > >
> > > I observe that the system freezes under load.
> > > Demonstration: https://youtu.be/1-gUrnEi2aU
> > > The GNOME shell stops responding, and even the clock in the GNOME
> > > status bar stops updating seconds.
> > > And this didn't happen when the v6.9 kernel was running. Second, I
> > > spotted high CPU usage by process kswapd0 when freezes occurred.
> > > Therefore, I decided to find the commit that led to high CPU
> > > consumption by the kswapd0 process.
> > > As we found out, this commit turned out to be 956a17d9d050.
> > >
> > > > I think no matter what we do, it's likely that kswapd will take more
> > > > time than before, because now there's extra work of going through
> > > > extent maps and dropping them.
> > > > We had to do it to prevent OOM situations because extent map creation
> > > > was unbounded.
> > >
> > > Unfortunately, the patch didn't improve anything.
> > > kswapd0 still consumes 100% CPU under load.
> > > And my system continues to freeze.
> >
> > Ok, the concerning part is the freezing and high cpu usage.
> >
> > So besides that patch, try 2 other patches on top of it:
> >
> > 1) https://gist.githubusercontent.com/fdmanana/5262e608b3eecb9a3b2631f8dad49863/raw/aaf4c00fd40aaee0ee2788cd9fdfe2f083328c39/btrfs-don-t-loop-again-over-pinned-extent-maps-when-.patch
> >     (this is the patch you tried before)
> >
> > 2) https://gist.githubusercontent.com/fdmanana/f2275050f04d1830adb811745bfd99d4/raw/1001d8154133d862e305959ee9eedebf55941669/gistfile1.txt
> >
> > 3) https://gist.githubusercontent.com/fdmanana/0a71b9e0fe71f38f67a50b7b53d520e6/raw/680cab70d2ef32337583bee6a4fb6519241b2faa/0003-btrfs-prevent-extent-map-shrinker-from-monopolizing-.patch
> >
> > Apply those patches on top of 6.10-rc in that order and let me know how it goes.
>
> Also, a 4th one:
>
> https://gist.githubusercontent.com/fdmanana/638d90142e4db7cd462121d812075de7/raw/acb90d92c1cab512414e0bd5461640c9015da4ec/0004-btrfs-use-delayed-iput-during-extent-map-shrinking.patch
>
> This one should apply in any order. Try all those 4 together please.
> Thanks!

I'm collecting all the patches in this branch:

https://git.kernel.org/pub/scm/linux/kernel/git/fdmanana/linux.git/log/?h=em_shrinker_6.10

They apply cleanly to 6.10-rc.

>
> > Thanks.
> >
> > >
> > > 6.10.0-0.rc6.51.fc41.x86_64+debug with patch
> > > up  1:00
> > > root         269 13.1  0.0      0     0 ?        S    12:24   7:53 [kswapd0]
> > > up  2:00
> > > root         269 29.9  0.0      0     0 ?        R    12:24  36:02 [kswapd0]
> > > up  3:00
> > > root         269 37.8  0.0      0     0 ?        S    12:24  68:19 [kswapd0]
> > > up  4:05
> > > root         269 39.3  0.0      0     0 ?        R    12:24  96:40 [kswapd0]
> > > up  5:01
> > > root         269 38.8  0.0      0     0 ?        R    12:24 117:00 [kswapd0]
> > > up  6:00
> > > root         269 40.3  0.0      0     0 ?        S    12:24 145:24 [kswapd0]
> > >
> > > --
> > > Best Regards,
> > > Mike Gavrilov.