linux-kernel - Re: 6.10/regression/bisected - after f1d97e769152 I spotted increased execution time of the kswapd0 process and symptoms as if there is not enough memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAL3q7H6vG6PEKjcsXtSuq=yks_g-MczAz_-V96QSZCs9ezRZpg@mail.gmail.com>
Date: Tue, 2 Jul 2024 18:22:27 +0100
From: Filipe Manana <fdmanana@...nel.org>
To: Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>
Cc: Linux List Kernel Mailing <linux-kernel@...r.kernel.org>, 
	Linux regressions mailing list <regressions@...ts.linux.dev>, Btrfs BTRFS <linux-btrfs@...r.kernel.org>, 
	dsterba@...e.com, josef@...icpanda.com
Subject: Re: 6.10/regression/bisected - after f1d97e769152 I spotted increased
 execution time of the kswapd0 process and symptoms as if there is not enough memory

On Tue, Jul 2, 2024 at 3:13 PM Mikhail Gavrilov
<mikhail.v.gavrilov@...il.com> wrote:
>
> On Mon, Jul 1, 2024 at 2:31 PM Filipe Manana <fdmanana@...nel.org> wrote:
> >
> > Try this:
> >
> > https://lore.kernel.org/linux-btrfs/cb12212b9c599817507f3978c9102767267625b2.1719825714.git.fdmanana@suse.com/
> >
> > That applies only to the "for-next", it will need conflict resolution
> > for 6.10-rc, as noted in the commnets.
> > For a version that cleanly applies to 6.10-rc:
> >
> > https://gist.githubusercontent.com/fdmanana/5262e608b3eecb9a3b2631f8dad49863/raw/1a82fe8eafbd5f6958dddf34d3c9648d7335018e/btrfs-don-t-loop-again-over-pinned-extent-maps-when-.patch
>
> I tested this patch on top of v6.10-rc6
>
> > Btw, besides the longer kswapd execution times, what else do you observe?
> > Is it impacting performance of any applications?
>
> I observe that the system freezes under load.
> Demonstration: https://youtu.be/1-gUrnEi2aU
> The GNOME shell stops responding, and even the clock in the GNOME
> status bar stops updating seconds.
> And this didn't happen when the v6.9 kernel was running. Second, I
> spotted high CPU usage by process kswapd0 when freezes occurred.
> Therefore, I decided to find the commit that led to high CPU
> consumption by the kswapd0 process.
> As we found out, this commit turned out to be 956a17d9d050.
>
> > I think no matter what we do, it's likely that kswapd will take more
> > time than before, because now there's extra work of going through
> > extent maps and dropping them.
> > We had to do it to prevent OOM situations because extent map creation
> > was unbounded.
>
> Unfortunately, the patch didn't improve anything.
> kswapd0 still consumes 100% CPU under load.
> And my system continues to freeze.

Ok, the concerning part is the freezing and high cpu usage.

So besides that patch, try 2 other patches on top of it:

1) https://gist.githubusercontent.com/fdmanana/5262e608b3eecb9a3b2631f8dad49863/raw/aaf4c00fd40aaee0ee2788cd9fdfe2f083328c39/btrfs-don-t-loop-again-over-pinned-extent-maps-when-.patch
    (this is the patch you tried before)

2) https://gist.githubusercontent.com/fdmanana/f2275050f04d1830adb811745bfd99d4/raw/1001d8154133d862e305959ee9eedebf55941669/gistfile1.txt

3) https://gist.githubusercontent.com/fdmanana/0a71b9e0fe71f38f67a50b7b53d520e6/raw/680cab70d2ef32337583bee6a4fb6519241b2faa/0003-btrfs-prevent-extent-map-shrinker-from-monopolizing-.patch

Apply those patches on top of 6.10-rc in that order and let me know how it goes.
Thanks.

>
> 6.10.0-0.rc6.51.fc41.x86_64+debug with patch
> up  1:00
> root         269 13.1  0.0      0     0 ?        S    12:24   7:53 [kswapd0]
> up  2:00
> root         269 29.9  0.0      0     0 ?        R    12:24  36:02 [kswapd0]
> up  3:00
> root         269 37.8  0.0      0     0 ?        S    12:24  68:19 [kswapd0]
> up  4:05
> root         269 39.3  0.0      0     0 ?        R    12:24  96:40 [kswapd0]
> up  5:01
> root         269 38.8  0.0      0     0 ?        R    12:24 117:00 [kswapd0]
> up  6:00
> root         269 40.3  0.0      0     0 ?        S    12:24 145:24 [kswapd0]
>
> --
> Best Regards,
> Mike Gavrilov.