linux-kernel - Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAJD7tkZrf_RVT1+LtBMrpL0T==1757WSjHiSOHp7Pio0T-F5OQ@mail.gmail.com>
Date: Wed, 4 Sep 2024 16:36:27 -0700
From: Yosry Ahmed <yosryahmed@...gle.com>
To: Nhat Pham <nphamcs@...il.com>
Cc: Piotr Oniszczuk <piotr.oniszczuk@...il.com>, Pedro Falcato <pedro.falcato@...il.com>, 
	Matthew Wilcox <willy@...radead.org>, 
	Linux regressions mailing list <regressions@...ts.linux.dev>, LKML <linux-kernel@...r.kernel.org>, 
	Johannes Weiner <hannes@...xchg.org>, Linux-MM <linux-mm@...ck.org>
Subject: Re: [regression] oops on heavy compilations ("kernel BUG at
 mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

On Tue, Sep 3, 2024 at 3:43 PM Nhat Pham <nphamcs@...il.com> wrote:
>
> On Tue, Sep 3, 2024 at 10:49 AM Yosry Ahmed <yosryahmed@...gle.com> wrote:
> >
> > On Mon, Sep 2, 2024 at 1:58 AM Piotr Oniszczuk
> > <piotr.oniszczuk@...il.com> wrote:
> > >
> > >
> > >
> > > > Wiadomość napisana przez Yosry Ahmed <yosryahmed@...gle.com> w dniu 31.08.2024, o godz. 19:23:
> > > >
> > > > On Sat, Aug 31, 2024 at 2:41 AM Piotr Oniszczuk
> > > > <piotr.oniszczuk@...il.com> wrote:
> > > >>
> > > >>
> > > >>
> > > >>> Wiadomość napisana przez Yosry Ahmed <yosryahmed@...gle.com> w dniu 29.08.2024, o godz. 23:54:
> > > >>>
> > > >>> I also noticed that you are using z3fold as the zpool. Is the problem
> > > >>> reproducible with zsmalloc? I wouldn't be surprised if there's a
> > > >>> z3fold bug somewhere.
> > > >>>
> > > >>
> > > >> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and …. after 16h of continuous tests I can’t reproduce issue.
> > > >> With zsmalloc 6.9.12 looks to me like stable.
> > > >
> > > > Interesting, and a little bit what I hoped for tbh.
> > >
> > > :-)
> > >
> > > I tested mainline 6.10.7 with 26h test and also it is stable with zsmalloc
> > >
> > > >
> > > >>
> > > >> With this - what will be your advice to move forward?
> > > >
> > > > Well, it's possible that some zswap change was not fully compatible
> > > > with z3fold, or surfaced a dormant bug in z3fold. Either way, my
> > > > recommendation is to use zsmalloc.
> > > > I have been trying to deprecate
> > >
> > > IMHO - isn’t bug in this report + difficulties to reproduce->fix enough to depreciate z3fold?
> >
> > I would say this bug report is yet another reason why we should deprecate it.
>
> +100000.
>
> This is precisely why I was asking which allocator was being used
> here. We have also accidentally selected z3fold internally a couple
> times in the past, which had bitten us as well.
>
> >
> > >
> > > > z3fold, and honestly you are the only person I have seen use z3fold in
> > > > a while -- which is probably why no one else reported such a problem.
> > >
> > > Well - in fact this is ArchLinux - not me.
> > > I’m using Arch and kernel in builder machine with ArchLinux config + packaging
> >
> > According to [1], zsmalloc should be the default allocator for zswap
> > on ArchLinux. Anyway, I initially thought that no one was using z3fold
> > and it was bitrot, but apparently some people are using it and it's
> > actively harming them.
> >
> > [1]https://wiki.archlinux.org/title/Zswap
> >
> > >
> > > >
> > >
> > > I see benefits already: on very memory demanding qtwebkit compile:
> > > z3fold: swap frequently gets 6..8G from 16G available
> > > zsmalloc: can’t see more than 1..2G
>
> Exactly :) zsmalloc is better than z3fold in a lot of workloads that I
> have observed.
>
> > >
> > > > doubt that you (or anyone) wants to spend time debugging a z3fold
> > > > problem :)
> > >
> > > lets depreciate it!
> >
> > I tried deprecating it before [2] and performed some analysis [3], but
> > there was some.. resistance. Maybe I will try again and use this bug
> > report as yet another argument for deprecating z3fold :)
> >
> > [2] https://lore.kernel.org/linux-mm/20240112193103.3798287-1-yosryahmed@google.com/
> > [3] https://lore.kernel.org/lkml/CAJD7tkbRF6od-2x_L8-A1QL3=2Ww13sCj4S3i4bNndqF+3+_Vg@mail.gmail.com/
>
> I don't wanna sound like a broken record. But this has been the nth
> time we need to spend extra engineering time and effort unnecessarily
> because we have not deprecated z3fold.
>
> If you need more datapoint - here's our last conversation where z3fold
> was a problem:
>
> https://lore.kernel.org/lkml/CAKEwX=Mo+EaaxBYcLMTHYADB4WhqC3QmWV3WQ0h2KM491FRuQA@mail.gmail.com/

I sent a v2 of the z3fold deprecation attempt:
https://lore.kernel.org/lkml/20240904233343.933462-1-yosryahmed@google.com/.