linux-kernel - Re: [regression] oops on heavy compilations ("kernel BUG at mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAKEwX=PiyvV7AxKESk5zc9TN1n5b0uFR0pr_HGxKsUFUgk0q+Q@mail.gmail.com>
Date: Tue, 3 Sep 2024 15:43:43 -0700
From: Nhat Pham <nphamcs@...il.com>
To: Yosry Ahmed <yosryahmed@...gle.com>
Cc: Piotr Oniszczuk <piotr.oniszczuk@...il.com>, Pedro Falcato <pedro.falcato@...il.com>, 
	Matthew Wilcox <willy@...radead.org>, 
	Linux regressions mailing list <regressions@...ts.linux.dev>, LKML <linux-kernel@...r.kernel.org>, 
	Johannes Weiner <hannes@...xchg.org>, Linux-MM <linux-mm@...ck.org>
Subject: Re: [regression] oops on heavy compilations ("kernel BUG at
 mm/zswap.c:1005!" and "Oops: invalid opcode: 0000")

On Tue, Sep 3, 2024 at 10:49 AM Yosry Ahmed <yosryahmed@...gle.com> wrote:
>
> On Mon, Sep 2, 2024 at 1:58 AM Piotr Oniszczuk
> <piotr.oniszczuk@...il.com> wrote:
> >
> >
> >
> > > Wiadomość napisana przez Yosry Ahmed <yosryahmed@...gle.com> w dniu 31.08.2024, o godz. 19:23:
> > >
> > > On Sat, Aug 31, 2024 at 2:41 AM Piotr Oniszczuk
> > > <piotr.oniszczuk@...il.com> wrote:
> > >>
> > >>
> > >>
> > >>> Wiadomość napisana przez Yosry Ahmed <yosryahmed@...gle.com> w dniu 29.08.2024, o godz. 23:54:
> > >>>
> > >>> I also noticed that you are using z3fold as the zpool. Is the problem
> > >>> reproducible with zsmalloc? I wouldn't be surprised if there's a
> > >>> z3fold bug somewhere.
> > >>>
> > >>
> > >> Hmm - yesterday i recompiled 6.9.12 with zsmalloc and …. after 16h of continuous tests I can’t reproduce issue.
> > >> With zsmalloc 6.9.12 looks to me like stable.
> > >
> > > Interesting, and a little bit what I hoped for tbh.
> >
> > :-)
> >
> > I tested mainline 6.10.7 with 26h test and also it is stable with zsmalloc
> >
> > >
> > >>
> > >> With this - what will be your advice to move forward?
> > >
> > > Well, it's possible that some zswap change was not fully compatible
> > > with z3fold, or surfaced a dormant bug in z3fold. Either way, my
> > > recommendation is to use zsmalloc.
> > > I have been trying to deprecate
> >
> > IMHO - isn’t bug in this report + difficulties to reproduce->fix enough to depreciate z3fold?
>
> I would say this bug report is yet another reason why we should deprecate it.

+100000.

This is precisely why I was asking which allocator was being used
here. We have also accidentally selected z3fold internally a couple
times in the past, which had bitten us as well.

>
> >
> > > z3fold, and honestly you are the only person I have seen use z3fold in
> > > a while -- which is probably why no one else reported such a problem.
> >
> > Well - in fact this is ArchLinux - not me.
> > I’m using Arch and kernel in builder machine with ArchLinux config + packaging
>
> According to [1], zsmalloc should be the default allocator for zswap
> on ArchLinux. Anyway, I initially thought that no one was using z3fold
> and it was bitrot, but apparently some people are using it and it's
> actively harming them.
>
> [1]https://wiki.archlinux.org/title/Zswap
>
> >
> > >
> >
> > I see benefits already: on very memory demanding qtwebkit compile:
> > z3fold: swap frequently gets 6..8G from 16G available
> > zsmalloc: can’t see more than 1..2G

Exactly :) zsmalloc is better than z3fold in a lot of workloads that I
have observed.

> >
> > > doubt that you (or anyone) wants to spend time debugging a z3fold
> > > problem :)
> >
> > lets depreciate it!
>
> I tried deprecating it before [2] and performed some analysis [3], but
> there was some.. resistance. Maybe I will try again and use this bug
> report as yet another argument for deprecating z3fold :)
>
> [2] https://lore.kernel.org/linux-mm/20240112193103.3798287-1-yosryahmed@google.com/
> [3] https://lore.kernel.org/lkml/CAJD7tkbRF6od-2x_L8-A1QL3=2Ww13sCj4S3i4bNndqF+3+_Vg@mail.gmail.com/

I don't wanna sound like a broken record. But this has been the nth
time we need to spend extra engineering time and effort unnecessarily
because we have not deprecated z3fold.

If you need more datapoint - here's our last conversation where z3fold
was a problem:

https://lore.kernel.org/lkml/CAKEwX=Mo+EaaxBYcLMTHYADB4WhqC3QmWV3WQ0h2KM491FRuQA@mail.gmail.com/