linux-kernel - Re: Zram writeback feature unstable with heavy swap utilization

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180724064726.GA138512@rodete-desktop-imager.corp.google.com>
Date:   Tue, 24 Jul 2018 15:47:26 +0900
From:   Minchan Kim <minchan@...nel.org>
To:     Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>,
        mawilcox@...rosoft.com
Cc:     Tino Lehnig <tino.lehnig@...tabo.de>, ngupta@...are.org,
        linux-kernel@...r.kernel.org,
        Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: Zram writeback feature unstable with heavy swap utilization -
 BUG: Bad page state in process...

On Tue, Jul 24, 2018 at 11:53:30AM +0900, Sergey Senozhatsky wrote:
> On (07/24/18 10:03), Minchan Kim wrote:
> > On Mon, Jul 23, 2018 at 02:29:32PM +0200, Tino Lehnig wrote:
> > > Hello,
> > > 
> > > after enabling the writeback feature in zram, I encountered the kernel bug
> > > below with heavy swap utilization. There is one specific workload that
> > > triggers the bug reliably and that is running Windows in KVM while
> > > overcommitting memory. The Windows VMs would fill all allocated memory with
> > > zero pages while booting. A few seconds after the host hits zram swap, the
> > > console on the host is flooded with the bug message. A few more seconds
> > > later I also encountered filesystem errors on the host causing the root
> > > filesystem to be mounted read-only. The filesystem errors do not occur when
> > > leaving RAM available for the host OS by limiting physical memory of the
> > > QEMU processes via cgroups.
> > > 
> > > I started three KVM instances with the following commands in my tests. Any
> > > Windows ISO or disk image can be used. Less instances and smaller allocated
> > > memory will also trigger the bug as long as swapping occurs. The type of
> > > writeback device does not seem to matter. I have tried a SATA SSD and an
> > > NVMe Optane drive so far. My test machine has 256 GB of RAM and one CPU. I
> > > saw the same behavior on another machine with two CPUs and 128 GB of RAM.
> > > 
> > > The bug does not occur when using zram as swap without "backing_dev" being
> > > set, but I had even more severe problems when running the same test on
> > > Ubuntu Kernels 4.15 and 4.17. Regardless of the writeback feature being used
> > > or not, the host would eventually lock up entirely when swap is in use on
> > > zram. The lockups may not be related directly to zram though and were
> > > apparently fixed in 4.18. I had absolutely no problems on Ubuntu Kernel 4.13
> > > either, before the writeback feature was introduced.
> > 
> > We didn't release v4.18 yet. Could you say what kernel tree/what version
> > you used?
> > 
> > Now I don't have enough time to dig in.
> > 
> > Sergey, I really appreciate if you could have availabe time to look into.
> > Anyway, I could try to see it asap if Sergey is not available.
> > No worry.
> 
> Interesting case.
> 
> Will take me several days to get to it.
> Sorry, quite busy at the moment.

I look though v4.18-rcX very quick what are changed for zsmalloc.
struct page fields were totally reordered. It could be a regression related to
that. Ccing Matthew