linux-kernel - powerpc hugepage leak caused by 576ed913 "block: use bio_add_page in bio_iov_iter_get

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Tue, 23 Apr 2019 15:41:31 +1000
From:   David Gibson <david@...son.dropbear.id.au>
To:     Christoph Hellwig <hch@....de>
Cc:     Jens Axboe <axboe@...nel.dk>,
        Michael Ellerman <michael@...erman.id.au>,
        Paul Mackerras <paulus@...ba.org>,
        linux-kernel@...r.kernel.org, linuxppc-dev@...ts.ozlabs.org,
        Nick Piggin <npiggin@...il.com>
Subject: powerpc hugepage leak caused by 576ed913 "block: use bio_add_page in
 bio_iov_iter_get_pages"

576ed913 "block: use bio_add_page in bio_iov_iter_get_pages", applied
late in the 4.19 cycle appears to introduce a regression causing a
huge page leak in a complicated set of circumstances I haven't fully
identified yet.

On a POWER8 machine with a kernel after the commit above, when I run a
KVM guest with RAM in hugetlbfs pages (and certain options, see below), a
handful of the hugepages used for RAM are not released after qemu and
the guest quit.  Usually 2 or 3 16MiB pages are leaked, though I've
seen anything from 0-8 occasionally.

There are a bunch of conditions on when it occurs, only some of which
I've pinned down:

 * It happens on a POWER8 8247-22L, but not a very similar 8247-21L,
   and I haven't been able to work out why, yet.

 * It only happens with certain combination of qemu block and caching
   options for the guest's root fs.  Specifically it appears to happen
   when the file used for the guest's root disk image is opened with
   O_DIRECT.

 * It depends somewhat on guest activity.
     - It doesn't occur if the guest is only booted to firmware
     - Booting only to initramfs without mounting the "real" root fs
       doesn't seem to trigger the problem
     - It appears to happen reliably with RHEL6 and RHEL7 guests, but
       only sometimes with RHEL8 guests, again, I don't know why at
       this stage

I pinned it down to this (host kernel) patch by bisection, and I've
double checked afterwards to confirm it really is this commit, not a
mistake during the bisection.

I've tried a bunch of instrumentation, but it hasn't been very
illuminating so far:

 * The leaked pages have non-zero count and are left in the
   hugepage_activelist

 * The leaked pages *don't* appear to be blocking release of the KVM
   VM or the qemu process owning it

 * The leaked pages *do* appear to be blocking release of the
   associated address_space and (anonymous) inode, though I'm not 100%
   certain about this.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)