linux-kernel - Re: kernel panic in reading /proc/kpageflags when enabling RAM-simulated PMEM

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 6 Jun 2018 05:16:24 +0000
From:   Naoya Horiguchi <n-horiguchi@...jp.nec.com>
To:     Matthew Wilcox <willy@...radead.org>
CC:     "linux-mm@...ck.org" <linux-mm@...ck.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Michal Hocko <mhocko@...nel.org>,
        "mingo@...nel.org" <mingo@...nel.org>,
        "dan.j.williams@...el.com" <dan.j.williams@...el.com>,
        Huang Ying <ying.huang@...el.com>,
        Pavel Tatashin <pasha.tatashin@...cle.com>
Subject: Re: kernel panic in reading /proc/kpageflags when enabling
 RAM-simulated PMEM

On Tue, Jun 05, 2018 at 07:35:01AM +0000, Horiguchi Naoya(堀口 直也) wrote:
> On Mon, Jun 04, 2018 at 06:18:36PM -0700, Matthew Wilcox wrote:
> > On Tue, Jun 05, 2018 at 12:54:03AM +0000, Naoya Horiguchi wrote:
> > > Reproduction precedure is like this:
> > >  - enable RAM based PMEM (with a kernel boot parameter like memmap=1G!4G)
> > >  - read /proc/kpageflags (or call tools/vm/page-types with no arguments)
> > >  (- my kernel config is attached)
> > > 
> > > I spent a few days on this, but didn't reach any solutions.
> > > So let me report this with some details below ...
> > > 
> > > In the critial page request, stable_page_flags() is called with an argument
> > > page whose ->compound_head was somehow filled with '0xffffffffffffffff'.
> > > And compound_head() returns (struct page *)(head - 1), which explains the
> > > address 0xfffffffffffffffe in the above message.
> > 
> > Hm.  compound_head shares with:
> > 
> >                         struct list_head lru;
> >                                 struct list_head slab_list;     /* uses lru */
> >                                 struct {        /* Partial pages */
> >                                         struct page *next;
> >                         unsigned long _compound_pad_1;  /* compound_head */
> >                         unsigned long _pt_pad_1;        /* compound_head */
> >                         struct dev_pagemap *pgmap;
> >                 struct rcu_head rcu_head;
> > 
> > None of them should be -1.
> > 
> > > It seems that this kernel panic happens when reading kpageflags of pfn range
> > > [0xbffd7, 0xc0000), which coresponds to a 'reserved' range.
> > > 
> > > [    0.000000] user-defined physical RAM map:
> > > [    0.000000] user: [mem 0x0000000000000000-0x000000000009fbff] usable
> > > [    0.000000] user: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> > > [    0.000000] user: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> > > [    0.000000] user: [mem 0x0000000000100000-0x00000000bffd6fff] usable
> > > [    0.000000] user: [mem 0x00000000bffd7000-0x00000000bfffffff] reserved
> > > [    0.000000] user: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> > > [    0.000000] user: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> > > [    0.000000] user: [mem 0x0000000100000000-0x000000013fffffff] persistent (type 12)
> > > 
> > > So I guess 'memmap=' parameter might badly affect the memory initialization process.
> > > 
> > > This problem doesn't reproduce on v4.17, so some pre-released patch introduces it.
> > > I hope this info helps you find the solution/workaround.
> > 
> > Can you try bisecting this?  It could be one of my patches to reorder struct
> > page, or it could be one of Pavel's deferred page initialisation patches.
> > Or something else ;-)
> 
> Thank you for the comment. I'm trying bisecting now, let you know the result later.
> 
> And I found that my statement "not reproduce on v4.17" was wrong (I used
> different kvm guests, which made some different test condition and misguided me),
> this seems an older (at least < 4.15) bug.

(Cc: Pavel)

Bisection showed that the following commit introduced this issue:

  commit f7f99100d8d95dbcf09e0216a143211e79418b9f
  Author: Pavel Tatashin <pasha.tatashin@...cle.com>
  Date:   Wed Nov 15 17:36:44 2017 -0800
  
      mm: stop zeroing memory during allocation in vmemmap

This patch postpones struct page zeroing to later stage of memory initialization.
My kernel config disabled CONFIG_DEFERRED_STRUCT_PAGE_INIT so two callsites of
__init_single_page() were never reached. So in such case, struct pages populated
by vmemmap_pte_populate() could be left uninitialized?
And I'm not sure yet how this issue becomes visible with memmap= setting.

Thanks,
Naoya Horiguchi